Asked By: Anonymous
Is it possible to concatenate 1000 CSV file that have header into one file with no duplicated header directly in Google Cloud Storage? I could easily do this by downloading the file into my local hard drive but I would prefer to do it natively in Cloud Storage.
They all have same columns, and have header row.
Answered By: Anonymous
I wrote an article to handle CSV files with BigQuery. To avoid several files, and if the volume is less than 1Gb, the recommended way is the following
- Create a temporary table in BigQuery with all your CSV.
- Use the Export API (not the export function)
Let me know if you need more guidance.