Using File Connector for Multiple CSV Uploads to Redshift

Slack Knowledge Base Q&A Connector Questions
, , , ,
1

Summary

User inquires about the capability of the File connector to upload multiple CSV files to a Redshift database, having successfully uploaded a single file but facing issues with file layout when using the S3 connector.


Question

Can I use the File connector (CSV, JSON, Excel, Feather, Parquet) to upload a set of 3 CSV files to a Redshift database?
Uploading a single file worked for me, but I need to transfer 3 files daily, which are stored in my S3 bucket.
I tried using the S3 connector by adding the CSV configuration, but it seems the component is slightly different, and I’m facing file layout issues that I don’t encounter when using the File connector.


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

['file-connector', 'csv', 'redshift', 's3-connector', 'file-layout-issues']
2

If it works for you, use it. Don’t ask anybody for permission :wink:

I think the biggest limitation of File source connector is that it’s using specific path/url to file, while S3 source connector allows you to use globs - file pattern that specifies which files should be selected from S3.

What do you mean by “file layout issues”?