Ingesting CSV files from S3 without csv extension

Slack Knowledge Base Q&A Connector Questions
, , , ,
1

Summary

Clarification on ingesting CSV files from S3 without csv extension


Question

Hi all, I’m trying to ingest csv files from S3 into my warehouse. However, the csv files don’t have the csv extension due to the way AWS creates part files. Do these need to have the csv extension in order to be ingested as the CSV file format?


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

["ingest", "csv-files", "s3", "warehouse", "csv-extension", "file-format"]
2

Have you tried S3 source connector https://docs.airbyte.com/integrations/sources/s3 ?

With Globs you should be to able to find S3 objects you need. I think file extension shouldn’t matter.

You can also use End-To-End Testing destination connector https://docs.airbyte.com/integrations/destinations/e2e-test to check if it works fine without saving data to warehouse.