S3 - CSV file with CR line breaks

Connector Questions & Issues
, ,
1
  • Is this your first time deploying Airbyte?: Yes
  • OS Version / Instance: Ubuntu
  • Memory / Disk: NA
  • Deployment: Kubernetes deployment
  • Airbyte Version: 0.44.2
  • Source name/version: S3 connector 3.0
  • Destination name/version: NA (postgres but has no impact)
  • Step: During sync
  • Description:

We are adding a source for CSV files from a S3 bucket, but the file on the bucket are formated with CR line breaks instead of LF line breaks. The configuration check is failing With the following error:

Error('new-line character seen in unquoted field - do you need to open the file in universal-newline mode?')

I couldn’t find any option in the options for the reader to handle this issue. The issue seems to only be that the new line in the file are made with \r (old MacOs format) instead of the standard \n

We could manually modify the files, but since they come from a process on which we don’t have control, it would require adding a process outside of airbyte which is obviously not ideal.