Error handling for different schemas in S3 to S3 connection

Slack Knowledge Base Q&A Connector Questions
, , , ,
1

Summary

Inquiring about the possibility of generating an error when source CSV files in an S3 to S3 connection have different schemas, as the records are currently being combined at the destination without error.


Question

Hello,
I am creating an S3 to S3 connection and I was wondering if it is possible to error when the source CSV files don’t follow the same schema. For example, in the S3 source, I create a stream with files of different schemas. I can’t make it error no matter what settings I use. It always moves the records to the destination, combining the schemas.


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

["s3-connector", "schema-error", "csv-files", "data-integration"]
2

Hi <@U01MMSDJGC9> is there anything you could say about this? This seems like a pretty fundamental feature that I would expect to be default. That is, when an s3 source has multiple schemas, we find that Airbyte simple merges the schemas and we have no recourse to raise an exception. Can you confirm?

3

Hi <@U01MMSDJGC9> bumping this question

4

I don’t think this is implemented today. The best path here is open a feature request to implement it in the future.
https://github.com/airbytehq/airbyte/issues/new/choose

5

Thank you for the reply!