Handling duplicated column names at source in Airbyte

Summary

Airbyte’s handling of duplicated column names at the source after standardization and user options for behavior selection.


Question

Also unrelated, how does Airbyte handle duplicated column names at source after standardization? Can users choose the behavior - exclude both, pick one, etc? (e.g. SecRegion_c vs Sec_Region__c both map to sec_region_c). Currently using a vendor and unhappy with the lack of options.



This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.

Join the conversation on Slack

["duplicated-column-names", "source", "standardization", "behavior-selection", "options"]

You have the column selection feature where you can pick one or exclude both.
In the image is one example where I excluded the group and creator_id

Is there a way to rename the destination column? Or change the behavior programmatically?

You can’t rename a column but you can use the column selection feature using the API.

https://airbyte-public-api-docs.s3.us-east-2.amazonaws.com/rapidoc-api-docs.html#post-/v1/connections/update

gotcha; thanks! can the rest api also tell me in a current connector, which fields are duplicated, and which ones are selected/blocked?

one of the excruciating pain with our current vendor is that the connector management api return exactly the same results for duplicated fields. they are both shown as “enabled”. there is no way for me to know which one was exposed and make any changes programmatically. imagine doing this hundreds of times in the UI!