How to Dynamically Determine String Length and Charset in Teradata Connector Using Airbyte V2 Java Framework?

Platform, Deploy & Infra Issues
1

Hi Airbyte Community,

We have implemented Typing and Deduping feature in Teradata Destination Connector to Support Airbyte V2. Teradata Destination connector is based on JDBC Destination.

Recently, we encountered an issue with the Airbyte Teradata Destination connector.

Our Challenge:

  • By default, string columns are created with the LATIN charset, which causes failures when special Unicode characters appear.
  • Switching all string columns to UNICODE solves this but doubles storage size unnecessarily for ASCII-only data.
  • We want to dynamically create final tables with proper VARCHAR length and charset (LATIN vs UNICODE) based on the actual data in raw tables.
  • The raw tables have the _airbyte_data JSON column defined as UNICODE always, so we can’t rely on that alone for charset decisions.
  • DefaultTyperDeduper doesn’t have support for constructing dynamic properties while creating final tables from raw tables.
  • We tried Implement dynamic character set in CreateTable method of TeradataSQLGenerator but this class is meant for only constructing SQL and has no support of connecting database.

** Questions:**

  • Is there any recommended best practice within Airbyte’s V2 framework or JDBCDestination for enabling data-driven typing and deduplication like this?

We need your guidance on this to implement proper solution for this issue.

Thanks in advance for your help!