New connectors not creating proper columns


Connectors are dumping data into a JSON object instead of creating proper columns. User questions if Airbyte is moving towards being a JSON blob mover based on AI bot’s information.


My new connectors aren’t creating proper columns anymore, and are just dumping data into a JSON object. The AI bot says that Airbyte is moving away from being a fully-featured replication tool towards being a JSON blob mover. Is this correct? It cites some|github issue, but that seems to be talking about dbt integration and hooks. (I’m hoping this isn’t the case because this would be a tragic step backwards in Airbyte being a valuable tool for analysts to quickly land data that can be queried as-is without manual engineering ETL support)

This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

["new-connectors", "json-object", "airbyte", "replication-tool", "json-blob-mover", "dbt-integration", "data-engineering", "ETL"]

Hi Alex, thanks for the response, and great that’s a relief to hear! Yes, we’re in the process of migrating from Snowflake to Databricks, so that would explain the difference in behavior I am seeing. I will look forward to Databricks support–thanks again!

Any time! Also, feel free to add information about your Databricks use case to the roadmap item if there is anything you think we should consider.

Hi Daniel! That is not the case. Airbyte still transforms the json data for <Upgrading to Destinations V2 | Airbyte Documentation of our destinations> (see note below).

It looks like Kapa confused…
normalization - which was our old process of transforming json data into tables, and has been replaced by <|Destinations V2>
custom normalization - which is a deprecated feature in OSS that allowed you to run custom dbt code after an Airbyte sync (<Deprecating Custom Normalization (dbt) in Airbyte OSS · airbytehq/airbyte · Discussion #34860 · GitHub issue link>)
Back to your use case - it sounds like you are using Databricks? Unfortunately, we haven’t implemented Destinations V2 for the Databricks destination yet, which is why you are only seeing the json data. The Databricks connector will be a focus for the team later this year (<Issues · airbytehq/airbyte · GitHub item>).