Summary
The user is experiencing an issue with the destination-postgresql connector where the staging table created by Airbyte gets truncated to 64 characters when the schema name + table name is longer. This leads to multiple streams writing to the same staging table and causing data mix-up. The user has raised an open issue on GitHub and is seeking clarification on whether this logic is controlled by the destination or if it’s a fundamental issue in Airbyte Core.
Question
Hi team, there is an issue we are seeing with the destination-postgresql connector (2.4.0) that I can’t tell is isolated to it , or how airbyte stages and writes data that is pulls from a source. Basically when the schema name + table name is longer than 64 characters, the staging table that the stream creates in airbtye_internal gets truncated to 64 characters. The problem is when two streams truncate to the same string, both write their streams to the same staging table, and both streams then try moving the data from that table to each of their final landed stream tables, leaving both streams a mess. I’ve documented what could be a better strategy in this open issue: https://github.com/airbytehq/airbyte/issues/45345. I’m open to helping implement a fix, but I’m not very familiar with this part of the architecture. Is this logic controlled by the destination or is this something more fundamental to Airbyte Core that would need looked at?
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.
Sharing here in the hope that the community or devs can help take this the rest of the way.