Trouble setting up data synchronization on new abctl instance

Slack Knowledge Base Q&A Connector Questions
, , , ,
1

Summary

User is facing issues with data synchronization on a new abctl instance despite source and destination connectivity working. Sync does not work when extending to a wider set of tables with no errors in logs or pending pods, destination tables created, but no data transfer happening.


Question

Re: https://airbytehq.slack.com/archives/C01AHCD885S/p1725679219551029

I am having trouble setting up data synchronization on a new abctl instance.

To the best of my knowledge:
• source connectivity is working (evidence: I have synced a single stream with this source connector)
• destination connectivity is working (evidence: I have synced a single stream with thus destination connector)
• the node is not under provisioned (evidence: 4 CPU, 16GB, no pending pods due to resource constraints)
• there are no errors in the connection specification (evidence: no error messages, of any type, in connection logs)
Yes, when I extend the set to a wider set of 8 tables, the sync does not work with these symptoms:

• no indication of errors in the connection logs or any k8s log
• no pending pods
• destination tables created
• orchestration, source and destination logs show idle all to be “running”, but idle even after 30 minutes of apparently doing nothing
• node CPU is idle
• there is no evidence of data being transferred nor indications of why data cannot be transferred


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

["data-synchronization", "abctl-instance", "source-connector", "destination-connector", "connection-logs", "k8s-logs"]
2

Ok, with some bisection, I think I found the issue. One of the streams, flat_bom_item is based on a view and I can reproduce a hang in the psql client with this simple command:

\dS+ queryv1.flat_bom_item;
So, that’ll be the issue. This might be a postgres 16.x thing, but not sure yet.

3

To close this out - there was a deadlock in the source database related to the drop and recreate of the views that airbyte was syncing from.

So, lesson learned - if syncing appears to be stalled, check for deadlocks!