Source Postgres - Sync completes without failure but not all streams get synced

  • Is this your first time deploying Airbyte?: No
  • EC2 Linux
  • Memory / Disk: 16Gb / 1 Tb
  • Deployment: Docker
  • Airbyte Version: 0.39.37
  • Source name/version: Postgres (0.4.34)
  • Destination name/version: S3 (0.3.11)
  • Step: Sync
  • Description: Sync completes without any failures, but there are some tables that are set up to sync that do not get synced. The syncs are set up as full refresh and the existing data does get deleted but the new data doesn’t get synced. After looking at the logs, I see these tables are included in the logs for table found, queing query, and deleting existing data but I don’t see anything in the log for these tables for the “Starting new buffer stream” step.

I uploaded the logs for an instance where the primary_diagnosis table is one that doesn’t get synced.

This issue happens occasionally and if I resync, sometimes its successful.

logs-2367.txt (354.7 KB)

Hi @ojaved-equip, thanks for your post and welcome to the community!

Could you share some more information about the tables that sometimes don’t sync properly? How big are these tables, how Furthermore, could you create a separate connection with the tables that sometimes don’t sync and share the logs of that sync? Just trying to isolate the issue to the tables that are not syncing to understand the issue a bit better.

The tables are pretty small. For example, in the logs I attached above, the sync never happens for the “primary_diagnosis” table which only has 8 records which is an ID column, and 3 string columns. Any table that comes after primary_diagnosis alphabetically doesn’t get synced either and they are also small reference tables. The weird thing is the table doesn’t show up at all in the ReplicationAttemptSummary in the log, but it does show up in the StandardSyncSummary at the end of the log without any failures.

I’ll create a separate connection for these tables as suggested. The issue doesn’t happen all the time (although it seems to consistently happen after every couple of syncs). I’ll keep you posted. Thanks for your help!