Source Postgres - Sync completes without failure but not all streams get synced

Connector Questions & Issues
, ,
1
  • Is this your first time deploying Airbyte?: No
  • EC2 Linux
  • Memory / Disk: 16Gb / 1 Tb
  • Deployment: Docker
  • Airbyte Version: 0.39.37
  • Source name/version: Postgres (0.4.34)
  • Destination name/version: S3 (0.3.11)
  • Step: Sync
  • Description: Sync completes without any failures, but there are some tables that are set up to sync that do not get synced. The syncs are set up as full refresh and the existing data does get deleted but the new data doesn’t get synced. After looking at the logs, I see these tables are included in the logs for table found, queing query, and deleting existing data but I don’t see anything in the log for these tables for the “Starting new buffer stream” step.

I uploaded the logs for an instance where the primary_diagnosis table is one that doesn’t get synced.

This issue happens occasionally and if I resync, sometimes its successful.

logs-2367.txt (354.7 KB)

2

Hi @ojaved-equip, thanks for your post and welcome to the community!

Could you share some more information about the tables that sometimes don’t sync properly? How big are these tables, how Furthermore, could you create a separate connection with the tables that sometimes don’t sync and share the logs of that sync? Just trying to isolate the issue to the tables that are not syncing to understand the issue a bit better.

3

Hi,
The tables are pretty small. For example, in the logs I attached above, the sync never happens for the “primary_diagnosis” table which only has 8 records which is an ID column, and 3 string columns. Any table that comes after primary_diagnosis alphabetically doesn’t get synced either and they are also small reference tables. The weird thing is the table doesn’t show up at all in the ReplicationAttemptSummary in the log, but it does show up in the StandardSyncSummary at the end of the log without any failures.

I’ll create a separate connection for these tables as suggested. The issue doesn’t happen all the time (although it seems to consistently happen after every couple of syncs). I’ll keep you posted. Thanks for your help!

4

This issue seems to be appearing again.

I’ve been having a weird issue where where Airbyte says a sync is successful but it doesn’t replicate one of the 10 streams some of the times. It’ll work for a while when I first create the connection but then stop working and won’t work again until I create a new connection. It is usually the same table that doesn’t get synced (which is a small table of like 30 records right now). I can see from the logs it finds the table and deletes the existing data (right now we are doing a full overwrite), but never queues up the query for the table or do anything after that. Anyone encounter something similar or have any idea what could be going on?

Airbyte version: 0.40.23
Source: Postgres 1.0.28 (I also tried with latest version and same issue)
Destination: S3 0.3.17

Should note that if I create a seperate connection with just the table, it always syncs fine.

6

33a235c5_4163_4eb8_8f8b_780358850334_logs_7455_txt.txt (110.3 KB)

7

In this log you can see equip_source table is included in the sync:
2023-02-14 14:53:20 [32mINFO[m i.a.w.g.DefaultReplicationWorker(run):150 - configured sync modes: {inquiryreferralservice.referral=full_refresh - overwrite, inquiryreferralservice.relation_to_patient=full_refresh - overwrite, inquiryreferralservice.relation_subcategory=full_refresh - overwrite, inquiryreferralservice.ad _tracking=full_refresh - overwrite, inquiryreferralservice.relation_to_patient_type=full_refresh - overwrite, inquiryreferralservice.information_source=full_refresh - overwrite, inquiryreferralservice.equip_source=full_refresh - overwrite, inquiryreferralservice.inquiry=full_refresh - overwrite, inquiryreferralservice.referral_carer=full_refresh - overwrite, inquiryreferralservice.patient=full_refresh - overwrite, inquiryreferralservice.referral_affiliation=full_refresh - overwrite, inquiryreferralservice.affiliation=full_refresh - overwrite}

8

But its not included in the final sync summary:

2023-02-14 14:53:54 [32mINFO[m i.a.w.g.DefaultReplicationWorker(getReplicationOutput):408 - sync summary: {
  "status" : "completed",
  "recordsSynced" : 37269,
  "bytesSynced" : 18108306,
  "startTime" : 1676386400316,
  "endTime" : 1676386434705,
  "totalStats" : {
    "recordsEmitted" : 37269,
    "bytesEmitted" : 18108306,
    "sourceStateMessagesEmitted" : 0,
    "destinationStateMessagesEmitted" : 0,
    "recordsCommitted" : 37269,
    "meanSecondsBeforeSourceStateMessageEmitted" : 0,
    "maxSecondsBeforeSourceStateMessageEmitted" : 0,
    "maxSecondsBetweenStateMessageEmittedandCommitted" : 0,
    "meanSecondsBetweenStateMessageEmittedandCommitted" : 0,
    "replicationStartTime" : 1676386400316,
    "replicationEndTime" : 1676386434705,
    "sourceReadStartTime" : 1676386400510,
    "sourceReadEndTime" : 1676386434024,
    "destinationWriteStartTime" : 1676386400617,
    "destinationWriteEndTime" : 1676386434704
  },
  "streamStats" : [ {
    "streamName" : "inquiry",
    "streamNamespace" : "inquiryreferralservice",
    "stats" : {
      "recordsEmitted" : 13911,
      "bytesEmitted" : 10196713,
      "recordsCommitted" : 13911
    }
  }, {
    "streamName" : "referral_carer",
    "streamNamespace" : "inquiryreferralservice",
    "stats" : {
      "recordsEmitted" : 841,
      "bytesEmitted" : 219441,
      "recordsCommitted" : 841
    }
  }, {
    "streamName" : "affiliation",
    "streamNamespace" : "inquiryreferralservice",
    "stats" : {
      "recordsEmitted" : 383,
      "bytesEmitted" : 82249,
      "recordsCommitted" : 383
    }
  }, {
    "streamName" : "ad_tracking",
    "streamNamespace" : "inquiryreferralservice",
    "stats" : {
      "recordsEmitted" : 5660,
      "bytesEmitted" : 1399433,
      "recordsCommitted" : 5660
    }
  }, {
    "streamName" : "information_source",
    "streamNamespace" : "inquiryreferralservice",
    "stats" : {
      "recordsEmitted" : 7,
      "bytesEmitted" : 1071,
      "recordsCommitted" : 7
    }
  }, {
    "streamName" : "referral",
    "streamNamespace" : "inquiryreferralservice",
    "stats" : {
      "recordsEmitted" : 841,
      "bytesEmitted" : 564709,
      "recordsCommitted" : 841
    }
  }, {
    "streamName" : "patient",
    "streamNamespace" : "inquiryreferralservice",
    "stats" : {
      "recordsEmitted" : 14752,
      "bytesEmitted" : 5238141,
      "recordsCommitted" : 14752
    }
  }, {
    "streamName" : "relation_subcategory",
    "streamNamespace" : "inquiryreferralservice",
    "stats" : {
      "recordsEmitted" : 3,
      "bytesEmitted" : 369,
      "recordsCommitted" : 3
    }
  }, {
    "streamName" : "relation_to_patient",
    "streamNamespace" : "inquiryreferralservice",
    "stats" : {
      "recordsEmitted" : 44,
      "bytesEmitted" : 9981,
      "recordsCommitted" : 44
    }
  }, {
    "streamName" : "referral_affiliation",
    "streamNamespace" : "inquiryreferralservice",
    "stats" : {
      "recordsEmitted" : 827,
      "bytesEmitted" : 396199,
      "recordsCommitted" : 827
    }
  } ]
}