Incremental Sync Successful but Rows in Table Duped

  • s this your first time deploying Airbyte: No
  • OS Version / Instance: EC2 t2.large
  • Memory / Disk: 16Gb
  • Deployment: Docker
  • Airbyte Version: 0.35.56-alpha
  • Source name/version: Postgres 0.4.10
  • Destination name/version: Snowflake 0.4.24
  • Step: Sync fails
  • Description: This job replicates about 70 tables that use incremental deduped history. However, for some reason two tables did not dedupe and the job replicated identical rows. This is the first time this has happened and not sure why. A few jobs have been failing due to invalid identifiers as well before this happened despite nothing changing in the source.

logs-6464.txt (1.6 MB)

Hey @kyle-cheung do you see any pattern in those rows which have not deduped?

No pattern, i did a hard reset and dropped the tables and they resynced with no issue. But still weird that it happened!