All Destinations: _scd tables not empty after reset

  • Is this your first time deploying Airbyte?: No
  • OS Version / Instance: Ubuntu
  • Deployment: Kubernetes v1.22.9
  • Airbyte Version: v0.39.17-alpha
  • Source name/version: Postgres v0.4.18
  • Destination name/version: Snowflake v0.4.30
  • Step: Reset
  • Description: I have a connection with five tables, all incremental + deduped history. I needed to change the key for tracking changes so I reset the data and re-synced. Then I was alerted by my team about duplicates in the data of these tables. I noticed that after running a reset, the _SCD tables were not being emptied and continued to have the old data. I truncated the _SCD tables after a reset and then duplicates did not appear.

I have attached an image of an _SCD table still having rows after running a reset.

Hello Andrew sorry the long delay tor reply here!
I created the issue in Github to solve the problem.
When this is solved I’ll return to you here.

1 Like

Just to bump this, it’s also happening with the BigQuery destination. I’ve created an issue here.

Isaac, the first issue is I raised the issue to cnnector team.