State hash XXX was already committed, likely indicating a state hash collision using Incremental Dedup

Connector Questions & Issues
1
  • Is this your first time deploying Airbyte?: No
  • OS Version / Instance: Ubuntu
  • Memory / Disk: 8 GB
  • Deployment: Docker
  • Airbyte Version: 0.39.23
  • Source name/version: MySQL
  • Destination name/version: Bigquery
  • Step: The issue is happening during sync
  • Description:

Since the upgrade to 0.39.23 (previously 0.35.55), there have been so many failed jobs with error “State hash XXX was already committed, likely indicating a state hash collision”.
The job already used incremental dedup + history. Kindly help.

2

@ben_nicole, I found an issue similar to yours here:
https://github.com/airbytehq/airbyte/issues/12844

Could you please recheck your tables and make sure everything is in order?

If that doesn’t help, please provide the full error logs you are getting.

3

Hi @natalyjazzviolin , thanks for your help. However, my problem is different from the one you posted. My job is sometimes failed, sometimes successful. and I use Incremental Dedup as well.

This happens very frequently. Kindly help. Thanks

4

Understood. Could you provide any and all error logs that you are getting - the ones from your servers, the ones from Airbyte?

5

@natalyjazzviolin here is the log. thanks!
logs-25780.txt (556.3 KB)

6

@natalyjazzviolin this error happen many times. kindly help :pray:

image
image1769×462 41 KB

7

Thank you for the logs! I don’t see any explicit errors in them, please let me know if I missed something. Is this causing duplication in the destination?

8

Nope. I used Incremental dedup. So no duplication in the destination.

9

What connector versions are you using? Could you also please update your Airbyte instance?