Airbyte Sync Issues Post Upgrade

Slack Knowledge Base Q&A Connector Questions
, , , ,
1

Summary

User reports multiple sync attempts required for successful completion after upgrading to Airbyte versions 0.63.3, 0.63.4, and 0.63.5. Errors indicate issues with destination processes, specifically with Snowflake connector version 3.11.0.


Question

Since upgrade to 0.63.3 I’m experiencing that Airbyte needs multiple attempts before successful sync. It’s happening on multiple sources (mysql, postgres, stripe api, shopify) without any clear pattern. Usually somewhere between 3rd to 6th attempt it successfully finishes, but this increases the sync times significantly (i.e. for a sync that usually took ~10 minutes it now needs 6x10min because of retries).

The situation did not improve with upgrade to 0.63.4 / 0.63.5

I’m seeing errors like this:

java.lang.IllegalStateException: Destination process is still alive, cannot retrieve exit value.
        at com.google.common.base.Preconditions.checkState(Preconditions.java:515) ~[guava-33.2.0-jre.jar:?]
        at io.airbyte.workers.internal.DefaultAirbyteDestination.getExitValue(DefaultAirbyteDestination.java:210) ~[io.airbyte-airbyte-commons-worker-0.63.5.jar:?]
        at io.airbyte.workers.general.BufferedReplicationWorker.readFromDestination(BufferedReplicationWorker.java:492) ~[io.airbyte-airbyte-commons-worker-0.63.5.jar:?]
        at io.airbyte.workers.general.BufferedReplicationWorker.lambda$runAsync$2(BufferedReplicationWorker.java:235) ~[io.airbyte-airbyte-commons-worker-0.63.5.jar:?]
        at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804) ~[?:?]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
        at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
2024-07-10 05:16:01 platform > readFromDestination: done. (writeToDestFailed:true, dest.isFinished:false)
2024-07-10 05:17:01 platform > airbyte-destination gobbler IOException: Stream closed. Typically happens when cancelling a job.
2024-07-10 05:18:01 platform > airbyte-source gobbler IOException: Stream closed. Typically happens when cancelling a job.
2024-07-10 05:18:01 platform > Closing StateCheckSumCountEventHandler```
Destination is Snowflake, I’m using latest snowflake connector 3.11.0

Anyone else experienced something similar? I’m not even sure where to start looking for a problem so any hint is more than welcome.

<br>

---

This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. [Click here](https://airbytehq.slack.com/archives/C021JANJ6TY/p1720599281960249) if you want
to access the original thread.

[Join the conversation on Slack](https://slack.airbyte.com)

<sub>
['airbyte', 'sync-issues', 'snowflake-connector', 'mysql', 'postgres', 'stripe-api', 'shopify', 'java-exception']
</sub>