Conflict Error in Airbyte Syncs

Summary

The user is facing a conflict error in Airbyte syncs while orchestrating a pipeline from Github to Snowflake using Airflow. Syncs are continuously running even when not scheduled, causing a ValueConflictKnownException. User intends to fetch data from multiple repositories, branches, and streams.


Question

Hi, I have created a pipeline from Github to Snowflake and orchestrated it using airflow. I’m reading from 1 repository, 2 branches and fetching data for 13 streams from each branch.
I’m getting following conflict error.

io.airbyte.commons.server.errors.ValueConflictKnownException: A sync is already running for: {connection_id}```
Also, syncs are continuously running even when they are not scheduled for the time. Syncs are getting succeeded but still getting conflict error.

My intend is to fetch data from 10 repos, 2 branch each and for 13 stream. Does any one have experience setting such pipeline? Any help is much appreciated.

<br>

---

This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. [Click here](https://airbytehq.slack.com/archives/C021JANJ6TY/p1718981130836709) if you want 
to access the original thread.

[Join the conversation on Slack](https://slack.airbyte.com)

<sub>
["conflict-error", "airbyte-syncs", "github", "snowflake", "airflow", "pipeline", "orchestration", "valueconflictknownexception"]
</sub>