Intermittent Stalling Issue with MySQL to Snowflake Data Pipeline in Airbyte


The user is facing intermittent stalling issues with the data pipeline loading Change Data Capture (CDC) from MySQL 5 into Snowflake using Airbyte. The loading process sometimes exceeds 1 hour, despite typically completing within 2-3 minutes. The pipeline triggers the connection every hour via API and checks the connection state before triggering, encountering errors when the connection is in a ‘running’ state.


Hi Airbyte Team,

Hope you are well.

I have established a data pipeline for loading Change Data Capture (CDC) from MySQL 5 into Snowflake.
Typically, this pipeline operates smoothly, completing the load within 2-3 minutes. However, intermittently, the loading process becomes stalled, resulting in a duration exceeding 1 hour. Currently, the pipeline is set to trigger the connection every hour via API. Before triggering the connection, it checks the connection state, in the event that the connection is in a ‘running’ state, it triggers an error, which is the current situation.

I’m using Airbyte Version 0.50.50 deployed in a EC2 Instance.

Attached is a screenshot of the job history showing the average loading time.
I’m also sharing the logs of the latest two jobs that took over 1 hour to complete.


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.

Join the conversation on Slack

["mysql-connector", "snowflake-connector", "data-pipeline", "stalling-issue", "airbyte-api", "job-history", "logs", "ec2-instance"]