Summary
The user is facing intermittent stalling issues with the data pipeline loading Change Data Capture (CDC) from MySQL 5 into Snowflake using Airbyte. The loading process sometimes exceeds 1 hour, despite typically completing within 2-3 minutes. The pipeline triggers the connection every hour via API and checks the connection state before triggering, encountering errors when the connection is in a ‘running’ state.
Question
Hi Airbyte Team,
Hope you are well.
I have established a data pipeline for loading Change Data Capture (CDC) from MySQL 5 into Snowflake.
Typically, this pipeline operates smoothly, completing the load within 2-3 minutes. However, intermittently, the loading process becomes stalled, resulting in a duration exceeding 1 hour. Currently, the pipeline is set to trigger the connection every hour via API. Before triggering the connection, it checks the connection state, in the event that the connection is in a ‘running’ state, it triggers an error, which is the current situation.
I’m using Airbyte Version 0.50.50 deployed in a EC2 Instance.
Attached is a screenshot of the job history showing the average loading time.
I’m also sharing the logs of the latest two jobs that took over 1 hour to complete.
Thanks!
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.