Summary
Connections stuck in ‘syncing’ status after restoring backup, unable to cancel syncs
Question
Hello,
We had to restore a backup of our airbyte machine and it seems the backup was made during a sync. As such we have 2 connections where the status is stuck to syncing
while no attempt is listed in the UI.
I can’t cancel the syncs, the button does nothing…
Is there a way I can fix this ? My pipelines have been stuck for 36 hours now
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.
Join the conversation on Slack
["stuck-sync-status", "restoring-backup", "syncing", "cancel-sync", "pipelines-stuck"]
Hi Xavier, I’ve had the same issue happen to me a couple of weeks ago. When hitting “cancel sync” and monitoring the logs you’ll notice that the temporal DB is throwing out multiple errors on Null connection references.
My solution was to completely drop the temporal and temporal visibility databases, remove bad workflows using temporal https://docs.temporal.io/cli|cli and clear running jobs in the airbyte db itself (they were not really running!). After conducting these actions Airbyte recovered and started syncing again.
P.S (edit):
Fixing this is not straightforward and do so on your own risk. On my end, I took great care in understanding which data and databases I’m removing to ensure I’m not experiencing any data-loss or making irreparable harm to the deployment.