Sync hangs until I cancel

Connector Questions & Issues
1
  • Is this your first time deploying Airbyte?: No
  • OS Version / Instance:
  • Memory / Disk: of what? server, worker, temporal?
  • Deployment: Kubernetes deployment
  • Airbyte Version: 0.35.29-alpha
  • Source name/version: Postgres SQL 11
  • Destination name/version: Big Query
  • Step: During sync
  • Description:

Hi, Ive been running an ETL from my GCP SQL database to Big Query for a few months (incremental, based on updatedAt).
Im seeing an issue where sometimes the job just hangs in “Running” state, very much indefinitely, for no apparent reason (hours), and I need to go and click “cancel” so that a new job takes over. It’s completely intermittent and a cancel usually “unblocks” it. Im wondering if there is a way to set a timeout so that it cancels itself after a certain amount of time or something of that effect.

2

There is an env variable you can configure to throw timeout.

But SYNC_JOB_MAX_TIMEOUT_DAYS=3 the minimal possible is 1 day.

3

Yeah 1 day is way too much. If there was a SYNC_JOB_MAX_TIMEOUT_HOURS that would be ok.

4

Hi there from the Community Assistance team.
We’re letting you know about an issue we discovered with the back-end process we use to handle topics and responses on the forum. If you experienced a situation where you posted the last message in a topic that did not receive any further replies, please open a new topic to continue the discussion. In addition, if you’re having a problem and find a closed topic on the subject, go ahead and open a new topic on it and we’ll follow up with you. We apologize for the inconvenience, and appreciate your willingness to work with us to provide a supportive community.