Sync hangs until I cancel

  • Is this your first time deploying Airbyte?: No
  • OS Version / Instance:
  • Memory / Disk: of what? server, worker, temporal?
  • Deployment: Kubernetes deployment
  • Airbyte Version: 0.35.29-alpha
  • Source name/version: Postgres SQL 11
  • Destination name/version: Big Query
  • Step: During sync
  • Description:

Hi, Ive been running an ETL from my GCP SQL database to Big Query for a few months (incremental, based on updatedAt).
Im seeing an issue where sometimes the job just hangs in “Running” state, very much indefinitely, for no apparent reason (hours), and I need to go and click “cancel” so that a new job takes over. It’s completely intermittent and a cancel usually “unblocks” it. Im wondering if there is a way to set a timeout so that it cancels itself after a certain amount of time or something of that effect.

There is an env variable you can configure to throw timeout.

But SYNC_JOB_MAX_TIMEOUT_DAYS=3 the minimal possible is 1 day.

Yeah 1 day is way too much. If there was a SYNC_JOB_MAX_TIMEOUT_HOURS that would be ok.