Sync hangs until I cancel

alexandresobolevski · April 19, 2022, 9:31pm

Is this your first time deploying Airbyte?: No
OS Version / Instance:
Memory / Disk: of what? server, worker, temporal?
Deployment: Kubernetes deployment
Airbyte Version: 0.35.29-alpha
Source name/version: Postgres SQL 11
Destination name/version: Big Query
Step: During sync
Description:

Hi, Ive been running an ETL from my GCP SQL database to Big Query for a few months (incremental, based on updatedAt).
Im seeing an issue where sometimes the job just hangs in “Running” state, very much indefinitely, for no apparent reason (hours), and I need to go and click “cancel” so that a new job takes over. It’s completely intermittent and a cancel usually “unblocks” it. Im wondering if there is a way to set a timeout so that it cancels itself after a certain amount of time or something of that effect.

marcosmarxm · April 19, 2022, 9:42pm

There is an env variable you can configure to throw timeout.

github.com

airbytehq/airbyte/blob/f065d361b1c1b2da91d64b6e4c07cf4fc313dc53/.env#L64

      
        
            TEMPORAL_HOST=airbyte-temporal:7233
            INTERNAL_API_HOST=airbyte-server:8001
            WEBAPP_URL=http://localhost:8000/
            # Although not present as an env var, required for webapp configuration.
            API_URL=/api/v1/
            
            

            
### JOBS ###
            # Relevant to scaling.
            SYNC_JOB_MAX_ATTEMPTS=3
            SYNC_JOB_MAX_TIMEOUT_DAYS=3
            JOB_MAIN_CONTAINER_CPU_REQUEST=
            JOB_MAIN_CONTAINER_CPU_LIMIT=
            JOB_MAIN_CONTAINER_MEMORY_REQUEST=
            JOB_MAIN_CONTAINER_MEMORY_LIMIT=
            
            

            
### LOGGING/MONITORING/TRACKING ###
            TRACKING_STRATEGY=segment
            # Although not present as an env var, expected by Log4J configuration.
            LOG_LEVEL=INFO

But SYNC_JOB_MAX_TIMEOUT_DAYS=3 the minimal possible is 1 day.

alexandresobolevski · April 19, 2022, 11:21pm

Yeah 1 day is way too much. If there was a SYNC_JOB_MAX_TIMEOUT_HOURS that would be ok.