Summary
Exploring the impact of implementing STATEMENT_TIMEOUT_IN_SECONDS in Snowflake on Airbyte jobs when a query is canceled
Question
several of my connections got stuck last night due to a transaction being blocked in Snowflake - they ended up running / queueing for over 9 hours. If I implement STATEMENT_TIMEOUT_IN_SECONDS and snowflake cancels an airbyte query, what happens to the job in airbyte?
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.
Join the conversation on Slack
["snowflake", "transaction-blocking", "statement-timeout-in-seconds", "airbyte-job"]
My impression is that Airbyte doesn’t see the cancelation and the job on the AIrbyte side has to timeout
hey <@U064R6L3ZBJ> thanks for the response!
is the suggested approach to set SYNC_JOB_MAX_TIMEOUT_DAYS to something like .1 to kill jobs that may be hanging?
how have you solved this, if you’ve encountered it?
I’m not sure if that is a valid value for that setting. We’ve encountered this issue before, but haven’t resolved it. Our main issue with Airbyte and Snowflake right now is we’re getting timeouts connecting to Snowflake over PrivateLink consistently and Airbyte is constantly retrying those connections