Summary
Some hourly jobs remain in sync mode for longer than expected. User wants to be alerted through email or slack.
Question
Has anybody noticed their hourly jobs not ending ? Over the last 5 days I’ve noticed 2 hourly jobs which remained in “sync mode” for way longer than expected (35 mins). Is there any feature that I can turn on that would make airbyte alert me about it somehow ? Through email, slack, etc.
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.
Join the conversation on Slack
["hourly-jobs", "sync-mode", "alert", "email", "slack"]
Hi Ahmed - Airbyte doesn’t have a notification for long running syncs today, but it has definitely come up as something that would be useful to have. It would be great if you could open a <https://github.com/airbytehq/airbyte/issues|github issue> for this, but it isn’t likely to happen until end of this year or early next year.
In the meantime, one option is to use the <https://reference.airbyte.com/reference/getjob|Airbyte API to monitor job status>. You could send a notification if the job is still running after a certain threshold.
Another option is to monitor data freshness in your destination. This is what my team currently does (using Metaplane), and we are alerted if tables are not updated in the usual timeframe. There are a lot of options for tools in this space, including dbt freshness tests.
thank you <@U03MTK52Z29> ! i’ve actually implemented the same now, freshness monitoring through dbt and elementary. i’ll open a ticket! thank you ! i’ll open another ticket about setting an time limit on jobs. i.e. if a job exceeds an hour, then terminate it.