Callbacks when jobs are finished


My team is using Airbyte to manage thousand of jobs every day. After those jobs are done, other processes are launched. To know if the jobs are finished we used the Airbyte API like (the Airflow sync operator is doing). However, we found it counter productive to make many call to monitor thousand of jobs every day. Is the Airbyte team planned to implement some sort of callback ?
I know there is a webhook that can be used (for slack for example). However, we are not sure if it is suitable to use it to trigger following processes at scale.

I second this.

External observability of jobs is something that Airbyte is still lacking IMO. Webhooks for whem jobs start, fail, success, etc would be very important for maintaining production environments with Airbyte.

1 Like