Improved Jobs REST API

Airbyte supports integrations with external orchestration systems, such as dagster and Prefect. Both of these systems work in a similar way: they create a job for an airbyte connection, and then poll until it completes. They use the following route to poll: https://airbyte-public-api-docs.s3.us-east-2.amazonaws.com/rapidoc-api-docs.html#post-/v1/jobs/get

This, unfortunately, doesn’t scale well. Why? This api route returns the job logs, not just job status, and provides no query param or body field to disable log inclusion. For any long running syncs, or particularly chatty connections, this API route will repeatedly slow down each time it is polled, as there are more and more logs to retrieve. I had this fail on me yesterday for a job after less than 10 minutes - the logs were so verbose (standard settings!) that the API route started taking 30+ seconds to return, well over the 15 second timeout per API call configured in the dagster airbyte integration. I of course could open a PR to dagster to change the timeout on airbyte API calls, but that really doesn’t strike me as an acceptable solution.

The job logs included in the status route seems like a convenience for the webapp, as its mechanism of getting the logs for each job.

To improve this, can we either make logs optional in the current route, or add a new route (a GET instead of a POST?) that returns only the status fields, without logs?

Thanks!

Hey @adam, thanks for the post and for the suggestion. I’ve gone ahead and created an issue on Github about this. Let me know if I misunderstood or forgot to include anything from your post. I’d recommend following the thread and adding any thoughts or points of discussion on the issue so that it’s easier for the engineering team to discuss.

Hi there from the Community Assistance team.
We’re letting you know about an issue we discovered with the back-end process we use to handle topics and responses on the forum. If you experienced a situation where you posted the last message in a topic that did not receive any further replies, please open a new topic to continue the discussion. In addition, if you’re having a problem and find a closed topic on the subject, go ahead and open a new topic on it and we’ll follow up with you. We apologize for the inconvenience, and appreciate your willingness to work with us to provide a supportive community.