Airbyte supports integrations with external orchestration systems, such as dagster and Prefect. Both of these systems work in a similar way: they create a job for an airbyte connection, and then poll until it completes. They use the following route to poll: https://airbyte-public-api-docs.s3.us-east-2.amazonaws.com/rapidoc-api-docs.html#post-/v1/jobs/get
This, unfortunately, doesn’t scale well. Why? This api route returns the job logs, not just job status, and provides no query param or body field to disable log inclusion. For any long running syncs, or particularly chatty connections, this API route will repeatedly slow down each time it is polled, as there are more and more logs to retrieve. I had this fail on me yesterday for a job after less than 10 minutes - the logs were so verbose (standard settings!) that the API route started taking 30+ seconds to return, well over the 15 second timeout per API call configured in the dagster airbyte integration. I of course could open a PR to dagster to change the timeout on airbyte API calls, but that really doesn’t strike me as an acceptable solution.
The job logs included in the status route seems like a convenience for the webapp, as its mechanism of getting the logs for each job.
To improve this, can we either make logs optional in the current route, or add a new route (a GET instead of a POST?) that returns only the status fields, without logs?