Summary
User is asking if there is a parameter in Airbyte to parallelize a connection while using Docker deployment.
Question
Hello everyone, I have a connection that takes another 10 hours. Is there any parameter in airbyte to parallelize this connection? I used docker deployment
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.
Join the conversation on Slack
["parallelizing-connection", "airbyte", "docker-deployment", "parameter"]
Streams in OSS run serially. I believe that AirByte Enterprise supports (or will support) parallelization of stream syncs in a single connection.
One work-around in OSS would be to partition your streams across several connection.
That said, if a single stream is taking 10 hours; I’m not sure much can be done, except perhaps to adjust the time-period of the sync to pull less data in one job.