Hey @stereowind,
JDBC connectors are fetching records in batches. The size of these batches depends on the fetchSize parameters we use in query made to the source database.
We recently shipped a feature that performs a dynamic computation of the fetchSize , and your connector version has this feature. Could you please share logs of your syncs so that we can check the value of this fetchSize?
- Is this sync speed within expected ranges? Or looks slow?
It does not look too slow from my perspective but there is definitely room for improvement on the Airbyte side to make this replication faster. We are currently focusing a lot of effort on this topic.
- If it looks slow, are there any ways to improve the performance?
It’d be great to identify the bottleneck in your context. I’m guessing that it might be the fetchSize, but it could also be the sizing of your database replica or the available resource on your host. Are you able to evaluate the load on Aurora when the sync runs, and check the Airbyte containers’ resource consumption when the sync happens?
I could improve the performance by splitting MySQL connectors into multiple ones and run them in parallel,
This is indeed a workaround to make the parallel replication happen, but we know that it’s not an ideal solution.