Optimized Postgres source connector performance

Connector Questions & Issues
, ,
1
  • Is this your first time deploying Airbyte?: No
  • OS Version / Instance: Ubuntu
  • Memory / Disk: you can use something like 16Gb / 300Gb
  • Deployment: Docker
  • Airbyte Version: 0.39.35-alpha
  • Source name/version: Postgres 0.4.31
  • Destination name/version: Redshift 0.3.46
  • Step: during sync
  • Description: Are there any parameters that I can tweak to optimize the sync speed? I have a table with 380Gb took 24hrs to complete the sync. I found even I distribute more resource to the container JOB_MAIN_CONTAINER_MEMORY_REQUEST there is no improvement.

There are still some room of memories for postgres connector I wondering if the slowness is relate to the idel memory resource?
The docker stats shows that source-postgres-read-… container still have a lot space of memory

Screen Shot 2022-07-22 at 8.18.34 AM
Screen Shot 2022-07-22 at 8.18.34 AM1409×226 19.7 KB

The Server hosting the airbyte server while the sync job is ruuning:

Screen Shot 2022-07-22 at 9.48.49 AM
Screen Shot 2022-07-22 at 9.48.49 AM2422×926 223 KB

No matter how many memory I give to the setting JOB_MAIN_CONTAINER_MEMORY_REQUEST the fetch_size is always 1000.

Screen Shot 2022-07-22 at 9.54.02 AM
Screen Shot 2022-07-22 at 9.54.02 AM737×144 53.6 KB

2

Hi @khungCU, it looks like fetch size is adjusted dynamically, ( https://github.com/airbytehq/airbyte/pull/12400 ) but this does not improve performance. I’m linking your post to an open issue regarding source connector performance:
https://github.com/airbytehq/airbyte/issues/12532

3

Hey, @khungCU! It looks like the GitHub issue has been resolved, could you update your Airbyte instance and check if the issue has been resolved?