Optimized Postgres source connector performance

  • Is this your first time deploying Airbyte?: No
  • OS Version / Instance: Ubuntu
  • Memory / Disk: you can use something like 16Gb / 300Gb
  • Deployment: Docker
  • Airbyte Version: 0.39.35-alpha
  • Source name/version: Postgres 0.4.31
  • Destination name/version: Redshift 0.3.46
  • Step: during sync
  • Description: Are there any parameters that I can tweak to optimize the sync speed? I have a table with 380Gb took 24hrs to complete the sync. I found even I distribute more resource to the container JOB_MAIN_CONTAINER_MEMORY_REQUEST there is no improvement.

There are still some room of memories for postgres connector I wondering if the slowness is relate to the idel memory resource?
The docker stats shows that source-postgres-read-… container still have a lot space of memory

The Server hosting the airbyte server while the sync job is ruuning:

No matter how many memory I give to the setting JOB_MAIN_CONTAINER_MEMORY_REQUEST the fetch_size is always 1000.

Hi @khungCU, it looks like fetch size is adjusted dynamically, ( https://github.com/airbytehq/airbyte/pull/12400 ) but this does not improve performance. I’m linking your post to an open issue regarding source connector performance:

Hey, @khungCU! It looks like the GitHub issue has been resolved, could you update your Airbyte instance and check if the issue has been resolved?