- Is this your first time deploying Airbyte?: No
- OS Version / Instance: Ubuntu
- Memory / Disk: you can use something like 16Gb / 300Gb
- Deployment: Docker
- Airbyte Version: 0.39.35-alpha
- Source name/version: Postgres 0.4.31
- Destination name/version: Redshift 0.3.46
- Step: during sync
- Description: Are there any parameters that I can tweak to optimize the sync speed? I have a table with 380Gb took 24hrs to complete the sync. I found even I distribute more resource to the container
JOB_MAIN_CONTAINER_MEMORY_REQUESTthere is no improvement.
There are still some room of memories for postgres connector I wondering if the slowness is relate to the idel memory resource?
The docker stats shows that source-postgres-read-… container still have a lot space of memory
The Server hosting the airbyte server while the sync job is ruuning:
No matter how many memory I give to the setting JOB_MAIN_CONTAINER_MEMORY_REQUEST the fetch_size is always 1000.