Slow data writing during large sync job from Oracle to Postgres

Slack Knowledge Base Q&A Platform Questions
, , , ,
1

Summary

Airbyte is slow when writing data from Oracle to Postgres during a large sync job, even with increased memory allocation. The sync job has been running for 16 hours and has not finished writing all data.


Question

i’m trying to sync a table contains +50 Million records, as per documentation I increased the memory for airbyte as much as i can, i gave it 5g and 2 cpu.
I ran a sync job yesterday and it still running for 16 hours and still didn’t finish writing all data.
why airbyte is slow when it comes to writing to the database.
I’m moving data from oracle to postgres


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

["slow-data-writing", "large-sync-job", "oracle", "postgres", "memory-allocation"]