Error regarding lack of resources when loading historical table with 400 million records

Slack Knowledge Base Q&A Connector Questions
, , , ,
1

Summary

The user is facing an error related to insufficient system resources when trying to load a historical table with 400 million records. They are specifically asking how to configure the batch size for the Oracle source in Airbyte.


Question

I’m trying to load a historical table with 400 million records and I receive the error below regarding lack of resources. How can I resolve this? How do I configure batch size for the Oracle source?Airbyte could not start the sync process. This may be due to insufficient system resources.
ResourceRequirements(claims=, limits={memory=30Gi, cpu=8}, requests={memory=30Gi, cpu=8}


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

["error", "lack-of-resources", "loading-historical-table", "400-million-records", "configure", "batch-size", "oracle-source", "system-resources"]
2

check this thread
https://airbytehq.slack.com/archives/C021JANJ6TY/p1729002808710989?thread_ts=1728999038.592549&cid=C021JANJ6TY