Scaling sync jobs horizontally on K8's for large CSV to Postgres sync job

[slack-user-airbyte]

Summary

How to scale source read and destination write pods horizontally on Kubernetes to handle a large CSV to Postgres sync job efficiently

Question

Hello Team,

Does any one know how to scale the sync jobs horizontally on K8’s.

Requirement: we are running 100MIllion CSV to postgres Sync job. We want to scale the source read, destination write pods horizontally to prevent more ram & cpu consumption on a single worker node. <#C021JANJ6TY|ask-community-for-troubleshooting>
We are ready to pay if any one of u solves the above issue.

Thanks

---

This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.

Join the conversation on Slack

_{["scaling", "sync-jobs", "horizontal-scaling", "kubernetes", "csv-to-postgres", "source-read", "destination-write", "ram-consumption", "cpu-consumption"]}