Java Heap Issue Syncing CSV File with Postgres using SFTP and Postgresql Connectors


The user is facing a Java heap issue while trying to sync a CSV file with 1 million rows from an SFTP source connector to a Postgresql destination connector. They provided details of their Kubernetes cluster setup and are seeking advice on whether they can make it work with the current hardware configuration or if they need to tune something in Airbyte.


Hey team , I have a CSV file with 1 million rows and trying to sync with postgres

Source connector: SFTP
Destination connector: Postgresql

Issue: I’m facing java heap issue

K8 master : 4 cpu and 16GB ram
2 worker Nodes : 4 cpu and 32GB ram

Can anyone suggest is there a way i can make it work with above hardware or do i need to tune something at airbyte

This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.

Join the conversation on Slack

["java-heap-issue", "csv-file", "postgres", "sftp-connector", "postgresql-connector", "kubernetes", "hardware-tuning", "airbyte"]

What resources does your container have?