Java Heap Issue with SFTP to PostgreSQL Sync

Slack Knowledge Base Q&A Connector Questions
, , , ,
1

Summary

User is experiencing a Java heap issue while syncing a large CSV file (1 million rows) from SFTP to PostgreSQL. They are inquiring about potential hardware adjustments or tuning required in Airbyte to resolve the issue.


Question

Hey team , I have a CSV file with 1 million rows and trying to sync with postgres

Source connector: SFTP
Destination connector: Postgresql

Issue: I’m facing java heap issue

K8 master : 4 cpu and 16GB ram
2 worker Nodes : 4 cpu and 32GB ram

Can anyone suggest is there a way i can make it work with above hardware or do i need to tune something at airbyte


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

['java-heap-issue', 'sftp-connector', 'postgresql-connector', 'airbyte-tuning', 'hardware-requirements']