Airbyte RAM usage issue during initial sync with PostgreSQL source and destination

Slack Knowledge Base Q&A Platform Questions
, , , ,
1

Summary

Airbyte running on a self-hosted setup with PostgreSQL source and destination using xmin column system freezes up during initial sync due to high RAM usage. Database is 15GB with 800 tables.


Question

Hi
I have a self hosted setup to evaluate the product.
I have a Posgresql source and a PostgreSQL destination, use the xmin column system.

I have airbyte running on a 4vcpu with 16gb ram.
When the sync ramps up (initial sync) the ram maxex out and the whole VM freezes up.
What am I doing Wrong?

The database id around 15GB in size and has 800 odd tables


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

["airbyte", "ram-usage", "initial-sync", "postgresql-source", "postgresql-destination", "self-hosted-setup"]
2

try chaning these variable in .env

JOB_MAIN_CONTAINER_CPU_REQUEST=15.0
JOB_MAIN_CONTAINER_CPU_LIMIT=15.0
JOB_MAIN_CONTAINER_MEMORY_REQUEST=122000MB
JOB_MAIN_CONTAINER_MEMORY_LIMIT=122000MB

NORMALIZATION_JOB_MAIN_CONTAINER_MEMORY_LIMIT=10000MB
NORMALIZATION_JOB_MAIN_CONTAINER_MEMORY_REQUEST=10000MB
NORMALIZATION_JOB_MAIN_CONTAINER_CPU_LIMIT=10.0
NORMALIZATION_JOB_MAIN_CONTAINER_CPU_REQUEST=10.0

3

it will limit the RAM and CPU Consumption