Java Heap Exception During Incremental Syncs

Slack Knowledge Base Q&A API, Terraform and Other Topics
, , , ,
1

Summary

User is experiencing a Java heap exception during incremental syncs of 300MB - 1GB after a full refresh in a self-hosted Airbyte setup using abctl. The configuration includes memory settings and sync modes.


Question

Hello,
I would like to ask for a little help. We`re hosting self-hosted Airbyte using abctl. The size of replicated data is about 130GB. The full refresh runs ok, but when it comes to next syncs that transfer data about size 300MB - 1GB it fails on java heap exception.

My current configuration is:
• all tables have Incremental Append + Depuded sync mode
• sync runs every 20 minutes
• VM - 64GB RAM
• memory settings in config map:
◦ JOB_MAIN_CONTAINER_MEMORY_LIMIT: 20Gi
◦ JOB_MAIN_CONTAINER_MEMORY_REQUEST 8Gi
• settings in worker and server:
◦ - name: DEFAULT_JVM_OPTIONS
◦ - value: -XX:+ExitOnOutOfMemoryError -XX:MaxRAMPercentage=75.0
Do you have any idea why it still gives me java heap exception?

Thank you


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

['java-heap-exception', 'incremental-sync', 'abctl', 'memory-settings', 'self-hosted-airbyte']
2
  1. What connectors, source and destination, do you use for that connection?
  2. Are there any other synchronizations running at the same time? 64GB RAM has to be divided for use by different pods.
  3. Have you tried to set the same values for JOB_MAIN_CONTAINER_MEMORY_LIMIT and JOB_MAIN_CONTAINER_MEMORY_REQUEST, e.g. 16Gi? If you don’t set other requests/limits, I’m not sure how pods behave exactly in case of an increased need for resources.
3
  1. What connectors, source and destination, do you use for that connection?
  2. Are there any other synchronizations running at the same time? 64GB RAM has to be divided for use by different pods.
  3. Have you tried to set the same values for JOB_MAIN_CONTAINER_MEMORY_LIMIT and JOB_MAIN_CONTAINER_MEMORY_REQUEST, e.g. 16Gi? If you don’t set other requests/limits, I’m not sure how pods behave exactly in case of an increased need for resources.