Java Heap Exception on Airbyte Sync

Slack Knowledge Base Q&A Platform Questions
, , , ,
1

Summary

The user is experiencing Java Heap Exception when running syncs on a self-hosted Airbyte instance. The issue occurs during smaller syncs after a successful full refresh. The user provided details of their current configuration and memory settings.


Question

Hello,
I would like to ask for a little help. We`re hosting self-hosted Airbyte using abctl. The size of replicated data is about 130GB. The full refresh runs ok, but when it comes to next syncs that transfer data about size 300MB - 1GB it fails on java heap exception.

My current configuration is:
• all tables have Incremental Append + Depuded sync mode
• sync runs every 20 minutes
• VM - 64GB RAM
• memory settings in config map:
◦ JOB_MAIN_CONTAINER_MEMORY_LIMIT: 20Gi
◦ JOB_MAIN_CONTAINER_MEMORY_REQUEST 8Gi
• settings in worker and server:
◦ - name: DEFAULT_JVM_OPTIONS
◦ - value: -XX:+ExitOnOutOfMemoryError -XX:MaxRAMPercentage=75.0
Do you have any idea why it still gives me java heap exception?

Thank you


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

["java-heap-exception", "sync", "self-hosted", "abctl", "configuration", "memory-settings"]
2
  1. What connectors, source and destination, do you use for that connection?
  2. Are there any other synchronizations running at the same time? 64GB RAM has to be divided for use by different pods.
  3. Have you tried to set the same values for JOB_MAIN_CONTAINER_MEMORY_LIMIT and JOB_MAIN_CONTAINER_MEMORY_REQUEST, e.g. 16Gi? If you don’t set other requests/limits, I’m not sure how pods behave exactly in case of an increased need for resources.