Summary
Custom builder connections with multiple parent streams causing Java out of memory error, specifically when calling Azure DevOps. Airbyte deployed to Kubernetes via Helm v1.2.0.
Question
Hey All,
I’m running a few different custom builder connections and it appears any that have multiple parent streams seem to consistently give me a java out of memory error. The example I’m working on currently is calling to Azure DevOps and only calling a couple thousand rows, which in my mind shouldn’t be causing this kind of error. Anyone else experienced this?
Just for context Airbyte is deployed to kubernetes via Helm and I’m running v1.2.0
2024-11-14 09:55:06,376 [pool-9-thread-2] INFO i.a.w.i.VersionedAirbyteStreamFactory(internalLog):314 - INFO pool-6-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):94 [ASYNC QUEUE INFO] Global: max: 1.5 GB, allocated: 20 MB (20.0 MB), %% used: 0.013020833333333334 | Queue ADO_Branches, num records: 2271, num bytes: 3.68 MB, allocated bytes: 10 MB | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0
2024-11-14 09:55:06,380 [pool-9-thread-2] INFO i.a.w.i.VersionedAirbyteStreamFactory(internalLog):314 - INFO pool-9-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):127 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
Terminating due to java.lang.OutOfMemoryError: Java heap space
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.