Summary
User reports a Java OutOfMemoryError when running custom builder connections with multiple parent streams in Airbyte, specifically when connecting to Azure DevOps. The issue occurs despite processing only a few thousand rows, indicating potential memory management issues in the connector’s implementation.
Question
Hey All,
I’m running a few different custom builder connections and it appears any that have multiple parent streams seem to consistently give me a java out of memory error. The example I’m working on currently is calling to Azure DevOps and only calling a couple thousand rows, which in my mind shouldn’t be causing this kind of error. Anyone else experienced this?
Just for context Airbyte is deployed to kubernetes via Helm and I’m running v1.2.0
2024-11-14 09:55:06,376 [pool-9-thread-2] INFO i.a.w.i.VersionedAirbyteStreamFactory(internalLog):314 - INFO pool-6-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):94 [ASYNC QUEUE INFO] Global: max: 1.5 GB, allocated: 20 MB (20.0 MB), %% used: 0.013020833333333334 | Queue ADO_Branches, num records: 2271, num bytes: 3.68 MB, allocated bytes: 10 MB | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0
2024-11-14 09:55:06,380 [pool-9-thread-2] INFO i.a.w.i.VersionedAirbyteStreamFactory(internalLog):314 - INFO pool-9-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):127 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
Terminating due to java.lang.OutOfMemoryError: Java heap space
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.