Summary
The user is facing Java heap space out of memory errors despite making database config changes for the source actor_definition. They are looking for guidance on increasing Java heapsize memory and determining the source’s total memory requirements.
Question
- Question is how to increase java jeapsize memory when the actor_defintion.resource_requirements json string doent seem to resolve source connector errors as shown below.
- How would one know if the settings have expanded the source’s total memory and or what amount of memory the source requires?
- Anyone run into this scenario with microsoftsql server sources? what did you change to resolve?
• abctl local version v0.18.0
source connector failure java heap out of memory keeps happening despite making database config changes to the source actor_definition via :
docker exec -it airbyte-abctl-control-plane /bin/bash
airbyte-db-0:/$ psql airbyte -d db-airbyte -t -A -c “UPDATE actor_definition SET resource_requirements=‘{"jobSpecific": [{"jobType": "sync", "resourceRequirements": {"cpu_limit": "2", "cpu_request": "2", "memory_limit": "3097Mi", "memory_request": "3097Mi"}}]}’ WHERE id = ‘b5ea17b1-f170-46dc-bc31-cc744ca984c1’;”
024-10-04 15:42:47 destination > 2024-10-04 15:42:47 INFO i.a.c.d.j.DefaultJdbcDatabase(lambda$unsafeQuery$1):115 - closing connection
835
2024-10-04 15:42:47 destination > 2024-10-04 15:42:47 INFO i.a.c.i.d.s.GeneralStagingFunctions(lambda$onStartFunction$0):70 - Preparing staging area in destination completed for schema airbyte_internal stream ResidualCalculationDataOnVolumeTransactions
836
2024-10-04 15:42:47 destination > 2024-10-04 15:42:47 INFO i.a.c.i.d.s.GeneralStagingFunctions(lambda$onStartFunction$0):53 - Preparing staging area in destination started for schema airbyte_internal stream ResidualCalculationDataInterchangeCreditOnReturns: target table: pps_source_cdc_raw__stream_ResidualCalculationDataInterchangeCreditOnReturns, stage: 2024/10/04/15/81D6C432-7334-4EAC-97DA-A2F44AEBC9A2/
837
2024-10-04 15:42:49 destination > 2024-10-04 15:42:49 INFO i.a.c.d.j.DefaultJdbcDatabase(lambda$unsafeQuery$1):115 - closing connection
838
2024-10-04 15:42:50 destination > 2024-10-04 15:42:50 INFO i.a.c.i.d.s.GeneralStagingFunctions(lambda$onStartFunction$0):70 - Preparing staging area in destination completed for schema airbyte_internal stream ResidualCalculationDataInterchangeCreditOnReturns
839
2024-10-04 15:42:50 destination > 2024-10-04 15:42:50 INFO i.a.c.i.d.s.GeneralStagingFunctions(lambda$onStartFunction$0):72 - Executing finalization of tables.
840
2024-10-04 15:43:24 destination > 2024-10-04 15:43:24 INFO i.a.c.i.d.b.BufferManager(printQueueInfo):118 - [ASYNC QUEUE INFO] Global: max: 1.01 GB, allocated: 10 MB (10.0 MB), % used: 0.009640877319836105 | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.000000
841
2024-10-04 15:43:24 destination > 2024-10-04 15:43:24 INFO i.a.c.i.d.FlushWorkers(printWorkerInfo):146 - [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
842
2024-10-04 15:44:24 destination > 2024-10-04 15:44:24 INFO i.a.c.i.d.b.BufferManager(printQueueInfo):118 - [ASYNC QUEUE INFO] Global: max: 1.01 GB, allocated: 10 MB (10.0 MB), % used: 0.009640877319836105 | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.000000
843
2024-10-04 15:44:24 destination > 2024-10-04 15:44:24 INFO i.a.c.i.d.FlushWorkers(printWorkerInfo):146 - [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
844
2024-10-04 15:45:24 destination > 2024-10-04 15:45:24 INFO i.a.c.i.d.b.BufferManager(printQueueInfo):118 - [ASYNC QUEUE INFO] Global: max: 1.01 GB, allocated: 10 MB (10.0 MB), % used: 0.009640877319836105 | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.000000
845
2024-10-04 15:45:24 destination > 2024-10-04 15:45:24 INFO i.a.c.i.d.FlushWorkers(printWorkerInfo):146 - [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
846
2024-10-04 15:46:04 source > Terminating due to java.lang.OutOfMemoryError: Java heap space
847
2024-10-04 14:00:00 replication-orchestrator > failures: [ {
992
“failureOrigin” : “source”,
993
“internalMessage” : “Source process exited with non-zero exit code 3”,
994
“externalMessage” : “Something went wrong within the source connector”,
995
“metadata” : {
996
“attemptNumber” : 0,
997
“jobId” : 13135,
998
“connector_command” : “read”
999
},
1000
“stacktrace” : “io.airbyte.workers.internal.exception.SourceException: Source process exited with non-zero exit code 3\n\tat io.airbyte.workers.general.BufferedReplicationWorker.readFromSource(BufferedReplicationWorker.java:364)\n\tat io.airbyte.workers.general.BufferedReplicationWorker.lambda$runAsyncWithHeartbeatCheck$3(BufferedReplicationWorker.java:222)\n\tat java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1583)\n”,
1001
“timestamp” : 1728050390635
1002
} ]
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.