Summary
The user is encountering a Java heap space OutOfMemoryError while running the SFTP source connector in Airbyte OSS deployed on Kubernetes. They are seeking guidelines on how to address this issue.
Question
getting OutOfMemoryError from source (SFTP source)
2024-02-20 13:49:27 source > java.lang.OutOfMemoryError: Java heap space
I am running on OSS deployed on kubernetes - any guidelines how to approach this issue?
2024-02-20 13:49:27 source > at org.apache.commons.io.IOUtils.byteArray(IOUtils.java:338) ~[commons-io-2.11.0.jar:2.11.0]
2024-02-20 13:49:27 source > at org.apache.commons.io.output.AbstractByteArrayOutputStream.toByteArrayImpl(AbstractByteArrayOutputStream.java:365) ~[commons-io-2.11.0.jar:2.11.0]
2024-02-20 13:49:27 source > at org.apache.commons.io.output.UnsynchronizedByteArrayOutputStream.toByteArray(UnsynchronizedByteArrayOutputStream.java:147) ~[commons-io-2.11.0.jar:2.11.0]
2024-02-20 13:49:27 source > at org.apache.commons.io.IOUtils.toByteArray(IOUtils.java:2406) ~[commons-io-2.11.0.jar:2.11.0]
2024-02-20 13:49:27 source > at io.airbyte.integrations.source.sftp.SftpClient.getFile(SftpClient.java:108) ~[io.airbyte.airbyte-integrations.connectors-source-sftp-0.50.45.jar:?]
2024-02-20 13:49:27 source > at io.airbyte.integrations.source.sftp.SftpCommand.getFileData(SftpCommand.java:115) ~[io.airbyte.airbyte-integrations.connectors-source-sftp-0.50.45.jar:?]
2024-02-20 13:49:27 source > at io.airbyte.integrations.source.sftp.SftpSource.lambda$getFileDataIterator$4(SftpSource.java:147) ~[io.airbyte.airbyte-integrations.connectors-source-sftp-0.50.45.jar:?]
2024-02-20 13:49:27 source > at io.airbyte.integrations.source.sftp.SftpSource$$Lambda$179/0x00007f92f128ecb8.get(Unknown Source) ~[?:?]
2024-02-20 13:49:27 source > at io.airbyte.commons.util.LazyAutoCloseableIterator.computeNext(LazyAutoCloseableIterator.java:41) ~[airbyte-cdk-airbyte-commons-0.13.2.jar:?]
2024-02-20 13:49:27 source > at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:146) ~[guava-31.1-jre.jar:?]
2024-02-20 13:49:27 source > at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:141) ~[guava-31.1-jre.jar:?]
2024-02-20 13:49:27 source > at com.google.common.collect.TransformedIterator.hasNext(TransformedIterator.java:46) ~[guava-31.1-jre.jar:?]
2024-02-20 13:49:27 source > at io.airbyte.commons.util.DefaultAutoCloseableIterator.computeNext(DefaultAutoCloseableIterator.java:42) ~[airbyte-cdk-airbyte-commons-0.13.2.jar:?]
2024-02-20 13:49:27 source > at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:146) ~[guava-31.1-jre.jar:?]
2024-02-20 13:49:27 source > at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:141) ~[guava-31.1-jre.jar:?]
2024-02-20 13:49:27 source > at io.airbyte.commons.util.CompositeIterator.computeNext(CompositeIterator.java:74) ~[airbyte-cdk-airbyte-commons-0.13.2.jar:?]
2024-02-20 13:49:27 source > at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:146) ~[guava-31.1-jre.jar:?]
2024-02-20 13:49:27 source > at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:141) ~[guava-31.1-jre.jar:?]
2024-02-20 13:49:27 source > at io.airbyte.commons.util.DefaultAutoCloseableIterator.computeNext(DefaultAutoCloseableIterator.java:42) ~[airbyte-cdk-airbyte-commons-0.13.2.jar:?]
2024-02-20 13:49:27 source > at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:146) ~[guava-31.1-jre.jar:?]
2024-02-20 13:49:27 source > at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:141) ~[guava-31.1-jre.jar:?]
2024-02-20 13:49:27 source > at java.base/java.util.Iterator.forEachRemaining(Iterator.java:132) ~[?:?]
2024-02-20 13:49:27 source > at io.airbyte.cdk.integrations.base.IntegrationRunner.produceMessages(IntegrationRunner.java:236) ~[airbyte-cdk-core-0.13.2.jar:?]
2024-02-20 13:49:27 source > at io.airbyte.cdk.integrations.base.IntegrationRunner.readSerial(IntegrationRunner.java:276) ~[airbyte-cdk-core-0.13.2.jar:?]
2024-02-20 13:49:27 source > at io.airbyte.cdk.integrations.base.IntegrationRunner.runInternal(IntegrationRunner.java:173) ~[airbyte-cdk-core-0.13.2.jar:?]
2024-02-20 13:49:27 source > at io.airbyte.cdk.integrations.base.IntegrationRunner.run(IntegrationRunner.java:125) ~[airbyte-cdk-core-0.13.2.jar:?]
2024-02-20 13:49:27 source > at io.airbyte.integrations.source.sftp.SftpSource.main(SftpSource.java:37) ~[io.airbyte.airbyte-integrations.connectors-source-sftp-0.50.45.jar:?]```
<br>
---
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. [Click here](https://airbytehq.slack.com/archives/C021JANJ6TY/p1708437463132769) if you want to access the original thread.
[Join the conversation on Slack](https://slack.airbyte.com)
<sub>
["outofmemoryerror", "sftp-source", "airbyte-oss", "kubernetes", "java-heap-space"]
</sub>