Summary
User seeks guidance on how to define Java heap space in the SFTP source connector due to an OutOfMemoryError indicating insufficient heap memory.
Question
hi community,
how to define java heap space in sftp source connector.
i am getting below error
Internal message: java.lang.OutOfMemoryError: Java heap space
Failure type: system_error
java.lang.OutOfMemoryError: Java heap space
at http://org.apache.commons.io|org.apache.commons.io.IOUtils.byteArray(IOUtils.java:368)
at http://org.apache.commons.io|org.apache.commons.io.output.AbstractByteArrayOutputStream.needNewBuffer(AbstractByteArrayOutputStream.java:140)
at http://org.apache.commons.io|org.apache.commons.io.output.AbstractByteArrayOutputStream.writeImpl(AbstractByteArrayOutputStream.java:331)
at http://org.apache.commons.io|org.apache.commons.io.output.UnsynchronizedByteArrayOutputStream.write(UnsynchronizedByteArrayOutputStream.java:197)
at http://org.apache.commons.io|org.apache.commons.io.output.ThresholdingOutputStream.write(ThresholdingOutputStream.java:249)
at http://org.apache.commons.io|org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1488)
at http://org.apache.commons.io|org.apache.commons.io.IOUtils.copy(IOUtils.java:1107)
at http://org.apache.commons.io|org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1460)
at http://org.apache.commons.io|org.apache.commons.io.IOUtils.copy(IOUtils.java:1085)
at http://org.apache.commons.io|org.apache.commons.io.IOUtils.toByteArray(IOUtils.java:2710)
at io.airbyte.integrations.source.sftp.SftpClient.getFile(SftpClient.java:109)
at io.airbyte.integrations.source.sftp.SftpCommand.lambda$getFilesSchemas$0(SftpCommand.java:74)
at io.airbyte.integrations.source.sftp.SftpCommand$$Lambda/0x00007f4d20234f88.accept(Unknown Source)
at java.base/java.lang.Iterable.forEach(Iterable.java:75)
at io.airbyte.integrations.source.sftp.SftpCommand.getFilesSchemas(SftpCommand.java:73)
at io.airbyte.integrations.source.sftp.SftpSource.discover(SftpSource.java:88)
at io.airbyte.cdk.integrations.base.IntegrationRunner.runInternal(IntegrationRunner.java:159)
at io.airbyte.cdk.integrations.base.IntegrationRunner.run(IntegrationRunner.java:125)
at io.airbyte.integrations.source.sftp.SftpSource.main(SftpSource.java:37)
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.