Increasing memory usage for source job in Airbyte

Slack Knowledge Base Q&A Connector Questions
, , , ,
1

Summary

The user is trying to increase memory usage for a source job in Airbyte, specifically for a MSSQL source. They have tried setting a limit for the container but it is not being fully utilized. They also attempted to increase fetchSize in the JDBC URL Params but it seems to be overridden by AdaptiveStreamingQueryConfig.


Question

hello,
I’m trying to move a table with a large number of rows in Airbyte. (use docker-compose on ec2)
I’m particularly curious about the source job.
Currently, in the picture below, I’ve set the Limit of 4GB MEM for the source-mssql-read-5-0-cpdjo Container, but it only uses around 300-400MiB in practice. Is there a way to increase this usage more efficiently?

I increased the fetchSize in the JDBC URL Params of the MSSQL Source Option, but it seemed to be overridden by the dynamic fetchSize due to the AdaptiveStreamingQueryConfig function. Therefore, it seems impossible to increase memory usage by directly specifying fetchSize.


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

["increasing-memory-usage", "source-job", "airbyte", "mssql-source", "fetchsize", "memory-limit"]