- Is this your first time deploying Airbyte?: No
- OS Version / Instance: Ubuntu
- Memory / Disk: 8Gb / 500Go
- Deployment: Over given docker-compose file
- Airbyte Version: latest
- Source name/version: I think it doesn’t matter (tested with Intercom / Salesforce / File)
- Destination name/version: MongoDB@latest
- Start the latest given docker-compose file (start Airbyte stack over docker)
- Create a source with high volume data to make sure you see the memory very high usage
- Create the MongoDB destination
- Create a connection between these two
- Start a sync and observe how mongodb writer memory usage is very-very high (>12Gb when having ~4Gb in source)
- Description: Using
ctopwe can observe memory in real-time over container created by Airbyte orchestrator (controller?), this is not an expected consumption and would most likely not fit with user requirements when dealing with big datas.
When it reach the max memory (8Gb in my case), the job will stop at the writing process without any useful debug informations.
We do known that the connector is in Alpha, hence not receiving support usually
On Github there were a big refacto on JDBC destination where there were things about data buffer max size, and MongoDB was not part of this refacto. I’m not sure if it is relevant to the actual issue, but it might be?
Also on Github, there are one issue that have the exact same issue, we already sent logs.
Thanks in advance,