Destination MongoDB: unexpected high memory usage

Connector Questions & Issues
1
  • Is this your first time deploying Airbyte?: No
  • OS Version / Instance: Ubuntu
  • Memory / Disk: 8Gb / 500Go
  • Deployment: Over given docker-compose file
  • Airbyte Version: latest
  • Source name/version: I think it doesn’t matter (tested with Intercom / Salesforce / File)
  • Destination name/version: MongoDB@latest
  • Step:
    • Start the latest given docker-compose file (start Airbyte stack over docker)
    • Create a source with high volume data to make sure you see the memory very high usage
    • Create the MongoDB destination
    • Create a connection between these two
    • Start a sync and observe how mongodb writer memory usage is very-very high (>12Gb when having ~4Gb in source)
  • Description: Using ctop we can observe memory in real-time over container created by Airbyte orchestrator (controller?), this is not an expected consumption and would most likely not fit with user requirements when dealing with big datas.
    When it reach the max memory (8Gb in my case), the job will stop at the writing process without any useful debug informations.

We do known that the connector is in Alpha, hence not receiving support usually

On Github there were a big refacto on JDBC destination where there were things about data buffer max size, and MongoDB was not part of this refacto. I’m not sure if it is relevant to the actual issue, but it might be?

Also on Github, there are one issue that have the exact same issue, we already sent logs.

Thanks in advance,

Romain.

2

Hey just trying to understand this better. We have a new mongodb-v2 is that the same you are using?

3

No, in fact, mongodb-v2 is only available on source integration, not as destination if I understood correctly @harshith

4

Hey have pinged the respective team about the issue. They will get back to you. Thanks for the patience.