Issue with MongoDB Source Connector on Sharded Cluster

Summary

User is experiencing issues with the MongoDB source connector version 1.2.15 not supporting sharded clusters, while version 0.2.5 works but encounters a memory limit error. The user seeks guidance on how to set the ‘allowDiskUse’ option to resolve the error.


Question

Hi All, Has anyone here successfully used mongodb source connector with MongoDB Atlas Sharded Cluster? Did some testing and attempted to use. But the latest version (1.2.15) doesn’t work with sharded cluster - only supports replica set. While the older version (0.2.5) can work with sharded cluster but getting the following error but I don’t know how to pass option ‘allowDiskUse: true’

“AirbyteExceptionHandler(uncaughtException):26 Something went wrong in the connector. See the logs for more details. com.mongodb.MongoCommandException: Command failed with error 292 (QueryExceededMemoryLimitNoDiskUseAllowed): ‘PlanExecutor error during aggregation :: caused by :: Sort exceeded memory limit of 104857600 bytes, but did not opt in to external sorting.’'”



This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

['mongodb', 'source-connector', 'sharded-cluster', 'allowDiskUse', 'airbyte-exception']