Memory Error when Migrating Data from MongoDB to ClickHouse

Slack Knowledge Base Q&A Connector Questions
, , , ,
1

Summary

Error occurs during normalization step when migrating data from MongoDB to ClickHouse due to memory limit exceeded in ClickHouse server.


Question

Hi everyone)
I’m trying to migrate data from mongodb to clickhouse on a self hosted instance, one of the tables is quite large, and during the normalization step, at some point i’m getting an error from clickhouse server that requested memory was exceeded. i tried to redeploy worker instance with various values of JOB_MAIN_CONTAINER_MEMORY_REQUEST, JOB_MAIN_CONTAINER_MEMORY_LIMIT, NORMALIZATION_JOB_MAIN_CONTAINER_MEMORY_REQUEST and NORMALIZATION_JOB_MAIN_CONTAINER_MEMORY_LIMIT - doesn’t seem to be working, it still tries to send chunks of 20-30 gb


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.

Join the conversation on Slack

["mongodb", "clickhouse", "data-migration", "memory-error", "normalization", "worker-instance"]
2

If I’m not wrong… I saw some other discussions happening in Github about Clickhouse throwing OOM errors :disappointed: Right now there are some works to move Clickhouse to Destination V2 basic https://github.com/airbytehq/airbyte/pull/34637 which could help with this issue