Error in Mongo -> Clickhouse integration

Slack Knowledge Base Q&A Connector Development
, , , ,
1

Summary

Error code 241: Memory limit exceeded while reading column in the integration process from Mongo to Clickhouse.


Question

Hi - I am facing the below error for a Mongo -> Clickhouse integration. Has anyone else faced this?

07:39:09.587062 [error] [MainThread]:   :HTTPDriver for  returned response code 500)
07:39:09.587296 [error] [MainThread]:    Code: 241. DB::Exception: Memory limit (total) exceeded: would use 7.20 GiB (attempt to allocate chunk of 8309560 bytes), maximum: 7.20 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker.: (while reading column,retryable=<null>,timestamp=1708587555511,additionalProperties={}], io.airbyte.config.FailureReason@6b8de9dc[failureOrigin=normalization,failureType=system_error,internalMessage=[0m07:39:09.587062 [error] [MainThread]:   :HTTPDriver for  returned response code 500),externalMessage=Normalization failed during the dbt run. ```


<br>

---

This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. [Click here](https://airbytehq.slack.com/archives/C027KKE4BCZ/p1708587900988479) if you want to access the original thread.

[Join the conversation on Slack](https://slack.airbyte.com)

<sub>
["error", "mongo-connector", "clickhouse-connector", "memory-limit", "integration"]
</sub>
2

Based on the error message you’ve shared, it seems like the memory limit is being exceeded during the data sync process. This is a common issue when dealing with large datasets or complex transformations. In one of the forum posts extracted above, a user faced a similar issue while trying to sync a 1GB table from MySQL to Clickhouse. The support agent suggested checking the memory configuration of the source worker as it reads up to 10,000 records in memory. This can present problems for database sources with tables that have large row sizes. The Java connectors in Airbyte follow Java’s default behaviour with container memory and will only use up to 1/4 of the host’s allocated memory. If the memory usage goes higher, it may result in Out-of-Memory exceptions. You may want to customize this by setting JOB_MAIN_CONTAINER_MEMORY_REQUEST and JOB_MAIN_CONTAINER_MEMORY_LIMIT environment variables to custom values. Here are the relevant documents for your reference: - Scaling Airbyte - Configuring Connector Resources Please review these documents and adjust your settings accordingly. If the issue persists, it would be helpful to know more about your instance and the connector versions you’re using. Source: Forum Post: Mysql to clickhouse fails with Memory exception for incremental dedup data for 1GB table > Answers Part 1

3

<@U04SE163WC8> Thanks - I am on Airbyte Cloud, so dont have direct access to change paramerters like JOB_MAIN_CONTAINER_MEMORY_REQUEST

4

New message text here

5

Hello <@U06KW7QQL8N> a new version of Clickhouse destination was released this week can you give a shoot?

6

Hey <@U01MMSDJGC9> Is it normal that the clickhouse destination inside of airbyte will only sync the raw stream data?