High RAM Usage after Upgrading Snowflake Destination

Summary

After upgrading the Snowflake destination and removing S3 staging, data seems to be stored in memory instead of on disk. RAM usage is high even after setting max per stream to 2GB. Data is being pulled from Postgres source to Snowflake destination.


Question

Hi All, I’m really struggling here. I upgraded the Snowflake destination to the most recent version, and removed S3 staging. But now it looks to be storing all the data to be transferred in memory instead of on disk (or previously S3).

One of those is 14GB, another 4GB. Other piplines today were at 15-20GB doing initial syncs.

I have set the max per stream to 2GB in the airbyte DB

> select * from actor_definition where name like ‘Snowflake’
> {“jobSpecific”: [{“jobType”: “sync”, “resourceRequirements”: {“memory_limit”: “2Gi”, “memory_request”: “2Gi”}}]}
Nothing seems to work. I’m pulling data from Postgres source (3.4.1) to Snowflake destination (3.8.0) and can’t see why its using so much Ram. We have had to bump it up to 64GB machine, and stagger slots to only run two concurrently.

Hoping someone has some wisdom for me :slightly_smiling_face:



This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

["snowflake-destination", "s3-staging", "ram-usage", "postgres-source", "airbyte-db"]