Summary
User reports high memory usage after upgrading Snowflake destination and removing S3 staging, despite setting max per stream to 2GB. The issue arises during data transfer from Postgres to Snowflake.
Question
Hi All, I’m really struggling here. I upgraded the Snowflake destination to the most recent version, and removed S3 staging. But now it looks to be storing all the data to be transferred in memory instead of on disk (or previously S3).
One of those is 14GB, another 4GB. Other piplines today were at 15-20GB doing initial syncs.
I have set the max per stream to 2GB in the airbyte DB
> select * from actor_definition where name like ‘Snowflake’
> {“jobSpecific”: [{“jobType”: “sync”, “resourceRequirements”: {“memory_limit”: “2Gi”, “memory_request”: “2Gi”}}]}
Nothing seems to work. I’m pulling data from Postgres source (3.4.1) to Snowflake destination (3.8.0) and can’t see why its using so much Ram. We have had to bump it up to 64GB machine, and stagger slots to only run two concurrently.
Hoping someone has some wisdom for me 
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.