Memory caching behavior in Airbyte worker with data synchronization

Slack Knowledge Base Q&A Platform Questions
, , , ,
1

Summary

Investigating if Airbyte worker caches data in memory when source reads more data than destination can write


Question

Hey folks,

is it possible that the airbyte-worker is caching a lot of data in the memory because the source is reading more data than the destination can write?
We are running airbyte on a VM with only 2 cores and I noticed that behaviour for snycs with a lot of data.


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

["airbyte-worker", "memory-caching", "data-synchronization", "performance-issue"]
2

Interested to know what you find out here. (Also it might help to say what the source and destination is?)