MySQL ingest huge amount of data

Troubleshooting
#1
  • Is this your first time deploying Airbyte?: Yes
  • OS Version / Instance: Ubuntu
  • Memory / Disk: 4Gb / 25 Gb
  • Deployment: Docker
  • Airbyte Version: 0.35.65-alpha
  • Source name/version: airbyte/source-mysql:0.5.6
  • Destination name/version: airbyte/destination-bigquery:0.3.12
  • Step: normal sync
  • ingestion type : CDC - Incremental Append
  • Description:
    When we run a connector which ingest several huge tables, all data seems to be put in cache.
    It generate too much data for my VM hosting my docker image.
    Is there any way to free this cache and manage each table one by one ?

Thanks in advance for your help.

#2

Hi @ftapin,
The syncs indeed leverage some memory buffering. There’s no safe way of emptying these buffers and the right way to go is to increase your VM memory. I would recommend at least 16Gb of RAM for an Airbyte instance.

#4

Hi there from the Community Assistance team.
We’re letting you know about an issue we discovered with the back-end process we use to handle topics and responses on the forum. If you experienced a situation where you posted the last message in a topic that did not receive any further replies, please open a new topic to continue the discussion. In addition, if you’re having a problem and find a closed topic on the subject, go ahead and open a new topic on it and we’ll follow up with you. We apologize for the inconvenience, and appreciate your willingness to work with us to provide a supportive community.

#5