MySQL ingest huge amount of data

Troubleshooting
#1
  • Is this your first time deploying Airbyte?: Yes
  • OS Version / Instance: Ubuntu
  • Memory / Disk: 4Gb / 25 Gb
  • Deployment: Docker
  • Airbyte Version: 0.35.65-alpha
  • Source name/version: airbyte/source-mysql:0.5.6
  • Destination name/version: airbyte/destination-bigquery:0.3.12
  • Step: normal sync
  • ingestion type : CDC - Incremental Append
  • Description:
    When we run a connector which ingest several huge tables, all data seems to be put in cache.
    It generate too much data for my VM hosting my docker image.
    Is there any way to free this cache and manage each table one by one ?

Thanks in advance for your help.

#2

Hi @ftapin,
The syncs indeed leverage some memory buffering. There’s no safe way of emptying these buffers and the right way to go is to increase your VM memory. I would recommend at least 16Gb of RAM for an Airbyte instance.