Summary
Users are facing issues with the internal PostgreSQL database size in Airbyte Helm chart version 0.53.3 when running sync jobs with MySQL source and BigQuery destination. They are experiencing a high volume of logs filling up the databases (airbyte-volume-db) and are unable to increase the size of the Postgres databases through the Helm chart. They have tried setting the variable TEMPORAL_HISTORY_RETENTION_IN_DAYS but it’s not working as expected. They are seeking help from others who may have encountered similar issues with MySQL source.
Question
Hi all,
we currently have an issue with the internal PostgreSQL of Airbyte (helm chart version 0.53.3).
We’re running a sync job with MySQL source (v3.3.3) and a BigQuery destination (v2.4.2), we run this sync hourly, all tablesare on incremental sync mode, each sync is between 250MB-7GB.
As a result, we’re noticing a very high volume of logs filling up the databases ( airbyte-volume-db )
Upon investigation, there’s no way through the Helm chart to increase the size of the Postgres databases - Therefore, we wanted to know if anyone else has encountered this issue.
Additionally, after looking for this issue in blogs, we decided to try to set the variable - TEMPORAL_HISTORY_RETENTION_IN_DAYS, but it seems like it’s not working as expected…
Has anyone else encounter similar issue using MYSQL source?
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.