Hey, This is still a problem.
We’ve now got our production Airbyte instance running and have created a few connections with production databases.
Our replication slots are configured via RDS parameter group with the following parameters.
- max_slot_wal_keep_size 2048 (MB) Sets the maximum WAL size that can be reserved by replication slots.
- max_wal_senders 20 Sets the maximum number of simultaneously running WAL sender processes.
- max_wal_size 2048 (MB) Sets the WAL size that triggers a checkpoint.
- min_wal_size 192 (MB) Sets the minimum size to shrink the WAL to.
- wal_compression 1 Compresses full-page writes written in WAL file.
- wal_keep_size 2048 (MB) Sets the size of WAL files held for standby servers.
We are only syncing 9 tables of data via the publication which are not updated overnight.
Airbyte is syncing once an hour.
Airbyte appears to be only resetting a replication slot if there are changes to the data, which means overnight the replication slot will eventually fill and then become lost, causing us to have to reset the replication slot and then perform the initial sync of all data again.
We’re at the point where creating a table which we update every 30 minutes and adding this into the publication should hopefully cause airbyte to clear the replication slot as a workaround.