Summary
User is seeking a method to reduce the sync time when there are no data changes in the source DB during CDC loading in a Docker container using Airbyte. Currently, sync time exceeds 2 minutes with no changes, while it is under 1 minute with changes.
Question
Hello everyone,
I’m in the process of loading cdc into a docker container using aribyte.
It takes more time when there is no changed data at all than when there is changed data in the source DB.
2024-12-03 07:16:06 source > INFO main i.a.c.i.d.i.DebeziumRecordIterator(computeNext):87 CDC events queue poll(): blocked for PT10.543918214S after its previous call which was also logged.
2024-12-03 07:16:06 source > INFO main i.a.c.i.d.i.DebeziumRecordIterator(computeNext):140 CDC events queue poll(): returned a heartbeat event: no progress since last heartbeat.
2024-12-03 07:16:16 source > INFO pool-3-thread-1 i.a.c.i.d.AirbyteDebeziumHandler$CapacityReportingBlockingQueue(reportQueueUtilization):48 CDC events queue stats: size=0, cap=10000, puts=10, polls=0
2024-12-03 07:16:16 source > INFO main i.a.c.i.d.i.DebeziumRecordIterator(computeNext):87 CDC events queue poll(): blocked for PT10.528857032S after its previous call which was also logged.
2024-12-03 07:16:16 source > INFO main i.a.c.i.d.i.DebeziumRecordIterator(computeNext):140 CDC events queue poll(): returned a heartbeat event: no progress since last heartbeat.
2024-12-03 07:16:26 source > INFO pool-3-thread-1 i.a.c.i.d.AirbyteDebeziumHandler$CapacityReportingBlockingQueue(reportQueueUtilization):48 CDC events queue stats: size=0, cap=10000, puts=11, polls=0
2024-12-03 07:16:26 source > INFO main i.a.c.i.d.i.DebeziumRecordIterator(computeNext):87 CDC events queue poll(): blocked for PT10.030039963S after its previous call which was also logged.
2024-12-03 07:16:26 source > INFO main i.a.c.i.d.i.DebeziumRecordIterator(computeNext):140 CDC events queue poll(): returned a heartbeat event: no progress since last heartbeat.
2024-12-03 07:16:37 source > INFO pool-3-thread-1 i.a.c.i.d.AirbyteDebeziumHandler$CapacityReportingBlockingQueue(reportQueueUtilization):48 CDC events queue stats: size=0, cap=10000, puts=12, polls=0
2024-12-03 07:16:37 source > INFO main i.a.c.i.d.i.DebeziumRecordIterator(computeNext):87 CDC events queue poll(): blocked for PT10.53084938S after its previous call which was also logged.
2024-12-03 07:16:37 source > INFO main i.a.c.i.d.i.DebeziumRecordIterator(computeNext):140 CDC events queue poll(): returned a heartbeat event: no progress since last heartbeat.
2024-12-03 07:16:47 source > INFO main i.a.c.i.d.i.DebeziumRecordIterator(computeNext):87 CDC events queue poll(): blocked for PT9.525437323S after its previous call which was also logged.
2024-12-03 07:16:47 source > INFO main i.a.c.i.d.i.DebeziumRecordIterator(computeNext):140 CDC events queue poll(): returned a heartbeat event: no progress since last heartbeat.```
If there is no change log because there are no data changes in the source db, is it possible to shorten the sync time within airbyte without repeating this log?
For now it takes over 2 min when there's no data changes. but when there's data changes, it takes within 1 minute.
<br>
---
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. [Click here](https://airbytehq.slack.com/archives/C021JANJ6TY/p1733205278822109) if you want
to access the original thread.
[Join the conversation on Slack](https://slack.airbyte.com)
<sub>
['cdc', 'docker', 'airbyte', 'sync-time', 'debezium']
</sub>