Summary
The user is experiencing longer sync times in a Docker container when using CDC with Airbyte, particularly when there are no data changes. They seek solutions to reduce polling or shorten sync times without unnecessary logging.
Question
Hello everyone,
I’m in the process of loading cdc into a docker container using aribyte.
It takes more time when there is no changed data at all than when there is changed data in the source DB.
2024-12-03 07:16:06 source > INFO main i.a.c.i.d.i.DebeziumRecordIterator(computeNext):87 CDC events queue poll(): blocked for PT10.543918214S after its previous call which was also logged.
2024-12-03 07:16:06 source > INFO main i.a.c.i.d.i.DebeziumRecordIterator(computeNext):140 CDC events queue poll(): returned a heartbeat event: no progress since last heartbeat.
2024-12-03 07:16:16 source > INFO pool-3-thread-1 i.a.c.i.d.AirbyteDebeziumHandler$CapacityReportingBlockingQueue(reportQueueUtilization):48 CDC events queue stats: size=0, cap=10000, puts=10, polls=0
2024-12-03 07:16:16 source > INFO main i.a.c.i.d.i.DebeziumRecordIterator(computeNext):87 CDC events queue poll(): blocked for PT10.528857032S after its previous call which was also logged.
2024-12-03 07:16:16 source > INFO main i.a.c.i.d.i.DebeziumRecordIterator(computeNext):140 CDC events queue poll(): returned a heartbeat event: no progress since last heartbeat.
2024-12-03 07:16:26 source > INFO pool-3-thread-1 i.a.c.i.d.AirbyteDebeziumHandler$CapacityReportingBlockingQueue(reportQueueUtilization):48 CDC events queue stats: size=0, cap=10000, puts=11, polls=0
2024-12-03 07:16:26 source > INFO main i.a.c.i.d.i.DebeziumRecordIterator(computeNext):87 CDC events queue poll(): blocked for PT10.030039963S after its previous call which was also logged.
2024-12-03 07:16:26 source > INFO main i.a.c.i.d.i.DebeziumRecordIterator(computeNext):140 CDC events queue poll(): returned a heartbeat event: no progress since last heartbeat.
2024-12-03 07:16:37 source > INFO pool-3-thread-1 i.a.c.i.d.AirbyteDebeziumHandler$CapacityReportingBlockingQueue(reportQueueUtilization):48 CDC events queue stats: size=0, cap=10000, puts=12, polls=0
2024-12-03 07:16:37 source > INFO main i.a.c.i.d.i.DebeziumRecordIterator(computeNext):87 CDC events queue poll(): blocked for PT10.53084938S after its previous call which was also logged.
2024-12-03 07:16:37 source > INFO main i.a.c.i.d.i.DebeziumRecordIterator(computeNext):140 CDC events queue poll(): returned a heartbeat event: no progress since last heartbeat.
2024-12-03 07:16:47 source > INFO main i.a.c.i.d.i.DebeziumRecordIterator(computeNext):87 CDC events queue poll(): blocked for PT9.525437323S after its previous call which was also logged.
2024-12-03 07:16:47 source > INFO main i.a.c.i.d.i.DebeziumRecordIterator(computeNext):140 CDC events queue poll(): returned a heartbeat event: no progress since last heartbeat.```
For now it takes over 2 min when there's no data changes. but when there's data changes, it takes within 1 minute.
Can I reduce the number of polling that is running every 10 seconds for 2 minutes? or
If there is no change log because there are no data changes in the source db, is it possible to shorten the sync time within airbyte without repeating this log?
<br>
---
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. [Click here](https://airbytehq.slack.com/archives/C021JANJ6TY/p1733277733546889) if you want
to access the original thread.
[Join the conversation on Slack](https://slack.airbyte.com)
<sub>
['cdc', 'docker', 'polling', 'sync-time', 'airbyte']
</sub>