Summary
The user is facing an issue where the data sync from Supabase (Postgres) to Aurora Postgres is not working as expected. The logs show that the batch contains 0 records and 0 bytes, with no data being pushed to the destination. The issue seems to be related to data not being pushed into the queue.
Question
So I am on 0.50.54 version of Airbyte, doing a sync from supabase(postgres) to Aurora postgres!
The logs are too confusing to understand but I would try to put those here, I would love to have a helping hand in understanding this better!
The last section Batch contains: 0 records, 0 bytes bytes. is repeated again and again with no data pushed to destination… Although the log says destination > , Is the issue related to data not being pushed into the queue? This is happening more often recently!
2024-10-03 05:41:26 destination > INFO pool-4-thread-2 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):138 Flush Worker (bc44d) -- Worker picked up work.
2024-10-03 05:41:26 destination > INFO pool-4-thread-2 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):144 Flush Worker (bc44d) -- Attempting to read from queue namespace: supabyte, stream: bio.
2024-10-03 05:41:26 destination > INFO pool-4-thread-2 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):166 Flush Worker (bc44d) -- Batch contains: 33998 records, 25 MB bytes.
2024-10-03 05:41:26 platform > Records read: 385000 (198 MB)
2024-10-03 05:41:31 destination > INFO pool-4-thread-2 i.a.c.i.d.a.s.GlobalAsyncStateManager(flushStates):159 Flushing states
2024-10-03 05:41:31 destination > INFO pool-4-thread-2 i.a.c.i.d.a.s.GlobalAsyncStateManager(flushStates):213 Flushing states complete
2024-10-03 05:41:31 destination > INFO pool-4-thread-2 i.a.c.i.d.a.GlobalMemoryManager(free):78 Freeing 0 bytes..
2024-10-03 05:41:31 destination > INFO pool-4-thread-2 i.a.c.i.d.a.GlobalMemoryManager(free):78 Freeing 26214291 bytes..
2024-10-03 05:41:31 destination > INFO pool-4-thread-2 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):184 Flush Worker (bc44d) -- Worker finished flushing. Current queue size: 9898
2024-10-03 05:41:33 destination > INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):106 [ASYNC QUEUE INFO] Global: max: 296.96 MB, allocated: 20.01 MB (20.00642967224121 MB), % used: 0.06737022249590743 | Queue `bio`, num records: 9898, num bytes: 7.24 MB, allocated bytes: 10.01 MB | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.000000
2024-10-03 05:41:33 destination > INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):130 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2024-10-03 05:41:57 source > INFO main i.a.c.i.s.r.AbstractDbSource$createReadIterator$3(invoke):507 Reading stream bio. Records read: 390000
2024-10-03 05:41:57 platform > Records read: 390000 (201 MB)
2024-10-03 05:41:58 platform > Records read: 395000 (203 MB)
2024-10-03 05:41:58 source > INFO main i.a.c.i.s.r.AbstractDbSource$createReadIterator$3(invoke):507 Reading stream bio. Records read: 400000
2024-10-03 05:41:59 platform > Records read: 400000 (206 MB)
2024-10-03 05:42:00 platform > Records read: 405000 (209 MB)
2024-10-03 05:42:00 source > INFO main i.a.c.i.s.r.AbstractDbSource$createReadIterator$3(invoke):507 Reading stream bio. Records read: 410000
2024-10-03 05:42:00 platform > Records read: 410000 (212 MB)
2024-10-03 05:42:01 platform > Records read: 415000 (215 MB)
2024-10-03 05:42:02 destination > INFO pool-5-thread-1 i.a.c.i.d.a.DetectStreamToFlush(getNextStreamToFlush):116 flushing: trigger info: supabyte - bio, time trigger: false , size trigger: true current threshold b: 25 MB, queue size b: 28.11 MB, penalty b: 0 bytes, after penalty b: 28.11 MB
2024-10-03 05:42:02 destination > INFO pool-4-thread-1 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):138 Flush Worker (5d8d2) -- Worker picked up work.
2024-10-03 05:42:02 destination > INFO pool-4-thread-1 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):144 Flush Worker (5d8d2) -- Attempting to read from queue namespace: supabyte, stream: bio.
2024-10-03 05:42:02 destination > INFO pool-4-thread-1 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):166 Flush Worker (5d8d2) -- Batch contains: 34783 records, 25 MB bytes.
2024-10-03 05:42:02 source > INFO main i.a.c.i.s.r.AbstractDbSource$createReadIterator$3(invoke):507 Reading stream bio. Records read: 420000
2024-10-03 05:42:02 platform > Records read: 420000 (217 MB)
2024-10-03 05:42:04 platform > Records read: 425000 (220 MB)
2024-10-03 05:42:05 source > INFO main i.a.c.i.s.r.AbstractDbSource$createReadIterator$3(invoke):507 Reading stream bio. Records read: 430000
2024-10-03 05:42:05 platform > Records read: 430000 (223 MB)
2024-10-03 05:42:07 platform > Records read: 435000 (226 MB)
2024-10-03 05:42:08 source > INFO main i.a.c.i.s.r.AbstractDbSource$createReadIterator$3(invoke):507 Reading stream bio. Records read: 440000
2024-10-03 05:42:08 platform > Records read: 440000 (229 MB)
2024-10-03 05:42:08 destination > INFO pool-4-thread-1 i.a.c.i.d.a.s.GlobalAsyncStateManager(flushStates):159 Flushing states
2024-10-03 05:42:08 destination > INFO pool-4-thread-1 i.a.c.i.d.a.s.GlobalAsyncStateManager(flushStates):213 Flushing states complete
2024-10-03 05:42:08 destination > INFO pool-4-thread-1 i.a.c.i.d.a.GlobalMemoryManager(free):78 Freeing 0 bytes..
2024-10-03 05:51:33 destination > INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):106 [ASYNC QUEUE INFO] Global: max: 296.96 MB, allocated: 10 MB (10.0 MB), % used: 0.03367428551701215 | Queue `bio`, num records: 0, num bytes: 0 bytes, allocated bytes: 0 bytes | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.000000
2024-10-03 05:51:33 destination > INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):130 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2024-10-03 05:52:12 destination > INFO pool-5-thread-1 i.a.c.i.d.a.DetectStreamToFlush(getNextStreamToFlush):116 flushing: trigger info: supabyte - bio, time trigger: true , size trigger: false current threshold b: 25 MB, queue size b: 0 bytes, penalty b: 0 bytes, after penalty b: 0 bytes
2024-10-03 05:52:12 destination > INFO pool-4-thread-2 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):138 Flush Worker (17205) -- Worker picked up work.
2024-10-03 05:52:12 destination > INFO pool-4-thread-2 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):144 Flush Worker (17205) -- Attempting to read from queue namespace: supabyte, stream: bio.
2024-10-03 05:52:12 destination > INFO pool-4-thread-2 i.a.c.i.d.a.GlobalMemoryManager(free):78 Freeing 0 bytes..
2024-10-03 05:52:12 destination > INFO pool-4-thread-2 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):166 Flush Worker (17205) -- Batch contains: 0 records, 0 bytes bytes.
2024-10-03 05:52:12 destination > INFO pool-4-thread-2 i.a.c.i.d.a.s.GlobalAsyncStateManager(flushStates):159 Flushing states
2024-10-03 05:52:12 destination > INFO pool-4-thread-2 i.a.c.i.d.a.s.GlobalAsyncStateManager(flushStates):213 Flushing states complete
2024-10-03 05:52:12 destination > INFO pool-4-thread-2 i.a.c.i.d.a.GlobalMemoryManager(free):78 Freeing 0 bytes..
2024-10-03 05:52:12 destination > INFO pool-4-thread-2 i.a.c.i.d.a.GlobalMemoryManager(free):78 Freeing 0 bytes..
2024-10-03 05:52:12 destination > INFO pool-4-thread-2 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):184 Flush Worker (17205) -- Worker finished flushing. Current queue size: 0
2024-10-03 05:52:33 destination > INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):106 [ASYNC QUEUE INFO] Global: max: 296.96 MB, allocated: 10 MB (10.0 MB), % used: 0.03367428551701215 | Queue `bio`, num records: 0, num bytes: 0 bytes, allocated bytes: 0 bytes | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.000000
2024-10-03 05:52:33 destination > INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):130 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2024-10-03 05:53:33 destination > INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):106 [ASYNC QUEUE INFO] Global: max: 296.96 MB, allocated: 10 MB (10.0 MB), % used: 0.03367428551701215 | Queue `bio`, num records: 0, num bytes: 0 bytes, allocated bytes: 0 bytes | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.000000
2024-10-03 05:53:33 destination > INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):130 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2024-10-03 05:54:33 destination > INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):106 [ASYNC QUEUE INFO] Global: max: 296.96 MB, allocated: 10 MB (10.0 MB), % used: 0.03367428551701215 | Queue `bio`, num records: 0, num bytes: 0 bytes, allocated bytes: 0 bytes | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.000000
2024-10-03 05:54:33 destination > INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):130 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2024-10-03 05:55:33 destination > INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):106 [ASYNC QUEUE INFO] Global: max: 296.96 MB, allocated: 10 MB (10.0 MB), % used: 0.03367428551701215 | Queue `bio`, num records: 0, num bytes: 0 bytes, allocated bytes: 0 bytes | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.000000
2024-10-03 05:55:33 destination > INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):130 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2024-10-03 05:56:33 destination > INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):106 [ASYNC QUEUE INFO] Global: max: 296.96 MB, allocated: 10 MB (10.0 MB), % used: 0.03367428551701215 | Queue `bio`, num records: 0, num bytes: 0 bytes, allocated bytes: 0 bytes | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.000000
2024-10-03 05:56:33 destination > INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):130 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2024-10-03 05:57:13 destination > INFO pool-5-thread-1 i.a.c.i.d.a.DetectStreamToFlush(getNextStreamToFlush):116 flushing: trigger info: supabyte - bio, time trigger: true , size trigger: false current threshold b: 25 MB, queue size b: 0 bytes, penalty b: 0 bytes, after penalty b: 0 bytes
2024-10-03 05:57:13 destination > INFO pool-4-thread-1 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):138 Flush Worker (1c3e2) -- Worker picked up work.
2024-10-03 05:57:13 destination > INFO pool-4-thread-1 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):144 Flush Worker (1c3e2) -- Attempting to read from queue namespace: supabyte, stream: bio.
2024-10-03 05:57:13 destination > INFO pool-4-thread-1 i.a.c.i.d.a.GlobalMemoryManager(free):78 Freeing 0 bytes..
2024-10-03 05:57:13 destination > INFO pool-4-thread-1 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):166 Flush Worker (1c3e2) -- Batch contains: 0 records, 0 bytes bytes.
2024-10-03 05:57:13 destination > INFO pool-4-thread-1 i.a.c.i.d.a.s.GlobalAsyncStateManager(flushStates):159 Flushing states
2024-10-03 05:57:13 destination > INFO pool-4-thread-1 i.a.c.i.d.a.s.GlobalAsyncStateManager(flushStates):213 Flushing states complete
2024-10-03 05:57:13 destination > INFO pool-4-thread-1 i.a.c.i.d.a.GlobalMemoryManager(free):78 Freeing 0 bytes..
2024-10-03 05:57:13 destination > INFO pool-4-thread-1 i.a.c.i.d.a.GlobalMemoryManager(free):78 Freeing 0 bytes..
2024-10-03 05:57:13 destination > INFO pool-4-thread-1 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):184 Flush Worker (1c3e2) -- Worker finished flushing. Current queue size: 0```
<br>
---
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. [Click here](https://airbytehq.slack.com/archives/C021JANJ6TY/p1727941611514019) if you want
to access the original thread.
[Join the conversation on Slack](https://slack.airbyte.com)
<sub>
["data-sync", "supabase", "aurora-postgres", "logs", "batch-records"]
</sub>