Summary
The user is experiencing an issue where the Postgres destination is not flushing data with the Jira source connector, leading to a loop where no items are being flushed despite items in the queue.
Question
I’m running into an issue with the postgres destination not flushing data with the Jira source connector:
2024-05-02 16:46:48 [43mdestination[0m > 2024-05-02 16:46:48 [32mINFO[m i.a.c.i.d.FlushWorkers(lambda$flush$1):176 - Flush Worker (24363) -- Worker finished flushing. Current queue size: 5642
2024-05-02 16:46:49 [43mdestination[0m > 2024-05-02 16:46:49 [32mINFO[m i.a.c.i.d.DetectStreamToFlush(getNextStreamToFlush):122 - flushing: trigger info: xxx - jira_issues, time trigger: false , size trigger: true current threshold b: 0 bytes, queue size b: 287.02 MB, penalty b: 0 bytes, after penalty b: 287.02 MB
2024-05-02 16:46:49 [43mdestination[0m > 2024-05-02 16:46:49 [32mINFO[m i.a.c.i.d.DetectStreamToFlush(getNextStreamToFlush):122 - flushing: trigger info: xxx - jira_issues, time trigger: false , size trigger: true current threshold b: 0 bytes, queue size b: 287.02 MB, penalty b: 25 MB, after penalty b: 262.02 MB
2024-05-02 16:46:49 [43mdestination[0m > 2024-05-02 16:46:49 [32mINFO[m i.a.c.i.d.FlushWorkers(lambda$flush$1):152 - Flush Worker (035a3) -- Worker picked up work.
2024-05-02 16:46:49 [43mdestination[0m > 2024-05-02 16:46:49 [32mINFO[m i.a.c.i.d.FlushWorkers(lambda$flush$1):154 - Flush Worker (035a3) -- Attempting to read from queue namespace: xxx, stream: jira_issues.
2024-05-02 16:46:49 [43mdestination[0m > 2024-05-02 16:46:49 [32mINFO[m i.a.c.i.d.FlushWorkers(lambda$flush$1):167 - Flush Worker (035a3) -- Batch contains: 0 records, 0 bytes bytes.
2024-05-02 16:46:49 [43mdestination[0m > 2024-05-02 16:46:49 [32mINFO[m i.a.c.i.d.GlobalMemoryManager(free):88 - Freeing 0 bytes..
2024-05-02 16:46:49 [43mdestination[0m > 2024-05-02 16:46:49 [32mINFO[m i.a.c.i.d.GlobalMemoryManager(free):88 - Freeing 0 bytes..
2024-05-02 16:46:49 [43mdestination[0m > 2024-05-02 16:46:49 [32mINFO[m i.a.c.i.d.FlushWorkers(lambda$flush$1):152 - Flush Worker (0e894) -- Worker picked up work.
2024-05-02 16:46:49 [43mdestination[0m > 2024-05-02 16:46:49 [32mINFO[m i.a.c.i.d.FlushWorkers(lambda$flush$1):176 - Flush Worker (035a3) -- Worker finished flushing. Current queue size: 5642
2024-05-02 16:46:49 [43mdestination[0m > 2024-05-02 16:46:49 [32mINFO[m i.a.c.i.d.FlushWorkers(lambda$flush$1):154 - Flush Worker (0e894) -- Attempting to read from queue namespace: xxx, stream: jira_issues.
2024-05-02 16:46:49 [43mdestination[0m > 2024-05-02 16:46:49 [32mINFO[m i.a.c.i.d.FlushWorkers(lambda$flush$1):167 - Flush Worker (0e894) -- Batch contains: 0 records, 0 bytes bytes.
2024-05-02 16:46:49 [43mdestination[0m > 2024-05-02 16:46:49 [32mINFO[m i.a.c.i.d.GlobalMemoryManager(free):88 - Freeing 0 bytes..
2024-05-02 16:46:49 [43mdestination[0m > 2024-05-02 16:46:49 [32mINFO[m i.a.c.i.d.GlobalMemoryManager(free):88 - Freeing 0 bytes..
2024-05-02 16:46:49 [43mdestination[0m > 2024-05-02 16:46:49 [32mINFO[m i.a.c.i.d.FlushWorkers(lambda$flush$1):176 - Flush Worker (0e894) -- Worker finished flushing. Current queue size: 5642```
has anyone run into this issue before? its stuck in this loop where nothing is being flushed even though there's items in the queue
<br>
---
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. [Click here](https://airbytehq.slack.com/archives/C021JANJ6TY/p1714685380113679) if you want to access the original thread.
[Join the conversation on Slack](https://slack.airbyte.com)
<sub>
["postgres-destination", "flushing-data", "jira-source-connector", "queue", "loop"]
</sub>