Summary
The user is facing issues with understanding how the Airbyte connector works in incremental mode using CDC, specifically with the postgres and mongodb source connectors. The user is confused about the behavior of the connector in incremental mode and why it hangs instead of terminating like in full refresh mode.
Question
Hi! I’m trying to use an Airbyte connector directly and having trouble understanding how it works, specifically with incremental mode using CDC. I am trying with the airbyte postgres and mongodb source connectors. Using full refresh mode it seems the read command terminates with my toy dataset (of like 2 items), but using the incremental mode it seems to just hang there. I’m trying to understand what this means and how I can interpret this behavior. I know that I get a state back as the last message so I could technically take that to get more. But I guess I’d like to understand why it is hanging there instead of terminating on its own like in the full refresh mode. I thought maybe that is because it was waiting indefinitely for more updates, but that doesn’t seem to be the case as no more new records are emitted when I insert new entries. Any ideas or explanations as to what is going on? Am I suppose to manually cut it off? Thank you!
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.