Mongodb incremental + append - no row after first sync

  • Is this your first time deploying Airbyte?: Yes
  • OS Version / Instance: Debian
  • Memory / Disk: 16Gb / 200 Gb SSD
  • Deployment: Hosted VM Compute Engine GCP
  • Airbyte Version: 0.39.23
  • Source name/version: 0.1.15
  • Destination name/version: 1.1.11
  • Step: The issue is happening during sync
  • Description:

Hello, we try to sync rows from a mongo collection, we have a field “date_scrap” of type date, which contains the date when the row has been scraped online. Since our rows are never modified on Mongo but only appended, “incremental + append” is the right solution for us. However when setting the cursor_field to “date_scrap” we don’t have anymore synced data after the first one when we setup the connection, all the next sync steps are “0 Bytes | no records | no records | 8m 24s | Sync | Succeeded”. I checked the doc and the cursor is the right type. I am missing something here ?

Thank you

Hey @oagudoperez thanks for the post and thanks for joining the community. Could you provide a few records from your mongo collection that you want appended and the last few records that have already been synced when you synced the first time?

Have you taken a look at this section of our documentation, outlining some of the limitations with the cursor based incremental appends? Let me know if it applies here:

Hi there from the Community Assistance team.
We’re letting you know about an issue we discovered with the back-end process we use to handle topics and responses on the forum. If you experienced a situation where you posted the last message in a topic that did not receive any further replies, please open a new topic to continue the discussion. In addition, if you’re having a problem and find a closed topic on the subject, go ahead and open a new topic on it and we’ll follow up with you. We apologize for the inconvenience, and appreciate your willingness to work with us to provide a supportive community.