Incremental | Append mode is not working correctly for notion> S3 bucket (File format->jsonl) connection

Connector Questions & Issues
1

Hi,
Connection: notion> S3 bucket (File format->jsonl)
Stream selected for replication: Databases
Primary key: id
Cursor Field: last_edited_time

0
: 
{stream: {name: "databases",…},…}
config
: 
{syncMode: "incremental", cursorField: ["last_edited_time"], destinationSyncMode: "append",…}
aliasName
: 
"databases"
cursorField
: 
["last_edited_time"]
destinationSyncMode
: 
"append"
primaryKey
: 
[["id"]]
selected
: 
true
syncMode
: 
"incremental"

Ran sync twice without making any changes in the databases stream data. The data should not be replicated again in 2nd but still one record (last record) is being replicated again.

Destination data:
First sync:

1st
1st1023×193 46.7 KB

first sync.txt (3.1 KB)
Second sync:
2nd sync.txt (1.4 KB)

n2d
n2d958×279 52.4 KB

2

Hello there! You are receiving this message because none of your fellow community members has stepped in to respond to your topic post. (If you are a community member and you are reading this response, feel free to jump in if you have the answer!) As a result, the Community Assistance Team has been made aware of this topic and will be investigating and responding as quickly as possible.
Some important considerations that will help your to get your issue solved faster:

  • It is best to use our topic creation template; if you haven’t yet, we recommend posting a followup with the requested information. With that information the team will be able to more quickly search for similar issues with connectors and the platform and troubleshoot more quickly your specific question or problem.
  • Make sure to upload the complete log file; a common investigation roadblock is that sometimes the error for the issue happens well before the problem is surfaced to the user, and so having the tail of the log is less useful than having the whole log to scan through.
  • Be as descriptive and specific as possible; when investigating it is extremely valuable to know what steps were taken to encounter the issue, what version of connector / platform / Java / Python / docker / k8s was used, etc. The more context supplied, the quicker the investigation can start on your topic and the faster we can drive towards an answer.
  • We in the Community Assistance Team are glad you’ve made yourself part of our community, and we’ll do our best to answer your questions and resolve the problems as quickly as possible. Expect to hear from a specific team member as soon as possible.

Thank you for your time and attention.
Best,
The Community Assistance Team

3

Hey @aqsa,

Thanks for your post. After looking at the two text files you shared, it looks like you’re performing a Full Refresh | Append instead of an Incremental | Append. Can you double check your connection and make sure you’re utilizing the right sync mode?

4

Hi @sajarin
Please have a look at the response I shared above it shows that sync mode was incremental append.
Moreover, according to my understanding, if it was full refresh append then both data fields would have been replicated in next sync. However, we can in second file only one data filed was replicated.

5

Hey @aqsa,

Thanks for the response. I only mentioned double checking because I saw a screenshot earlier of a Full Refresh sync (I think you edited your post since then though)

I am a bit confused on what the issue is here. I agree that this is not a full refresh | append sync because you would have had 3 pieces of data in the second sync. What is the difference between the first piece of data and the second piece of data?

6

Hey @sajarin
Basically, I Ran sync twice without making any changes in the stream data. Since, no updation / insertion is done, the data should not be replicated again in 2nd sync but still one record (last record) is being replicated again.
I have attached the screenshot that shows two files are generated in second sync. The second file should be generated only if any change is made in data after first sync.
In my case, after second sync ,only one file (file generated during first sync ) was expected . If you open both files you will see that the second file contains duplicate data.

7

Hey @aqsa,

Apologies for the long delay. Are you still experiencing issues with the incremental append mode with the notion source connector?

8

Hey, @sajarin
Yes, I have just executed same steps. Still experiencing aforementioned issue.

9

Hey @aqsa,

Can you provide any more info? What version connectors are you using and have you upgraded since before? Are there any new logs to share or new errors? This might be a unresolved bug with the source.