Incremental | Append mode is not working correctly for salesforce> S3 bucket (File format->Parquet) connection

Connector Questions & Issues
1

Hi,
Connection: salesforce> S3 bucket (File format->Parquet)
Stream selected for replication: Accounts
Primary key: id
Cursor Field: SystemModstamp

Ran sync twice without making any changes in the accounts stream data. The data should not be replicated again in 2nd but still one record (10th record) is being replicated again.

Destination Data:

First sync:

1stparquetfile
1stparquetfile903×143 34.8 KB

second sync:

2ndsync
2ndsync876×257 43.8 KB

2

Hi @aqsa,

This probably has to do with the cursor field, could you give me an example of what that looks like? What is the reason that you’ve decided to use this sync mode as opposed to Incremental Sync - Deduped History, the recommended one?

3

Hi @natalyjazzviolin ,
The cursor field was generated by the system when I selected the incremental-append mode.
cursorfield
I did tried the same flow with timestamp field: LastModifiedDate and its behavior was similar.

I am trying with incremental-append as opposed to Incremental Sync - Deduped History as I need this mode in my use case where I can have new/modified data appended without having the history table.

4

OK, got it! Digging more into this -

  1. Have you changed the schema since setting up the connection? The incremental sync mode cannot handle schema changes yet.

  2. Do you have any nested streams or fields in your source data?

5

@natalyjazzviolin
Apologies for late response.

  1. No, I didn’t changed the schema data since setting up connector. I just re-ran sync for it.
  2. Yes, it seems that the account stream that I mentioned is a nested stream.
    Thank you