Hi @sajarin,
Hope you can help!
I tried to apply Incremental | Append methods since my data full refresh are getting bigger and bigger (take hours to query if I use Full Refresh | Overwrite).
So I have read about cursor field value in this doc. And in Known Limitations Part, it talks about max cursor field value.
Thus, I have a question about where the ‘last_sync_max_cursor_field_value’ stored? Is there anyway I can modify this value without writing custom code?
FYI: I did try to delete today’s records in destination (in my case is Big Query), then sync again using Incremental | Append, but it always returns no new records.
Here some logs:
-
First sync return all records up to today’s date which is 03/10/2022. Then I deleted all 03/10/2022 record in destination
-
Second sync when using Incremental | Append
2022-10-03 03:23:22 source > Starting syncing SourceGoogleAnalyticsDataApi
2022-10-03 03:23:22 source > Syncing stream: revenue by ad events
2022-10-03 03:23:22 source > Setting state of revenue by ad events stream to {'date': '2022-10-03'}
2022-10-03 03:23:22 source > Read 0 records from revenue by ad events stream
2022-10-03 03:23:22 source > Finished syncing revenue by ad events
2022-10-03 03:23:22 source > SourceGoogleAnalyticsDataApi runtimes:
- Try to sync again:
2022-10-03 03:32:04 source > Starting syncing SourceGoogleAnalyticsDataApi
2022-10-03 03:32:04 source > Syncing stream: revenue by ad events
2022-10-03 03:32:04 source > Setting state of revenue by ad events stream to {'date': '2022-10-04'}
2022-10-03 03:32:04 source > Read 0 records from revenue by ad events stream
2022-10-03 03:32:04 source > Finished syncing revenue by ad events
2022-10-03 03:32:04 source > SourceGoogleAnalyticsDataApi runtimes:
The date cursor field automatedly increase to tomorrow’s date!!
If you have any solution, hope to hear it too!