Incremental Syncs for Partitioned Data

To enable your partitioned streams to share the same logic as your parent stream with “Incremental | Append + Deduped” sync mode, you need to ensure that they share the same cursor field. This is the field that Airbyte uses to identify new or updated records in the source data. In your case, if you’re pulling orders in the parent stream and using the orderId in the partitioned streams to get products and shipping information, you would set the “Parent Key” to id and the “Current Parent Key Value Identifier” to order. This configuration is done in the Parent Stream component for the responses stream. Here’s an example of how you would configure it: - “Parent Key” is set to id - “Current Parent Key Value Identifier” is set to order - In the general section of the stream configuration, the “URL Path” is set to /orders/{{ stream_partition.order }}/notes When triggering a sync, the connector will first fetch all records of the orders stream. The records will look like this: { "id": 123, "currency": "EUR", "shipping_total": "12.23", ... } { "id": 456, "currency": "EUR", "shipping_total": "45.56", ... } { "id": 789, "currency": "EUR", "shipping_total": "78.89", ... } To turn a record into a partition value, the “parent key” is extracted, resulting in the partition values 123, 456 and 789. This results in the following requests to fetch the records of the notes stream: curl -X GET <https://example.com/wp-json/wc/v3/orders/123/notes> curl -X GET <https://example.com/wp-json/wc/v3/orders/456/notes> curl -X GET <https://example.com/wp-json/wc/v3/orders/789/notes> For the partitioned streams to share the same logic, you need to define your own stream_slices method on the child stream, reading the parent stream with SyncMode.incremental [^1^]. Please note that the current behavior of Incremental is not able to handle source schema changes yet, for example, when a column is added, renamed or deleted from an existing table etc. It is recommended to trigger a Full refresh - Overwrite to correctly replicate the data to the destination with the new schema changes [^2^]. [^1^]: (Source: Forum Post: How does sync mode work with sub stream / nested stream?) [^2^]: (Source: Documentation > Using-airbyte > Core-concepts > Sync-modes > Incremental Sync - Append > Known Limitations Part 2)