Incremental Load Issue with Customized Xero Connector in Airbyte CDK

Slack Knowledge Base Q&A Connector Development
, , , ,
1

Summary

The user is facing an issue with the incremental load in a customized version of the Xero connector in Airbyte CDK. They have successfully added the Quotes stream but are confused about the incremental load behavior, which seems to sync the last record from the previous run. The user has provided configuration details and a snapshot for reference.


Question

Hi team,
I’m new to Airbyte CDK and am currently working on implementing incremental loads in a customized version for your new Xero low-code source. This is because your connector doesn’t yet include the Quotes stream, which is something we need.
I’ve added the Quote stream successfully, and the full load is working well. I simply copied the configuration from another stream (e.g., Invoices) and modified it for Quotes.

However, I’m currently confused regarding the incremental load. It seems that the connection always syncs the last record from the previous run. I’ve provided a detailed example in the snapshot for reference.

I’d be really grateful for your help with this!

Fyi, here is the configuration of my quotes_stream in the manifest.yaml file, I haven’t modified the incremental sync base:

    $ref: "#/definitions/base_stream"
    name: "quotes"
    schema_loader:
      type: InlineSchemaLoader
      schema:
        $ref: "#/definitions/quotes_schema"
    retriever:
      $ref: "#/definitions/retriever"
      paginator:
        $ref: "#/definitions/paginator_base"
      record_selector:
        $ref: "#/definitions/selector"
        record_filter:
          condition: "{{ record['UpdatedDateUTC'] >= stream_interval['start_time'] }}"
    $parameters:
      primary_key: "QuoteID"
      path: "/Quotes"
      extractor_path: "Quotes"
    incremental_sync:
      $ref: "#/definitions/incremental_sync_base"```

<br>

---

This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. [Click here](https://airbytehq.slack.com/archives/C027KKE4BCZ/p1717497006726249) if you want 
to access the original thread.

[Join the conversation on Slack](https://slack.airbyte.com)

<sub>
["incremental-load", "customized-xero-connector", "airbyte-cdk", "quotes-stream", "manifest.yaml"]
</sub>