Setting up parent stream for Brightpearl ERP system integration

Slack Knowledge Base Q&A Connector Development
, , , ,
1

Summary

Need guidance on setting up a parent stream using the OPTIONS method to retrieve orders in batches from Brightpearl ERP system API


Question

Hello everyone,

Im trying to set up a new integration with an ERP systems Brightpearl.
To retrieve all the orders i need to iterate through them in batches of 200

The api offers an OPTIONS method to request the orders in the least possible requests.

Id like to set up a parent stream using this options method to use in the orders stream.

wondering if anyone can point me in the right direction as im struggling to find anything in the documentation.

Below is the docs for the API
https://api-docs.brightpearl.com/order/order/get.html


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.

Join the conversation on Slack

["brightpearl", "erp-system", "integration", "parent-stream", "options-method", "api-documentation"]
2

Hey <@U06TGMUUPJL>! Thank you for trying out Airbyte.

~Hmmm, I’m lost in their docs a bit, and I don’t quite see where they’re saying you should use HTTP OPTIONS. Can you point me at it?~

UPD: I see it, give me a second, looking.

Otherwise, I do see they want an OAuth2 token (that’s fine, Airbyte can do that), and they have a relatively fine API — you should be able to build that in Connector Builder rather easily.

3

Okay, I understand now.

Hypothetically, you could build a custom python component for a connector (custom paginator or partition router something something, I know, unhelpful) that would do that. Not rocket science.

But it looks to me that the use case for that OPTIONS call is to get the least amount of API calls possible, and it would help if your orders are with sparse IDs. I have a sneaky suspicion that you would not win a whole lot by doing it.

Here’s what I would do instead:

  1. Use Order Search endpoint: https://api-docs.brightpearl.com/order/order/search.html
  2. In search, filter by placedOn . Make the stream incremental, use placedOn as cursor.
  3. For pagination, use offset paginator. Search endpoint will give you 500 records per request. Airbyte will get placedOn from the last record each page, and make the next request asking for records starting with that date.
    Does that make sense?
4

The less custom python code you write the better, generally. As we make improvements to the low-code CDK (such as concurrent sync support for example), not having components will make it easier for you to adopt new features in the future.

5

Wait maybe I’m lying — maybe Parametrized Requests is what you want: https://docs.airbyte.com/connector-development/connector-builder-ui/partitioning#parameterized-requests

6

Hey Natik,
Thanks for your reply

I have gone down the Order_search method and trying to use the firstResult parameter for pagination.

Having limited success. Can return the first set of results, however the records are nested inside an array of arrays and im not sure how is best to access these using the record selector:

  [
    1402,
    1,
    5,
    4,
    3,
    "2022-07-12T10:37:46.000+01:00",
    4,
    "WS-#1165",
    1,
    "2022-07-15T11:54:05.000+01:00",
    0,
    4,
    "WS-#1165",
    null,
    2,
    0,
    "2022-07-12T01:00:00.000+01:00",
    3,
    null
  ]
]```
7

Hmmm. I’m not entirely sure either. I’d try and do [0] as a record selector?

8

thanks Natik, Sadly that hasnt worked either.