Amazon Seller Partner Source : Approach for "GET /finances/v0/financialEvents"

Hi All,

I’m trying to add a new stream on Amazon Seller Partner source to get the Financial Events data by using “GET /finances/v0/financialEvents” API.

The response format for this API is complex, where we have nested objects and arrays

I used this schema on connector to get the data and dump it to local json.
I have put only first level attributes.

JSON output view

Question: Do we need to map the complete response with schema? ( like each attribute and objects ).

Please suggest the best way forward.

Thanks a ton.

Hey can you share one record you are returning as a file ?

list-financial-event-airbyte-record.txt (20.7 KB)

Attached is 1 record, which has multiple event lists with multiple underlying records.

If there is NextPageToken in response, similar more records are saved in the output file ( please see the output screenshot shared in first post )

Hey Ideally you need to generate the complete json schema to individual attribute to get the right normalisation. You can take one record and use some online services to generate the schema

After syncing the data with Snowflake with above schema, Airbyte created columns for each of the first level fields like “ShipmentEventList”, “RefundEventList” & so on. Data inside those columns were string formatted JSON objects.

This is exactly the format we were looking for the data to get saved in snowflake.

Will try to create full schema and see if there is any changes in data storage views.

How much of normalisation we would want to happen?

  1. Normalisation with complete schema: Generated ~364 models and took 40 mins for normalisation process.

  2. Normalisation with above schema ( only first level fields ): Generated ~4 models and took 7 mins for normalisation process.

  3. Without Normalisation: Sync was completed in 2 mins.

Hey would suggest keeping atleast 2 levels otherwise normalisation doesn’t make any sense for users who are opting for that in this connector. But at a higher level you can also use custom DBT

Have started a PR here : 🎉 Source Amazon Seller Partner: add financial streams by ganpatagarwal · Pull Request #13364 · airbytehq/airbyte · GitHub

Is there a way to configure desired levels of normalization?

PR has been updated with schema up to 2 levels for better normalisation. PTAL

Sure thank you team will look into it