Hi All,
I’m trying to add a new stream on Amazon Seller Partner source to get the Financial Events data by using “GET /finances/v0/financialEvents” API.
https://developer-docs.amazon.com/sp-api/docs/finances-api-reference#get-financesv0financialevents
The response format for this API is complex, where we have nested objects and arrays
I used this schema on connector to get the data and dump it to local json.
I have put only first level attributes.
JSON output view
Question: Do we need to map the complete response with schema? ( like each attribute and objects ).
Please suggest the best way forward.
Thanks a ton.
Hey can you share one record you are returning as a file ?
list-financial-event-airbyte-record.txt (20.7 KB)
Attached is 1 record, which has multiple event lists with multiple underlying records.
If there is NextPageToken in response, similar more records are saved in the output file ( please see the output screenshot shared in first post )
Hey Ideally you need to generate the complete json schema to individual attribute to get the right normalisation. You can take one record and use some online services to generate the schema
After syncing the data with Snowflake with above schema, Airbyte created columns for each of the first level fields like “ShipmentEventList”, “RefundEventList” & so on. Data inside those columns were string formatted JSON objects.
This is exactly the format we were looking for the data to get saved in snowflake.
Will try to create full schema and see if there is any changes in data storage views.
How much of normalisation we would want to happen?
-
Normalisation with complete schema: Generated ~364 models and took 40 mins for normalisation process.
-
Normalisation with above schema ( only first level fields ): Generated ~4 models and took 7 mins for normalisation process.
-
Without Normalisation: Sync was completed in 2 mins.
Hey would suggest keeping atleast 2 levels otherwise normalisation doesn’t make any sense for users who are opting for that in this connector. But at a higher level you can also use custom DBT
Is there a way to configure desired levels of normalization?
PR has been updated with schema up to 2 levels for better normalisation. PTAL
Sure thank you team will look into it
Hi there from the Community Assistance team.
We’re letting you know about an issue we discovered with the back-end process we use to handle topics and responses on the forum. If you experienced a situation where you posted the last message in a topic that did not receive any further replies, please open a new topic to continue the discussion. In addition, if you’re having a problem and find a closed topic on the subject, go ahead and open a new topic on it and we’ll follow up with you. We apologize for the inconvenience, and appreciate your willingness to work with us to provide a supportive community.
