Summary
The user is facing a fatal error while trying to sync data to S3 using the Stripe connector. The error is related to Avro conversion and seems to be caused by a mismatch in the expected field types. The issue occurs on specific customers, especially those who have had Stripe for a longer time. The user suspects that the lack of logic in the connector source code to manage API version compatibility could be the cause.
Question
I am trying to sync to S3 using the Stripe connector and get the following fatal error. The destination creates Avro files.
tech.allegro.schema.json2avro.converter.AvroConversionException: Failed to convert JSON to Avro: Could not evaluate union, field setup_error is expected to be one of these: NULL, RECORD. If this is a complex type, check if offending field (path: setup_error.payment_method.customer) adheres to schema: { some stripe data }
I get the above error with json as well, just not fatal.
This issue happens to us on specific customers but not all, and particularly customer who have had Stripe for longer time. There is no resolution that I could find on Slack or git.
One potential cause could be; in Stripe world each account has a certain api version linked to it. If you don’t pass the account version it’ll look up the linked version and use that to send the response. I do not see a logic in the Stripe connector source code that manages for this compatibility, so makes me think that could be the culprit.
My current Airbyte Stripe connector version is 4.4.0 which corresponds to 2023-10-04 for Stripe.
When I try to curl for a response using native Stripe API for the above customer is, I get 2018-05-21 as the version.
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.