- Is this your first time deploying Airbyte?: No
- Deployment: Kubernetes
- Airbyte Version: What version are you using now? 0.39.34-alpha
- Source name/version: MSSQL 0.4.20
- Destination name/version: S3 0.3.16 → parquet, snappy
- Step: Incremental Sync
trying to sync data from MSSQL (CDC) to S3. When I set the sync to be a full sync, it works fine, but as soon as I put it to incremental, it fails with the following message:
2022-10-12 09:53:41 - Additional Failure Information: tech.allegro.schema.json2avro.converter.AvroConversionException: Failed to convert JSON to Avro: Could not evaluate union, field Location is expected to be one of these: NULL, DOUBLE. If this is a complex type, check if offending field (path: Location) adheres to schema: 9.447
It fails on a number of tables, all with the same type of data (double) that looks numeric.
Hi @Huib! First things first - could you please upgrade your Airbyte version? We get updates so often that just doing that might fix the issue.
A few ideas:
Have you seen the docs on Avro - JSON conversion?
As a workaround, could you try using JSON or CSV while this gets resolved?
I’ve found a related issue on GitHub:
If upgrading Airbyte doesn’t resolve this, I would suggest using the workaround for now as I can’t tell you where this is on the roadmap!
Thanks! I’ll try updating first, I’ll report back with the results.
Great, looking forward to it!
Updating Airbyte and moving to the Databricks destination connector fixed the issue - I am now able to sync the large tables reliably to our data lake Thanks!
That is wonderful to hear! Thank you for the update