Stripe /Zendesk to MySQL 8 destination schema validation errors

  • Is this your first time deploying Airbyte?: Yes
  • OS Version / Instance: Ubuntu 22.04 LTS
  • Memory / Disk: 32GB / 100GB
  • Deployment: Docker
  • Airbyte Version: What version are you using now?
  • Source name/version:
  • Destination name/version:
  • Step: The issue is happening during sync.
  • Description:

This is a brand new setup. Ubuntu 22.04 LTS with the default docker compose file from Airbyte.

When syncing Stripe or Zendesk into MySQL 8, I run into errors like this:

  compiled SQL at ../build/run/airbyte_utils/models/generated/airbyte_tables/airbyte_zendesk_support/tickets.sql,retryable=<null>,timestamp=1675332845916], io.airbyte.config.FailureReason@678c78f8[failureOrigin=normalization,failureType=system_error,internalMessage=1292 (22007): Truncated incorrect CHAR(1646) value: 'Hi x,,externalMessage=Normalization failed during the dbt run. This may indicate a problem with the data itself.,metadata=io.airbyte.config.Metadata@4ffe2fd[additionalProperties={attemptNumber=2, jobId=280, from_trace_message=true}],stacktrace=AirbyteDbtError: 
1 of 7 ERROR creating table model [ERROR in 0.24s]
Database Error in model tickets (models/generated/airbyte_tables/airbyte_zendesk_support/tickets.sql)
  1292 (22007): Truncated incorrect CHAR(1646) value: 'Hi c
2023-02-02 10:13:37 e[33mWARNe[m i.a.w.g.DefaultReplicationWorker(lambda$readFromSrcAndWriteToDstRunnable$5):395 - Schema validation errors found for stream _charges. Error messages: [$.dispute is of an incorrect type. Expected it to be [null, string]]

2023-02-02 10:13:56 e[42mnormalizatione[0m > e[33mDatabase Error in model charges (models/generated/airbyte_tables/airbyte_stripe_temp/charges.sql)e[0m
2023-02-02 10:13:56 e[42mnormalizatione[0m >   1292 (22007): Truncated incorrect CHAR(1024) value: '{"id": "du_1v798svxckhjj", "amount": 29000, "charge": "ch_skdjfh8927345hjk", "object": "dispute", "reason": "'
2023-02-02 10:13:56 e[42mnormalizatione[0m >   compiled SQL at ../build/run/airbyte_utils/models/generated/airbyte_tables/airbyte_stripe_temp/charges.sql

I’m starting to think flattening JSONs into MySQL is perhaps not the best option. My goal is to have data from Stripe and Zendesk in a database so I can visualize it with Grafana. Would MongoDB or Postgres be a better solution?

Probably ElasticSearch where you can plug Kibana easily, not sure about if Grafana connects to ElasticSearch.

Grafana has support for ElasticSearch, so it is possible. It is less likely to run into issues when you use ELK as the destination?

Are the errors I see with MySQL common?

MySQL sometimes can be complicated to handle some data types :frowning: but the connectoin btw Zendesk to Mysql should be stable though