Stripe /Zendesk to MySQL 8 destination schema validation errors

Connector Questions & Issues
, , ,
1
  • Is this your first time deploying Airbyte?: Yes
  • OS Version / Instance: Ubuntu 22.04 LTS
  • Memory / Disk: 32GB / 100GB
  • Deployment: Docker
  • Airbyte Version: What version are you using now?
  • Source name/version:
  • Destination name/version:
  • Step: The issue is happening during sync.
  • Description:

This is a brand new setup. Ubuntu 22.04 LTS with the default docker compose file from Airbyte.

When syncing Stripe or Zendesk into MySQL 8, I run into errors like this:

  compiled SQL at ../build/run/airbyte_utils/models/generated/airbyte_tables/airbyte_zendesk_support/tickets.sql,retryable=<null>,timestamp=1675332845916], io.airbyte.config.FailureReason@678c78f8[failureOrigin=normalization,failureType=system_error,internalMessage=1292 (22007): Truncated incorrect CHAR(1646) value: 'Hi x,,externalMessage=Normalization failed during the dbt run. This may indicate a problem with the data itself.,metadata=io.airbyte.config.Metadata@4ffe2fd[additionalProperties={attemptNumber=2, jobId=280, from_trace_message=true}],stacktrace=AirbyteDbtError: 
1 of 7 ERROR creating table model airbyte_zendesk_support.tickets....................................................... [ERROR in 0.24s]
Database Error in model tickets (models/generated/airbyte_tables/airbyte_zendesk_support/tickets.sql)
  1292 (22007): Truncated incorrect CHAR(1646) value: 'Hi c

2023-02-02 10:13:37 e[33mWARNe[m i.a.w.g.DefaultReplicationWorker(lambda$readFromSrcAndWriteToDstRunnable$5):395 - Schema validation errors found for stream _charges. Error messages: [$.dispute is of an incorrect type. Expected it to be [null, string]]

2023-02-02 10:13:56 e[42mnormalizatione[0m > e[33mDatabase Error in model charges (models/generated/airbyte_tables/airbyte_stripe_temp/charges.sql)e[0m
2023-02-02 10:13:56 e[42mnormalizatione[0m >   1292 (22007): Truncated incorrect CHAR(1024) value: '{"id": "du_1v798svxckhjj", "amount": 29000, "charge": "ch_skdjfh8927345hjk", "object": "dispute", "reason": "'
2023-02-02 10:13:56 e[42mnormalizatione[0m >   compiled SQL at ../build/run/airbyte_utils/models/generated/airbyte_tables/airbyte_stripe_temp/charges.sql

I’m starting to think flattening JSONs into MySQL is perhaps not the best option. My goal is to have data from Stripe and Zendesk in a database so I can visualize it with Grafana. Would MongoDB or Postgres be a better solution?

2

Probably ElasticSearch where you can plug Kibana easily, not sure about if Grafana connects to ElasticSearch.

3

Grafana has support for ElasticSearch, so it is possible. It is less likely to run into issues when you use ELK as the destination?

Are the errors I see with MySQL common?

4

MySQL sometimes can be complicated to handle some data types :frowning: but the connectoin btw Zendesk to Mysql should be stable though