JSON schema validation failed while syncing MySQL CDC -> BigQuery

Troubleshooting
, , ,
#1
  • Is this your first time deploying Airbyte?: Yes
  • OS Version / Instance: Debian
  • Memory / Disk: 4GB / 50GB
  • Deployment: Docker
  • Airbyte Version: 0.36.1-alpha
  • Source name/version: MySQL 0.5.6
  • Destination name/version: BigQuery (denormalized typed struct) 0.3.1
  • Step: Sync

Description:

When I first set up my connection, it appeared to hang at the 2022-04-22 14:22:06 source > 2022-04-22 14:22:05 INFO i.a.i.s.m.MySqlSource(main):208 - completed source: class io.airbyte.integrations.source.mysql.MySqlSource step of the initial sync.

I cancelled it after 23 hours to restart a sync from scratch, and it took two attempts for succeeding. I say “succeed” as it’s the final state of the sync, but the last two lines of the logs are the following:

2022-04-23 13:33:07 INFO i.a.v.j.JsonSchemaValidator(test):56 - JSON schema validation failed. 
errors: $.credential: is not defined in the schema and the schema does not allow additional properties, $.part_size_mb: is not defined in the schema and the schema does not allow additional properties, $.gcs_bucket_name: is not defined in the schema and the schema does not allow additional properties, $.gcs_bucket_path: is not defined in the schema and the schema does not allow additional properties, $.keep_files_in_gcs-bucket: is not defined in the schema and the schema does not allow additional properties, $.method: must be a constant value Standard

Even since that first sync, none of my subsequent syncs (I’ve set it up as Incremental | Append) appear to work, as I encounter the errors: $.credential: is not defined in the schema and the schema does not allow additional properties, $.part_size_mb: is not defined in the schema and the schema does not allow additional properties, $.gcs_bucket_name: is not defined in the schema and the schema does not allow additional properties, $.gcs_bucket_path: is not defined in the schema and the schema does not allow additional properties, $.keep_files_in_gcs-bucket: is not defined in the schema and the schema does not allow additional properties, $.method: must be a constant value Standard again each time.

I’m attaching success logs for the initial sync (logs-2) along with the logs for the last incremental sync (logs-4).

Thanks!

logs-2.txt (117.4 KB)
logs-4.txt (325.8 KB)

#2

@xavierosee looks to me like a resource issue. Can you try increasing the memory and try syncing it again?

#3

Hi @harshith and thank you!

I’ve upgraded memory to 8GB, and ran a new ìnitial sync` yesterday, 18 hours ago.

It’s now stuck to 2022-04-25 16:02:12 source > 2022-04-25 16:02:11 INFO i.a.i.s.m.MySqlSource(main):208 - completed source: class io.airbyte.integrations.source.mysql.MySqlSource since yesterday. Do you still believe it’s a memory issue ?

#4

Hey @xavierosee do you see DEADLINE_EXCEEDED similar to log_4 ?

#5

@harshith no, but the sync is still on running… I could try cancelling it and relaunching a new one to check…

#6

yeah do you mind retrying it

#7

It did succeed right now, even though I still have an error on the last line :face_with_head_bandage: (logs-9)

I manually triggered another sync right after and that worked as well though still with the same error (logs-10). I’ll keep monitoring but it’s looking like it’s flowing =)
logs-10.txt (219.3 KB)
logs-9.txt (677.3 KB)

#8

@xavierosee is the sync getting successful ?

#9

Yes absolutely.

Do you know what this error is about though ? I don’t understand how it can be a successful sync if they all end with this error…

errors: 
$.credential: is not defined in the schema and the schema does not allow additional properties, 
$.part_size_mb: is not defined in the schema and the schema does not allow additional properties, 
$.gcs_bucket_name: is not defined in the schema and the schema does not allow additional properties, 
$.gcs_bucket_path: is not defined in the schema and the schema does not allow additional properties, 
$.keep_files_in_gcs-bucket: is not defined in the schema and the schema does not allow additional properties, 
$.method: must be a constant value Standard
#10

Got it. There is an issue created around this All connections are failing to BigQuery · Issue #12162 · airbytehq/airbyte · GitHub kindly request you to follow it up.

1 Like