Destination BigQuery: 400 Bad Request Invalid credential

Troubleshooting
, , ,
#1

Is this your first time deploying Airbyte : No
OS Version / Instance: EC2 t2.large
Deployment : EC2 Docker
Airbyte Version : 0.35.4-alpha
Source name/version : Postgres 0.4.14
Destination name/version : BigQuery 1.0.1
Description : BigQuery destination fails to close and causes sync failures

Hi,
Since this morning my Postgres to BQ connection is failing because of the below error. Strangely the other connections to BQ are not failing even though the error seems to be BQ related.

It is an Incremental Dedup connection with the standard Loading Method.

Do you know how to fix this ?

Thanks

The error is the following:

Failed to close AbstractBigQueryUploader{table=_airbyte_raw_share, tmpTable=_airbyte_tmp_dig_share, syncMode=WRITE_APPEND, writer=class io.airbyte.integrations.destination.bigquery.writer.BigQueryTableWriter, recordFormatter=class io.airbyte.integrations.destination.bigquery.formatter.DefaultBigQueryRecordFormatter} writer, 
details: 400 Bad Request
PUT https://www.googleapis.com/upload/bigquery/v2/projects/pledg-331014/jobs?uploadType=resumable&upload_id=ADPycdsKhyT9oLRZsfCeIIpq9Znkgpuz5wAz2PLhx1-f1_937uNcnu5GHAqJ2Hcmv33gZp1c2ADSZiBJ8JKESguZACSg3Q
i.a.v.j.JsonSchemaValidator(test):56 - JSON schema validation failed. 
errors: $: string found, object expected
2022-05-12 07:51:18 e[1;31mERRORe[m i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$1):70 - Validation failed: "error"
2022-05-12 07:51:18 e[32mINFOe[m i.a.v.j.JsonSchemaValidator(test):56 - JSON schema validation failed. 
errors: $: string found, object expected
Airbyte/destination-bigquery 400 Bad Request Invalid credential
#2

Hey can you share the complete logs

#3

Note that I have the same error after upgrading the destination connector to BigQuery 1.1.4
It’s happening approx 75% of the time. And it corrupts the destination data because the cursor gets updated but not the data in the destination so the data since the first occurence of this bug is lost.
Is there any way to manually override the value of the cursor ?
Thanks

#4

logs-24695-2.txt (523.2 KB)

Here you are @harshith

Bigquery details: 400 Bad Request
#5

Hey ! Since this night, I encounter an issue almost similar (throwing a JSON schema validation failed error) for my postgreSQL <-> BigQuery connection.

I tried to re-run many times but all my jobs failed since then. It’s odd because I did not make any update on my stack since at least a week so I don’t see what could be the cause.

Not sure if it’s the same error or not but as I first encountered this issue last night and @anatole posted it a few hours ago (+ both logs mention JSON validation error) , I felt it might exits a link between our issues so I thought my logfile could add some input to find a solution.

Here is my logs :

logs-286.txt (729.4 KB)

#6

I have/had the same issue. Airbyte is behaving very strangely and the behaviour of Full Refresh | Overwrite syncs are not consistent, even when source data is identical.

#7

You can manually override the value of the cursor by running the query mentioned in this thread: Update the cursor manually breaks my connection - #6 by boggdan

#8

Can you describe more the issue you’re having @maxkrog ? (if possible create another topic for that)