Is this your first time deploying Airbyte : No
OS Version / Instance: EC2 t2.large
Deployment : EC2 Docker
Airbyte Version : 0.35.4-alpha
Source name/version : Postgres 0.4.14
Destination name/version : BigQuery 1.0.1
Description : BigQuery destination fails to close and causes sync failures
Hi,
Since this morning my Postgres to BQ connection is failing because of the below error. Strangely the other connections to BQ are not failing even though the error seems to be BQ related.
It is an Incremental Dedup connection with the standard Loading Method.
Do you know how to fix this ?
Thanks
The error is the following:
Failed to close AbstractBigQueryUploader{table=_airbyte_raw_share, tmpTable=_airbyte_tmp_dig_share, syncMode=WRITE_APPEND, writer=class io.airbyte.integrations.destination.bigquery.writer.BigQueryTableWriter, recordFormatter=class io.airbyte.integrations.destination.bigquery.formatter.DefaultBigQueryRecordFormatter} writer,
details: 400 Bad Request
PUT https://www.googleapis.com/upload/bigquery/v2/projects/pledg-331014/jobs?uploadType=resumable&upload_id=ADPycdsKhyT9oLRZsfCeIIpq9Znkgpuz5wAz2PLhx1-f1_937uNcnu5GHAqJ2Hcmv33gZp1c2ADSZiBJ8JKESguZACSg3Q
i.a.v.j.JsonSchemaValidator(test):56 - JSON schema validation failed.
errors: $: string found, object expected
2022-05-12 07:51:18 e[1;31mERRORe[m i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$1):70 - Validation failed: "error"
2022-05-12 07:51:18 e[32mINFOe[m i.a.v.j.JsonSchemaValidator(test):56 - JSON schema validation failed.
errors: $: string found, object expected
Hey can you share the complete logs
Note that I have the same error after upgrading the destination connector to BigQuery 1.1.4
It’s happening approx 75% of the time. And it corrupts the destination data because the cursor gets updated but not the data in the destination so the data since the first occurence of this bug is lost.
Is there any way to manually override the value of the cursor ?
Thanks
logs-24695-2.txt (523.2 KB)
Here you are @harshith
Hey ! Since this night, I encounter an issue almost similar (throwing a JSON schema validation failed error) for my postgreSQL <-> BigQuery connection.
I tried to re-run many times but all my jobs failed since then. It’s odd because I did not make any update on my stack since at least a week so I don’t see what could be the cause.
Not sure if it’s the same error or not but as I first encountered this issue last night and @anatole posted it a few hours ago (+ both logs mention JSON validation error) , I felt it might exits a link between our issues so I thought my logfile could add some input to find a solution.
Here is my logs :
logs-286.txt (729.4 KB)
I have/had the same issue. Airbyte is behaving very strangely and the behaviour of Full Refresh | Overwrite syncs are not consistent, even when source data is identical.
You can manually override the value of the cursor by running the query mentioned in this thread: Update the cursor manually breaks my connection - #6 by boggdan
Can you describe more the issue you’re having @maxkrog ? (if possible create another topic for that)
Hi there from the Community Assistance team.
We’re letting you know about an issue we discovered with the back-end process we use to handle topics and responses on the forum. If you experienced a situation where you posted the last message in a topic that did not receive any further replies, please open a new topic to continue the discussion. In addition, if you’re having a problem and find a closed topic on the subject, go ahead and open a new topic on it and we’ll follow up with you. We apologize for the inconvenience, and appreciate your willingness to work with us to provide a supportive community.