- Is this your first time deploying Airbyte?: No
- OS Version / Instance: Ubuntu 20.04 / e2-standard-8
- Memory / Disk: 32GB / 50GB
- Deployment: Docker
- Airbyte Version: 0.38.1
- Source name/version: mysql 0.4.8, postgresql 0.4.15, mssql 0.3.22
- Destination name/version: bigquery 1.1.5
- Step: The issue is happening during sync
- Description:
Airbyte started throwing error on Wednesday, May 11th 2022. Every connection with BigQuery as destination are affected by this error. Sometimes the sync run successfully, but it was failed most of the time.
The errors:
2022-05-13 10:43:28 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):313 - Records read: 5273000 (2 GB)
2022-05-13 10:43:28 e[44msourcee[0m > 2022-05-13 10:43:27 e[32mINFOe[m i.a.i.s.r.AbstractDbSource(lambda$read$2):121 - {} - Closing database connection pool.
2022-05-13 10:43:28 e[44msourcee[0m > 2022-05-13 10:43:27 e[32mINFOe[m i.a.i.s.r.AbstractDbSource(lambda$read$2):123 - {} - Closed database connection pool.
2022-05-13 10:43:28 e[44msourcee[0m > 2022-05-13 10:43:27 e[32mINFOe[m i.a.i.b.IntegrationRunner(run):133 - {} - Completed integration: io.airbyte.integrations.base.ssh.SshWrappedSource
2022-05-13 10:43:28 e[44msourcee[0m > 2022-05-13 10:43:27 e[32mINFOe[m i.a.i.s.m.MySqlSource(main):249 - {} - completed source: class io.airbyte.integrations.source.mysql.MySqlSource
2022-05-13 10:43:28 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):324 - Total records read: 5273315 (2 GB)
2022-05-13 10:43:28 e[32mINFOe[m i.a.w.DefaultReplicationWorker(run):166 - One of source or destination thread complete. Waiting on the other.
2022-05-13 10:43:28 e[43mdestinatione[0m > 2022-05-13 10:43:28 e[32mINFOe[m i.a.i.b.FailureTrackingAirbyteMessageConsumer(close):65 - Airbyte message consumer: succeeded.
2022-05-13 10:43:28 e[43mdestinatione[0m > 2022-05-13 10:43:28 e[32mINFOe[m i.a.i.d.b.BigQueryRecordConsumer(close):58 - Started closing all connections
2022-05-13 10:43:28 e[43mdestinatione[0m > 2022-05-13 10:43:28 e[32mINFOe[m i.a.i.d.b.u.AbstractBigQueryUploader(close):78 - Closing connector: AbstractBigQueryUploader{table=_airbyte_raw_phone_verification, tmpTable=_airbyte_tmp_yph_phone_verification, syncMode=WRITE_TRUNCATE, writer=class io.airbyte.integrations.destination.bigquery.writer.BigQueryTableWriter, recordFormatter=class io.airbyte.integrations.destination.bigquery.formatter.DefaultBigQueryRecordFormatter}
2022-05-13 10:43:28 e[43mdestinatione[0m > 2022-05-13 10:43:28 e[1;31mERRORe[m i.a.i.d.b.u.AbstractBigQueryUploader(close):87 - Failed to close AbstractBigQueryUploader{table=_airbyte_raw_phone_verification, tmpTable=_airbyte_tmp_yph_phone_verification, syncMode=WRITE_TRUNCATE, writer=class io.airbyte.integrations.destination.bigquery.writer.BigQueryTableWriter, recordFormatter=class io.airbyte.integrations.destination.bigquery.formatter.DefaultBigQueryRecordFormatter} writer,
2022-05-13 10:43:28 e[43mdestinatione[0m > details: 400 Bad Request
2022-05-13 10:43:28 e[43mdestinatione[0m > PUT https://www.googleapis.com/upload/bigquery/v2/projects/myproject/jobs?uploadType=resumable&upload_id=ADPycds0Jt0MYHFKyN9k0gKGG4aoWtzr8jscxBuPyUtsvQ_0LN2NebXn9WnT0R57lJMFpQgn4nlCb_m18cxkFUN_WsCDkjyKli92
2022-05-13 10:43:32 e[43mdestinatione[0m > {
2022-05-13 10:43:32 e[43mdestinatione[0m > "error": {
2022-05-13 10:43:32 e[43mdestinatione[0m > "code": 400,
2022-05-13 10:43:32 e[43mdestinatione[0m > "message": "Invalid credential",
2022-05-13 10:43:32 e[43mdestinatione[0m > "errors": [
2022-05-13 10:43:32 e[43mdestinatione[0m > {
2022-05-13 10:43:32 e[43mdestinatione[0m > "message": "Invalid credential",
2022-05-13 10:43:32 e[43mdestinatione[0m > "domain": "global",
2022-05-13 10:43:32 e[43mdestinatione[0m > "reason": "invalid"
2022-05-13 10:43:32 e[43mdestinatione[0m > }
2022-05-13 10:43:32 e[43mdestinatione[0m > ],
2022-05-13 10:43:32 e[43mdestinatione[0m > "status": "INVALID_ARGUMENT"
2022-05-13 10:43:32 e[43mdestinatione[0m > }
2022-05-13 10:43:32 e[43mdestinatione[0m > }
2022-05-13 10:43:32 e[43mdestinatione[0m >
2022-05-13 10:43:32 e[43mdestinatione[0m > java.lang.RuntimeException: com.google.cloud.bigquery.BigQueryException: 400 Bad Request
logs-3205.txt (977.0 KB)