Failed to load data to BigQuery destination due to missing table

Troubleshooting
,
#1

Hi there!

I am having an issue when syncing data from Postgres to BigQuery.

  • Is this your first time deploying Airbyte?: Yes
  • OS Version / Instance: Ubuntu
  • Memory / Disk: you can use something like 8Gb / 1 Tb
  • Deployment: EC2 with docker-compose
  • Airbyte Version: 0.36.0-alpha
  • Source name/version: Postgres [airbyte/source-postgres] 0.4.10
  • Destination name/version: [airbyte/destination-bigquery] 1.1.0
  • Step: the sync fails because Airbyte complains about a table it should have created itself does not exist.
  • Description: When syncing Postgres with BigQuery, Airbyte fails during the transformation step because it complains about a raw table does not exist. This table, as far as I can tell, should have been created by Airbyte itself in a previous step.
404 Not found: Table recap-dev-347108:staging._airbyte_raw_company_contract was not found in location US

You can find the logs attached.

logs-4.txt (257.7 KB)

#2

Are you using GCS Staging? If yes, could you change to default Insert method?
From your logs errors:

2022-04-19 10:29:31 e[43mdestinatione[0m > com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: null; S3 Extended Request ID: null; Proxy: null)

We use S3 Java library to handle permissions/credentials for Cloud Buckets.

#4

I am also facing this same issue and I am NOT using GCS Staging.
In the dataset few tables are created with data but getting error in some table

[0m > 09:19:17.259653 [error] [MainThread]: 404 Not found: Table xxxx-realm-122345:stripe_data._airbyte_raw_disputes was not found in location US