Failed to fetch schema. Please try again (Postgres to BigQuery)

  • Is this your first time deploying Airbyte?: No
  • OS Version / Instance: GCP
  • Memory / Disk: Node 1: 40GB, Node 2: 7.77 GB, Node 3: 7.77 GB, Node 4: 7.77 GB
  • Deployment: Plural.sh uses Kubernetes
  • Airbyte Version: OSS * 0.35.5-alpha
  • Source name/version: Postgres
  • Destination name/version: BigQuery
  • Step: The issue is happening during setup connection
  • Description: The error says “Failed to fetch schema. Please try again.”

The whole log file is 80 MB can’t upload here.

So I just searched for the keyword ERROR. Hope this helps!

Please help! Thank you!
airbyte-log-ERROR-only.txt (6.6 KB)

Hello,
Could you please check the network calls in your browser’s inspector and check the response of the discover_schema calls?
Could you also double-check the permissions of your Postgres user, it must of reading permissions on the tables you want to replicate?
Feel free to restart the server and try again to have a lighter and shareable log file.
I can’t provide much more help if I don’t have the logs.

Hi @alafanechere ,

Here’s the response for discover_schema:

Yes I’m actually using like an admin user to our Postgres so I have access to everything.

How can I restart the server? I’m using plural.sh to spin up Airbyte OSS.

Thanks!

Hi @alafanechere ,

Here’s the full log.

airbyte-logs-postgres-to-bigquery.txt (8.4 KB)

Hi @alafanechere ,

Here’s another helpful logs. Seems like we hit the max resource allocated.

airbyte-logs-max-resource.txt (4.2 KB)

There might be some specifities around the deployment of Airbyte that I’m not aware of.
It would be great if you could get the logs of the discover container that was spawned when you set up the connection.
I’m not sure that the error from airbyte-logs-max-resource.txt is related to the discover schema error, but it might be worth fixing it first for peace of mind:
“message” : “Invalid bucket name: ‘{env:GCS_LOG_BUCKET}’” it means you did not correctly set the GCS_LOG_BUCKET value.

Thanks @alafanechere ! Let’s separate some concerns.

Where and how can we fix the GCS_LOG_BUCKET issue?

Regarding the max resource consumption, is this something we can fix in Airbyte?

Where and how can we fix the GCS_LOG_BUCKET issue?

GCS_LOG_BUCKET is an environment variable that is mapped to the scheduler, server and worker pods.
I think you can edit this value from Helm, but it might also be a default set by Plural, so please reach out to them to check if this is something you can fix yourself or if it’s a known bug from their side.

Regarding the max resource consumption, is this something we can fix in Airbyte?

Here’s another helpful logs. Seems like we hit the max resource allocated.

I can’t find this error in your logs, could you please show mewhere you found it?

Hi @alafanechere ,

Here’s the max resource error log.

Hi @alafanechere ,

Here’s an image of max resource for your reference:

Could you please share the full file of the logs you screenshoted?