Failed to fetch schema. Please try again (Postgres to BigQuery)

Connector Questions & Issues
1
  • Is this your first time deploying Airbyte?: No
  • OS Version / Instance: GCP
  • Memory / Disk: Node 1: 40GB, Node 2: 7.77 GB, Node 3: 7.77 GB, Node 4: 7.77 GB
  • Deployment: Plural.sh uses Kubernetes
  • Airbyte Version: OSS * 0.35.5-alpha
  • Source name/version: Postgres
  • Destination name/version: BigQuery
  • Step: The issue is happening during setup connection
  • Description: The error says “Failed to fetch schema. Please try again.”

image
image1920×1098 33 KB

The whole log file is 80 MB can’t upload here.

So I just searched for the keyword ERROR. Hope this helps!

Please help! Thank you!
airbyte-log-ERROR-only.txt (6.6 KB)

2

Hello,
Could you please check the network calls in your browser’s inspector and check the response of the discover_schema calls?
Could you also double-check the permissions of your Postgres user, it must of reading permissions on the tables you want to replicate?
Feel free to restart the server and try again to have a lighter and shareable log file.
I can’t provide much more help if I don’t have the logs.

3

Hi @alafanechere ,

Here’s the response for discover_schema:

Screen Shot 2022-05-06 at 9.38.53 AM
Screen Shot 2022-05-06 at 9.38.53 AM1072×70 11.7 KB

Yes I’m actually using like an admin user to our Postgres so I have access to everything.

How can I restart the server? I’m using plural.sh to spin up Airbyte OSS.

Thanks!

4

Hi @alafanechere ,

Here’s the full log.

airbyte-logs-postgres-to-bigquery.txt (8.4 KB)

5

Hi @alafanechere ,

Here’s another helpful logs. Seems like we hit the max resource allocated.

airbyte-logs-max-resource.txt (4.2 KB)

6

There might be some specifities around the deployment of Airbyte that I’m not aware of.
It would be great if you could get the logs of the discover container that was spawned when you set up the connection.
I’m not sure that the error from airbyte-logs-max-resource.txt is related to the discover schema error, but it might be worth fixing it first for peace of mind:
“message” : “Invalid bucket name: ‘{env:GCS_LOG_BUCKET}’” it means you did not correctly set the GCS_LOG_BUCKET value.

7

Thanks @alafanechere ! Let’s separate some concerns.

Where and how can we fix the GCS_LOG_BUCKET issue?

8

Regarding the max resource consumption, is this something we can fix in Airbyte?

9

Where and how can we fix the GCS_LOG_BUCKET issue?

GCS_LOG_BUCKET is an environment variable that is mapped to the scheduler, server and worker pods.
I think you can edit this value from Helm, but it might also be a default set by Plural, so please reach out to them to check if this is something you can fix yourself or if it’s a known bug from their side.

Regarding the max resource consumption, is this something we can fix in Airbyte?

Here’s another helpful logs. Seems like we hit the max resource allocated.

I can’t find this error in your logs, could you please show mewhere you found it?

10

Hi @alafanechere ,

Here’s the max resource error log.

11

Hi @alafanechere ,

Here’s an image of max resource for your reference:

Screen Shot 2022-05-09 at 7.42.21 PM
Screen Shot 2022-05-09 at 7.42.21 PM1920×988 282 KB

12

Could you please share the full file of the logs you screenshoted?

13

Hi there from the Community Assistance team.
We’re letting you know about an issue we discovered with the back-end process we use to handle topics and responses on the forum. If you experienced a situation where you posted the last message in a topic that did not receive any further replies, please open a new topic to continue the discussion. In addition, if you’re having a problem and find a closed topic on the subject, go ahead and open a new topic on it and we’ll follow up with you. We apologize for the inconvenience, and appreciate your willingness to work with us to provide a supportive community.