Schema Discovery Issue in Local Docker Environment vs. Airbyte Cloud

Slack Knowledge Base Q&A API, Terraform and Other Topics
, , , ,
1

Summary

The user is facing a schema discovery failure in a local Docker environment when setting up a connector built using the Builder tool. However, the schema discovery succeeds when the connector is exported and imported into Airbyte cloud. The issue was discovered when trying to add new tables to an active production pipeline.


Question

I have a connector that I built using the Builder. If I build it in a local docker environment, the schema discovery fails during connector setup. I have tried this on multiple builds. If I export the connector and import it into Airbyte cloud then the schema discovery doesn’t fail. Any ideas on why it fails in local but succeeds in cloud? <@U01MMSDJGC9> <@U01AB6V6NMQ>

I have been using this connector for a few months in an active production pipeline. I discovered the issue when I tried to add some new tables, deployed the connector but can’t pull in the new tables because the schema won’t refresh.


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.

Join the conversation on Slack

["schema-discovery", "local-docker-environment", "builder", "connector-setup", "airbyte-cloud", "production-pipeline"]
2

What version are you running locally?

Can you try and make a new connection using the new connector version? Would schemas work there?

3

I ran into the same issue in both 0.50.54 (the instance I have been running for a while) and 0.57.3 (a fresh install). Same issue in both

4

It seems like a timeout or limits issue. The connection is hitting a rest endpoint and parsing the returning JSON. The connection has 33 endpoints/feeds. I found that if I break it down into groups of 5 or 6 then the schema refresh works. If you get up to 10 or 11 then it fails again

5

The schema refresh did work in airbyte cloud.

Thanks <@U069EMNRPA4>!

6

I haven’t been able to update my production instance from 0.50.54 to 0.57.3 because of the docker breaking change and I haven’t figured out how to bypass the permissions issue.

7

So schema refresh in Builder fails?

8

Definitely file an issue. I’ll be happy to groom and work on this with the team.

9

Happy to! Is there any particular procedure I should follow?

10

just a gh issue on https://github.com/airbytehq/airbyte|https://github.com/airbytehq/airbyte