Issue with MSSQL to BigQuery Connection in Airbyte

Slack Knowledge Base Q&A API, Terraform and Other Topics
, , , ,
1

Summary

User is experiencing a failure during the discovery step when creating a connection from MSSQL to BigQuery due to a large number of tables (over 2700) in the source database while using Airbyte in a Docker environment.


Question

Hi All!!
I am running an airbyte docker image with docker compose in an e2-standard-8 compute engine vm, I am trying to create a connection with a mssql source to a bigquery destination (both source and destination were created without any issue), but the tool never completes the task, it breaks on the discovery step for all the streams in the database due to the source database having more that 2700 tables. Any ideas on how can I resolve this issue?


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

['airbyte', 'docker', 'mssql', 'bigquery', 'discovery-step', 'connection-issue']
2

Being precise and providing details is useful in increasing your chances to get your issue solved :wink:
How does the discovery step break? Is it timeout? Out of memory issue? Do you have logs from container doing discovery, worker? Do you use load balancer in front of your Airbyte deployment? Which Airbyte version do you use?