Airbyte Docker Image Connection Issue with MSSQL Source to BigQuery Destination

Summary

The user is facing an issue where the Airbyte tool breaks on the discovery step when trying to create a connection between a MSSQL source and a BigQuery destination, likely due to the source database having more than 2700 tables.


Question

Hi All!!
I am running an airbyte docker image with docker compose in an e2-standard-8 compute engine vm, I am trying to create a connection with a mssql source to a bigquery destination (both source and destination were created without any issue), but the tool never completes the task, it breaks on the discovery step for all the streams in the database due to the source database having more that 2700 tables. Any ideas on how can I resolve this issue?



This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

["airbyte-docker-image", "mssql-source", "bigquery-destination", "discovery-step", "2700-tables"]

Being precise and providing details is useful in increasing your chances to get your issue solved :wink:
How does the discovery step break? Is it timeout? Out of memory issue? Do you have logs from container doing discovery, worker? Do you use load balancer in front of your Airbyte deployment? Which Airbyte version do you use?