Error during MSSQL schema discovery in Azure Synapse connection

Slack Knowledge Base Q&A Connector Questions
, , , ,
1

Summary

Error occurs during MSSQL schema discovery in Azure Synapse connection due to nested SQL statement issue. User suspects it may be related to the number of tables in the database, but even with a limited schema, the error persists.


Question

I’m trying to set up a connection to a MSSQL connection to Azure Synapse. The Airbyte source is created correctly but during the schema discovery step I get the error Some part of your SQL statement is nested too deeply. Rewrite the query or break it up into smaller queries..
I thought this could be caused by the number of tables in the database but even when I limited it to a small schema in the source configuration the error persists. Does anyone have a solution for it or it should be considered as a bug?
I’m running Airbyte 0.63.8 deployed with helm chart.

I found some old github issue https://github.com/airbytehq/airbyte/issues/4546 with the same error that was pointed to generic large schema fetching issue: https://github.com/airbytehq/airbyte/issues/4564. But that one seems to be related to API timeouts or HTTP server limits whereas the MS SQL schema discovery clearly returns an SQL error.


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

["mssql-connector", "azure-synapse", "schema-discovery", "nested-sql-statement", "error", "bug", "helm-chart"]
2 
        at com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDatabaseError(SQLServerException.java:261)
        at com.microsoft.sqlserver.jdbc.SQLServerStatement.getNextResult(SQLServerStatement.java:1752)
        at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.doExecutePreparedStatement(SQLServerPreparedStatement.java:675)
        at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement$PrepStmtExecCmd.doExecute(SQLServerPreparedStatement.java:594)
        at com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:7739)
        at com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:4384)
        at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeCommand(SQLServerStatement.java:293)
        at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeStatement(SQLServerStatement.java:263)
        at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.executeQuery(SQLServerPreparedStatement.java:509)
        at com.microsoft.sqlserver.jdbc.SQLServerDatabaseMetaData.getColumns(SQLServerDatabaseMetaData.java:759)
        at com.zaxxer.hikari.pool.ProxyDatabaseMetaData.getColumns(ProxyDatabaseMetaData.java:107)
        at com.zaxxer.hikari.pool.HikariProxyDatabaseMetaData.getColumns(HikariProxyDatabaseMetaData.java)
        at io.airbyte.cdk.integrations.source.jdbc.AbstractJdbcSource.discoverInternal$lambda$8(AbstractJdbcSource.kt:375)
        at io.airbyte.cdk.db.jdbc.DefaultJdbcDatabase.bufferedResultSetQuery(DefaultJdbcDatabase.kt:39)
        at io.airbyte.cdk.integrations.source.jdbc.AbstractJdbcSource.discoverInternal(AbstractJdbcSource.kt:373)
        at io.airbyte.cdk.integrations.source.jdbc.AbstractJdbcSource.discoverInternal(AbstractJdbcSource.kt:487)
        at io.airbyte.integrations.source.mssql.MssqlSource.discoverInternal(MssqlSource.java:235)
        at io.airbyte.integrations.source.mssql.MssqlSource.discoverInternal(MssqlSource.java:77)
        at io.airbyte.cdk.integrations.source.relationaldb.AbstractDbSource.discoverWithoutSystemTables(AbstractDbSource.kt:302)
        at io.airbyte.cdk.integrations.source.relationaldb.AbstractDbSource.discover(AbstractDbSource.kt:93)
        at io.airbyte.cdk.integrations.source.jdbc.AbstractJdbcSource.discover(AbstractJdbcSource.kt:118)
        at io.airbyte.integrations.source.mssql.MssqlSource.discover(MssqlSource.java:216)
        at io.airbyte.cdk.integrations.base.ssh.SshWrappedSource.discover$lambda$1(SshWrappedSource.kt:58)
        at io.airbyte.cdk.integrations.base.ssh.SshTunnel$Companion.sshWrap(SshTunnel.kt:545)
        at io.airbyte.cdk.integrations.base.ssh.SshWrappedSource.discover(SshWrappedSource.kt:58)
        at io.airbyte.cdk.integrations.base.IntegrationRunner.runInternal(IntegrationRunner.kt:175)
        at io.airbyte.cdk.integrations.base.IntegrationRunner.run(IntegrationRunner.kt:119)
        at io.airbyte.cdk.integrations.base.IntegrationRunner.run$default(IntegrationRunner.kt:113)
        at io.airbyte.cdk.integrations.base.IntegrationRunner.run(IntegrationRunner.kt)
        at io.airbyte.integrations.source.mssql.MssqlSource.main(MssqlSource.java:583)```
3

I created a github issue: https://github.com/airbytehq/airbyte/issues/42491