Destination Databricks - Missing Unity Catalog

Connector Questions & Issues
,
1
  • Is this your first time deploying Airbyte?: No
  • OS Version / Instance: MacOS Montery 12.6
  • Memory / Disk: 8Gb
  • Deployment: Docker
  • Airbyte Version: 0.40.18
  • Source name/version: -
  • Destination name/version: databricks / 0.3.1
  • Step: The issue is happening during sync

The connector does not work with the unity catalog feature of databricks. i guess the new three tier namespace is missing here, because the tables that the connector will create, are only at database.table level instead of catalog.database.table.

I just did a quick test and the sync fails when creating the table.CREATE TABLE public._airbyte_tmp_yhm_orders USING parquet LOCATION 's3://<bucket-xyz>/data/6fed6526-b350-40a6-8719-1907a58cfe65/orders'. I would have expected that it would be possible to specify the catalogue in the configuration, but currently I don’t know in which catalogue the table would be created.

3

Hello Hinnenkamp, it’s been a while without an update from us. Are you still having problems or did you find a solution?

4

The connector does seem to work with the Unity Catalog. Our expectation was simply different, so we thought a new catalogue would be created with the configured database and table name of the stream. However, the connector puts the new database and table in the root bucket.

The ticket can therefore be closed, as the general functionality with the Unity Catalog is given.