Custom DBT not working for SQL Server destination

Connector Questions & Issues
1
  • Is this your first time deploying Airbyte?: Yes
  • OS Version / Instance: Windows
  • Memory / Disk: you can use something like 200GB
  • Deployment: Docker
  • Airbyte Version: 0.40.26
  • Source name/version: airbyte/source-file 0.2.31
  • Destination name/version: airbyte/destination-mssql 0.1.22
  • Step: The issue is happening during custom transformation
  • Description: I wanted to perform custom transformation after data replication to the destination database table. But the custom transformation did not work. I have also tested the destination with Postgres and it was successful.

Log is attached below:
2023-01-06 15:36:50 �[32mINFO�[m i.a.v.j.JsonSchemaValidator(test):130 - JSON schema validation failed.
errors: $.storage: must be a constant value HTTPS
2023-01-06 15:36:50 �[32mINFO�[m i.a.v.j.JsonSchemaValidator(test):130 - JSON schema validation failed.
errors: $.storage: must be a constant value GCS
2023-01-06 15:36:50 �[32mINFO�[m i.a.v.j.JsonSchemaValidator(test):130 - JSON schema validation failed.
errors: $.storage: must be a constant value S3
2023-01-06 15:36:50 �[32mINFO�[m i.a.v.j.JsonSchemaValidator(test):130 - JSON schema validation failed.
errors: .storage:mustbeaconstantvalueAzBlob,.storage_account: is missing but it is required
2023-01-06 15:36:50 �[32mINFO�[m i.a.v.j.JsonSchemaValidator(test):130 - JSON schema validation failed.
errors: .storage:mustbeaconstantvalueSSH,.password: object found, string expected
2023-01-06 15:36:50 �[32mINFO�[m i.a.v.j.JsonSchemaValidator(test):130 - JSON schema validation failed.
errors: .storage:mustbeaconstantvalueSCP,.password: object found, string expected
2023-01-06 15:36:50 �[32mINFO�[m i.a.v.j.JsonSchemaValidator(test):130 - JSON schema validation failed.
errors: $.password: object found, string expected
2023-01-06 15:36:50 �[32mINFO�[m i.a.v.j.JsonSchemaValidator(test):130 - JSON schema validation failed.
errors: $.storage: must be a constant value local
2023-01-06 15:36:51 �[32mINFO�[m i.a.w.t.TemporalAttemptExecution(get):136 - Docker volume job log path: /tmp/workspace/13/0/logs.log
2023-01-06 15:36:51 �[32mINFO�[m i.a.w.t.TemporalAttemptExecution(get):141 - Executing worker wrapper. Airbyte version: 0.40.26
2023-01-06 15:36:51 �[32mINFO�[m i.a.a.c.AirbyteApiClient(retryWithJitter):172 - Attempt 0 to save workflow id for cancellation
2023-01-06 15:36:52 �[32mINFO�[m i.a.c.i.LineGobbler(voidCall):114 -
2023-01-06 15:36:52 �[32mINFO�[m i.a.c.i.LineGobbler(voidCall):114 - ----- START CHECK -----
2023-01-06 15:36:52 �[32mINFO�[m i.a.c.i.LineGobbler(voidCall):114 -
2023-01-06 15:36:52 �[32mINFO�[m i.a.c.i.LineGobbler(voidCall):114 - Checking if airbyte/source-file:0.2.31 exists…
2023-01-06 15:36:52 �[32mINFO�[m i.a.c.i.LineGobbler(voidCall):114 - airbyte/source-file:0.2.31 was found locally.
2023-01-06 15:36:52 �[32mINFO�[m i.a.w.p.DockerProcessFactory(create):120 - Creating docker container = source-file-check-13-0-pmxyt with resources io.airbyte.config.ResourceRequirements@1034c6e7[cpuRequest=,cpuLimit=,memoryRequest=,memoryLimit=]
2023-01-06 15:36:52 �[32mINFO�[m i.a.w.p.DockerProcessFactory(create):164 - Preparing command: docker run --rm --init -i -w /data/13/0 --log-driver none --name source-file-check-13-0-pmxyt --network host -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -e DEPLOYMENT_MODE=OSS -e USE_STREAM_CAPABLE_STATE=true -e WORKER_ENVIRONMENT=DOCKER -e AIRBYTE_ROLE= -e WORKER_JOB_ATTEMPT=0 -e WORKER_CONNECTOR_IMAGE=airbyte/source-file:0.2.31 -e AUTO_DETECT_SCHEMA=false -e AIRBYTE_VERSION=0.40.26 -e WORKER_JOB_ID=13 airbyte/source-file:0.2.31 check --config source_config.json
2023-01-06 15:36:52 �[32mINFO�[m i.a.w.i.VersionedAirbyteStreamFactory(create):97 - Reading messages from protocol version 0.2.0
2023-01-06 15:36:57 �[33mWARN�[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):117 - ignoring unsupported keyword arguments: [‘connect_kwargs’]
2023-01-06 15:37:00 �[33mWARN�[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):117 - ignoring unsupported keyword arguments: [‘connect_kwargs’]
2023-01-06 15:37:03 �[32mINFO�[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - Check succeeded
2023-01-06 15:37:04 �[32mINFO�[m i.a.c.i.LineGobbler(voidCall):114 -
2023-01-06 15:37:04 �[32mINFO�[m i.a.c.i.LineGobbler(voidCall):114 - ----- END CHECK -----
2023-01-06 15:37:04 �[32mINFO�[m i.a.c.i.LineGobbler(voidCall):114 -
2023-01-06 15:37:04 �[32mINFO�[m i.a.w.t.TemporalAttemptExecution(get):163 - Stopping cancellation check scheduling…
2023-01-06 15:37:04 �[32mINFO�[m i.a.w.t.TemporalAttemptExecution(get):136 - Docker volume job log path: /tmp/workspace/13/0/logs.log
2023-01-06 15:37:04 �[32mINFO�[m i.a.w.t.TemporalAttemptExecution(get):141 - Executing worker wrapper. Airbyte version: 0.40.26
2023-01-06 15:37:04 �[32mINFO�[m i.a.a.c.AirbyteApiClient(retryWithJitter):172 - Attempt 0 to save workflow id for cancellation
2023-01-06 15:37:04 �[32mINFO�[m i.a.c.i.LineGobbler(voidCall):114 -
2023-01-06 15:37:04 �[32mINFO�[m i.a.c.i.LineGobbler(voidCall):114 - ----- START CHECK -----
2023-01-06 15:37:04 �[32mINFO�[m i.a.c.i.LineGobbler(voidCall):114 -
2023-01-06 15:37:04 �[32mINFO�[m i.a.c.i.LineGobbler(voidCall):114 - Checking if airbyte/destination-mssql:0.1.22 exists…
2023-01-06 15:37:04 �[32mINFO�[m i.a.c.i.LineGobbler(voidCall):114 - airbyte/destination-mssql:0.1.22 was found locally.
2023-01-06 15:37:04 �[32mINFO�[m i.a.w.p.DockerProcessFactory(create):120 - Creating docker container = destination-mssql-check-13-0-iidws with resources io.airbyte.config.ResourceRequirements@1034c6e7[cpuRequest=,cpuLimit=,memoryRequest=,memoryLimit=]
2023-01-06 15:37:04 �[32mINFO�[m i.a.w.p.DockerProcessFactory(create):164 - Preparing command: docker run --rm --init -i -w /data/13/0 --log-driver none --name destination-mssql-check-13-0-iidws --network host -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -e DEPLOYMENT_MODE=OSS -e USE_STREAM_CAPABLE_STATE=true -e WORKER_ENVIRONMENT=DOCKER -e AIRBYTE_ROLE= -e WORKER_JOB_ATTEMPT=0 -e WORKER_CONNECTOR_IMAGE=airbyte/destination-mssql:0.1.22 -e AUTO_DETECT_SCHEMA=false -e AIRBYTE_VERSION=0.40.26 -e WORKER_JOB_ID=13 airbyte/destination-mssql:0.1.22 check --config source_config.json
2023-01-06 15:37:04 �[32mINFO�[m i.a.w.i.VersionedAirbyteStreamFactory(create):97 - Reading messages from protocol version 0.2.0
2023-01-06 15:37:08 �[32mINFO�[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - starting destination: class io.airbyte.integrations.destination.mssql.MSSQLDestination
2023-01-06 15:37:09 �[32mINFO�[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - integration args: {check=null, config=source_config.json}
2023-01-06 15:37:09 �[32mINFO�[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - Running integration: io.airbyte.integrations.base.ssh.SshWrappedDestination
2023-01-06 15:37:09 �[32mINFO�[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - Command: CHECK
2023-01-06 15:37:09 �[32mINFO�[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - Integration config: IntegrationConfig{command=CHECK, configPath=‘source_config.json’, catalogPath=‘null’, statePath=‘null’}
2023-01-06 15:37:09 �[33mWARN�[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):117 - Unknown keyword order - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2023-01-06 15:37:09 �[33mWARN�[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):117 - Unknown keyword airbyte_secret - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2023-01-06 15:37:09 �[32mINFO�[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - Starting connection with method: NO_TUNNEL
2023-01-06 15:37:09 �[32mINFO�[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - HikariPool-1 - Starting…
2023-01-06 15:37:09 �[32mINFO�[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - HikariPool-1 - Start completed.
2023-01-06 15:37:12 �[32mINFO�[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - HikariPool-1 - Shutdown initiated…
2023-01-06 15:37:12 �[32mINFO�[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - HikariPool-1 - Shutdown completed.
2023-01-06 15:37:12 �[32mINFO�[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - Completed integration: io.airbyte.integrations.base.ssh.SshWrappedDestination
2023-01-06 15:37:12 �[32mINFO�[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - completed destination: class io.airbyte.integrations.destination.mssql.MSSQLDestination
2023-01-06 15:37:13 �[32mINFO�[m i.a.w.t.TemporalAttemptExecution(get):163 - Stopping cancellation check scheduling…
2023-01-06 15:37:13 �[32mINFO�[m i.a.c.i.LineGobbler(voidCall):114 -
2023-01-06 15:37:13 �[32mINFO�[m i.a.c.i.LineGobbler(voidCall):114 - ----- END CHECK -----
2023-01-06 15:37:13 �[32mINFO�[m i.a.c.i.LineGobbler(voidCall):114 -
2023-01-06 15:37:14 �[32mINFO�[m i.a.w.t.TemporalAttemptExecution(get):136 - Docker volume job log path: /tmp/workspace/13/0/logs.log
2023-01-06 15:37:14 �[32mINFO�[m i.a.w.t.TemporalAttemptExecution(get):141 - Executing worker wrapper. Airbyte version: 0.40.26
2023-01-06 15:37:14 �[32mINFO�[m i.a.a.c.AirbyteApiClient(retryWithJitter):172 - Attempt 0 to save workflow id for cancellation
2023-01-06 15:37:15 �[32mINFO�[m i.a.c.f.EnvVariableFeatureFlags(getEnvOrDefault):64 - Using default value for environment variable LOG_CONNECTOR_MESSAGES: ‘false’
2023-01-06 15:37:15 �[32mINFO�[m i.a.c.EnvConfigs(getEnvOrDefault):1161 - Using default value for environment variable METRIC_CLIENT: ‘’
2023-01-06 15:37:15 �[33mWARN�[m i.a.m.l.MetricClientFactory(initialize):60 - Metric client is already initialized to
2023-01-06 15:37:15 �[32mINFO�[m i.a.c.f.EnvVariableFeatureFlags(getEnvOrDefault):64 - Using default value for environment variable LOG_CONNECTOR_MESSAGES: ‘false’
2023-01-06 15:37:15 �[32mINFO�[m i.a.w.g.DefaultReplicationWorker(run):142 - start sync worker. job id: 13 attempt id: 0
2023-01-06 15:37:15 �[32mINFO�[m i.a.w.g.DefaultReplicationWorker(run):157 - configured sync modes: {null.imported_data=full_refresh - overwrite}
2023-01-06 15:37:15 �[32mINFO�[m i.a.c.i.LineGobbler(voidCall):114 -
2023-01-06 15:37:15 �[32mINFO�[m i.a.c.i.LineGobbler(voidCall):114 - ----- START REPLICATION -----
2023-01-06 15:37:15 �[32mINFO�[m i.a.w.i.DefaultAirbyteDestination(start):72 - Running destination…
2023-01-06 15:37:15 �[32mINFO�[m i.a.c.i.LineGobbler(voidCall):114 -
2023-01-06 15:37:15 �[32mINFO�[m i.a.c.i.LineGobbler(voidCall):114 - Checking if airbyte/destination-mssql:0.1.22 exists…
2023-01-06 15:37:15 �[32mINFO�[m i.a.c.i.LineGobbler(voidCall):114 - airbyte/destination-mssql:0.1.22 was found locally.
2023-01-06 15:37:15 �[32mINFO�[m i.a.w.p.DockerProcessFactory(create):120 - Creating docker container = destination-mssql-write-13-0-qnapz with resources io.airbyte.config.ResourceRequirements@71a9c604[cpuRequest=,cpuLimit=,memoryRequest=,memoryLimit=]
2023-01-06 15:37:15 �[32mINFO�[m i.a.w.p.DockerProcessFactory(create):164 - Preparing command: docker run --rm --init -i -w /data/13/0 --log-driver none --name destination-mssql-write-13-0-qnapz --network host -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -e DEPLOYMENT_MODE=OSS -e USE_STREAM_CAPABLE_STATE=true -e WORKER_ENVIRONMENT=DOCKER -e AIRBYTE_ROLE= -e WORKER_JOB_ATTEMPT=0 -e WORKER_CONNECTOR_IMAGE=airbyte/destination-mssql:0.1.22 -e AUTO_DETECT_SCHEMA=false -e AIRBYTE_VERSION=0.40.26 -e WORKER_JOB_ID=13 airbyte/destination-mssql:0.1.22 write --config destination_config.json --catalog destination_catalog.json

2

Full log here: airbyte custom dbt error - Google Docs

3

Hello there! You are receiving this message because none of your fellow community members has stepped in to respond to your topic post. (If you are a community member and you are reading this response, feel free to jump in if you have the answer!) As a result, the Community Assistance Team has been made aware of this topic and will be investigating and responding as quickly as possible.
Some important considerations that will help your to get your issue solved faster:

  • It is best to use our topic creation template; if you haven’t yet, we recommend posting a followup with the requested information. With that information the team will be able to more quickly search for similar issues with connectors and the platform and troubleshoot more quickly your specific question or problem.
  • Make sure to upload the complete log file; a common investigation roadblock is that sometimes the error for the issue happens well before the problem is surfaced to the user, and so having the tail of the log is less useful than having the whole log to scan through.
  • Be as descriptive and specific as possible; when investigating it is extremely valuable to know what steps were taken to encounter the issue, what version of connector / platform / Java / Python / docker / k8s was used, etc. The more context supplied, the quicker the investigation can start on your topic and the faster we can drive towards an answer.
  • We in the Community Assistance Team are glad you’ve made yourself part of our community, and we’ll do our best to answer your questions and resolve the problems as quickly as possible. Expect to hear from a specific team member as soon as possible.

Thank you for your time and attention.
Best,
The Community Assistance Team

4

Hello meta-research-jack, it’s been a while without an update from us. Are you still having problems or did you find a solution?