JsonSchemaValidator doesn't respect oneOf in credentials

yli · May 4, 2022, 3:56pm

airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-snowflake/src/main/resources/spec.json#L13

      
        
            "connectionSpecification": {
              "$schema": "http://json-schema.org/draft-07/schema#",
              "title": "Snowflake Source Spec",
              "type": "object",
              "required": ["host", "role", "warehouse", "database", "schema"],
              "additionalProperties": true,
              "properties": {
                "credentials": {
                  "title": "Authorization Method",
                  "type": "object",
                  "oneOf": [
                    {
                      "type": "object",
                      "title": "OAuth2.0",
                      "order": 0,
                      "required": ["client_id", "client_secret", "auth_type"],
                      "properties": {
                        "auth_type": {
                          "type": "string",
                          "const": "OAuth",
                          "default": "OAuth",

According to the recent update at source-snowflake, oauth2 is supported, thanks for the new feature.

However, I am still using account/password in my local development, i.e. the credential is The sync and normalization work fine. The destination is nothing but postgres. However, the JsonSchemaValidator throws an error at the end.

errors: $.client_id: is missing but it is required, $.client_secret: is missing but it is required, $.auth_type: must be a constant value OAuth

I am using the sample database / schema when I created a trail account from snowflake.

Airbyte version: 0.36.8-alpha
source_snowflake version: 0.1.12

more logs:

2022-05-04 15:43:15 e[32mINFOe[m i.a.w.w.WorkerRun(call):49 - Executing worker wrapper. Airbyte version: 0.36.8-alpha
2022-05-04 15:43:15 e[32mINFOe[m i.a.w.t.TemporalAttemptExecution(get):108 - Docker volume job log path: /tmp/workspace/1/0/logs.log
2022-05-04 15:43:15 e[32mINFOe[m i.a.w.t.TemporalAttemptExecution(get):113 - Executing worker wrapper. Airbyte version: 0.36.8-alpha
2022-05-04 15:43:15 e[33mWARNe[m c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword example - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2022-05-04 15:43:15 e[32mINFOe[m i.a.w.DefaultReplicationWorker(run):104 - start sync worker. job id: 1 attempt id: 0
2022-05-04 15:43:15 e[32mINFOe[m i.a.w.DefaultReplicationWorker(run):116 - configured sync modes: {TPCH_SF1.NATION=incremental - append}
2022-05-04 15:43:15 e[32mINFOe[m i.a.w.p.a.DefaultAirbyteDestination(start):69 - Running destination...
2022-05-04 15:43:15 e[32mINFOe[m i.a.c.i.LineGobbler(voidCall):82 - Checking if airbyte/destination-postgres:0.3.19 exists...
2022-05-04 15:43:15 e[32mINFOe[m i.a.c.i.LineGobbler(voidCall):82 - airbyte/destination-postgres:0.3.19 was found locally.
2022-05-04 15:43:15 e[32mINFOe[m i.a.w.p.DockerProcessFactory(create):107 - Creating docker job ID: 1
2022-05-04 15:43:15 e[32mINFOe[m i.a.w.p.DockerProcessFactory(create):162 - Preparing command: docker run --rm --init -i -w /data/1/0 --log-driver none --name destination-postgres-sync-1-0-bhudo --network host -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -e WORKER_CONNECTOR_IMAGE=airbyte/destination-postgres:0.3.19 -e WORKER_JOB_ATTEMPT=0 -e AIRBYTE_ROLE= -e WORKER_ENVIRONMENT=DOCKER -e AIRBYTE_VERSION=0.36.8-alpha -e WORKER_JOB_ID=1 airbyte/destination-postgres:0.3.19 write --config destination_config.json --catalog destination_catalog.json
2022-05-04 15:43:15 e[32mINFOe[m i.a.c.i.LineGobbler(voidCall):82 - Checking if airbyte/source-snowflake:0.1.12 exists...
2022-05-04 15:43:15 e[32mINFOe[m i.a.c.i.LineGobbler(voidCall):82 - airbyte/source-snowflake:0.1.12 was found locally.
2022-05-04 15:43:15 e[32mINFOe[m i.a.w.p.DockerProcessFactory(create):107 - Creating docker job ID: 1
2022-05-04 15:43:15 e[32mINFOe[m i.a.w.p.DockerProcessFactory(create):162 - Preparing command: docker run --rm --init -i -w /data/1/0 --log-driver none --name source-snowflake-sync-1-0-henmc --network host -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -e WORKER_CONNECTOR_IMAGE=airbyte/source-snowflake:0.1.12 -e WORKER_JOB_ATTEMPT=0 -e AIRBYTE_ROLE= -e WORKER_ENVIRONMENT=DOCKER -e AIRBYTE_VERSION=0.36.8-alpha -e WORKER_JOB_ID=1 airbyte/source-snowflake:0.1.12 read --config source_config.json --catalog source_catalog.json
2022-05-04 15:43:15 e[32mINFOe[m i.a.w.DefaultReplicationWorker(run):158 - Waiting for source and destination threads to complete.
2022-05-04 15:43:15 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):279 - Replication thread started.
2022-05-04 15:43:15 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getDestinationOutputRunnable$6):339 - Destination output thread started.
2022-05-04 15:43:15 e[43mdestinatione[0m > SLF4J: Class path contains multiple SLF4J bindings.
2022-05-04 15:43:15 e[43mdestinatione[0m > SLF4J: Found binding in [jar:file:/airbyte/lib/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
2022-05-04 15:43:15 e[43mdestinatione[0m > SLF4J: Found binding in [jar:file:/airbyte/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
2022-05-04 15:43:15 e[43mdestinatione[0m > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
2022-05-04 15:43:16 e[43mdestinatione[0m > SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
2022-05-04 15:43:16 e[44msourcee[0m > 2022-05-04 15:43:16 e[32mINFOe[m i.a.i.s.s.SnowflakeSource(main):42 - starting source: class io.airbyte.integrations.source.snowflake.SnowflakeSource
2022-05-04 15:43:16 e[43mdestinatione[0m > 2022-05-04 15:43:16 e[32mINFOe[m i.a.i.d.p.PostgresDestination(main):89 - starting destination: class io.airbyte.integrations.destination.postgres.PostgresDestination
2022-05-04 15:43:16 e[43mdestinatione[0m > 2022-05-04 15:43:16 e[32mINFOe[m i.a.i.b.IntegrationCliParser(parseOptions):118 - integration args: {catalog=destination_catalog.json, write=null, config=destination_config.json}
2022-05-04 15:43:16 e[43mdestinatione[0m > 2022-05-04 15:43:16 e[32mINFOe[m i.a.i.b.IntegrationRunner(runInternal):121 - Running integration: io.airbyte.integrations.base.ssh.SshWrappedDestination
2022-05-04 15:43:16 e[43mdestinatione[0m > 2022-05-04 15:43:16 e[32mINFOe[m i.a.i.b.IntegrationRunner(runInternal):122 - Command: WRITE
2022-05-04 15:43:16 e[43mdestinatione[0m > 2022-05-04 15:43:16 e[32mINFOe[m i.a.i.b.IntegrationRunner(runInternal):123 - Integration config: IntegrationConfig{command=WRITE, configPath='destination_config.json', catalogPath='destination_catalog.json', statePath='null'}
2022-05-04 15:43:16 e[44msourcee[0m > 2022-05-04 15:43:16 e[32mINFOe[m i.a.i.b.IntegrationCliParser(parseOptions):118 - integration args: {read=null, catalog=source_catalog.json, config=source_config.json}
2022-05-04 15:43:16 e[44msourcee[0m > 2022-05-04 15:43:16 e[32mINFOe[m i.a.i.b.IntegrationRunner(runInternal):121 - Running integration: io.airbyte.integrations.source.snowflake.SnowflakeSource
2022-05-04 15:43:16 e[44msourcee[0m > 2022-05-04 15:43:16 e[32mINFOe[m i.a.i.b.IntegrationRunner(runInternal):122 - Command: READ
2022-05-04 15:43:16 e[44msourcee[0m > 2022-05-04 15:43:16 e[32mINFOe[m i.a.i.b.IntegrationRunner(runInternal):123 - Integration config: IntegrationConfig{command=READ, configPath='source_config.json', catalogPath='source_catalog.json', statePath='null'}
2022-05-04 15:43:17 e[43mdestinatione[0m > 2022-05-04 15:43:17 e[33mWARNe[m c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword order - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2022-05-04 15:43:17 e[43mdestinatione[0m > 2022-05-04 15:43:17 e[33mWARNe[m c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword examples - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2022-05-04 15:43:17 e[43mdestinatione[0m > 2022-05-04 15:43:17 e[33mWARNe[m c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword airbyte_secret - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2022-05-04 15:43:17 e[43mdestinatione[0m > 2022-05-04 15:43:17 e[33mWARNe[m c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword multiline - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2022-05-04 15:43:17 e[44msourcee[0m > 2022-05-04 15:43:17 e[33mWARNe[m c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword order - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2022-05-04 15:43:17 e[44msourcee[0m > 2022-05-04 15:43:17 e[33mWARNe[m c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword airbyte_secret - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2022-05-04 15:43:17 e[44msourcee[0m > 2022-05-04 15:43:17 e[33mWARNe[m c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword examples - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2022-05-04 15:43:17 e[44msourcee[0m > 2022-05-04 15:43:17 e[32mINFOe[m i.a.i.s.r.CdcStateManager(<init>):26 - Initialized CDC state with: null
2022-05-04 15:43:17 e[44msourcee[0m > 2022-05-04 15:43:17 e[32mINFOe[m i.a.i.s.r.StateManager(createCursorInfoForStream):118 - No cursor field set in catalog but not present in state. Stream: AirbyteStreamNameNamespacePair{name='NATION', namespace='TPCH_SF1'}, New Cursor Field: N_NATIONKEY. Resetting cursor value
2022-05-04 15:43:17 e[43mdestinatione[0m > 2022-05-04 15:43:17 e[32mINFOe[m i.a.i.b.s.SshTunnel(getInstance):170 - Starting connection with method: NO_TUNNEL
2022-05-04 15:43:17 e[44msourcee[0m > 2022-05-04 15:43:17 e[32mINFOe[m i.a.i.s.s.SnowflakeDataSourceUtils(createDataSource):70 - Authorization mode is 'Username and password'
2022-05-04 15:43:17 e[44msourcee[0m > 2022-05-04 15:43:17 e[32mINFOe[m c.z.h.HikariDataSource(getConnection):110 - HikariPool-1 - Starting...
2022-05-04 15:43:17 e[43mdestinatione[0m > 2022-05-04 15:43:17 e[32mINFOe[m i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$toWriteConfig$0):98 - Write config: WriteConfig{streamName=NATION, namespace=TPCH_SF1, outputSchemaName=TPCH_SF1, tmpTableName=_airbyte_tmp_ukc_NATION, outputTableName=_airbyte_raw_NATION, syncMode=append}
2022-05-04 15:43:17 e[43mdestinatione[0m > 2022-05-04 15:43:17 e[32mINFOe[m i.a.i.d.b.BufferedStreamConsumer(startTracked):116 - class io.airbyte.integrations.destination.buffered_stream_consumer.BufferedStreamConsumer started.
2022-05-04 15:43:17 e[43mdestinatione[0m > 2022-05-04 15:43:17 e[32mINFOe[m i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onStartFunction$1):123 - Preparing tmp tables in destination started for 1 streams
2022-05-04 15:43:17 e[43mdestinatione[0m > 2022-05-04 15:43:17 e[32mINFOe[m i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onStartFunction$1):127 - Preparing tmp table in destination started for stream NATION. schema: TPCH_SF1, tmp table name: _airbyte_tmp_ukc_NATION
2022-05-04 15:43:17 e[43mdestinatione[0m > 2022-05-04 15:43:17 e[32mINFOe[m i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onStartFunction$1):133 - Preparing tables in destination completed.
2022-05-04 15:43:19 e[44msourcee[0m > 2022-05-04 15:43:19 e[32mINFOe[m c.z.h.p.HikariPool(checkFailFast):565 - HikariPool-1 - Added connection net.snowflake.client.jdbc.SnowflakeConnectionV1@58f39564
2022-05-04 15:43:19 e[44msourcee[0m > 2022-05-04 15:43:19 e[32mINFOe[m c.z.h.HikariDataSource(getConnection):123 - HikariPool-1 - Start completed.
2022-05-04 15:43:19 e[44msourcee[0m > 2022-05-04 15:43:19 e[32mINFOe[m i.a.i.s.s.SnowflakeSource(buildUsernamePasswordConfig):103 - jdbc:snowflake://io49756.us-east-2.aws.snowflakecomputing.com/?role=AIRBYTE_ROLE&warehouse=COMPUTE_WH&database=SNOWFLAKE_SAMPLE_DATA&schema=TPCH_SF1&JDBC_QUERY_RESULT_FORMAT=JSON&CLIENT_SESSION_KEEP_ALIVE=true
2022-05-04 15:43:21 e[44msourcee[0m > 2022-05-04 15:43:21 e[32mINFOe[m i.a.i.s.j.AbstractJdbcSource(lambda$discoverInternal$5):138 - Table CUSTOMER column C_CUSTKEY (type NUMBER[38]) -> Json type io.airbyte.protocol.models.JsonSchemaType@1f1ff879
2022-05-04 15:43:21 e[44msourcee[0m > 2022-05-04 15:43:21 e[32mINFOe[m i.a.i.s.j.AbstractJdbcSource(lambda$discoverInternal$5):138 - Table CUSTOMER column C_NAME (type VARCHAR[25]) -> Json type io.airbyte.protocol.models.JsonSchemaType@24f870ee
2022-05-04 15:43:21 e[44msourcee[0m > 2022-05-04 15:43:21 e[32mINFOe[m i.a.i.s.j.AbstractJdbcSource(lambda$discoverInternal$5):138 - Table CUSTOMER column C_ADDRESS (type VARCHAR[40]) -> Json type io.airbyte.protocol.models.JsonSchemaType@24f870ee
2022-05-04 15:43:21 e[44msourcee[0m > 2022-05-04 15:43:21 e[32mINFOe[m i.a.i.s.j.AbstractJdbcSource(lambda$discoverInternal$5):138 - Table CUSTOMER column C_NATIONKEY (type NUMBER[38]) -> Json type io.airbyte.protocol.models.JsonSchemaType@1f1ff879
2022-05-04 15:43:21 e[44msourcee[0m > 2022-05-04 15:43:21 e[32mINFOe[m i.a.i.s.j.AbstractJdbcSource(lambda$discoverInternal$5):138 - Table CUSTOMER column C_PHONE (type VARCHAR[15]) -> Json type io.airbyte.protocol.models.JsonSchemaType@24f870ee
...
2022-05-04 15:43:21 e[44msourcee[0m > 2022-05-04 15:43:21 e[32mINFOe[m i.a.i.s.j.AbstractJdbcSource(lambda$discoverInternal$5):138 - Table PARTSUPP column PS_COMMENT (type VARCHAR[199]) -> Json type io.airbyte.protocol.models.JsonSchemaType@24f870ee
2022-05-04 15:43:21 e[44msourcee[0m > 2022-05-04 15:43:21 e[32mINFOe[m i.a.i.s.r.AbstractRelationalDbSource(queryTableFullRefresh):35 - Queueing query for table: NATION
2022-05-04 15:43:21 e[44msourcee[0m > 2022-05-04 15:43:21 e[32mINFOe[m i.a.d.j.s.AdaptiveStreamingQueryConfig(initialize):32 - Set initial fetch size: 10 rows
2022-05-04 15:43:22 e[44msourcee[0m > 2022-05-04 15:43:22 e[32mINFOe[m i.a.d.j.s.AdaptiveStreamingQueryConfig(accept):42 - Updated fetch size: 100000 rows
2022-05-04 15:43:22 e[44msourcee[0m > 2022-05-04 15:43:22 e[32mINFOe[m i.a.i.s.r.StateDecoratingIterator(computeNext):60 - State Report: stream name: AirbyteStreamNameNamespacePair{name='NATION', namespace='TPCH_SF1'}, original cursor field: null, original cursor null, cursor field: N_NATIONKEY, new cursor: 24
2022-05-04 15:43:22 e[44msourcee[0m > 2022-05-04 15:43:22 e[32mINFOe[m i.a.i.s.r.AbstractDbSource(lambda$read$2):124 - Closing database connection pool.
2022-05-04 15:43:22 e[44msourcee[0m > 2022-05-04 15:43:22 e[32mINFOe[m c.z.h.HikariDataSource(close):350 - HikariPool-1 - Shutdown initiated...
2022-05-04 15:43:23 e[44msourcee[0m > 2022-05-04 15:43:23 e[32mINFOe[m c.z.h.HikariDataSource(close):352 - HikariPool-1 - Shutdown completed.
2022-05-04 15:43:23 e[44msourcee[0m > 2022-05-04 15:43:23 e[32mINFOe[m i.a.i.s.r.AbstractDbSource(lambda$read$2):126 - Closed database connection pool.
2022-05-04 15:43:23 e[44msourcee[0m > 2022-05-04 15:43:23 e[32mINFOe[m i.a.i.b.IntegrationRunner(runInternal):169 - Completed integration: io.airbyte.integrations.source.snowflake.SnowflakeSource
2022-05-04 15:43:23 e[44msourcee[0m > 2022-05-04 15:43:23 e[32mINFOe[m i.a.i.s.s.SnowflakeSource(main):45 - completed source: class io.airbyte.integrations.source.snowflake.SnowflakeSource
2022-05-04 15:43:24 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):305 - Total records read: 26 (3 KB)
2022-05-04 15:43:24 e[32mINFOe[m i.a.w.DefaultReplicationWorker(run):163 - One of source or destination thread complete. Waiting on the other.
2022-05-04 15:43:24 e[43mdestinatione[0m > 2022-05-04 15:43:24 e[32mINFOe[m i.a.i.b.FailureTrackingAirbyteMessageConsumer(close):65 - Airbyte message consumer: succeeded.
2022-05-04 15:43:24 e[43mdestinatione[0m > 2022-05-04 15:43:24 e[32mINFOe[m i.a.i.d.b.BufferedStreamConsumer(close):170 - executing on success close procedure.
2022-05-04 15:43:24 e[43mdestinatione[0m > 2022-05-04 15:43:24 e[32mINFOe[m i.a.i.d.r.InMemoryRecordBufferingStrategy(lambda$flushAll$1):84 - Flushing NATION: 25 records (11 KB)
2022-05-04 15:43:24 e[43mdestinatione[0m > 2022-05-04 15:43:24 e[32mINFOe[m i.a.i.d.j.SqlOperations(onDestinationCloseOperations):136 - No onDestinationCloseOperations required for this destination.
2022-05-04 15:43:24 e[43mdestinatione[0m > 2022-05-04 15:43:24 e[32mINFOe[m i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onCloseFunction$3):163 - Finalizing tables in destination started for 1 streams
2022-05-04 15:43:24 e[43mdestinatione[0m > 2022-05-04 15:43:24 e[32mINFOe[m i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onCloseFunction$3):168 - Finalizing stream NATION. schema TPCH_SF1, tmp table _airbyte_tmp_ukc_NATION, final table _airbyte_raw_NATION
2022-05-04 15:43:24 e[43mdestinatione[0m > 2022-05-04 15:43:24 e[32mINFOe[m i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onCloseFunction$3):181 - Executing finalization of tables.
2022-05-04 15:43:24 e[43mdestinatione[0m > 2022-05-04 15:43:24 e[32mINFOe[m i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onCloseFunction$3):183 - Finalizing tables in destination completed.
2022-05-04 15:43:24 e[43mdestinatione[0m > 2022-05-04 15:43:24 e[32mINFOe[m i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onCloseFunction$3):186 - Cleaning tmp tables in destination started for 1 streams
2022-05-04 15:43:24 e[43mdestinatione[0m > 2022-05-04 15:43:24 e[32mINFOe[m i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onCloseFunction$3):190 - Cleaning tmp table in destination started for stream NATION. schema TPCH_SF1, tmp table name: _airbyte_tmp_ukc_NATION
2022-05-04 15:43:24 e[43mdestinatione[0m > 2022-05-04 15:43:24 e[32mINFOe[m i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onCloseFunction$3):195 - Cleaning tmp tables in destination completed.
2022-05-04 15:43:24 e[32mINFOe[m i.a.w.DefaultReplicationWorker(lambda$getDestinationOutputRunnable$6):349 - State in DefaultReplicationWorker from destination: io.airbyte.protocol.models.AirbyteMessage@12d8196e[type=STATE,log=<null>,spec=<null>,connectionStatus=<null>,catalog=<null>,record=<null>,state=io.airbyte.protocol.models.AirbyteStateMessage@16e78fed[data={"cdc":false,"streams":[{"stream_name":"NATION","stream_namespace":"TPCH_SF1","cursor_field":["N_NATIONKEY"],"cursor":"24"}]},additionalProperties={}],trace=<null>,additionalProperties={}]
2022-05-04 15:43:24 e[43mdestinatione[0m > 2022-05-04 15:43:24 e[32mINFOe[m i.a.i.b.IntegrationRunner(runInternal):169 - Completed integration: io.airbyte.integrations.base.ssh.SshWrappedDestination
2022-05-04 15:43:24 e[43mdestinatione[0m > 2022-05-04 15:43:24 e[32mINFOe[m i.a.i.d.p.PostgresDestination(main):91 - completed destination: class io.airbyte.integrations.destination.postgres.PostgresDestination
2022-05-04 15:43:24 e[32mINFOe[m i.a.w.DefaultReplicationWorker(run):165 - Source and destination threads complete.
2022-05-04 15:43:24 e[32mINFOe[m i.a.w.DefaultReplicationWorker(run):228 - sync summary: io.airbyte.config.ReplicationAttemptSummary@3dfda6ee[status=completed,recordsSynced=25,bytesSynced=3553,startTime=1651678995380,endTime=1651679004460,totalStats=io.airbyte.config.SyncStats@2e0b0d75[recordsEmitted=25,bytesEmitted=3553,stateMessagesEmitted=1,recordsCommitted=25],streamStats=[io.airbyte.config.StreamSyncStats@53cb2aaf[streamName=NATION,stats=io.airbyte.config.SyncStats@af81d44[recordsEmitted=25,bytesEmitted=3553,stateMessagesEmitted=<null>,recordsCommitted=25]]]]
2022-05-04 15:43:24 e[32mINFOe[m i.a.w.DefaultReplicationWorker(run):248 - Source output at least one state message
2022-05-04 15:43:24 e[32mINFOe[m i.a.w.DefaultReplicationWorker(run):254 - State capture: Updated state to: Optional[io.airbyte.config.State@2cf4cebf[state={"cdc":false,"streams":[{"stream_name":"NATION","stream_namespace":"TPCH_SF1","cursor_field":["N_NATIONKEY"],"cursor":"24"}]}]]
2022-05-04 15:43:24 e[32mINFOe[m i.a.w.t.TemporalAttemptExecution(get):134 - Stopping cancellation check scheduling...
2022-05-04 15:43:24 e[32mINFOe[m i.a.w.t.s.ReplicationActivityImpl(lambda$replicate$3):155 - sync summary: io.airbyte.config.StandardSyncOutput@16eecd92[standardSyncSummary=io.airbyte.config.StandardSyncSummary@746ba92f[status=completed,recordsSynced=25,bytesSynced=3553,startTime=1651678995380,endTime=1651679004460,totalStats=io.airbyte.config.SyncStats@2e0b0d75[recordsEmitted=25,bytesEmitted=3553,stateMessagesEmitted=1,recordsCommitted=25],streamStats=[io.airbyte.config.StreamSyncStats@53cb2aaf[streamName=NATION,stats=io.airbyte.config.SyncStats@af81d44[recordsEmitted=25,bytesEmitted=3553,stateMessagesEmitted=<null>,recordsCommitted=25]]]],normalizationSummary=<null>,state=io.airbyte.config.State@2cf4cebf[state={"cdc":false,"streams":[{"stream_name":"NATION","stream_namespace":"TPCH_SF1","cursor_field":["N_NATIONKEY"],"cursor":"24"}]}],outputCatalog=io.airbyte.protocol.models.ConfiguredAirbyteCatalog@8451f74[streams=[io.airbyte.protocol.models.ConfiguredAirbyteStream@517effdd[stream=io.airbyte.protocol.models.AirbyteStream@319c3a3d[name=NATION,jsonSchema={"type":"object","properties":{"N_NAME":{"type":"string"},"N_COMMENT":{"type":"string"},"N_NATIONKEY":{"type":"number"},"N_REGIONKEY":{"type":"number"}}},supportedSyncModes=[full_refresh, incremental],sourceDefinedCursor=<null>,defaultCursorField=[],sourceDefinedPrimaryKey=[],namespace=TPCH_SF1,additionalProperties={}],syncMode=incremental,cursorField=[N_NATIONKEY],destinationSyncMode=append,primaryKey=[],additionalProperties={}]],additionalProperties={}],failures=[]]
2022-05-04 15:43:24 e[32mINFOe[m i.a.w.t.TemporalUtils(withBackgroundHeartbeat):236 - Stopping temporal heartbeating...
2022-05-04 15:43:24 e[32mINFOe[m i.a.c.p.ConfigRepository(updateConnectionState):774 - Updating connection 5c4c92e4-e267-465c-871d-6ab039ee90f4 state: io.airbyte.config.State@222a3be0[state={"cdc":false,"streams":[{"stream_name":"NATION","stream_namespace":"TPCH_SF1","cursor_field":["N_NATIONKEY"],"cursor":"24"}]}]
2022-05-04 15:43:24 e[32mINFOe[m i.a.w.t.TemporalAttemptExecution(get):108 - Docker volume job log path: /tmp/workspace/1/0/logs.log
2022-05-04 15:43:24 e[32mINFOe[m i.a.w.t.TemporalAttemptExecution(get):113 - Executing worker wrapper. Airbyte version: 0.36.8-alpha
2022-05-04 15:43:24 e[32mINFOe[m i.a.w.DefaultNormalizationWorker(run):47 - Running normalization.
2022-05-04 15:43:24 e[32mINFOe[m i.a.w.n.DefaultNormalizationRunner(runProcess):122 - Running with normalization version: airbyte/normalization:0.1.77
2022-05-04 15:43:24 e[32mINFOe[m i.a.c.i.LineGobbler(voidCall):82 - Checking if airbyte/normalization:0.1.77 exists...
2022-05-04 15:43:24 e[32mINFOe[m i.a.c.i.LineGobbler(voidCall):82 - airbyte/normalization:0.1.77 was found locally.
2022-05-04 15:43:24 e[32mINFOe[m i.a.w.p.DockerProcessFactory(create):107 - Creating docker job ID: 1
2022-05-04 15:43:24 e[32mINFOe[m i.a.w.p.DockerProcessFactory(create):162 - Preparing command: docker run --rm --init -i -w /data/1/0/normalize --log-driver none --name normalization-sync-1-0-fwtnz --network host -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -e AIRBYTE_ROLE= -e WORKER_ENVIRONMENT=DOCKER -e AIRBYTE_VERSION=0.36.8-alpha airbyte/normalization:0.1.77 run --integration-type postgres --config destination_config.json --catalog destination_catalog.json
2022-05-04 15:43:24 e[42mnormalizatione[0m > Running: transform-config --config destination_config.json --integration-type postgres --out /data/1/0/normalize
2022-05-04 15:43:25 e[42mnormalizatione[0m > Namespace(config='destination_config.json', integration_type=<DestinationType.POSTGRES: 'postgres'>, out='/data/1/0/normalize')
2022-05-04 15:43:25 e[42mnormalizatione[0m > transform_postgres
2022-05-04 15:43:25 e[42mnormalizatione[0m > Running: transform-catalog --integration-type postgres --profile-config-dir /data/1/0/normalize --catalog destination_catalog.json --out /data/1/0/normalize/models/generated/ --json-column _airbyte_data
2022-05-04 15:43:25 e[42mnormalizatione[0m > Processing destination_catalog.json...
2022-05-04 15:43:25 e[42mnormalizatione[0m >   Generating airbyte_ctes/tpch_sf1/nation_ab1.sql from NATION
2022-05-04 15:43:25 e[42mnormalizatione[0m >   Generating airbyte_ctes/tpch_sf1/nation_ab2.sql from NATION
2022-05-04 15:43:25 e[42mnormalizatione[0m >   Generating airbyte_ctes/tpch_sf1/nation_ab3.sql from NATION
2022-05-04 15:43:25 e[42mnormalizatione[0m >   Generating airbyte_incremental/tpch_sf1/nation.sql from NATION
2022-05-04 15:43:25 e[42mnormalizatione[0m > detected no config file for ssh, assuming ssh is off.
2022-05-04 15:43:27 e[42mnormalizatione[0m >            [--event-buffer-size EVENT_BUFFER_SIZE]
2022-05-04 15:43:27 e[42mnormalizatione[0m >   --event-buffer-size EVENT_BUFFER_SIZE
2022-05-04 15:43:27 e[42mnormalizatione[0m > 
2022-05-04 15:43:27 e[42mnormalizatione[0m > DBT >=1.0.0 detected; using 10K event buffer size
2022-05-04 15:43:27 e[42mnormalizatione[0m > 
2022-05-04 15:43:30 e[42mnormalizatione[0m > 15:43:30  Running with dbt=1.0.0
2022-05-04 15:43:30 e[42mnormalizatione[0m > 15:43:30  Partial parse save file not found. Starting full parse.
2022-05-04 15:43:31 e[42mnormalizatione[0m > 15:43:31  [WARNING]: Configuration paths exist in your dbt_project.yml file which do not apply to any resources.
2022-05-04 15:43:31 e[42mnormalizatione[0m > There are 2 unused configuration paths:
2022-05-04 15:43:31 e[42mnormalizatione[0m > - models.airbyte_utils.generated.airbyte_views
2022-05-04 15:43:31 e[42mnormalizatione[0m > - models.airbyte_utils.generated.airbyte_tables
2022-05-04 15:43:31 e[42mnormalizatione[0m > 
2022-05-04 15:43:31 e[42mnormalizatione[0m > 15:43:31  Found 4 models, 0 tests, 0 snapshots, 0 analyses, 521 macros, 0 operations, 0 seed files, 1 source, 0 exposures, 0 metrics
2022-05-04 15:43:31 e[42mnormalizatione[0m > 15:43:31  
2022-05-04 15:43:31 e[42mnormalizatione[0m > 15:43:31  Concurrency: 8 threads (target='prod')
2022-05-04 15:43:31 e[42mnormalizatione[0m > 15:43:31  
2022-05-04 15:43:32 e[42mnormalizatione[0m > 15:43:32  1 of 1 START incremental model tpch_sf1.nation.......................................................................... [RUN]
2022-05-04 15:43:32 e[42mnormalizatione[0m > 15:43:32  15:43:32 + "ship_week".tpch_sf1."nation"._airbyte_ab_id does not exist yet. The table will be created or rebuilt with dbt.full_refresh
2022-05-04 15:43:32 e[42mnormalizatione[0m > 15:43:32  1 of 1 OK created incremental model tpch_sf1.nation..................................................................... [SELECT 25 in 0.16s]
2022-05-04 15:43:32 e[42mnormalizatione[0m > 15:43:32  
2022-05-04 15:43:32 e[42mnormalizatione[0m > 15:43:32  Finished running 1 incremental model in 0.40s.
2022-05-04 15:43:32 e[42mnormalizatione[0m > 15:43:32  
2022-05-04 15:43:32 e[42mnormalizatione[0m > 15:43:32  Completed successfully
2022-05-04 15:43:32 e[42mnormalizatione[0m > 15:43:32  
2022-05-04 15:43:32 e[42mnormalizatione[0m > 15:43:32  Done. PASS=1 WARN=0 ERROR=0 SKIP=0 TOTAL=1
2022-05-04 15:43:32 e[32mINFOe[m i.a.w.DefaultNormalizationWorker(run):71 - Normalization executed in 7 seconds.
2022-05-04 15:43:32 e[32mINFOe[m i.a.w.DefaultNormalizationWorker(run):77 - Normalization summary: io.airbyte.config.NormalizationSummary@59d6cb4a[startTime=1651679004623,endTime=1651679012513]
2022-05-04 15:43:32 e[32mINFOe[m i.a.w.t.TemporalAttemptExecution(get):134 - Stopping cancellation check scheduling...
2022-05-04 15:43:32 e[32mINFOe[m i.a.w.t.TemporalUtils(withBackgroundHeartbeat):236 - Stopping temporal heartbeating...
2022-05-04 15:43:32 e[32mINFOe[m i.a.v.j.JsonSchemaValidator(test):56 - JSON schema validation failed. 
errors: $.client_id: is missing but it is required, $.client_secret: is missing but it is required, $.auth_type: must be a constant value OAuth

marcosmarxm · May 5, 2022, 2:13am

Thanks for reporting this Yli, I raised the discussion to our engineer team to investigate and return to you.

marcosmarxm · July 13, 2022, 12:00am

Hi there from the Community Assistance team.
We’re letting you know about an issue we discovered with the back-end process we use to handle topics and responses on the forum. If you experienced a situation where you posted the last message in a topic that did not receive any further replies, please open a new topic to continue the discussion. In addition, if you’re having a problem and find a closed topic on the subject, go ahead and open a new topic on it and we’ll follow up with you. We apologize for the inconvenience, and appreciate your willingness to work with us to provide a supportive community.

Topic		Replies	Views
JSON schema validation failed (Redshift -> Snowflake) Connector Questions & Issues data-loading	12	765	July 14, 2022
JSON schema validation failed - Oracle to Snowflake Connector Questions & Issues normalization , connectors	3	307	August 5, 2022
JSON schema validation failed Connector Questions & Issues source-salesforce , connectors	2	680	March 12, 2023
JSON schema validation failed while syncing MySQL CDC -> BigQuery Connector Questions & Issues source-mysql , destination-bigquery , normalization , cdc	11	745	July 14, 2022
JSON schema validation failed Amazon Seller Partner Connector Questions & Issues normalization , connectors	5	486	July 14, 2022

JsonSchemaValidator doesn't respect oneOf in credentials

Related Topics