2022-05-03 14:34:13 INFO i.a.w.w.WorkerRun(call):49 - Executing worker wrapper. Airbyte version: 0.36.1-alpha
2022-05-03 14:34:13 INFO i.a.w.t.TemporalAttemptExecution(get):105 - Docker volume job log path: /tmp/workspace/672/0/logs.log
2022-05-03 14:34:13 INFO i.a.w.t.TemporalAttemptExecution(get):110 - Executing worker wrapper. Airbyte version: 0.36.1-alpha
2022-05-03 14:34:13 INFO i.a.w.DefaultReplicationWorker(run):104 - start sync worker. job id: 672 attempt id: 0
2022-05-03 14:34:13 INFO i.a.w.DefaultReplicationWorker(run):116 - configured sync modes: {null.segment_tracking=full_refresh - append}
2022-05-03 14:34:13 INFO i.a.w.p.a.DefaultAirbyteDestination(start):69 - Running destination...
2022-05-03 14:34:13 INFO i.a.c.i.LineGobbler(voidCall):82 - Checking if airbyte/destination-bigquery-denormalized:0.3.3 exists...
2022-05-03 14:34:13 INFO i.a.c.i.LineGobbler(voidCall):82 - airbyte/destination-bigquery-denormalized:0.3.3 was found locally.
2022-05-03 14:34:13 INFO i.a.w.p.DockerProcessFactory(create):106 - Creating docker job ID: 672
2022-05-03 14:34:13 INFO i.a.w.p.DockerProcessFactory(create):158 - Preparing command: docker run --rm --init -i -w /data/672/0 --log-driver none --network host -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -e WORKER_JOB_ATTEMPT=0 -e WORKER_CONNECTOR_IMAGE=airbyte/destination-bigquery-denormalized:0.3.3 -e AIRBYTE_ROLE= -e WORKER_ENVIRONMENT=DOCKER -e AIRBYTE_VERSION=0.36.1-alpha -e WORKER_JOB_ID=672 airbyte/destination-bigquery-denormalized:0.3.3 write --config destination_config.json --catalog destination_catalog.json
2022-05-03 14:34:13 INFO i.a.c.i.LineGobbler(voidCall):82 - Checking if airbyte/source-kafka:0.1.5 exists...
2022-05-03 14:34:13 INFO i.a.c.i.LineGobbler(voidCall):82 - airbyte/source-kafka:0.1.5 was found locally.
2022-05-03 14:34:13 INFO i.a.w.p.DockerProcessFactory(create):106 - Creating docker job ID: 672
2022-05-03 14:34:13 INFO i.a.w.p.DockerProcessFactory(create):158 - Preparing command: docker run --rm --init -i -w /data/672/0 --log-driver none --network host -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -e WORKER_JOB_ATTEMPT=0 -e WORKER_CONNECTOR_IMAGE=airbyte/source-kafka:0.1.5 -e AIRBYTE_ROLE= -e WORKER_ENVIRONMENT=DOCKER -e AIRBYTE_VERSION=0.36.1-alpha -e WORKER_JOB_ID=672 airbyte/source-kafka:0.1.5 read --config source_config.json --catalog source_catalog.json --state input_state.json
2022-05-03 14:34:13 INFO i.a.w.DefaultReplicationWorker(lambda$getDestinationOutputRunnable$6):339 - Destination output thread started.
2022-05-03 14:34:13 INFO i.a.w.DefaultReplicationWorker(run):158 - Waiting for source and destination threads to complete.
2022-05-03 14:34:13 INFO i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):279 - Replication thread started.
2022-05-03 14:34:14 destination > SLF4J: Class path contains multiple SLF4J bindings.
2022-05-03 14:34:14 destination > SLF4J: Found binding in [jar:file:/airbyte/lib/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
2022-05-03 14:34:14 destination > SLF4J: Found binding in [jar:file:/airbyte/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
2022-05-03 14:34:14 destination > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
2022-05-03 14:34:14 destination > SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
2022-05-03 14:34:15 source > 2022-05-03 14:34:15 INFO i.a.i.s.k.KafkaSource(main):134 - Starting source: class io.airbyte.integrations.source.kafka.KafkaSource
2022-05-03 14:34:15 destination > 2022-05-03 14:34:15 INFO i.a.i.b.IntegrationCliParser(parseOptions):118 - integration args: {catalog=destination_catalog.json, write=null, config=destination_config.json}
2022-05-03 14:34:15 destination > 2022-05-03 14:34:15 INFO i.a.i.b.IntegrationRunner(runInternal):121 - Running integration: io.airbyte.integrations.destination.bigquery.BigQueryDenormalizedDestination
2022-05-03 14:34:15 destination > 2022-05-03 14:34:15 INFO i.a.i.b.IntegrationRunner(runInternal):122 - Command: WRITE
2022-05-03 14:34:15 source > 2022-05-03 14:34:15 INFO i.a.i.b.IntegrationCliParser(parseOptions):118 - integration args: {read=null, catalog=source_catalog.json, state=input_state.json, config=source_config.json}
2022-05-03 14:34:15 source > 2022-05-03 14:34:15 INFO i.a.i.b.IntegrationRunner(runInternal):121 - Running integration: io.airbyte.integrations.source.kafka.KafkaSource
2022-05-03 14:34:15 source > 2022-05-03 14:34:15 INFO i.a.i.b.IntegrationRunner(runInternal):122 - Command: READ
2022-05-03 14:34:15 destination > 2022-05-03 14:34:15 INFO i.a.i.b.IntegrationRunner(runInternal):123 - Integration config: IntegrationConfig{command=WRITE, configPath='destination_config.json', catalogPath='destination_catalog.json', statePath='null'}
2022-05-03 14:34:15 source > 2022-05-03 14:34:15 INFO i.a.i.b.IntegrationRunner(runInternal):123 - Integration config: IntegrationConfig{command=READ, configPath='source_config.json', catalogPath='source_catalog.json', statePath='input_state.json'}
2022-05-03 14:34:15 source > 2022-05-03 14:34:15 WARN c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword examples - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2022-05-03 14:34:15 source > 2022-05-03 14:34:15 WARN c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword airbyte_secret - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2022-05-03 14:34:15 source > 2022-05-03 14:34:15 INFO i.a.i.s.k.KafkaSourceConfig(propertiesByProtocol):81 - Kafka protocol config: {"security_protocol":"PLAINTEXT"}
2022-05-03 14:34:15 source > 2022-05-03 14:34:15 WARN o.a.k.c.CommonClientConfigs(warnIfDeprecatedDnsLookupValue):192 - Configuration 'client.dns.lookup' with value 'default' is deprecated and will be removed in future version. Please use 'use_all_dns_ips' or another non-deprecated value.
2022-05-03 14:34:15 destination > 2022-05-03 14:34:15 WARN c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword examples - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2022-05-03 14:34:15 destination > 2022-05-03 14:34:15 WARN c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword airbyte_secret - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2022-05-03 14:34:15 destination > 2022-05-03 14:34:15 INFO i.a.i.d.b.BigQueryUtils(getLoadingMethod):295 - Selected loading method is set to: GCS
2022-05-03 14:34:16 destination > 2022-05-03 14:34:16 INFO i.a.i.d.s.S3FormatConfigs(getS3FormatConfig):22 - S3 format config: {"format_type":"AVRO","flattening":"No flattening","part_size_mb":"5"}
2022-05-03 14:34:16 destination > 2022-05-03 14:34:16 INFO i.a.i.d.b.BigQueryUtils(isKeepFilesInGcs):311 - All tmp files will be removed from GCS when replication is finished
2022-05-03 14:34:16 source > 2022-05-03 14:34:16 INFO i.a.i.s.k.KafkaSource(check):57 - Successfully connected to Kafka brokers for topic 'alert_feature_affecting_events'.
2022-05-03 14:34:16 source > 2022-05-03 14:34:16 INFO i.a.i.s.k.KafkaSourceConfig(propertiesByProtocol):81 - Kafka protocol config: {"security_protocol":"PLAINTEXT"}
2022-05-03 14:34:16 source > 2022-05-03 14:34:16 WARN o.a.k.c.CommonClientConfigs(warnIfDeprecatedDnsLookupValue):192 - Configuration 'client.dns.lookup' with value 'default' is deprecated and will be removed in future version. Please use 'use_all_dns_ips' or another non-deprecated value.
2022-05-03 14:34:16 source > 2022-05-03 14:34:16 INFO i.a.i.s.k.KafkaSourceConfig(getConsumer):105 - Kafka subscribe method: {"topic_pattern":"segment_tracking","subscription_type":"subscribe"}
2022-05-03 14:34:16 destination > 2022-05-03 14:34:16 INFO i.a.i.d.b.BigQueryDestination(getGcsRecordConsumer):288 - Creating BigQuery staging message consumer with staging ID 966f3d7b-8f64-4f3b-8379-0933736854ac at 2022-05-03T14:34:16.138Z
2022-05-03 14:34:16 destination > 2022-05-03 14:34:16 INFO i.a.i.d.b.f.DefaultBigQueryDenormalizedRecordFormatter(formatAllOfAndAnyOfFields):68 - getSchemaFields : {"type":"object","properties":{"value":{"type":"string"}}} namingResolver io.airbyte.integrations.destination.bigquery.BigQuerySQLNameTransformer@38a1a26
2022-05-03 14:34:16 destination > 2022-05-03 14:34:16 INFO i.a.i.d.b.f.DefaultBigQueryDenormalizedRecordFormatter(getSchemaFields):222 - getSchemaFields : {"type":"object","properties":{"value":{"type":"string"}}} namingResolver io.airbyte.integrations.destination.bigquery.BigQuerySQLNameTransformer@38a1a26
2022-05-03 14:34:16 destination > 2022-05-03 14:34:16 INFO i.a.i.d.b.f.DefaultBigQueryDenormalizedRecordFormatter(getBigQuerySchema):217 - Airbyte Schema is transformed from {"type":"object","properties":{"value":{"type":"string"}}} to [Field{name=value, type=STRING, mode=null, description=null, policyTags=null}, Field{name=_airbyte_ab_id, type=STRING, mode=null, description=null, policyTags=null}, Field{name=_airbyte_emitted_at, type=TIMESTAMP, mode=null, description=null, policyTags=null}].
2022-05-03 14:34:16 destination > 2022-05-03 14:34:16 INFO i.a.i.d.b.BigQueryStagingConsumerFactory(lambda$createWriteConfigs$1):86 - BigQuery write config: BigQueryWriteConfig[streamName=kafka_segment_tracking, namespace=e2e_poc, datasetId=e2e_poc, datasetLocation=europe-west1, tmpTableId=GenericData{classInfo=[datasetId, projectId, tableId], {datasetId=e2e_poc, tableId=_airbyte_tmp_dof_kafka_segment_tracking}}, targetTableId=GenericData{classInfo=[datasetId, projectId, tableId], {datasetId=e2e_poc, tableId=kafka_segment_tracking}}, tableSchema=Schema{fields=[Field{name=value, type=STRING, mode=null, description=null, policyTags=null}, Field{name=_airbyte_ab_id, type=STRING, mode=null, description=null, policyTags=null}, Field{name=_airbyte_emitted_at, type=TIMESTAMP, mode=null, description=null, policyTags=null}]}, syncMode=append, stagedFiles=[]]
2022-05-03 14:34:16 destination > 2022-05-03 14:34:16 INFO i.a.i.d.b.BufferedStreamConsumer(startTracked):116 - class io.airbyte.integrations.destination.buffered_stream_consumer.BufferedStreamConsumer started.
2022-05-03 14:34:16 destination > 2022-05-03 14:34:16 INFO i.a.i.d.b.BigQueryStagingConsumerFactory(lambda$onStartFunction$3):98 - Preparing tmp tables in destination started for 1 streams
2022-05-03 14:34:16 destination > 2022-05-03 14:34:16 INFO i.a.i.d.b.BigQueryGcsOperations(createSchemaIfNotExists):86 - Creating dataset e2e_poc
2022-05-03 14:34:17 destination > 2022-05-03 14:34:17 INFO i.a.i.d.b.BigQueryGcsOperations(createTmpTableIfNotExists):94 - Creating tmp table GenericData{classInfo=[datasetId, projectId, tableId], {datasetId=e2e_poc, tableId=_airbyte_tmp_dof_kafka_segment_tracking}}
2022-05-03 14:34:18 destination > 2022-05-03 14:34:18 INFO i.a.i.d.b.BigQueryUtils(createPartitionedTable):131 - Partitioned table created successfully: GenericData{classInfo=[datasetId, projectId, tableId], {datasetId=e2e_poc, tableId=_airbyte_tmp_dof_kafka_segment_tracking}}
2022-05-03 14:34:18 destination > 2022-05-03 14:34:18 INFO i.a.i.d.b.BigQueryGcsOperations(createStageIfNotExists):101 - Creating staging path for stream kafka_segment_tracking (dataset e2e_poc): airbyte/mini_poc/e2e_poc_kafka_segment_tracking/2022/05/03/14/966f3d7b-8f64-4f3b-8379-0933736854ac/
2022-05-03 14:34:18 destination > 2022-05-03 14:34:18 INFO i.a.i.d.s.S3StorageOperations(createBucketObjectIfNotExists):94 - Storage Object mini-poc-staging/airbyte/mini_poc/e2e_poc_kafka_segment_tracking/2022/05/03/14/966f3d7b-8f64-4f3b-8379-0933736854ac/ does not exist in bucket; creating...
2022-05-03 14:34:18 destination > 2022-05-03 14:34:18 INFO i.a.i.d.s.S3StorageOperations(createBucketObjectIfNotExists):96 - Storage Object mini-poc-staging/airbyte/mini_poc/e2e_poc_kafka_segment_tracking/2022/05/03/14/966f3d7b-8f64-4f3b-8379-0933736854ac/ has been created in bucket.
2022-05-03 14:34:18 destination > 2022-05-03 14:34:18 INFO i.a.i.d.b.BigQueryStagingConsumerFactory(lambda$onStartFunction$3):107 - Preparing tmp tables in destination completed.
2022-05-03 14:34:20 source > 2022-05-03 14:34:20 INFO i.a.i.b.IntegrationRunner(runInternal):169 - Completed integration: io.airbyte.integrations.source.kafka.KafkaSource
2022-05-03 14:34:20 source > 2022-05-03 14:34:20 INFO i.a.i.s.k.KafkaSource(main):136 - Completed source: class io.airbyte.integrations.source.kafka.KafkaSource
2022-05-03 14:34:20 INFO i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):305 - Total records read: 0 (0 bytes)
2022-05-03 14:34:20 INFO i.a.w.DefaultReplicationWorker(run):163 - One of source or destination thread complete. Waiting on the other.
2022-05-03 14:34:20 destination > 2022-05-03 14:34:20 INFO i.a.i.b.FailureTrackingAirbyteMessageConsumer(close):65 - Airbyte message consumer: succeeded.
2022-05-03 14:34:20 destination > 2022-05-03 14:34:20 INFO i.a.i.d.b.BufferedStreamConsumer(close):170 - executing on success close procedure.
2022-05-03 14:34:20 destination > 2022-05-03 14:34:20 INFO i.a.i.d.r.SerializedBufferingStrategy(flushAll):92 - Flushing all 0 current buffers (0 bytes in total)
2022-05-03 14:34:20 destination > 2022-05-03 14:34:20 INFO i.a.i.d.b.BigQueryStagingConsumerFactory(lambda$onCloseFunction$5):137 - Copying into tables in destination started for 1 streams
2022-05-03 14:34:20 destination > 2022-05-03 14:34:20 INFO i.a.i.d.b.BigQueryGcsOperations(copyIntoTmpTableFromStage):122 - Uploading records from staging files to tmp table GenericData{classInfo=[datasetId, projectId, tableId], {datasetId=e2e_poc, tableId=_airbyte_tmp_dof_kafka_segment_tracking}} (dataset e2e_poc): []
2022-05-03 14:34:20 destination > 2022-05-03 14:34:20 INFO i.a.i.d.b.BigQueryGcsOperations(copyIntoTargetTable):162 - Copying data from tmp table GenericData{classInfo=[datasetId, projectId, tableId], {datasetId=e2e_poc, tableId=_airbyte_tmp_dof_kafka_segment_tracking}} to target table GenericData{classInfo=[datasetId, projectId, tableId], {datasetId=e2e_poc, tableId=kafka_segment_tracking}} (dataset e2e_poc, sync mode append)
2022-05-03 14:34:23 destination > 2022-05-03 14:34:23 INFO i.a.i.d.b.u.AbstractBigQueryUploader(copyTable):185 - successfully copied table: GenericData{classInfo=[datasetId, projectId, tableId], {datasetId=e2e_poc, tableId=_airbyte_tmp_dof_kafka_segment_tracking}} to table: GenericData{classInfo=[datasetId, projectId, tableId], {datasetId=e2e_poc, tableId=kafka_segment_tracking}}
2022-05-03 14:34:23 destination > 2022-05-03 14:34:23 INFO i.a.i.d.b.BigQueryStagingConsumerFactory(lambda$onCloseFunction$5):156 - Finalizing tables in destination completed
2022-05-03 14:34:23 destination > 2022-05-03 14:34:23 INFO i.a.i.d.b.BigQueryStagingConsumerFactory(lambda$onCloseFunction$5):159 - Cleaning up destination started for 1 streams
2022-05-03 14:34:23 destination > 2022-05-03 14:34:23 INFO i.a.i.d.b.BigQueryGcsOperations(dropTableIfExists):172 - Deleting tmp table GenericData{classInfo=[datasetId, projectId, tableId], {datasetId=e2e_poc, tableId=_airbyte_tmp_dof_kafka_segment_tracking}} (dataset e2e_poc)
2022-05-03 14:34:24 destination > 2022-05-03 14:34:24 INFO i.a.i.d.b.BigQueryGcsOperations(dropStageIfExists):183 - Cleaning up staging path for stream kafka_segment_tracking (dataset e2e_poc): airbyte/mini_poc/e2e_poc_kafka_segment_tracking
2022-05-03 14:34:24 destination > 2022-05-03 14:34:24 INFO i.a.i.d.g.GcsStorageOperations(cleanUpObjects):41 - Deleting object airbyte/mini_poc/e2e_poc_kafka_segment_tracking/2022/05/03/14/966f3d7b-8f64-4f3b-8379-0933736854ac/
2022-05-03 14:34:24 destination > 2022-05-03 14:34:24 INFO i.a.i.d.s.S3StorageOperations(cleanUpBucketObject):253 - Storage bucket airbyte/mini_poc/e2e_poc_kafka_segment_tracking has been cleaned-up (1 objects were deleted)...
2022-05-03 14:34:24 destination > 2022-05-03 14:34:24 INFO i.a.i.d.b.BigQueryStagingConsumerFactory(lambda$onCloseFunction$5):164 - Cleaning up destination completed.
2022-05-03 14:34:24 destination > 2022-05-03 14:34:24 INFO i.a.i.b.IntegrationRunner(runInternal):169 - Completed integration: io.airbyte.integrations.destination.bigquery.BigQueryDenormalizedDestination
2022-05-03 14:34:24 INFO i.a.w.DefaultReplicationWorker(run):165 - Source and destination threads complete.
2022-05-03 14:34:24 INFO i.a.w.DefaultReplicationWorker(run):228 - sync summary: io.airbyte.config.ReplicationAttemptSummary@7c0faca5[status=completed,recordsSynced=0,bytesSynced=0,startTime=1651588453648,endTime=1651588464313,totalStats=io.airbyte.config.SyncStats@4b3bcc14[recordsEmitted=0,bytesEmitted=0,stateMessagesEmitted=0,recordsCommitted=0],streamStats=[]]
2022-05-03 14:34:24 INFO i.a.w.DefaultReplicationWorker(run):250 - Source did not output any state messages
2022-05-03 14:34:24 WARN i.a.w.DefaultReplicationWorker(run):258 - State capture: No new state, falling back on input state: io.airbyte.config.State@27b406c4[state={}]
2022-05-03 14:34:24 INFO i.a.w.t.TemporalAttemptExecution(get):131 - Stopping cancellation check scheduling...
2022-05-03 14:34:24 INFO i.a.w.t.s.ReplicationActivityImpl(lambda$replicate$1):147 - sync summary: io.airbyte.config.StandardSyncOutput@283ed09f[standardSyncSummary=io.airbyte.config.StandardSyncSummary@44b984a6[status=completed,recordsSynced=0,bytesSynced=0,startTime=1651588453648,endTime=1651588464313,totalStats=io.airbyte.config.SyncStats@4b3bcc14[recordsEmitted=0,bytesEmitted=0,stateMessagesEmitted=0,recordsCommitted=0],streamStats=[]],normalizationSummary=<null>,state=io.airbyte.config.State@27b406c4[state={}],outputCatalog=io.airbyte.protocol.models.ConfiguredAirbyteCatalog@50eb1aba[streams=[io.airbyte.protocol.models.ConfiguredAirbyteStream@15a9ef70[stream=io.airbyte.protocol.models.AirbyteStream@4bcd6f72[name=kafka_segment_tracking,jsonSchema={"type":"object","properties":{"value":{"type":"string"}}},supportedSyncModes=[full_refresh, incremental],sourceDefinedCursor=<null>,defaultCursorField=[],sourceDefinedPrimaryKey=[],namespace=e2e_poc,additionalProperties={}],syncMode=full_refresh,cursorField=[],destinationSyncMode=append,primaryKey=[],additionalProperties={}]],additionalProperties={}],failures=[]]
2022-05-03 14:34:24 INFO i.a.w.t.TemporalUtils(withBackgroundHeartbeat):235 - Stopping temporal heartbeating...
2022-05-03 14:34:24 INFO i.a.c.p.ConfigRepository(updateConnectionState):773 - Updating connection b5212ece-8e95-43e8-ac6d-cbf552a053be state: io.airbyte.config.State@11e71660[state={}]
2022-05-03 14:34:24 INFO i.a.v.j.JsonSchemaValidator(test):56 - JSON schema validation failed. 
errors: $.topic_partitions: is missing but it is required, $.subscription_type: does not have a value in the enumeration [assign], $.subscription_type: must be a constant value assign
2022-05-03 14:34:24 INFO i.a.v.j.JsonSchemaValidator(test):56 - JSON schema validation failed. 
errors: $.credential: is not defined in the schema and the schema does not allow additional properties, $.part_size_mb: is not defined in the schema and the schema does not allow additional properties, $.gcs_bucket_name: is not defined in the schema and the schema does not allow additional properties, $.gcs_bucket_path: is not defined in the schema and the schema does not allow additional properties, $.keep_files_in_gcs-bucket: is not defined in the schema and the schema does not allow additional properties, $.method: must be a constant value Standard