2023-03-22 05:56:11 INFO i.a.a.c.AirbyteApiClient(retryWithJitter):172 - Attempt 0 to get state 2023-03-22 05:56:11 INFO i.a.a.c.AirbyteApiClient(retryWithJitter):172 - Attempt 0 to set attempt sync config 2023-03-22 05:56:11 INFO i.a.v.j.JsonSchemaValidator(test):129 - JSON schema validation failed. errors: $.filetype: must be a constant value csv 2023-03-22 05:56:11 INFO i.a.v.j.JsonSchemaValidator(test):129 - JSON schema validation failed. errors: $.filetype: must be a constant value parquet 2023-03-22 05:56:11 INFO i.a.v.j.JsonSchemaValidator(test):129 - JSON schema validation failed. errors: $.filetype: must be a constant value avro 2023-03-22 05:56:12 INFO i.a.w.t.TemporalAttemptExecution(get):142 - Docker volume job log path: /tmp/workspace/182/0/logs.log 2023-03-22 05:56:12 INFO i.a.w.t.TemporalAttemptExecution(get):147 - Executing worker wrapper. Airbyte version: 0.42.0 2023-03-22 05:56:12 INFO i.a.a.c.AirbyteApiClient(retryWithJitter):172 - Attempt 0 to save workflow id for cancellation 2023-03-22 05:56:12 INFO i.a.c.EnvConfigs(getEnvOrDefault):1222 - Using default value for environment variable METRIC_CLIENT: '' 2023-03-22 05:56:12 WARN i.a.m.l.MetricClientFactory(initialize):60 - Metric client is already initialized to 2023-03-22 05:56:12 INFO i.a.w.g.DefaultReplicationWorker(run):164 - start sync worker. job id: 182 attempt id: 0 2023-03-22 05:56:12 INFO i.a.w.g.DefaultReplicationWorker(run):166 - Committing states from persistState activity 2023-03-22 05:56:12 INFO i.a.c.i.LineGobbler(voidCall):149 - 2023-03-22 05:56:12 INFO i.a.w.g.DefaultReplicationWorker(run):182 - configured sync modes: {null.actions2=incremental - append} 2023-03-22 05:56:12 INFO i.a.c.i.LineGobbler(voidCall):149 - ----- START REPLICATION ----- 2023-03-22 05:56:12 INFO i.a.c.i.LineGobbler(voidCall):149 - 2023-03-22 05:56:12 INFO i.a.w.i.DefaultAirbyteDestination(start):87 - Running destination... 2023-03-22 05:56:12 INFO i.a.c.EnvConfigs(getEnvOrDefault):1222 - Using default value for environment variable SIDECAR_KUBE_CPU_LIMIT: '2.0' 2023-03-22 05:56:12 INFO i.a.c.EnvConfigs(getEnvOrDefault):1222 - Using default value for environment variable SOCAT_KUBE_CPU_LIMIT: '2.0' 2023-03-22 05:56:12 INFO i.a.c.EnvConfigs(getEnvOrDefault):1222 - Using default value for environment variable SIDECAR_KUBE_CPU_REQUEST: '0.1' 2023-03-22 05:56:12 INFO i.a.c.EnvConfigs(getEnvOrDefault):1222 - Using default value for environment variable SOCAT_KUBE_CPU_REQUEST: '0.1' 2023-03-22 05:56:12 INFO i.a.c.EnvConfigs(getEnvOrDefault):1222 - Using default value for environment variable LAUNCHDARKLY_KEY: '' 2023-03-22 05:56:12 INFO i.a.c.EnvConfigs(getEnvOrDefault):1222 - Using default value for environment variable FEATURE_FLAG_CLIENT: '' 2023-03-22 05:56:12 INFO i.a.c.i.LineGobbler(voidCall):149 - Checking if airbyte/destination-clickhouse:0.2.2 exists... 2023-03-22 05:56:12 INFO i.a.c.i.LineGobbler(voidCall):149 - airbyte/destination-clickhouse:0.2.2 was found locally. 2023-03-22 05:56:12 INFO i.a.w.p.DockerProcessFactory(create):130 - Creating docker container = destination-clickhouse-write-182-0-gktew with resources io.airbyte.config.ResourceRequirements@45358673[cpuRequest=,cpuLimit=,memoryRequest=,memoryLimit=] and allowedHosts null 2023-03-22 05:56:12 INFO i.a.w.p.DockerProcessFactory(create):175 - Preparing command: docker run --rm --init -i -w /data/182/0 --log-driver none --name destination-clickhouse-write-182-0-gktew --network host -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -e DEPLOYMENT_MODE=OSS -e STRICT_COMPARISON_NORMALIZATION_WORKSPACES= -e WORKER_CONNECTOR_IMAGE=airbyte/destination-clickhouse:0.2.2 -e AUTO_DETECT_SCHEMA=true -e LAUNCHDARKLY_KEY= -e SOCAT_KUBE_CPU_REQUEST=0.1 -e STRICT_COMPARISON_NORMALIZATION_TAG=strict_comparison2 -e SOCAT_KUBE_CPU_LIMIT=2.0 -e USE_STREAM_CAPABLE_STATE=true -e FIELD_SELECTION_WORKSPACES= -e AIRBYTE_ROLE= -e WORKER_ENVIRONMENT=DOCKER -e APPLY_FIELD_SELECTION=false -e WORKER_JOB_ATTEMPT=0 -e FEATURE_FLAG_CLIENT= -e AIRBYTE_VERSION=0.42.0 -e WORKER_JOB_ID=182 airbyte/destination-clickhouse:0.2.2 write --config destination_config.json --catalog destination_catalog.json 2023-03-22 05:56:12 INFO i.a.w.i.VersionedAirbyteMessageBufferedWriterFactory(createWriter):41 - Writing messages to protocol version 0.2.0 2023-03-22 05:56:12 INFO i.a.w.i.VersionedAirbyteStreamFactory(create):107 - Reading messages from protocol version 0.2.0 2023-03-22 05:56:12 INFO i.a.c.EnvConfigs(getEnvOrDefault):1222 - Using default value for environment variable SIDECAR_KUBE_CPU_LIMIT: '2.0' 2023-03-22 05:56:12 INFO i.a.c.EnvConfigs(getEnvOrDefault):1222 - Using default value for environment variable SOCAT_KUBE_CPU_LIMIT: '2.0' 2023-03-22 05:56:12 INFO i.a.c.EnvConfigs(getEnvOrDefault):1222 - Using default value for environment variable SIDECAR_KUBE_CPU_REQUEST: '0.1' 2023-03-22 05:56:12 INFO i.a.c.EnvConfigs(getEnvOrDefault):1222 - Using default value for environment variable SOCAT_KUBE_CPU_REQUEST: '0.1' 2023-03-22 05:56:12 INFO i.a.c.EnvConfigs(getEnvOrDefault):1222 - Using default value for environment variable LAUNCHDARKLY_KEY: '' 2023-03-22 05:56:12 INFO i.a.c.EnvConfigs(getEnvOrDefault):1222 - Using default value for environment variable FEATURE_FLAG_CLIENT: '' 2023-03-22 05:56:12 INFO i.a.c.i.LineGobbler(voidCall):149 - Checking if airbyte/source-s3:2.0.3 exists... 2023-03-22 05:56:12 INFO i.a.c.i.LineGobbler(voidCall):149 - airbyte/source-s3:2.0.3 was found locally. 2023-03-22 05:56:12 INFO i.a.w.p.DockerProcessFactory(create):130 - Creating docker container = source-s3-read-182-0-cseyi with resources io.airbyte.config.ResourceRequirements@68c14b9b[cpuRequest=,cpuLimit=,memoryRequest=,memoryLimit=] and allowedHosts io.airbyte.config.AllowedHosts@dffb332[hosts=[*.s3.amazonaws.com, *.datadoghq.com, *.datadoghq.eu, *.sentry.io],additionalProperties={}] 2023-03-22 05:56:12 INFO i.a.w.p.DockerProcessFactory(create):175 - Preparing command: docker run --rm --init -i -w /data/182/0 --log-driver none --name source-s3-read-182-0-cseyi --network host -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -e DEPLOYMENT_MODE=OSS -e STRICT_COMPARISON_NORMALIZATION_WORKSPACES= -e WORKER_CONNECTOR_IMAGE=airbyte/source-s3:2.0.3 -e AUTO_DETECT_SCHEMA=true -e LAUNCHDARKLY_KEY= -e SOCAT_KUBE_CPU_REQUEST=0.1 -e STRICT_COMPARISON_NORMALIZATION_TAG=strict_comparison2 -e SOCAT_KUBE_CPU_LIMIT=2.0 -e USE_STREAM_CAPABLE_STATE=true -e FIELD_SELECTION_WORKSPACES= -e AIRBYTE_ROLE= -e WORKER_ENVIRONMENT=DOCKER -e APPLY_FIELD_SELECTION=false -e WORKER_JOB_ATTEMPT=0 -e FEATURE_FLAG_CLIENT= -e AIRBYTE_VERSION=0.42.0 -e WORKER_JOB_ID=182 airbyte/source-s3:2.0.3 read --config source_config.json --catalog source_catalog.json --state input_state.json 2023-03-22 05:56:12 INFO i.a.w.i.VersionedAirbyteStreamFactory(create):107 - Reading messages from protocol version 0.2.0 2023-03-22 05:56:12 INFO i.a.w.g.DefaultReplicationWorker(replicate):268 - Waiting for source and destination threads to complete. 2023-03-22 05:56:12 INFO i.a.w.g.DefaultReplicationWorker(lambda$readFromDstRunnable$4):312 - Destination output thread started. 2023-03-22 05:56:12 INFO i.a.w.g.DefaultReplicationWorker(lambda$readFromSrcAndWriteToDstRunnable$7):377 - Replication thread started. 2023-03-22 05:56:15 source > initialised stream with format: 2023-03-22 05:56:15 source > Starting syncing SourceS3 2023-03-22 05:56:15 source > initialised stream with format: {'filetype': 'jsonl', 'block_size': 10000, 'newlines_in_values': False, 'unexpected_field_behavior': 'infer'} 2023-03-22 05:56:15 source > Syncing stream: actions2 2023-03-22 05:56:15 source > Iterating S3 bucket 'elt-tests' 2023-03-22 05:56:15 destination > starting destination: class io.airbyte.integrations.destination.clickhouse.ClickhouseDestination 2023-03-22 05:56:15 source > finished reading a stream slice 2023-03-22 05:56:15 source > Read 17 records from actions2 stream 2023-03-22 05:56:15 source > Finished syncing actions2 2023-03-22 05:56:15 source > SourceS3 runtimes: Syncing stream actions2 0:00:00.722202 2023-03-22 05:56:15 source > Finished syncing SourceS3 2023-03-22 05:56:15 destination > integration args: {catalog=destination_catalog.json, write=null, config=destination_config.json} 2023-03-22 05:56:15 destination > Running integration: io.airbyte.integrations.base.ssh.SshWrappedDestination 2023-03-22 05:56:15 destination > Command: WRITE 2023-03-22 05:56:15 destination > Integration config: IntegrationConfig{command=WRITE, configPath='destination_config.json', catalogPath='destination_catalog.json', statePath='null'} 2023-03-22 05:56:16 destination > Unknown keyword order - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword 2023-03-22 05:56:16 destination > Unknown keyword airbyte_secret - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword 2023-03-22 05:56:16 INFO i.a.w.g.DefaultReplicationWorker(lambda$readFromSrcAndWriteToDstRunnable$7):433 - Source has no more messages, closing connection. 2023-03-22 05:56:16 INFO i.a.w.g.DefaultReplicationWorker(lambda$readFromSrcAndWriteToDstRunnable$7):442 - Total records read: 18 (16 KB) 2023-03-22 05:56:16 INFO i.a.w.g.DefaultReplicationWorker(replicate):273 - One of source or destination thread complete. Waiting on the other. 2023-03-22 05:56:16 destination > Starting connection with method: NO_TUNNEL 2023-03-22 05:56:16 destination > HikariPool-1 - Starting... 2023-03-22 05:56:16 destination > HikariPool-1 - Start completed. 2023-03-22 05:56:16 destination > Write config: WriteConfig{streamName=mindobox2actions2, namespace=null, outputSchemaName=dwh_prod, tmpTableName=_airbyte_tmp_xui_mindobox2actions2, outputTableName=_airbyte_raw_mindobox2actions2, syncMode=append} 2023-03-22 05:56:16 destination > class io.airbyte.integrations.destination.buffered_stream_consumer.BufferedStreamConsumer started. 2023-03-22 05:56:16 destination > Preparing tmp tables in destination started for 1 streams 2023-03-22 05:56:16 destination > Preparing tmp table in destination started for stream mindobox2actions2. schema: dwh_prod, tmp table name: _airbyte_tmp_xui_mindobox2actions2 2023-03-22 05:56:17 destination > Preparing tables in destination completed. 2023-03-22 05:56:17 destination > Airbyte message consumer: succeeded. 2023-03-22 05:56:17 destination > executing on success close procedure. 2023-03-22 05:56:17 destination > Flushing mindobox2actions2: 17 records (75 KB) 2023-03-22 05:56:17 destination > actual size of batch: 17 2023-03-22 05:56:17 destination > No onDestinationCloseOperations required for this destination. 2023-03-22 05:56:17 destination > Finalizing tables in destination started for 1 streams 2023-03-22 05:56:17 destination > Finalizing stream mindobox2actions2. schema dwh_prod, tmp table _airbyte_tmp_xui_mindobox2actions2, final table _airbyte_raw_mindobox2actions2 2023-03-22 05:56:17 destination > Executing finalization of tables. 2023-03-22 05:56:17 destination > Finalizing tables in destination completed. 2023-03-22 05:56:17 destination > Cleaning tmp tables in destination started for 1 streams 2023-03-22 05:56:17 destination > Cleaning tmp table in destination started for stream mindobox2actions2. schema dwh_prod, tmp table name: _airbyte_tmp_xui_mindobox2actions2 2023-03-22 05:56:17 destination > Cleaning tmp tables in destination completed. 2023-03-22 05:56:17 INFO i.a.w.g.DefaultReplicationWorker(lambda$readFromDstRunnable$4):323 - State in DefaultReplicationWorker from destination: io.airbyte.protocol.models.AirbyteMessage@4384ac47[type=STATE,log=,spec=,connectionStatus=,catalog=,record=,state=io.airbyte.protocol.models.AirbyteStateMessage@4d435a47[type=STREAM,stream=io.airbyte.protocol.models.AirbyteStreamState@497cc737[streamDescriptor=io.airbyte.protocol.models.StreamDescriptor@c7e507c[name=actions2,namespace=,additionalProperties={}],streamState={"_ab_source_file_last_modified":"2023-03-21T12:41:08Z","history":{"2023-03-21":["actions_20230321021610.json"]}},additionalProperties={}],global=,data={"actions2":{"_ab_source_file_last_modified":"2023-03-21T12:41:08Z","history":{"2023-03-21":["actions_20230321021610.json"]}}},additionalProperties={}],trace=,control=,additionalProperties={}] 2023-03-22 05:56:17 destination > Completed integration: io.airbyte.integrations.base.ssh.SshWrappedDestination 2023-03-22 05:56:17 destination > completed destination: class io.airbyte.integrations.destination.clickhouse.ClickhouseDestination 2023-03-22 05:56:17 INFO i.a.w.g.DefaultReplicationWorker(replicate):275 - Source and destination threads complete. 2023-03-22 05:56:17 INFO i.a.w.g.DefaultReplicationWorker(prepStateForLaterSaving):605 - Source output at least one state message 2023-03-22 05:56:17 INFO i.a.w.g.DefaultReplicationWorker(prepStateForLaterSaving):611 - State capture: Updated state to: Optional[io.airbyte.config.State@6c60ebb[state=[{"type":"STREAM","stream":{"stream_descriptor":{"name":"actions2"},"stream_state":{"_ab_source_file_last_modified":"2023-03-21T12:41:08Z","history":{"2023-03-21":["actions_20230321021610.json"]}}}}]]] 2023-03-22 05:56:17 INFO i.a.w.g.DefaultReplicationWorker(getReplicationOutput):538 - sync summary: { "status" : "completed", "recordsSynced" : 17, "bytesSynced" : 17135, "startTime" : 1679464572400, "endTime" : 1679464577663, "totalStats" : { "bytesEmitted" : 17135, "destinationStateMessagesEmitted" : 1, "destinationWriteEndTime" : 1679464577662, "destinationWriteStartTime" : 1679464572624, "meanSecondsBeforeSourceStateMessageEmitted" : 0, "maxSecondsBeforeSourceStateMessageEmitted" : 0, "maxSecondsBetweenStateMessageEmittedandCommitted" : 2, "meanSecondsBetweenStateMessageEmittedandCommitted" : 2, "recordsEmitted" : 17, "recordsCommitted" : 17, "replicationEndTime" : 1679464577663, "replicationStartTime" : 1679464572400, "sourceReadEndTime" : 1679464576368, "sourceReadStartTime" : 1679464572445, "sourceStateMessagesEmitted" : 1 }, "streamStats" : [ { "streamName" : "mindobox2actions2", "stats" : { "bytesEmitted" : 17135, "recordsEmitted" : 17, "recordsCommitted" : 17 } } ] } 2023-03-22 05:56:17 INFO i.a.w.g.DefaultReplicationWorker(getReplicationOutput):539 - failures: [ ] 2023-03-22 05:56:17 INFO i.a.w.t.TemporalAttemptExecution(get):169 - Stopping cancellation check scheduling... 2023-03-22 05:56:17 INFO i.a.w.t.s.ReplicationActivityImpl(lambda$replicate$4):259 - sync summary: io.airbyte.config.StandardSyncOutput@71f87464[standardSyncSummary=io.airbyte.config.StandardSyncSummary@1c7537dc[status=completed,recordsSynced=17,bytesSynced=17135,startTime=1679464572400,endTime=1679464577663,totalStats=io.airbyte.config.SyncStats@470e59b4[bytesEmitted=17135,destinationStateMessagesEmitted=1,destinationWriteEndTime=1679464577662,destinationWriteStartTime=1679464572624,estimatedBytes=,estimatedRecords=,meanSecondsBeforeSourceStateMessageEmitted=0,maxSecondsBeforeSourceStateMessageEmitted=0,maxSecondsBetweenStateMessageEmittedandCommitted=2,meanSecondsBetweenStateMessageEmittedandCommitted=2,recordsEmitted=17,recordsCommitted=17,replicationEndTime=1679464577663,replicationStartTime=1679464572400,sourceReadEndTime=1679464576368,sourceReadStartTime=1679464572445,sourceStateMessagesEmitted=1,additionalProperties={}],streamStats=[io.airbyte.config.StreamSyncStats@6ec4b422[streamName=mindobox2actions2,streamNamespace=,stats=io.airbyte.config.SyncStats@5cadb6ca[bytesEmitted=17135,destinationStateMessagesEmitted=,destinationWriteEndTime=,destinationWriteStartTime=,estimatedBytes=,estimatedRecords=,meanSecondsBeforeSourceStateMessageEmitted=,maxSecondsBeforeSourceStateMessageEmitted=,maxSecondsBetweenStateMessageEmittedandCommitted=,meanSecondsBetweenStateMessageEmittedandCommitted=,recordsEmitted=17,recordsCommitted=17,replicationEndTime=,replicationStartTime=,sourceReadEndTime=,sourceReadStartTime=,sourceStateMessagesEmitted=,additionalProperties={}]]]],normalizationSummary=,webhookOperationSummary=,state=io.airbyte.config.State@6c60ebb[state=[{"type":"STREAM","stream":{"stream_descriptor":{"name":"actions2"},"stream_state":{"_ab_source_file_last_modified":"2023-03-21T12:41:08Z","history":{"2023-03-21":["actions_20230321021610.json"]}}}}]],outputCatalog=io.airbyte.protocol.models.ConfiguredAirbyteCatalog@69d3a154[streams=[io.airbyte.protocol.models.ConfiguredAirbyteStream@5ca58623[stream=io.airbyte.protocol.models.AirbyteStream@178b4c31[name=mindobox2actions2,jsonSchema={"type":"object","properties":{"rootAction":{"type":["null","object"]},"dateTimeUtc":{"type":["null","string"]},"actionTemplate":{"type":["null","object"]},"channel":{"type":["null","object"]},"ids":{"type":["null","object"]},"_ab_source_file_last_modified":{"format":"date-time","type":"string"},"creationDateTimeUtc":{"type":["null","string"]},"mailing":{"type":["null","object"]},"_ab_additional_properties":{"type":"object"},"brand":{"type":["null","object"]},"_ab_source_file_url":{"type":"string"},"customer":{"type":["null","object"]}}},supportedSyncModes=[full_refresh, incremental],sourceDefinedCursor=true,defaultCursorField=[_ab_source_file_last_modified],sourceDefinedPrimaryKey=[],namespace=,additionalProperties={}],syncMode=incremental,cursorField=[_ab_source_file_last_modified],destinationSyncMode=append,primaryKey=[[_ab_source_file_url]],additionalProperties={}]],additionalProperties={}],failures=[],commitStateAsap=false] 2023-03-22 05:56:17 INFO i.a.w.t.s.ReplicationActivityImpl(lambda$replicate$4):264 - Sync summary length: 3078 2023-03-22 05:56:17 INFO i.a.c.t.TemporalUtils(withBackgroundHeartbeat):307 - Stopping temporal heartbeating... 2023-03-22 05:56:17 INFO i.a.c.i.LineGobbler(voidCall):149 - 2023-03-22 05:56:17 INFO i.a.c.i.LineGobbler(voidCall):149 - ----- END REPLICATION ----- 2023-03-22 05:56:17 INFO i.a.c.i.LineGobbler(voidCall):149 - 2023-03-22 05:56:17 INFO i.a.a.c.AirbyteApiClient(retryWithJitter):172 - Attempt 0 to get state 2023-03-22 05:56:17 INFO i.a.a.c.AirbyteApiClient(retryWithJitter):172 - Attempt 0 to create or update state 2023-03-22 05:56:17 INFO i.a.v.j.JsonSchemaValidator(test):129 - JSON schema validation failed. errors: $.filetype: must be a constant value csv 2023-03-22 05:56:17 INFO i.a.v.j.JsonSchemaValidator(test):129 - JSON schema validation failed. errors: $.filetype: must be a constant value parquet 2023-03-22 05:56:17 INFO i.a.v.j.JsonSchemaValidator(test):129 - JSON schema validation failed. errors: $.filetype: must be a constant value avro 2023-03-22 05:56:17 INFO i.a.c.t.StreamResetRecordsHelper(deleteStreamResetRecordsForJob):50 - deleteStreamResetRecordsForJob was called for job 182 with config type sync. Returning, as config type is not resetConnection.