2022-10-07 13:18:03 INFO i.a.v.j.JsonSchemaValidator(test):71 - JSON schema validation failed. errors: $.storage: must be a constant value HTTPS 2022-10-07 13:18:04 INFO i.a.v.j.JsonSchemaValidator(test):71 - JSON schema validation failed. errors: $.storage: must be a constant value GCS 2022-10-07 13:18:04 INFO i.a.v.j.JsonSchemaValidator(test):71 - JSON schema validation failed. errors: $.storage: must be a constant value S3 2022-10-07 13:18:33 destination > 2022-10-07 13:18:28 INFO i.a.i.b.IntegrationCliParser(parseOptions):118 - integration args: {catalog=destination_catalog.json, write=null, config=destination_config.json} 2022-10-07 13:18:04 INFO i.a.v.j.JsonSchemaValidator(test):71 - JSON schema validation failed. errors: $.storage_account: is missing but it is required, $.storage: must be a constant value AzBlob 2022-10-07 13:18:04 INFO i.a.v.j.JsonSchemaValidator(test):71 - JSON schema validation failed. errors: $.password: object found, string expected, $.storage: must be a constant value SSH 2022-10-07 13:18:04 INFO i.a.v.j.JsonSchemaValidator(test):71 - JSON schema validation failed. errors: $.password: object found, string expected, $.storage: must be a constant value SCP 2022-10-07 13:18:04 INFO i.a.v.j.JsonSchemaValidator(test):71 - JSON schema validation failed. errors: $.password: object found, string expected 2022-10-07 13:18:04 INFO i.a.v.j.JsonSchemaValidator(test):71 - JSON schema validation failed. errors: $.storage: must be a constant value local 2022-10-07 13:18:04 INFO i.a.v.j.JsonSchemaValidator(test):71 - JSON schema validation failed. errors: $.compression_codec: is missing but it is required, $.format_type: does not have a value in the enumeration [Avro] 2022-10-07 13:18:04 INFO i.a.w.t.TemporalAttemptExecution(get):108 - Cloud storage job log path: /workspace/868/0/logs.log 2022-10-07 13:18:04 INFO i.a.w.t.TemporalAttemptExecution(get):111 - Executing worker wrapper. Airbyte version: 0.40.6 2022-10-07 13:18:04 INFO i.a.w.p.KubeProcessFactory(create):100 - Attempting to start pod = source-file-check-868-0-hqjkz for airbyte/source-file:0.2.20 with resources io.airbyte.config.ResourceRequirements@9f6d799[cpuRequest=,cpuLimit=,memoryRequest=,memoryLimit=] 2022-10-07 13:18:04 INFO i.a.w.p.KubeProcessFactory(create):103 - source-file-check-868-0-hqjkz stdoutLocalPort = 9019 2022-10-07 13:18:04 INFO i.a.w.p.KubeProcessFactory(create):106 - source-file-check-868-0-hqjkz stderrLocalPort = 9021 2022-10-07 13:18:04 INFO i.a.c.i.LineGobbler(voidCall):114 - 2022-10-07 13:18:04 INFO i.a.c.i.LineGobbler(voidCall):114 - ----- START CHECK ----- 2022-10-07 13:18:04 INFO i.a.c.i.LineGobbler(voidCall):114 - 2022-10-07 13:18:04 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$9):589 - Creating stdout socket server... 2022-10-07 13:18:04 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$10):607 - Creating stderr socket server... 2022-10-07 13:18:04 INFO i.a.w.p.KubePodProcess():520 - Creating pod source-file-check-868-0-hqjkz... 2022-10-07 13:18:04 INFO i.a.w.p.KubePodProcess(waitForInitPodToRun):309 - Waiting for init container to be ready before copying files... 2022-10-07 13:18:04 INFO i.a.w.p.KubePodProcess(waitForInitPodToRun):313 - Init container present.. 2022-10-07 13:18:06 INFO i.a.w.p.KubePodProcess(waitForInitPodToRun):316 - Init container ready.. 2022-10-07 13:18:06 INFO i.a.w.p.KubePodProcess():551 - Copying files... 2022-10-07 13:18:06 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):258 - Uploading file: source_config.json 2022-10-07 13:18:06 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):266 - kubectl cp /tmp/2affc26f-70bd-4a81-bc6f-6412c7b487ef/source_config.json dev/source-file-check-868-0-hqjkz:/config/source_config.json -c init 2022-10-07 13:18:06 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):269 - Waiting for kubectl cp to complete 2022-10-07 13:18:07 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):283 - kubectl cp complete, closing process 2022-10-07 13:18:07 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):258 - Uploading file: FINISHED_UPLOADING 2022-10-07 13:18:07 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):266 - kubectl cp /tmp/1a7bf7ea-810c-4d98-b841-c7c525066517/FINISHED_UPLOADING dev/source-file-check-868-0-hqjkz:/config/FINISHED_UPLOADING -c init 2022-10-07 13:18:07 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):269 - Waiting for kubectl cp to complete 2022-10-07 13:18:08 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):283 - kubectl cp complete, closing process 2022-10-07 13:18:08 INFO i.a.w.p.KubePodProcess():554 - Waiting until pod is ready... 2022-10-07 13:18:08 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$9):598 - Setting stdout... 2022-10-07 13:18:09 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$10):610 - Setting stderr... 2022-10-07 13:18:09 INFO i.a.w.p.KubePodProcess():570 - Reading pod IP... 2022-10-07 13:18:09 INFO i.a.w.p.KubePodProcess():572 - Pod IP: 14.0.3.80 2022-10-07 13:18:09 INFO i.a.w.p.KubePodProcess():579 - Using null stdin output stream... 2022-10-07 13:18:11 INFO i.a.w.i.DefaultAirbyteStreamFactory(internalLog):103 - Checking access to ****:///uploads/jsontocsvn.json... 2022-10-07 13:18:11 WARN i.a.w.i.DefaultAirbyteStreamFactory(internalLog):100 - ignoring unsupported keyword arguments: ['connect_kwargs'] 2022-10-07 13:18:11 INFO i.a.w.i.DefaultAirbyteStreamFactory(internalLog):103 - Check succeeded 2022-10-07 13:18:13 INFO i.a.w.p.KubePodProcess(close):737 - (pod: dev / source-file-check-868-0-hqjkz) - Closed all resources for pod 2022-10-07 13:18:13 INFO i.a.w.t.TemporalAttemptExecution(get):132 - Stopping cancellation check scheduling... 2022-10-07 13:18:13 INFO i.a.c.i.LineGobbler(voidCall):114 - 2022-10-07 13:18:13 INFO i.a.c.i.LineGobbler(voidCall):114 - ----- END CHECK ----- 2022-10-07 13:18:13 INFO i.a.c.i.LineGobbler(voidCall):114 - 2022-10-07 13:18:13 INFO i.a.w.t.TemporalAttemptExecution(get):108 - Cloud storage job log path: /workspace/868/0/logs.log 2022-10-07 13:18:13 INFO i.a.w.t.TemporalAttemptExecution(get):111 - Executing worker wrapper. Airbyte version: 0.40.6 2022-10-07 13:18:13 INFO i.a.w.p.KubeProcessFactory(create):100 - Attempting to start pod = destination-s3-check-868-0-llbwu for airbyte/destination-s3:0.3.15 with resources io.airbyte.config.ResourceRequirements@9f6d799[cpuRequest=,cpuLimit=,memoryRequest=,memoryLimit=] 2022-10-07 13:18:13 INFO i.a.w.p.KubeProcessFactory(create):103 - destination-s3-check-868-0-llbwu stdoutLocalPort = 9023 2022-10-07 13:18:13 INFO i.a.w.p.KubeProcessFactory(create):106 - destination-s3-check-868-0-llbwu stderrLocalPort = 9015 2022-10-07 13:18:13 INFO i.a.c.i.LineGobbler(voidCall):114 - 2022-10-07 13:18:13 INFO i.a.c.i.LineGobbler(voidCall):114 - ----- START CHECK ----- 2022-10-07 13:18:13 INFO i.a.c.i.LineGobbler(voidCall):114 - 2022-10-07 13:18:13 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$10):607 - Creating stderr socket server... 2022-10-07 13:18:13 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$9):589 - Creating stdout socket server... 2022-10-07 13:18:13 INFO i.a.w.p.KubePodProcess():520 - Creating pod destination-s3-check-868-0-llbwu... 2022-10-07 13:18:13 INFO i.a.w.p.KubePodProcess(waitForInitPodToRun):309 - Waiting for init container to be ready before copying files... 2022-10-07 13:18:13 INFO i.a.w.p.KubePodProcess(waitForInitPodToRun):313 - Init container present.. 2022-10-07 13:18:15 INFO i.a.w.p.KubePodProcess(waitForInitPodToRun):316 - Init container ready.. 2022-10-07 13:18:15 INFO i.a.w.p.KubePodProcess():551 - Copying files... 2022-10-07 13:18:15 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):258 - Uploading file: source_config.json 2022-10-07 13:18:15 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):266 - kubectl cp /tmp/ae4fe965-45b8-4655-a3b4-336549613ef1/source_config.json dev/destination-s3-check-868-0-llbwu:/config/source_config.json -c init 2022-10-07 13:18:15 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):269 - Waiting for kubectl cp to complete 2022-10-07 13:18:16 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):283 - kubectl cp complete, closing process 2022-10-07 13:18:16 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):258 - Uploading file: FINISHED_UPLOADING 2022-10-07 13:18:16 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):266 - kubectl cp /tmp/898d6d23-92c3-49d4-a61a-a57166037eec/FINISHED_UPLOADING dev/destination-s3-check-868-0-llbwu:/config/FINISHED_UPLOADING -c init 2022-10-07 13:18:16 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):269 - Waiting for kubectl cp to complete 2022-10-07 13:18:16 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):283 - kubectl cp complete, closing process 2022-10-07 13:18:16 INFO i.a.w.p.KubePodProcess():554 - Waiting until pod is ready... 2022-10-07 13:18:17 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$9):598 - Setting stdout... 2022-10-07 13:18:17 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$10):610 - Setting stderr... 2022-10-07 13:18:18 INFO i.a.w.p.KubePodProcess():570 - Reading pod IP... 2022-10-07 13:18:18 INFO i.a.w.p.KubePodProcess():572 - Pod IP: 14.0.3.81 2022-10-07 13:18:18 INFO i.a.w.p.KubePodProcess():579 - Using null stdin output stream... 2022-10-07 13:18:18 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2022-10-07 13:18:18 INFO i.a.i.b.IntegrationCliParser(parseOptions):118 - integration args: {check=null, config=source_config.json} 2022-10-07 13:18:18 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2022-10-07 13:18:18 INFO i.a.i.b.IntegrationRunner(runInternal):104 - Running integration: io.airbyte.integrations.destination.s3.S3Destination 2022-10-07 13:18:18 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2022-10-07 13:18:18 INFO i.a.i.b.IntegrationRunner(runInternal):105 - Command: CHECK 2022-10-07 13:18:18 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2022-10-07 13:18:18 INFO i.a.i.b.IntegrationRunner(runInternal):106 - Integration config: IntegrationConfig{command=CHECK, configPath='source_config.json', catalogPath='null', statePath='null'} 2022-10-07 13:18:19 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2022-10-07 13:18:19 WARN c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword airbyte_secret - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword 2022-10-07 13:18:19 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2022-10-07 13:18:19 WARN c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword examples - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword 2022-10-07 13:18:19 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2022-10-07 13:18:19 WARN c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword order - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword 2022-10-07 13:18:19 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2022-10-07 13:18:19 WARN c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword requires - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword 2022-10-07 13:18:19 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2022-10-07 13:18:19 INFO i.a.i.d.s.S3FormatConfigs(getS3FormatConfig):22 - S3 format config: {"flattening":"Root level flattening","compression":{"compression_type":"No Compression"},"format_type":"CSV"} 2022-10-07 13:18:19 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2022-10-07 13:18:19 INFO i.a.i.d.s.S3DestinationConfig(createS3Client):190 - Creating S3 client... 2022-10-07 13:18:20 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2022-10-07 13:18:20 INFO i.a.i.d.s.S3StorageOperations(createBucketObjectIfNotExists):102 - Storage Object redshift-dev-s3-bucket/csv_path_dev does not exist in bucket; creating... 2022-10-07 13:18:20 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2022-10-07 13:18:20 INFO i.a.i.d.s.S3StorageOperations(createBucketObjectIfNotExists):104 - Storage Object redshift-dev-s3-bucket/csv_path_dev has been created in bucket. 2022-10-07 13:18:20 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2022-10-07 13:18:20 INFO i.a.i.d.s.S3Destination(testIAMUserHasListObjectPermission):156 - Started testing if IAM user can call listObjects on the destination bucket 2022-10-07 13:18:20 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2022-10-07 13:18:20 INFO i.a.i.d.s.S3Destination(testIAMUserHasListObjectPermission):159 - Finished checking for listObjects permission 2022-10-07 13:18:20 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2022-10-07 13:18:20 INFO i.a.i.d.s.S3Destination(testSingleUpload):81 - Started testing if all required credentials assigned to user for single file uploading 2022-10-07 13:18:20 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2022-10-07 13:18:20 INFO i.a.i.d.s.S3Destination(testSingleUpload):91 - Finished checking for normal upload mode 2022-10-07 13:18:20 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2022-10-07 13:18:20 INFO i.a.i.d.s.S3Destination(testMultipartUpload):95 - Started testing if all required credentials assigned to user for multipart upload 2022-10-07 13:18:20 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2022-10-07 13:18:20 INFO a.m.s.StreamTransferManager(getMultiPartOutputStreams):329 - Initiated multipart upload to redshift-dev-s3-bucket/csv_path_dev/test_1665148700661 with full ID 2.Qa68PdXXNChM7Xt1s0z7h4DGY1T1f_ehLCt7sbmZ2YLS5vmtyK6cOgrxuCIRSITubReXXB13BMx_GeLY69vpSps0v7NTOGbZigvojQd0KU2HsQqiopd1NN4R6IcRu8 2022-10-07 13:18:20 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2022-10-07 13:18:20 INFO a.m.s.MultiPartOutputStream(close):158 - Called close() on [MultipartOutputStream for parts 1 - 10000] 2022-10-07 13:18:20 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2022-10-07 13:18:20 INFO a.m.s.MultiPartOutputStream(close):158 - Called close() on [MultipartOutputStream for parts 1 - 10000] 2022-10-07 13:18:20 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2022-10-07 13:18:20 WARN a.m.s.MultiPartOutputStream(close):160 - [MultipartOutputStream for parts 1 - 10000] is already closed 2022-10-07 13:18:20 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2022-10-07 13:18:20 INFO a.m.s.StreamTransferManager(complete):367 - [Manager uploading to redshift-dev-s3-bucket/csv_path_dev/test_1665148700661 with id 2.Qa68PdX...N4R6IcRu8]: Uploading leftover stream [Part number 1 containing 3.34 MB] 2022-10-07 13:18:21 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2022-10-07 13:18:21 INFO a.m.s.StreamTransferManager(uploadStreamPart):558 - [Manager uploading to redshift-dev-s3-bucket/csv_path_dev/test_1665148700661 with id 2.Qa68PdX...N4R6IcRu8]: Finished uploading [Part number 1 containing 3.34 MB] 2022-10-07 13:18:21 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2022-10-07 13:18:21 INFO a.m.s.StreamTransferManager(complete):395 - [Manager uploading to redshift-dev-s3-bucket/csv_path_dev/test_1665148700661 with id 2.Qa68PdX...N4R6IcRu8]: Completed 2022-10-07 13:18:21 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2022-10-07 13:18:21 INFO i.a.i.d.s.S3Destination(testMultipartUpload):119 - Finished verification for multipart upload mode 2022-10-07 13:18:21 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2022-10-07 13:18:21 INFO i.a.i.b.IntegrationRunner(runInternal):152 - Completed integration: io.airbyte.integrations.destination.s3.S3Destination 2022-10-07 13:18:21 INFO i.a.w.p.KubePodProcess(close):737 - (pod: dev / destination-s3-check-868-0-llbwu) - Closed all resources for pod 2022-10-07 13:18:21 INFO i.a.c.i.LineGobbler(voidCall):114 - 2022-10-07 13:18:21 INFO i.a.c.i.LineGobbler(voidCall):114 - ----- END CHECK ----- 2022-10-07 13:18:21 INFO i.a.c.i.LineGobbler(voidCall):114 - 2022-10-07 13:18:21 INFO i.a.w.t.TemporalAttemptExecution(get):132 - Stopping cancellation check scheduling... 2022-10-07 13:18:21 INFO i.a.w.t.TemporalAttemptExecution(get):108 - Cloud storage job log path: /workspace/868/0/logs.log 2022-10-07 13:18:21 INFO i.a.w.t.TemporalAttemptExecution(get):111 - Executing worker wrapper. Airbyte version: 0.40.6 2022-10-07 13:18:21 INFO i.a.c.f.EnvVariableFeatureFlags(getEnvOrDefault):49 - Using default value for environment variable LOG_CONNECTOR_MESSAGES: 'false' 2022-10-07 13:18:21 INFO i.a.c.EnvConfigs(getEnvOrDefault):1096 - Using default value for environment variable METRIC_CLIENT: '' 2022-10-07 13:18:21 WARN i.a.m.l.MetricClientFactory(initialize):60 - Metric client is already initialized to 2022-10-07 13:18:22 INFO i.a.c.f.EnvVariableFeatureFlags(getEnvOrDefault):49 - Using default value for environment variable LOG_CONNECTOR_MESSAGES: 'false' 2022-10-07 13:18:22 INFO i.a.w.g.DefaultReplicationWorker(run):122 - start sync worker. job id: 868 attempt id: 0 2022-10-07 13:18:22 INFO i.a.w.g.DefaultReplicationWorker(run):135 - configured sync modes: {null.typeupdated=full_refresh - overwrite} 2022-10-07 13:18:22 INFO i.a.c.i.LineGobbler(voidCall):114 - 2022-10-07 13:18:22 INFO i.a.w.i.DefaultAirbyteDestination(start):69 - Running destination... 2022-10-07 13:18:22 INFO i.a.c.i.LineGobbler(voidCall):114 - ----- START REPLICATION ----- 2022-10-07 13:18:22 INFO i.a.c.i.LineGobbler(voidCall):114 - 2022-10-07 13:18:22 INFO i.a.w.p.KubeProcessFactory(create):100 - Attempting to start pod = destination-s3-write-868-0-dghhd for airbyte/destination-s3:0.3.15 with resources io.airbyte.config.ResourceRequirements@49855f8a[cpuRequest=,cpuLimit=,memoryRequest=,memoryLimit=] 2022-10-07 13:18:22 INFO i.a.w.p.KubeProcessFactory(create):103 - destination-s3-write-868-0-dghhd stdoutLocalPort = 9034 2022-10-07 13:18:22 INFO i.a.w.p.KubeProcessFactory(create):106 - destination-s3-write-868-0-dghhd stderrLocalPort = 9035 2022-10-07 13:18:22 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$9):589 - Creating stdout socket server... 2022-10-07 13:18:22 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$10):607 - Creating stderr socket server... 2022-10-07 13:18:22 INFO i.a.w.p.KubePodProcess():520 - Creating pod destination-s3-write-868-0-dghhd... 2022-10-07 13:18:22 INFO i.a.w.p.KubePodProcess(waitForInitPodToRun):309 - Waiting for init container to be ready before copying files... 2022-10-07 13:18:22 INFO i.a.w.p.KubePodProcess(waitForInitPodToRun):313 - Init container present.. 2022-10-07 13:18:24 INFO i.a.w.p.KubePodProcess(waitForInitPodToRun):316 - Init container ready.. 2022-10-07 13:18:24 INFO i.a.w.p.KubePodProcess():551 - Copying files... 2022-10-07 13:18:24 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):258 - Uploading file: destination_config.json 2022-10-07 13:18:24 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):266 - kubectl cp /tmp/78a101e0-1220-428e-9e65-488ad8744c62/destination_config.json dev/destination-s3-write-868-0-dghhd:/config/destination_config.json -c init 2022-10-07 13:18:24 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):269 - Waiting for kubectl cp to complete 2022-10-07 13:18:24 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):283 - kubectl cp complete, closing process 2022-10-07 13:18:24 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):258 - Uploading file: destination_catalog.json 2022-10-07 13:18:24 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):266 - kubectl cp /tmp/05a1a8b4-f2b9-481d-a1fc-911a427712d9/destination_catalog.json dev/destination-s3-write-868-0-dghhd:/config/destination_catalog.json -c init 2022-10-07 13:18:24 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):269 - Waiting for kubectl cp to complete 2022-10-07 13:18:25 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):283 - kubectl cp complete, closing process 2022-10-07 13:18:25 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):258 - Uploading file: FINISHED_UPLOADING 2022-10-07 13:18:25 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):266 - kubectl cp /tmp/eaaccb04-2173-4570-b7d7-96c0dc0a4af4/FINISHED_UPLOADING dev/destination-s3-write-868-0-dghhd:/config/FINISHED_UPLOADING -c init 2022-10-07 13:18:25 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):269 - Waiting for kubectl cp to complete 2022-10-07 13:18:25 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):283 - kubectl cp complete, closing process 2022-10-07 13:18:25 INFO i.a.w.p.KubePodProcess():554 - Waiting until pod is ready... 2022-10-07 13:18:26 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$9):598 - Setting stdout... 2022-10-07 13:18:26 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$10):610 - Setting stderr... 2022-10-07 13:18:27 INFO i.a.w.p.KubePodProcess():570 - Reading pod IP... 2022-10-07 13:18:27 INFO i.a.w.p.KubePodProcess():572 - Pod IP: 14.0.3.82 2022-10-07 13:18:27 INFO i.a.w.p.KubePodProcess():575 - Creating stdin socket... 2022-10-07 13:18:27 INFO i.a.w.p.KubeProcessFactory(create):100 - Attempting to start pod = source-file-read-868-0-oopll for airbyte/source-file:0.2.20 with resources io.airbyte.config.ResourceRequirements@712a9db7[cpuRequest=,cpuLimit=,memoryRequest=,memoryLimit=] 2022-10-07 13:18:27 INFO i.a.w.p.KubeProcessFactory(create):103 - source-file-read-868-0-oopll stdoutLocalPort = 9012 2022-10-07 13:18:27 INFO i.a.w.p.KubeProcessFactory(create):106 - source-file-read-868-0-oopll stderrLocalPort = 9013 2022-10-07 13:18:27 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$9):589 - Creating stdout socket server... 2022-10-07 13:18:27 INFO i.a.w.p.KubePodProcess():520 - Creating pod source-file-read-868-0-oopll... 2022-10-07 13:18:27 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$10):607 - Creating stderr socket server... 2022-10-07 13:18:27 INFO i.a.w.p.KubePodProcess(waitForInitPodToRun):309 - Waiting for init container to be ready before copying files... 2022-10-07 13:18:27 INFO i.a.w.p.KubePodProcess(waitForInitPodToRun):313 - Init container present.. 2022-10-07 13:18:30 INFO i.a.w.p.KubePodProcess(waitForInitPodToRun):316 - Init container ready.. 2022-10-07 13:18:30 INFO i.a.w.p.KubePodProcess():551 - Copying files... 2022-10-07 13:18:30 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):258 - Uploading file: source_config.json 2022-10-07 13:18:30 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):266 - kubectl cp /tmp/a28b4a96-99c6-4491-9731-8afcc4cf1684/source_config.json dev/source-file-read-868-0-oopll:/config/source_config.json -c init 2022-10-07 13:18:30 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):269 - Waiting for kubectl cp to complete 2022-10-07 13:18:31 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):283 - kubectl cp complete, closing process 2022-10-07 13:18:31 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):258 - Uploading file: source_catalog.json 2022-10-07 13:18:31 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):266 - kubectl cp /tmp/3887696f-3678-4d89-96ff-c4625e7c9202/source_catalog.json dev/source-file-read-868-0-oopll:/config/source_catalog.json -c init 2022-10-07 13:18:31 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):269 - Waiting for kubectl cp to complete 2022-10-07 13:18:31 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):283 - kubectl cp complete, closing process 2022-10-07 13:18:31 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):258 - Uploading file: FINISHED_UPLOADING 2022-10-07 13:18:31 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):266 - kubectl cp /tmp/91ea9bc4-aa08-417a-a9e9-419a87e2be0a/FINISHED_UPLOADING dev/source-file-read-868-0-oopll:/config/FINISHED_UPLOADING -c init 2022-10-07 13:18:31 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):269 - Waiting for kubectl cp to complete 2022-10-07 13:18:31 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):283 - kubectl cp complete, closing process 2022-10-07 13:18:31 INFO i.a.w.p.KubePodProcess():554 - Waiting until pod is ready... 2022-10-07 13:18:32 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$9):598 - Setting stdout... 2022-10-07 13:18:33 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$10):610 - Setting stderr... 2022-10-07 13:18:33 INFO i.a.w.p.KubePodProcess():570 - Reading pod IP... 2022-10-07 13:18:33 INFO i.a.w.p.KubePodProcess():572 - Pod IP: 14.0.3.68 2022-10-07 13:18:33 INFO i.a.w.p.KubePodProcess():579 - Using null stdin output stream... 2022-10-07 13:18:33 INFO i.a.w.g.DefaultReplicationWorker(lambda$getDestinationOutputRunnable$7):435 - Destination output thread started. 2022-10-07 13:18:33 INFO i.a.w.g.DefaultReplicationWorker(run):177 - Waiting for source and destination threads to complete. 2022-10-07 13:18:33 INFO i.a.w.g.DefaultReplicationWorker(lambda$getReplicationRunnable$6):322 - Replication thread started. 2022-10-07 13:18:35 INFO i.a.w.g.DefaultReplicationWorker(lambda$getReplicationRunnable$6):355 - Source has no more messages, closing connection. 2022-10-07 13:18:37 INFO i.a.w.p.KubePodProcess(close):737 - (pod: dev / source-file-read-868-0-oopll) - Closed all resources for pod 2022-10-07 13:18:37 INFO i.a.w.g.DefaultReplicationWorker(lambda$getReplicationRunnable$6):363 - Total records read: 1 (128 bytes) 2022-10-07 13:18:37 INFO i.a.w.g.DefaultReplicationWorker(run):182 - One of source or destination thread complete. Waiting on the other. 2022-10-07 13:18:38 INFO i.a.w.p.KubePodProcess(close):737 - (pod: dev / destination-s3-write-868-0-dghhd) - Closed all resources for pod 2022-10-07 13:18:38 INFO i.a.w.g.DefaultReplicationWorker(run):184 - Source and destination threads complete. 2022-10-07 13:18:38 INFO i.a.w.g.DefaultReplicationWorker(run):281 - Source did not output any state messages 2022-10-07 13:18:38 WARN i.a.w.g.DefaultReplicationWorker(run):292 - State capture: No state retained. 2022-10-07 13:18:38 INFO i.a.w.g.DefaultReplicationWorker(run):300 - sync summary: { "status" : "completed", "recordsSynced" : 1, "bytesSynced" : 128, "startTime" : 1665148702091, "endTime" : 1665148718227, "totalStats" : { "recordsEmitted" : 1, "bytesEmitted" : 128, "sourceStateMessagesEmitted" : 0, "destinationStateMessagesEmitted" : 0, "recordsCommitted" : 1, "meanSecondsBeforeSourceStateMessageEmitted" : 0, "maxSecondsBeforeSourceStateMessageEmitted" : 0, "maxSecondsBetweenStateMessageEmittedandCommitted" : 0, "meanSecondsBetweenStateMessageEmittedandCommitted" : 0 }, "streamStats" : [ { "streamName" : "typeupdated", "stats" : { "recordsEmitted" : 1, "bytesEmitted" : 128, "recordsCommitted" : 1 } } ] } 2022-10-07 13:18:38 INFO i.a.w.g.DefaultReplicationWorker(run):301 - failures: [ ] 2022-10-07 13:18:38 INFO i.a.w.t.TemporalAttemptExecution(get):132 - Stopping cancellation check scheduling... 2022-10-07 13:18:38 INFO i.a.w.t.s.ReplicationActivityImpl(lambda$replicate$3):166 - sync summary: io.airbyte.config.StandardSyncOutput@7687c406[standardSyncSummary=io.airbyte.config.StandardSyncSummary@6d99faac[status=completed,recordsSynced=1,bytesSynced=128,startTime=1665148702091,endTime=1665148718227,totalStats=io.airbyte.config.SyncStats@1625e223[recordsEmitted=1,bytesEmitted=128,sourceStateMessagesEmitted=0,destinationStateMessagesEmitted=0,recordsCommitted=1,meanSecondsBeforeSourceStateMessageEmitted=0,maxSecondsBeforeSourceStateMessageEmitted=0,maxSecondsBetweenStateMessageEmittedandCommitted=0,meanSecondsBetweenStateMessageEmittedandCommitted=0,additionalProperties={}],streamStats=[io.airbyte.config.StreamSyncStats@44ae206e[streamName=typeupdated,stats=io.airbyte.config.SyncStats@c417b09[recordsEmitted=1,bytesEmitted=128,sourceStateMessagesEmitted=,destinationStateMessagesEmitted=,recordsCommitted=1,meanSecondsBeforeSourceStateMessageEmitted=,maxSecondsBeforeSourceStateMessageEmitted=,maxSecondsBetweenStateMessageEmittedandCommitted=,meanSecondsBetweenStateMessageEmittedandCommitted=,additionalProperties={}]]]],normalizationSummary=,state=,outputCatalog=io.airbyte.protocol.models.ConfiguredAirbyteCatalog@2e2facb7[streams=[io.airbyte.protocol.models.ConfiguredAirbyteStream@17a267ab[stream=io.airbyte.protocol.models.AirbyteStream@223a8861[name=typeupdated,jsonSchema={"type":"object","$schema":"http://json-schema.org/draft-07/schema#","required":["Entry1","Entry2"],"properties":{"Entry1":{"type":"array","items":{"type":"object","properties":{"size":{"type":"string"},"color":{"type":"string"},"fruit":{"type":"string"}}}},"Entry2":{"type":"array","items":{"type":"object","properties":{"size":{"type":"string"},"color":{"type":"string"},"fruit":{"type":"string"}}}}}},supportedSyncModes=[full_refresh],sourceDefinedCursor=,defaultCursorField=[],sourceDefinedPrimaryKey=[],namespace=,additionalProperties={}],syncMode=full_refresh,cursorField=[],destinationSyncMode=overwrite,primaryKey=[],additionalProperties={}]],additionalProperties={}],failures=[]] 2022-10-07 13:18:38 INFO i.a.w.t.TemporalUtils(withBackgroundHeartbeat):312 - Stopping temporal heartbeating... 2022-10-07 13:18:38 INFO i.a.c.i.LineGobbler(voidCall):114 - 2022-10-07 13:18:38 INFO i.a.c.i.LineGobbler(voidCall):114 - ----- END REPLICATION ----- 2022-10-07 13:18:38 INFO i.a.c.i.LineGobbler(voidCall):114 - 2022-10-07 13:18:33 destination > 2022-10-07 13:18:28 INFO i.a.i.b.IntegrationRunner(runInternal):104 - Running integration: io.airbyte.integrations.destination.s3.S3Destination 2022-10-07 13:18:33 destination > 2022-10-07 13:18:28 INFO i.a.i.b.IntegrationRunner(runInternal):105 - Command: WRITE 2022-10-07 13:18:33 destination > 2022-10-07 13:18:28 INFO i.a.i.b.IntegrationRunner(runInternal):106 - Integration config: IntegrationConfig{command=WRITE, configPath='destination_config.json', catalogPath='destination_catalog.json', statePath='null'} 2022-10-07 13:18:33 destination > 2022-10-07 13:18:29 WARN c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword airbyte_secret - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword 2022-10-07 13:18:33 destination > 2022-10-07 13:18:29 WARN c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword examples - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword 2022-10-07 13:18:33 destination > 2022-10-07 13:18:29 WARN c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword order - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword 2022-10-07 13:18:33 destination > 2022-10-07 13:18:29 WARN c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword requires - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword 2022-10-07 13:18:33 destination > 2022-10-07 13:18:29 INFO i.a.i.d.s.S3FormatConfigs(getS3FormatConfig):22 - S3 format config: {"flattening":"Root level flattening","compression":{"compression_type":"No Compression"},"format_type":"CSV"} 2022-10-07 13:18:33 destination > 2022-10-07 13:18:29 INFO i.a.i.d.s.S3DestinationConfig(createS3Client):190 - Creating S3 client... 2022-10-07 13:18:33 destination > 2022-10-07 13:18:30 INFO i.a.i.d.s.SerializedBufferFactory(getCreateFunction):49 - S3 format config: S3CsvFormatConfig{flattening=ROOT_LEVEL, compression=NO_COMPRESSION} 2022-10-07 13:18:33 destination > 2022-10-07 13:18:30 INFO i.a.i.d.s.S3ConsumerFactory(lambda$toWriteConfig$0):83 - Write config: WriteConfig{streamName=typeupdated, namespace=null, outputBucketPath=csv_path_dev, pathFormat=csv_path_dev/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_, fullOutputPath=csv_path_dev/typeupdated/2022_10_07_1665148709249_, syncMode=overwrite} 2022-10-07 13:18:33 destination > 2022-10-07 13:18:30 INFO i.a.i.d.b.BufferedStreamConsumer(startTracked):116 - class io.airbyte.integrations.destination.buffered_stream_consumer.BufferedStreamConsumer started. 2022-10-07 13:18:33 destination > 2022-10-07 13:18:30 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):90 - Preparing bucket in destination started for 1 streams 2022-10-07 13:18:33 destination > 2022-10-07 13:18:30 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):97 - Clearing storage area in destination started for namespace null stream typeupdated bucketObject csv_path_dev pathFormat csv_path_dev/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_ 2022-10-07 13:18:33 destination > 2022-10-07 13:18:30 INFO i.a.i.d.s.S3StorageOperations(cleanUpBucketObject):250 - Storage bucket csv_path_dev has been cleaned-up (0 objects matching csv_path_dev/typeupdated/[0-9]{4}_[0-9]{2}_[0-9]{2}_[0-9]+_.* were deleted)... 2022-10-07 13:18:33 destination > 2022-10-07 13:18:30 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):100 - Clearing storage area in destination completed for namespace null stream typeupdated bucketObject csv_path_dev 2022-10-07 13:18:33 destination > 2022-10-07 13:18:30 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):104 - Preparing storage area in destination completed. 2022-10-07 13:18:35 source > Reading typeupdated (****:///uploads/jsontocsvn.json)... 2022-10-07 13:18:35 source > ignoring unsupported keyword arguments: ['connect_kwargs'] 2022-10-07 13:18:37 destination > 2022-10-07 13:18:37 INFO i.a.i.d.r.SerializedBufferingStrategy(lambda$addRecord$0):47 - Starting a new buffer for stream typeupdated (current state: 0 bytes in 0 buffers) 2022-10-07 13:18:37 destination > 2022-10-07 13:18:37 INFO i.a.i.b.FailureTrackingAirbyteMessageConsumer(close):62 - Airbyte message consumer: succeeded. 2022-10-07 13:18:37 destination > 2022-10-07 13:18:37 INFO i.a.i.d.b.BufferedStreamConsumer(close):171 - executing on success close procedure. 2022-10-07 13:18:37 destination > 2022-10-07 13:18:37 INFO i.a.i.d.r.SerializedBufferingStrategy(flushAll):100 - Flushing all 1 current buffers (0 bytes in total) 2022-10-07 13:18:37 destination > 2022-10-07 13:18:37 INFO i.a.i.d.r.SerializedBufferingStrategy(flushAll):102 - Flushing buffer of stream typeupdated (0 bytes) 2022-10-07 13:18:37 destination > 2022-10-07 13:18:37 INFO i.a.i.d.s.S3ConsumerFactory(lambda$flushBufferFunction$2):121 - Flushing buffer for stream typeupdated (0 bytes) to storage 2022-10-07 13:18:37 destination > 2022-10-07 13:18:37 INFO i.a.i.d.r.BaseSerializedBuffer(flush):138 - Finished writing data to dfcba479-2f36-4c02-8ce8-c17bd171db627719455680914303240.csv (249 bytes) 2022-10-07 13:18:37 destination > 2022-10-07 13:18:37 INFO a.m.s.StreamTransferManager(getMultiPartOutputStreams):329 - Initiated multipart upload to redshift-dev-s3-bucket/csv_path_dev/typeupdated/2022_10_07_1665148709249_0.csv with full ID huETl92HENCSstyYSARMhe9WgTUJAjqfJqG3kqbvWXWReLQVgAax1USRQ3_DIDJm7UoPNdQCD08P2kbr526KT4NglJkqK.b8TUbFBHgZg1FeHoFjwfO3E3BjPwUocpAW 2022-10-07 13:18:37 destination > 2022-10-07 13:18:37 INFO a.m.s.MultiPartOutputStream(close):158 - Called close() on [MultipartOutputStream for parts 1 - 10000] 2022-10-07 13:18:37 destination > 2022-10-07 13:18:37 INFO a.m.s.StreamTransferManager(complete):367 - [Manager uploading to redshift-dev-s3-bucket/csv_path_dev/typeupdated/2022_10_07_1665148709249_0.csv with id huETl92HE...jPwUocpAW]: Uploading leftover stream [Part number 1 containing 0.00 MB] 2022-10-07 13:18:37 destination > 2022-10-07 13:18:37 INFO a.m.s.StreamTransferManager(uploadStreamPart):558 - [Manager uploading to redshift-dev-s3-bucket/csv_path_dev/typeupdated/2022_10_07_1665148709249_0.csv with id huETl92HE...jPwUocpAW]: Finished uploading [Part number 1 containing 0.00 MB] 2022-10-07 13:18:38 destination > 2022-10-07 13:18:38 INFO a.m.s.StreamTransferManager(complete):395 - [Manager uploading to redshift-dev-s3-bucket/csv_path_dev/typeupdated/2022_10_07_1665148709249_0.csv with id huETl92HE...jPwUocpAW]: Completed 2022-10-07 13:18:38 destination > 2022-10-07 13:18:38 INFO i.a.i.d.s.S3StorageOperations(loadDataIntoBucket):198 - Uploaded buffer file to storage: dfcba479-2f36-4c02-8ce8-c17bd171db627719455680914303240.csv -> csv_path_dev/typeupdated/2022_10_07_1665148709249_0.csv (filename: 2022_10_07_1665148709249_0.csv) 2022-10-07 13:18:38 destination > 2022-10-07 13:18:38 INFO i.a.i.d.r.FileBuffer(deleteFile):90 - Deleting tempFile data dfcba479-2f36-4c02-8ce8-c17bd171db627719455680914303240.csv 2022-10-07 13:18:38 destination > 2022-10-07 13:18:38 INFO i.a.i.d.r.SerializedBufferingStrategy(close):122 - Closing buffer for stream typeupdated 2022-10-07 13:18:38 destination > 2022-10-07 13:18:38 INFO i.a.i.b.IntegrationRunner(runInternal):152 - Completed integration: io.airbyte.integrations.destination.s3.S3Destination