2022-09-04 11:26:41 INFO i.a.w.w.WorkerRun(call):49 - Executing worker wrapper. Airbyte version: 0.35.31-alpha 2022-09-04 11:26:41 INFO i.a.w.t.TemporalAttemptExecution(get):107 - Cloud storage job log path: /workspace/414/0/logs.log 2022-09-04 11:26:44 destination > SLF4J: Class path contains multiple SLF4J bindings. 2022-09-04 11:26:41 INFO i.a.w.t.TemporalAttemptExecution(get):110 - Executing worker wrapper. Airbyte version: 0.35.31-alpha 2022-09-04 11:26:41 INFO i.a.w.DefaultReplicationWorker(run):103 - start sync worker. job id: 414 attempt id: 0 2022-09-04 11:26:41 INFO i.a.w.DefaultReplicationWorker(run):115 - configured sync modes: {cdpaas-demo-test.people=incremental - append} 2022-09-04 11:26:41 INFO i.a.w.p.a.DefaultAirbyteDestination(start):69 - Running destination... 2022-09-04 11:26:41 INFO i.a.w.p.KubeProcessFactory(create):111 - Attempting to start pod = destination-redshift-sync-414-0-iykoj 2022-09-04 11:26:41 INFO i.a.w.p.KubeProcessFactory(create):114 - destination-redshift-sync-414-0-iykoj stdoutLocalPort = 9022 2022-09-04 11:26:41 INFO i.a.w.p.KubeProcessFactory(create):117 - destination-redshift-sync-414-0-iykoj stderrLocalPort = 9023 2022-09-04 11:26:41 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$8):521 - Creating stdout socket server... 2022-09-04 11:26:41 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$9):532 - Creating stderr socket server... 2022-09-04 11:26:41 INFO i.a.w.p.KubePodProcess():480 - Creating pod... 2022-09-04 11:26:41 INFO i.a.w.p.KubePodProcess(waitForInitPodToRun):290 - Waiting for init container to be ready before copying files... 2022-09-04 11:26:41 INFO i.a.w.p.KubePodProcess(waitForInitPodToRun):293 - Init container present.. 2022-09-04 11:26:42 INFO i.a.w.p.KubePodProcess(waitForInitPodToRun):296 - Init container ready.. 2022-09-04 11:26:42 INFO i.a.w.p.KubePodProcess():485 - Copying files... 2022-09-04 11:26:42 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):240 - Uploading file: destination_config.json 2022-09-04 11:26:42 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):248 - kubectl cp /tmp/2aadcd61-6cfa-47cb-aed0-652b6af02721/destination_config.json dev/destination-redshift-sync-414-0-iykoj:/config/destination_config.json -c init 2022-09-04 11:26:42 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):251 - Waiting for kubectl cp to complete 2022-09-04 11:26:42 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):265 - kubectl cp complete, closing process 2022-09-04 11:26:42 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):240 - Uploading file: destination_catalog.json 2022-09-04 11:26:42 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):248 - kubectl cp /tmp/3d0da6ec-d410-4f7c-b004-97a01753e3a0/destination_catalog.json dev/destination-redshift-sync-414-0-iykoj:/config/destination_catalog.json -c init 2022-09-04 11:26:42 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):251 - Waiting for kubectl cp to complete 2022-09-04 11:26:43 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):265 - kubectl cp complete, closing process 2022-09-04 11:26:43 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):240 - Uploading file: FINISHED_UPLOADING 2022-09-04 11:26:43 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):248 - kubectl cp /tmp/c6cc75fd-ea43-4428-bf26-a8070b57496a/FINISHED_UPLOADING dev/destination-redshift-sync-414-0-iykoj:/config/FINISHED_UPLOADING -c init 2022-09-04 11:26:43 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):251 - Waiting for kubectl cp to complete 2022-09-04 11:26:43 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):265 - kubectl cp complete, closing process 2022-09-04 11:26:43 INFO i.a.w.p.KubePodProcess():488 - Waiting until pod is ready... 2022-09-04 11:26:44 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$8):523 - Setting stdout... 2022-09-04 11:26:44 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$9):534 - Setting stderr... 2022-09-04 11:26:44 INFO i.a.w.p.KubePodProcess():502 - Reading pod IP... 2022-09-04 11:26:44 INFO i.a.w.p.KubePodProcess():504 - Pod IP: 12.0.3.112 2022-09-04 11:26:44 INFO i.a.w.p.KubePodProcess():507 - Creating stdin socket... 2022-09-04 11:26:44 INFO i.a.w.p.KubeProcessFactory(create):111 - Attempting to start pod = source-mongodb-v2-sync-414-0-fkhfh 2022-09-04 11:26:44 INFO i.a.w.p.KubeProcessFactory(create):114 - source-mongodb-v2-sync-414-0-fkhfh stdoutLocalPort = 9018 2022-09-04 11:26:44 INFO i.a.w.p.KubeProcessFactory(create):117 - source-mongodb-v2-sync-414-0-fkhfh stderrLocalPort = 9019 2022-09-04 11:26:44 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$8):521 - Creating stdout socket server... 2022-09-04 11:26:44 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$9):532 - Creating stderr socket server... 2022-09-04 11:26:44 INFO i.a.w.p.KubePodProcess():480 - Creating pod... 2022-09-04 11:26:44 INFO i.a.w.p.KubePodProcess(waitForInitPodToRun):290 - Waiting for init container to be ready before copying files... 2022-09-04 11:26:44 INFO i.a.w.p.KubePodProcess(waitForInitPodToRun):293 - Init container present.. 2022-09-04 11:26:47 INFO i.a.w.p.KubePodProcess(waitForInitPodToRun):296 - Init container ready.. 2022-09-04 11:26:47 INFO i.a.w.p.KubePodProcess():485 - Copying files... 2022-09-04 11:26:47 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):240 - Uploading file: input_state.json 2022-09-04 11:26:47 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):248 - kubectl cp /tmp/616021fe-0fae-428b-a046-b2c3a64475d0/input_state.json dev/source-mongodb-v2-sync-414-0-fkhfh:/config/input_state.json -c init 2022-09-04 11:26:47 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):251 - Waiting for kubectl cp to complete 2022-09-04 11:26:47 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):265 - kubectl cp complete, closing process 2022-09-04 11:26:47 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):240 - Uploading file: source_config.json 2022-09-04 11:26:47 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):248 - kubectl cp /tmp/7b1664e2-7c7c-4036-9e46-ddf9141c7be0/source_config.json dev/source-mongodb-v2-sync-414-0-fkhfh:/config/source_config.json -c init 2022-09-04 11:26:47 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):251 - Waiting for kubectl cp to complete 2022-09-04 11:26:47 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):265 - kubectl cp complete, closing process 2022-09-04 11:26:47 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):240 - Uploading file: source_catalog.json 2022-09-04 11:26:47 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):248 - kubectl cp /tmp/08ed6942-149a-4e83-b712-156de65108a0/source_catalog.json dev/source-mongodb-v2-sync-414-0-fkhfh:/config/source_catalog.json -c init 2022-09-04 11:26:47 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):251 - Waiting for kubectl cp to complete 2022-09-04 11:26:47 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):265 - kubectl cp complete, closing process 2022-09-04 11:26:47 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):240 - Uploading file: FINISHED_UPLOADING 2022-09-04 11:26:47 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):248 - kubectl cp /tmp/cc09000e-79ff-4b24-ba90-6a730bc0d78f/FINISHED_UPLOADING dev/source-mongodb-v2-sync-414-0-fkhfh:/config/FINISHED_UPLOADING -c init 2022-09-04 11:26:47 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):251 - Waiting for kubectl cp to complete 2022-09-04 11:26:48 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):265 - kubectl cp complete, closing process 2022-09-04 11:26:48 INFO i.a.w.p.KubePodProcess():488 - Waiting until pod is ready... 2022-09-04 11:26:49 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$8):523 - Setting stdout... 2022-09-04 11:26:49 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$9):534 - Setting stderr... 2022-09-04 11:26:50 INFO i.a.w.p.KubePodProcess():502 - Reading pod IP... 2022-09-04 11:26:50 INFO i.a.w.p.KubePodProcess():504 - Pod IP: 12.0.3.215 2022-09-04 11:26:50 INFO i.a.w.p.KubePodProcess():511 - Using null stdin output stream... 2022-09-04 11:26:50 INFO i.a.w.DefaultReplicationWorker(run):157 - Waiting for source and destination threads to complete. 2022-09-04 11:26:50 INFO i.a.w.DefaultReplicationWorker(lambda$getDestinationOutputRunnable$6):338 - Destination output thread started. 2022-09-04 11:26:50 INFO i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):278 - Replication thread started. 2022-09-04 11:27:23 INFO i.a.w.p.KubePodProcess(getReturnCode):696 - Exit code for pod source-mongodb-v2-sync-414-0-fkhfh is 0 2022-09-04 11:27:23 INFO i.a.w.p.KubePodProcess(exitValue):711 - Closed all resources for pod source-mongodb-v2-sync-414-0-fkhfh 2022-09-04 11:27:23 INFO i.a.w.DefaultReplicationWorker(lambda$getReplicationRunnable$5):304 - Total records read: 2 2022-09-04 11:27:23 INFO i.a.w.p.KubePodProcess(exitValue):711 - Closed all resources for pod source-mongodb-v2-sync-414-0-fkhfh 2022-09-04 11:27:23 INFO i.a.w.p.KubePodProcess(exitValue):711 - Closed all resources for pod source-mongodb-v2-sync-414-0-fkhfh 2022-09-04 11:27:23 INFO i.a.w.DefaultReplicationWorker(run):162 - One of source or destination thread complete. Waiting on the other. 2022-09-04 11:27:24 INFO i.a.w.DefaultReplicationWorker(lambda$getDestinationOutputRunnable$6):348 - State in DefaultReplicationWorker from destination: io.airbyte.protocol.models.AirbyteMessage@5ae2db2[type=STATE,log=,spec=,connectionStatus=,catalog=,record=,state=io.airbyte.protocol.models.AirbyteStateMessage@62856f88[data={"cdc":false,"streams":[{"stream_name":"people","stream_namespace":"cdpaas-demo-test","cursor_field":["name"],"cursor":"Jane Doe UPDATED"}]},additionalProperties={}],additionalProperties={}] 2022-09-04 11:26:44 destination > SLF4J: Found binding in [jar:file:/airbyte/lib/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] 2022-09-04 11:26:44 destination > SLF4J: Found binding in [jar:file:/airbyte/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] 2022-09-04 11:26:44 destination > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. 2022-09-04 11:26:44 destination > SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] 2022-09-04 11:26:50 destination > 2022-09-04 11:26:45 INFO i.a.i.d.r.RedshiftDestination(main):77 - starting destination: class io.airbyte.integrations.destination.redshift.RedshiftDestination 2022-09-04 11:26:50 destination > 2022-09-04 11:26:46 INFO i.a.i.b.IntegrationCliParser(parseOptions):118 - integration args: {catalog=destination_catalog.json, write=null, config=destination_config.json} 2022-09-04 11:26:50 destination > 2022-09-04 11:26:46 INFO i.a.i.b.IntegrationRunner(runInternal):105 - Running integration: io.airbyte.integrations.destination.redshift.RedshiftDestination 2022-09-04 11:26:50 destination > 2022-09-04 11:26:46 INFO i.a.i.b.IntegrationRunner(runInternal):106 - Command: WRITE 2022-09-04 11:26:50 destination > 2022-09-04 11:26:46 INFO i.a.i.b.IntegrationRunner(runInternal):107 - Integration config: IntegrationConfig{command=WRITE, configPath='destination_config.json', catalogPath='destination_catalog.json', statePath='null'} 2022-09-04 11:26:50 destination > 2022-09-04 11:26:46 WARN c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword examples - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword 2022-09-04 11:26:50 destination > 2022-09-04 11:26:46 WARN c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword airbyte_secret - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword 2022-09-04 11:26:50 destination > 2022-09-04 11:26:46 INFO i.a.i.d.j.c.SwitchingDestination(getConsumer):65 - Using destination type: INSERT 2022-09-04 11:26:50 destination > 2022-09-04 11:26:46 INFO i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$toWriteConfig$0):96 - Write config: WriteConfig{streamName=mongoDB_people, namespace=_mehak_pipeline_mongo_db_1d6287abb53046e1b1cf59071f7622e4, outputSchemaName=_mehak_pipeline_mongo_db_1d6287abb53046e1b1cf59071f7622e4, tmpTableName=_airbyte_tmp_wdg_mongodb_people, outputTableName=_airbyte_raw_mongodb_people, syncMode=append} 2022-09-04 11:26:50 destination > 2022-09-04 11:26:46 INFO i.a.i.d.b.BufferedStreamConsumer(startTracked):140 - class io.airbyte.integrations.destination.buffered_stream_consumer.BufferedStreamConsumer started. 2022-09-04 11:26:50 destination > 2022-09-04 11:26:46 INFO i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onStartFunction$1):121 - Preparing tmp tables in destination started for 1 streams 2022-09-04 11:26:50 destination > 2022-09-04 11:26:46 INFO i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onStartFunction$1):125 - Preparing tmp table in destination started for stream mongoDB_people. schema: _mehak_pipeline_mongo_db_1d6287abb53046e1b1cf59071f7622e4, tmp table name: _airbyte_tmp_wdg_mongodb_people 2022-09-04 11:26:50 destination > 2022-09-04 11:26:47 INFO i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onStartFunction$1):131 - Preparing tables in destination completed. 2022-09-04 11:26:50 source > 2022-09-04 11:26:50 INFO i.a.i.s.m.MongoDbSource(main):64 - starting source: class io.airbyte.integrations.source.mongodb.MongoDbSource 2022-09-04 11:26:50 source > 2022-09-04 11:26:50 INFO i.a.i.b.IntegrationRunner(run):76 - Running integration: io.airbyte.integrations.source.mongodb.MongoDbSource 2022-09-04 11:26:50 source > 2022-09-04 11:26:50 INFO i.a.i.b.IntegrationCliParser(parseOptions):118 - integration args: {read=null, catalog=source_catalog.json, state=input_state.json, config=source_config.json} 2022-09-04 11:26:50 source > 2022-09-04 11:26:50 INFO i.a.i.b.IntegrationRunner(run):80 - Command: READ 2022-09-04 11:26:50 source > 2022-09-04 11:26:50 INFO i.a.i.b.IntegrationRunner(run):81 - Integration config: IntegrationConfig{command=READ, configPath='source_config.json', catalogPath='source_catalog.json', statePath='input_state.json'} 2022-09-04 11:26:50 source > 2022-09-04 11:26:50 WARN c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword order - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword 2022-09-04 11:26:50 source > 2022-09-04 11:26:50 WARN c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword examples - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword 2022-09-04 11:26:50 source > 2022-09-04 11:26:50 WARN c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword airbyte_secret - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword 2022-09-04 11:26:50 source > 2022-09-04 11:26:50 INFO i.a.i.s.r.CdcStateManager():26 - Initialized CDC state with: null 2022-09-04 11:26:50 source > 2022-09-04 11:26:50 INFO i.a.i.s.r.StateManager(createCursorInfoForStream):108 - Found matching cursor in state. Stream: AirbyteStreamNameNamespacePair{name='people', namespace='cdpaas-demo-test'}. Cursor Field: name Value: Jane Doe 2022-09-04 11:26:51 source > 2022-09-04 11:26:51 INFO c.m.d.l.SLF4JLogger(info):71 - Cluster created with settings {hosts=[127.0.0.1:27017], srvHost=cluster1.fdwxj2a.mongodb.net, mode=MULTIPLE, requiredClusterType=REPLICA_SET, serverSelectionTimeout='30000 ms', requiredReplicaSetName='atlas-elgph3-shard-0'} 2022-09-04 11:26:51 source > 2022-09-04 11:26:51 INFO c.m.d.l.SLF4JLogger(info):71 - Adding discovered server ac-djgyhho-shard-00-01.fdwxj2a.mongodb.net:27017 to client view of cluster 2022-09-04 11:26:51 source > 2022-09-04 11:26:51 INFO c.m.d.l.SLF4JLogger(info):71 - Cluster description not yet available. Waiting for 30000 ms before timing out 2022-09-04 11:26:51 source > 2022-09-04 11:26:51 INFO c.m.d.l.SLF4JLogger(info):71 - Adding discovered server ac-djgyhho-shard-00-02.fdwxj2a.mongodb.net:27017 to client view of cluster 2022-09-04 11:26:51 source > 2022-09-04 11:26:51 INFO c.m.d.l.SLF4JLogger(info):71 - Adding discovered server ac-djgyhho-shard-00-00.fdwxj2a.mongodb.net:27017 to client view of cluster 2022-09-04 11:26:51 source > 2022-09-04 11:26:51 INFO c.m.d.l.SLF4JLogger(info):71 - No server chosen by com.mongodb.client.internal.MongoClientDelegate$1@91c4a3f from cluster description ClusterDescription{type=REPLICA_SET, connectionMode=MULTIPLE, serverDescriptions=[ServerDescription{address=ac-djgyhho-shard-00-02.fdwxj2a.mongodb.net:27017, type=UNKNOWN, state=CONNECTING}, ServerDescription{address=ac-djgyhho-shard-00-00.fdwxj2a.mongodb.net:27017, type=UNKNOWN, state=CONNECTING}, ServerDescription{address=ac-djgyhho-shard-00-01.fdwxj2a.mongodb.net:27017, type=UNKNOWN, state=CONNECTING}]}. Waiting for 30000 ms before timing out 2022-09-04 11:26:51 source > 2022-09-04 11:26:51 INFO c.m.d.l.SLF4JLogger(info):71 - Opened connection [connectionId{localValue:2, serverValue:80949}] to ac-djgyhho-shard-00-02.fdwxj2a.mongodb.net:27017 2022-09-04 11:26:51 source > 2022-09-04 11:26:51 INFO c.m.d.l.SLF4JLogger(info):71 - Opened connection [connectionId{localValue:1, serverValue:79370}] to ac-djgyhho-shard-00-01.fdwxj2a.mongodb.net:27017 2022-09-04 11:26:51 source > 2022-09-04 11:26:51 INFO c.m.d.l.SLF4JLogger(info):71 - Opened connection [connectionId{localValue:3, serverValue:80949}] to ac-djgyhho-shard-00-02.fdwxj2a.mongodb.net:27017 2022-09-04 11:26:51 source > 2022-09-04 11:26:51 INFO c.m.d.l.SLF4JLogger(info):71 - Opened connection [connectionId{localValue:4, serverValue:80173}] to ac-djgyhho-shard-00-00.fdwxj2a.mongodb.net:27017 2022-09-04 11:26:51 source > 2022-09-04 11:26:51 INFO c.m.d.l.SLF4JLogger(info):71 - Opened connection [connectionId{localValue:6, serverValue:80173}] to ac-djgyhho-shard-00-00.fdwxj2a.mongodb.net:27017 2022-09-04 11:26:51 source > 2022-09-04 11:26:51 INFO c.m.d.l.SLF4JLogger(info):71 - Opened connection [connectionId{localValue:5, serverValue:79370}] to ac-djgyhho-shard-00-01.fdwxj2a.mongodb.net:27017 2022-09-04 11:26:51 source > 2022-09-04 11:26:51 INFO c.m.d.l.SLF4JLogger(info):71 - Monitor thread successfully connected to server with description ServerDescription{address=ac-djgyhho-shard-00-01.fdwxj2a.mongodb.net:27017, type=REPLICA_SET_SECONDARY, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=13, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=366240198, setName='atlas-elgph3-shard-0', canonicalAddress=ac-djgyhho-shard-00-01.fdwxj2a.mongodb.net:27017, hosts=[ac-djgyhho-shard-00-00.fdwxj2a.mongodb.net:27017, ac-djgyhho-shard-00-01.fdwxj2a.mongodb.net:27017, ac-djgyhho-shard-00-02.fdwxj2a.mongodb.net:27017], passives=[], arbiters=[], primary='ac-djgyhho-shard-00-02.fdwxj2a.mongodb.net:27017', tagSet=TagSet{[Tag{name='nodeType', value='ELECTABLE'}, Tag{name='provider', value='AWS'}, Tag{name='region', value='US_WEST_2'}, Tag{name='workloadType', value='OPERATIONAL'}]}, electionId=null, setVersion=7, topologyVersion=TopologyVersion{processId=630fc0928e80c3c796cfe974, counter=3}, lastWriteDate=Sun Sep 04 11:26:51 UTC 2022, lastUpdateTimeNanos=1971882290446565} 2022-09-04 11:26:51 source > 2022-09-04 11:26:51 INFO c.m.d.l.SLF4JLogger(info):71 - Monitor thread successfully connected to server with description ServerDescription{address=ac-djgyhho-shard-00-02.fdwxj2a.mongodb.net:27017, type=REPLICA_SET_PRIMARY, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=13, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=372128507, setName='atlas-elgph3-shard-0', canonicalAddress=ac-djgyhho-shard-00-02.fdwxj2a.mongodb.net:27017, hosts=[ac-djgyhho-shard-00-00.fdwxj2a.mongodb.net:27017, ac-djgyhho-shard-00-01.fdwxj2a.mongodb.net:27017, ac-djgyhho-shard-00-02.fdwxj2a.mongodb.net:27017], passives=[], arbiters=[], primary='ac-djgyhho-shard-00-02.fdwxj2a.mongodb.net:27017', tagSet=TagSet{[Tag{name='nodeType', value='ELECTABLE'}, Tag{name='provider', value='AWS'}, Tag{name='region', value='US_WEST_2'}, Tag{name='workloadType', value='OPERATIONAL'}]}, electionId=7fffffff0000000000000049, setVersion=7, topologyVersion=TopologyVersion{processId=630fbfb580dd0d9f98e2e20f, counter=6}, lastWriteDate=Sun Sep 04 11:26:51 UTC 2022, lastUpdateTimeNanos=1971882286969333} 2022-09-04 11:26:51 source > 2022-09-04 11:26:51 INFO c.m.d.l.SLF4JLogger(info):71 - Monitor thread successfully connected to server with description ServerDescription{address=ac-djgyhho-shard-00-00.fdwxj2a.mongodb.net:27017, type=REPLICA_SET_SECONDARY, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=13, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=382457549, setName='atlas-elgph3-shard-0', canonicalAddress=ac-djgyhho-shard-00-00.fdwxj2a.mongodb.net:27017, hosts=[ac-djgyhho-shard-00-00.fdwxj2a.mongodb.net:27017, ac-djgyhho-shard-00-01.fdwxj2a.mongodb.net:27017, ac-djgyhho-shard-00-02.fdwxj2a.mongodb.net:27017], passives=[], arbiters=[], primary='ac-djgyhho-shard-00-02.fdwxj2a.mongodb.net:27017', tagSet=TagSet{[Tag{name='nodeType', value='ELECTABLE'}, Tag{name='provider', value='AWS'}, Tag{name='region', value='US_WEST_2'}, Tag{name='workloadType', value='OPERATIONAL'}]}, electionId=null, setVersion=7, topologyVersion=TopologyVersion{processId=630fbe9c143d0f6f18ccf70a, counter=4}, lastWriteDate=Sun Sep 04 11:26:51 UTC 2022, lastUpdateTimeNanos=1971882296340022} 2022-09-04 11:26:51 source > 2022-09-04 11:26:51 INFO c.m.d.l.SLF4JLogger(info):71 - Setting max election id to 7fffffff0000000000000049 from replica set primary ac-djgyhho-shard-00-02.fdwxj2a.mongodb.net:27017 2022-09-04 11:26:51 source > 2022-09-04 11:26:51 INFO c.m.d.l.SLF4JLogger(info):71 - Setting max set version to 7 from replica set primary ac-djgyhho-shard-00-02.fdwxj2a.mongodb.net:27017 2022-09-04 11:26:51 source > 2022-09-04 11:26:51 INFO c.m.d.l.SLF4JLogger(info):71 - Discovered replica set primary ac-djgyhho-shard-00-02.fdwxj2a.mongodb.net:27017 2022-09-04 11:26:52 source > 2022-09-04 11:26:52 INFO c.m.d.l.SLF4JLogger(info):71 - Opened connection [connectionId{localValue:7, serverValue:80901}] to ac-djgyhho-shard-00-02.fdwxj2a.mongodb.net:27017 2022-09-04 11:26:52 source > 2022-09-04 11:26:52 INFO i.a.i.s.r.StateDecoratingIterator(computeNext):60 - State Report: stream name: AirbyteStreamNameNamespacePair{name='people', namespace='cdpaas-demo-test'}, original cursor field: name, original cursor Jane Doe, cursor field: name, new cursor: Jane Doe UPDATED 2022-09-04 11:26:52 source > 2022-09-04 11:26:52 INFO i.a.i.s.r.AbstractDbSource(lambda$read$2):123 - Closing database connection pool. 2022-09-04 11:26:52 source > 2022-09-04 11:26:52 INFO i.a.i.s.r.AbstractDbSource(lambda$read$2):125 - Closed database connection pool. 2022-09-04 11:26:52 source > 2022-09-04 11:26:52 INFO i.a.i.b.IntegrationRunner(run):133 - Completed integration: io.airbyte.integrations.source.mongodb.MongoDbSource 2022-09-04 11:26:52 source > 2022-09-04 11:26:52 INFO i.a.i.s.m.MongoDbSource(main):66 - completed source: class io.airbyte.integrations.source.mongodb.MongoDbSource 2022-09-04 11:27:23 destination > 2022-09-04 11:27:23 INFO i.a.i.b.FailureTrackingAirbyteMessageConsumer(close):63 - Airbyte message consumer: succeeded. 2022-09-04 11:27:23 destination > 2022-09-04 11:27:23 INFO i.a.i.d.b.BufferedStreamConsumer(close):217 - executing on success close procedure. 2022-09-04 11:27:23 destination > 2022-09-04 11:27:23 INFO i.a.i.d.r.RedshiftSqlOperations(insertRecordsInternal):42 - actual size of batch: 1 2022-09-04 11:27:23 destination > 2022-09-04 11:27:23 INFO i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onCloseFunction$3):160 - Finalizing tables in destination started for 1 streams 2022-09-04 11:27:23 destination > 2022-09-04 11:27:23 INFO i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onCloseFunction$3):165 - Finalizing stream mongoDB_people. schema _mehak_pipeline_mongo_db_1d6287abb53046e1b1cf59071f7622e4, tmp table _airbyte_tmp_wdg_mongodb_people, final table _airbyte_raw_mongodb_people 2022-09-04 11:27:23 destination > 2022-09-04 11:27:23 INFO i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onCloseFunction$3):178 - Executing finalization of tables. 2022-09-04 11:27:24 destination > 2022-09-04 11:27:24 INFO i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onCloseFunction$3):180 - Finalizing tables in destination completed. 2022-09-04 11:27:24 destination > 2022-09-04 11:27:24 INFO i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onCloseFunction$3):183 - Cleaning tmp tables in destination started for 1 streams 2022-09-04 11:27:24 destination > 2022-09-04 11:27:24 INFO i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onCloseFunction$3):187 - Cleaning tmp table in destination started for stream mongoDB_people. schema _mehak_pipeline_mongo_db_1d6287abb53046e1b1cf59071f7622e4, tmp table name: _airbyte_tmp_wdg_mongodb_people 2022-09-04 11:27:24 destination > 2022-09-04 11:27:24 INFO i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onCloseFunction$3):192 - Cleaning tmp tables in destination completed. 2022-09-04 11:27:24 destination > 2022-09-04 11:27:24 INFO i.a.i.b.IntegrationRunner(runInternal):153 - Completed integration: io.airbyte.integrations.destination.redshift.RedshiftDestination 2022-09-04 11:27:24 destination > 2022-09-04 11:27:24 INFO i.a.i.d.r.RedshiftDestination(main):79 - completed destination: class io.airbyte.integrations.destination.redshift.RedshiftDestination 2022-09-04 11:28:05 INFO i.a.v.j.JsonSchemaValidator(test):56 - JSON schema validation failed. errors: $.host: is missing but it is required, $.port: is missing but it is required, $.instance: does not have a value in the enumeration [standalone] 2022-09-04 11:28:05 INFO i.a.v.j.JsonSchemaValidator(test):56 - JSON schema validation failed. errors: $.server_addresses: is missing but it is required, $.instance: does not have a value in the enumeration [replica] 2022-09-04 11:27:54 INFO i.a.w.p.KubePodProcess(getReturnCode):696 - Exit code for pod destination-redshift-sync-414-0-iykoj is 0 2022-09-04 11:27:54 INFO i.a.w.p.KubePodProcess(exitValue):711 - Closed all resources for pod destination-redshift-sync-414-0-iykoj 2022-09-04 11:27:54 INFO i.a.w.p.KubePodProcess(exitValue):711 - Closed all resources for pod destination-redshift-sync-414-0-iykoj 2022-09-04 11:27:54 INFO i.a.w.p.KubePodProcess(exitValue):711 - Closed all resources for pod destination-redshift-sync-414-0-iykoj 2022-09-04 11:27:54 INFO i.a.w.DefaultReplicationWorker(run):164 - Source and destination threads complete. 2022-09-04 11:27:54 INFO i.a.w.p.KubePodProcess(exitValue):711 - Closed all resources for pod source-mongodb-v2-sync-414-0-fkhfh 2022-09-04 11:27:54 INFO i.a.w.p.KubePodProcess(exitValue):711 - Closed all resources for pod source-mongodb-v2-sync-414-0-fkhfh 2022-09-04 11:27:54 INFO i.a.w.p.KubePodProcess(exitValue):711 - Closed all resources for pod source-mongodb-v2-sync-414-0-fkhfh 2022-09-04 11:27:54 INFO i.a.w.p.KubePodProcess(exitValue):711 - Closed all resources for pod source-mongodb-v2-sync-414-0-fkhfh 2022-09-04 11:27:54 INFO i.a.w.p.KubePodProcess(exitValue):711 - Closed all resources for pod destination-redshift-sync-414-0-iykoj 2022-09-04 11:27:54 INFO i.a.w.p.KubePodProcess(exitValue):711 - Closed all resources for pod destination-redshift-sync-414-0-iykoj 2022-09-04 11:27:54 INFO i.a.w.p.KubePodProcess(exitValue):711 - Closed all resources for pod destination-redshift-sync-414-0-iykoj 2022-09-04 11:27:54 INFO i.a.w.p.KubePodProcess(exitValue):711 - Closed all resources for pod destination-redshift-sync-414-0-iykoj 2022-09-04 11:27:54 INFO i.a.w.DefaultReplicationWorker(run):227 - sync summary: io.airbyte.config.ReplicationAttemptSummary@2a62c5c9[status=completed,recordsSynced=1,bytesSynced=136,startTime=1662290801716,endTime=1662290874562,totalStats=io.airbyte.config.SyncStats@4aba9faf[recordsEmitted=1,bytesEmitted=136,stateMessagesEmitted=1,recordsCommitted=1],streamStats=[io.airbyte.config.StreamSyncStats@63bdd5[streamName=mongoDB_people,stats=io.airbyte.config.SyncStats@43f12db4[recordsEmitted=1,bytesEmitted=136,stateMessagesEmitted=,recordsCommitted=1]]]] 2022-09-04 11:27:54 INFO i.a.w.DefaultReplicationWorker(run):247 - Source output at least one state message 2022-09-04 11:27:54 INFO i.a.w.DefaultReplicationWorker(run):253 - State capture: Updated state to: Optional[io.airbyte.config.State@464b76b4[state={"cdc":false,"streams":[{"stream_name":"people","stream_namespace":"cdpaas-demo-test","cursor_field":["name"],"cursor":"Jane Doe UPDATED"}]}]] 2022-09-04 11:27:54 INFO i.a.w.t.TemporalAttemptExecution(get):131 - Stopping cancellation check scheduling... 2022-09-04 11:27:54 INFO i.a.w.t.s.ReplicationActivityImpl(lambda$replicate$1):144 - sync summary: io.airbyte.config.StandardSyncOutput@32c6de86[standardSyncSummary=io.airbyte.config.StandardSyncSummary@54642737[status=completed,recordsSynced=1,bytesSynced=136,startTime=1662290801716,endTime=1662290874562,totalStats=io.airbyte.config.SyncStats@4aba9faf[recordsEmitted=1,bytesEmitted=136,stateMessagesEmitted=1,recordsCommitted=1],streamStats=[io.airbyte.config.StreamSyncStats@63bdd5[streamName=mongoDB_people,stats=io.airbyte.config.SyncStats@43f12db4[recordsEmitted=1,bytesEmitted=136,stateMessagesEmitted=,recordsCommitted=1]]]],state=io.airbyte.config.State@464b76b4[state={"cdc":false,"streams":[{"stream_name":"people","stream_namespace":"cdpaas-demo-test","cursor_field":["name"],"cursor":"Jane Doe UPDATED"}]}],outputCatalog=io.airbyte.protocol.models.ConfiguredAirbyteCatalog@5923839d[streams=[io.airbyte.protocol.models.ConfiguredAirbyteStream@fedf271[stream=io.airbyte.protocol.models.AirbyteStream@bdb3fe4[name=mongoDB_people,jsonSchema={"type":"object","properties":{"_id":{"type":"string"},"age":{"type":"number"},"name":{"type":"string"},"email":{"type":"string"},"hobbies":{"type":"array"}}},supportedSyncModes=[full_refresh, incremental],sourceDefinedCursor=false,defaultCursorField=[],sourceDefinedPrimaryKey=[],namespace=_mehak_pipeline_mongo_db_1d6287abb53046e1b1cf59071f7622e4,additionalProperties={}],syncMode=incremental,cursorField=[name],destinationSyncMode=append,primaryKey=[],additionalProperties={}]],additionalProperties={}],failures=[]] 2022-09-04 11:27:54 INFO i.a.w.t.TemporalUtils(withBackgroundHeartbeat):235 - Stopping temporal heartbeating... 2022-09-04 11:27:54 INFO i.a.c.p.ConfigRepository(updateConnectionState):551 - Updating connection f4ee33de-e327-410c-a080-21ee89ca1651 state: io.airbyte.config.State@78870bc2[state={"cdc":false,"streams":[{"stream_name":"people","stream_namespace":"cdpaas-demo-test","cursor_field":["name"],"cursor":"Jane Doe UPDATED"}]}] 2022-09-04 11:27:54 INFO i.a.w.t.TemporalAttemptExecution(get):107 - Cloud storage job log path: /workspace/414/0/logs.log 2022-09-04 11:27:54 INFO i.a.w.t.TemporalAttemptExecution(get):110 - Executing worker wrapper. Airbyte version: 0.35.31-alpha 2022-09-04 11:27:54 INFO i.a.w.DefaultNormalizationWorker(run):46 - Running normalization. 2022-09-04 11:27:54 INFO i.a.w.n.DefaultNormalizationRunner(runProcess):122 - Running with normalization version: airbyte/normalization:0.1.67 2022-09-04 11:27:54 INFO i.a.w.p.KubeProcessFactory(create):111 - Attempting to start pod = normalization-sync-414-0-vlwsz 2022-09-04 11:27:54 INFO i.a.w.p.KubeProcessFactory(create):114 - normalization-sync-414-0-vlwsz stdoutLocalPort = 9038 2022-09-04 11:27:54 INFO i.a.w.p.KubeProcessFactory(create):117 - normalization-sync-414-0-vlwsz stderrLocalPort = 9039 2022-09-04 11:27:54 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$8):521 - Creating stdout socket server... 2022-09-04 11:27:54 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$9):532 - Creating stderr socket server... 2022-09-04 11:27:54 INFO i.a.w.p.KubePodProcess():480 - Creating pod... 2022-09-04 11:27:54 INFO i.a.w.p.KubePodProcess(waitForInitPodToRun):290 - Waiting for init container to be ready before copying files... 2022-09-04 11:27:54 INFO i.a.w.p.KubePodProcess(waitForInitPodToRun):293 - Init container present.. 2022-09-04 11:27:55 INFO i.a.w.p.KubePodProcess(waitForInitPodToRun):296 - Init container ready.. 2022-09-04 11:27:55 INFO i.a.w.p.KubePodProcess():485 - Copying files... 2022-09-04 11:27:55 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):240 - Uploading file: destination_config.json 2022-09-04 11:27:55 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):248 - kubectl cp /tmp/94c4fac5-6113-4cc9-984e-32e45839ce08/destination_config.json dev/normalization-sync-414-0-vlwsz:/config/destination_config.json -c init 2022-09-04 11:27:55 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):251 - Waiting for kubectl cp to complete 2022-09-04 11:27:55 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):265 - kubectl cp complete, closing process 2022-09-04 11:27:55 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):240 - Uploading file: destination_catalog.json 2022-09-04 11:27:55 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):248 - kubectl cp /tmp/3258f128-4375-4422-a5ed-750c206faeb0/destination_catalog.json dev/normalization-sync-414-0-vlwsz:/config/destination_catalog.json -c init 2022-09-04 11:27:55 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):251 - Waiting for kubectl cp to complete 2022-09-04 11:27:55 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):265 - kubectl cp complete, closing process 2022-09-04 11:27:55 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):240 - Uploading file: FINISHED_UPLOADING 2022-09-04 11:27:55 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):248 - kubectl cp /tmp/8ed117bb-7f35-4e25-96ea-e6f5cb45e209/FINISHED_UPLOADING dev/normalization-sync-414-0-vlwsz:/config/FINISHED_UPLOADING -c init 2022-09-04 11:27:55 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):251 - Waiting for kubectl cp to complete 2022-09-04 11:27:56 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):265 - kubectl cp complete, closing process 2022-09-04 11:27:56 INFO i.a.w.p.KubePodProcess():488 - Waiting until pod is ready... 2022-09-04 11:27:56 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$8):523 - Setting stdout... 2022-09-04 11:27:56 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$9):534 - Setting stderr... 2022-09-04 11:27:57 INFO i.a.w.p.KubePodProcess():502 - Reading pod IP... 2022-09-04 11:27:57 INFO i.a.w.p.KubePodProcess():504 - Pod IP: 12.0.3.215 2022-09-04 11:27:57 INFO i.a.w.p.KubePodProcess():511 - Using null stdin output stream... 2022-09-04 11:28:05 INFO i.a.w.p.KubePodProcess(getReturnCode):696 - Exit code for pod normalization-sync-414-0-vlwsz is 0 2022-09-04 11:28:05 INFO i.a.w.p.KubePodProcess(exitValue):711 - Closed all resources for pod normalization-sync-414-0-vlwsz 2022-09-04 11:28:05 INFO i.a.w.p.KubePodProcess(exitValue):711 - Closed all resources for pod normalization-sync-414-0-vlwsz 2022-09-04 11:28:05 INFO i.a.w.p.KubePodProcess(exitValue):711 - Closed all resources for pod normalization-sync-414-0-vlwsz 2022-09-04 11:28:05 INFO i.a.w.p.KubePodProcess(exitValue):711 - Closed all resources for pod normalization-sync-414-0-vlwsz 2022-09-04 11:28:05 INFO i.a.w.p.KubePodProcess(exitValue):711 - Closed all resources for pod normalization-sync-414-0-vlwsz 2022-09-04 11:28:05 INFO i.a.w.p.KubePodProcess(exitValue):711 - Closed all resources for pod normalization-sync-414-0-vlwsz 2022-09-04 11:28:05 INFO i.a.w.DefaultNormalizationWorker(run):69 - Normalization executed in 10 seconds. 2022-09-04 11:28:05 INFO i.a.w.t.TemporalAttemptExecution(get):131 - Stopping cancellation check scheduling... 2022-09-04 11:28:05 INFO i.a.w.t.TemporalUtils(withBackgroundHeartbeat):235 - Stopping temporal heartbeating... 2022-09-04 11:27:57 normalization > Running: transform-config --config destination_config.json --integration-type redshift --out /config 2022-09-04 11:27:57 normalization > Namespace(config='destination_config.json', integration_type=, out='/config') 2022-09-04 11:27:57 normalization > transform_redshift 2022-09-04 11:27:57 normalization > Running: transform-catalog --integration-type redshift --profile-config-dir /config --catalog destination_catalog.json --out /config/models/generated/ --json-column _airbyte_data 2022-09-04 11:27:57 normalization > Processing destination_catalog.json... 2022-09-04 11:27:57 normalization > Generating airbyte_ctes/_mehak_pipeline_mongo_db_1d6287abb53046e1b1cf59071f7622e4/mongodb_people_ab1.sql from mongoDB_people 2022-09-04 11:27:57 normalization > Generating airbyte_ctes/_mehak_pipeline_mongo_db_1d6287abb53046e1b1cf59071f7622e4/mongodb_people_ab2.sql from mongoDB_people 2022-09-04 11:27:57 normalization > Generating airbyte_ctes/_mehak_pipeline_mongo_db_1d6287abb53046e1b1cf59071f7622e4/mongodb_people_ab3.sql from mongoDB_people 2022-09-04 11:27:57 normalization > Generating airbyte_incremental/_mehak_pipeline_mongo_db_1d6287abb53046e1b1cf59071f7622e4/mongodb_people.sql from mongoDB_people 2022-09-04 11:27:57 normalization > detected no config file for ssh, assuming ssh is off. 2022-09-04 11:27:58 normalization > Running with dbt=0.21.1 2022-09-04 11:27:59 normalization > Unable to do partial parsing because ../build/partial_parse.msgpack not found 2022-09-04 11:28:00 normalization > [WARNING]: Configuration paths exist in your dbt_project.yml file which do not apply to any resources. 2022-09-04 11:28:00 normalization > There are 2 unused configuration paths: 2022-09-04 11:28:00 normalization > - models.airbyte_utils.generated.airbyte_tables 2022-09-04 11:28:00 normalization > - models.airbyte_utils.generated.airbyte_views 2022-09-04 11:28:00 normalization > 2022-09-04 11:28:00 normalization > Found 4 models, 0 tests, 0 snapshots, 0 analyses, 520 macros, 0 operations, 0 seed files, 1 source, 0 exposures 2022-09-04 11:28:00 normalization > 2022-09-04 11:28:00 normalization > 11:28:00 | Concurrency: 4 threads (target='prod') 2022-09-04 11:28:00 normalization > 11:28:00 | 2022-09-04 11:28:01 normalization > 11:28:01 | 1 of 1 START incremental model _mehak_pipeline_mongo_db_1d6287abb53046e1b1cf59071f7622e4.mongodb_people...... [RUN] 2022-09-04 11:28:03 normalization > 11:28:03 | 1 of 1 OK created incremental model _mehak_pipeline_mongo_db_1d6287abb53046e1b1cf59071f7622e4.mongodb_people. [INSERT 0 2 in 2.36s] 2022-09-04 11:28:04 normalization > 11:28:04 | 2022-09-04 11:28:04 normalization > 11:28:04 | Finished running 1 incremental model in 4.05s. 2022-09-04 11:28:04 normalization > 2022-09-04 11:28:04 normalization > Completed successfully 2022-09-04 11:28:04 normalization > 2022-09-04 11:28:04 normalization > Done. PASS=1 WARN=0 ERROR=0 SKIP=0 TOTAL=1 2023-03-20 07:11:10 INFO i.a.v.j.JsonSchemaValidator(test):71 - JSON schema validation failed. errors: $.compression_codec: is missing but it is required, $.format_type: does not have a value in the enumeration [Avro] 2023-03-20 07:11:30 destination > 2023-03-20 07:11:27 INFO i.a.i.b.IntegrationCliParser(parseOptions):118 - integration args: {catalog=destination_catalog.json, write=null, config=destination_config.json} 2023-03-20 07:11:30 source > Starting syncing SourceSendgrid 2023-03-20 07:11:30 source > Syncing stream: blocks 2023-03-20 07:11:30 destination > 2023-03-20 07:11:27 INFO i.a.i.b.IntegrationRunner(runInternal):104 - Running integration: io.airbyte.integrations.destination.s3.S3Destination 2023-03-20 07:11:30 destination > 2023-03-20 07:11:27 INFO i.a.i.b.IntegrationRunner(runInternal):105 - Command: WRITE 2023-03-20 07:11:30 destination > 2023-03-20 07:11:27 INFO i.a.i.b.IntegrationRunner(runInternal):106 - Integration config: IntegrationConfig{command=WRITE, configPath='destination_config.json', catalogPath='destination_catalog.json', statePath='null'} 2023-03-20 07:11:30 destination > 2023-03-20 07:11:27 WARN c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword airbyte_secret - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword 2023-03-20 07:11:30 destination > 2023-03-20 07:11:27 WARN c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword examples - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword 2023-03-20 07:11:30 destination > 2023-03-20 07:11:27 WARN c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword order - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword 2023-03-20 07:11:30 destination > 2023-03-20 07:11:27 WARN c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword requires - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword 2023-03-20 07:11:30 destination > 2023-03-20 07:11:27 INFO i.a.i.d.s.S3FormatConfigs(getS3FormatConfig):22 - S3 format config: {"flattening":"No flattening","compression":{"compression_type":"No Compression"},"format_type":"CSV"} 2023-03-20 07:11:30 destination > 2023-03-20 07:11:27 INFO i.a.i.d.s.S3DestinationConfig(createS3Client):190 - Creating S3 client... 2023-03-20 07:11:30 destination > 2023-03-20 07:11:27 INFO i.a.i.d.s.SerializedBufferFactory(getCreateFunction):49 - S3 format config: S3CsvFormatConfig{flattening=NO, compression=NO_COMPRESSION} 2023-03-20 07:11:30 destination > 2023-03-20 07:11:27 INFO i.a.i.d.s.S3ConsumerFactory(lambda$toWriteConfig$0):83 - Write config: WriteConfig{streamName=blocks, namespace=null, outputBucketPath=testAutomation, pathFormat=testAutomation/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_, fullOutputPath=testAutomation/blocks/2023_03_20_1679296287341_, syncMode=overwrite} 2023-03-20 07:11:30 destination > 2023-03-20 07:11:27 INFO i.a.i.d.s.S3ConsumerFactory(lambda$toWriteConfig$0):83 - Write config: WriteConfig{streamName=bounces, namespace=null, outputBucketPath=testAutomation, pathFormat=testAutomation/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_, fullOutputPath=testAutomation/bounces/2023_03_20_1679296287341_, syncMode=overwrite} 2023-03-20 07:11:30 destination > 2023-03-20 07:11:27 INFO i.a.i.d.s.S3ConsumerFactory(lambda$toWriteConfig$0):83 - Write config: WriteConfig{streamName=campaigns, namespace=null, outputBucketPath=testAutomation, pathFormat=testAutomation/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_, fullOutputPath=testAutomation/campaigns/2023_03_20_1679296287341_, syncMode=overwrite} 2023-03-20 07:11:30 destination > 2023-03-20 07:11:27 INFO i.a.i.d.s.S3ConsumerFactory(lambda$toWriteConfig$0):83 - Write config: WriteConfig{streamName=contacts, namespace=null, outputBucketPath=testAutomation, pathFormat=testAutomation/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_, fullOutputPath=testAutomation/contacts/2023_03_20_1679296287341_, syncMode=overwrite} 2023-03-20 07:11:30 destination > 2023-03-20 07:11:27 INFO i.a.i.d.s.S3ConsumerFactory(lambda$toWriteConfig$0):83 - Write config: WriteConfig{streamName=global_suppressions, namespace=null, outputBucketPath=testAutomation, pathFormat=testAutomation/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_, fullOutputPath=testAutomation/global_suppressions/2023_03_20_1679296287341_, syncMode=overwrite} 2023-03-20 07:11:30 destination > 2023-03-20 07:11:27 INFO i.a.i.d.s.S3ConsumerFactory(lambda$toWriteConfig$0):83 - Write config: WriteConfig{streamName=invalid_emails, namespace=null, outputBucketPath=testAutomation, pathFormat=testAutomation/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_, fullOutputPath=testAutomation/invalid_emails/2023_03_20_1679296287341_, syncMode=overwrite} 2023-03-20 07:11:30 destination > 2023-03-20 07:11:27 INFO i.a.i.d.s.S3ConsumerFactory(lambda$toWriteConfig$0):83 - Write config: WriteConfig{streamName=lists, namespace=null, outputBucketPath=testAutomation, pathFormat=testAutomation/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_, fullOutputPath=testAutomation/lists/2023_03_20_1679296287341_, syncMode=overwrite} 2023-03-20 07:11:30 destination > 2023-03-20 07:11:27 INFO i.a.i.d.s.S3ConsumerFactory(lambda$toWriteConfig$0):83 - Write config: WriteConfig{streamName=messages, namespace=null, outputBucketPath=testAutomation, pathFormat=testAutomation/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_, fullOutputPath=testAutomation/messages/2023_03_20_1679296287341_, syncMode=overwrite} 2023-03-20 07:11:30 destination > 2023-03-20 07:11:27 INFO i.a.i.d.s.S3ConsumerFactory(lambda$toWriteConfig$0):83 - Write config: WriteConfig{streamName=segments, namespace=null, outputBucketPath=testAutomation, pathFormat=testAutomation/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_, fullOutputPath=testAutomation/segments/2023_03_20_1679296287341_, syncMode=overwrite} 2023-03-20 07:11:30 destination > 2023-03-20 07:11:27 INFO i.a.i.d.s.S3ConsumerFactory(lambda$toWriteConfig$0):83 - Write config: WriteConfig{streamName=single_sends, namespace=null, outputBucketPath=testAutomation, pathFormat=testAutomation/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_, fullOutputPath=testAutomation/single_sends/2023_03_20_1679296287341_, syncMode=overwrite} 2023-03-20 07:11:30 destination > 2023-03-20 07:11:27 INFO i.a.i.d.s.S3ConsumerFactory(lambda$toWriteConfig$0):83 - Write config: WriteConfig{streamName=spam_reports, namespace=null, outputBucketPath=testAutomation, pathFormat=testAutomation/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_, fullOutputPath=testAutomation/spam_reports/2023_03_20_1679296287341_, syncMode=overwrite} 2023-03-20 07:11:30 destination > 2023-03-20 07:11:27 INFO i.a.i.d.s.S3ConsumerFactory(lambda$toWriteConfig$0):83 - Write config: WriteConfig{streamName=stats_automations, namespace=null, outputBucketPath=testAutomation, pathFormat=testAutomation/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_, fullOutputPath=testAutomation/stats_automations/2023_03_20_1679296287341_, syncMode=overwrite} 2023-03-20 07:11:30 destination > 2023-03-20 07:11:28 INFO i.a.i.d.s.S3ConsumerFactory(lambda$toWriteConfig$0):83 - Write config: WriteConfig{streamName=suppression_group_members, namespace=null, outputBucketPath=testAutomation, pathFormat=testAutomation/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_, fullOutputPath=testAutomation/suppression_group_members/2023_03_20_1679296287341_, syncMode=overwrite} 2023-03-20 07:11:30 destination > 2023-03-20 07:11:28 INFO i.a.i.d.s.S3ConsumerFactory(lambda$toWriteConfig$0):83 - Write config: WriteConfig{streamName=suppression_groups, namespace=null, outputBucketPath=testAutomation, pathFormat=testAutomation/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_, fullOutputPath=testAutomation/suppression_groups/2023_03_20_1679296287341_, syncMode=overwrite} 2023-03-20 07:11:30 destination > 2023-03-20 07:11:28 INFO i.a.i.d.s.S3ConsumerFactory(lambda$toWriteConfig$0):83 - Write config: WriteConfig{streamName=templates, namespace=null, outputBucketPath=testAutomation, pathFormat=testAutomation/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_, fullOutputPath=testAutomation/templates/2023_03_20_1679296287341_, syncMode=overwrite} 2023-03-20 07:11:30 destination > 2023-03-20 07:11:28 INFO i.a.i.d.b.BufferedStreamConsumer(startTracked):116 - class io.airbyte.integrations.destination.buffered_stream_consumer.BufferedStreamConsumer started. 2023-03-20 07:11:30 destination > 2023-03-20 07:11:28 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):90 - Preparing bucket in destination started for 15 streams 2023-03-20 07:11:30 destination > 2023-03-20 07:11:28 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):97 - Clearing storage area in destination started for namespace null stream blocks bucketObject testAutomation pathFormat testAutomation/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_ 2023-03-20 07:11:30 destination > 2023-03-20 07:11:28 INFO i.a.i.d.s.S3StorageOperations(cleanUpBucketObject):250 - Storage bucket testAutomation has been cleaned-up (0 objects matching testAutomation/blocks/[0-9]{4}_[0-9]{2}_[0-9]{2}_[0-9]+_.* were deleted)... 2023-03-20 07:11:30 destination > 2023-03-20 07:11:28 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):100 - Clearing storage area in destination completed for namespace null stream blocks bucketObject testAutomation 2023-03-20 07:11:30 destination > 2023-03-20 07:11:28 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):97 - Clearing storage area in destination started for namespace null stream bounces bucketObject testAutomation pathFormat testAutomation/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_ 2023-03-20 07:11:30 destination > 2023-03-20 07:11:28 INFO i.a.i.d.s.S3StorageOperations(cleanUpBucketObject):250 - Storage bucket testAutomation has been cleaned-up (0 objects matching testAutomation/bounces/[0-9]{4}_[0-9]{2}_[0-9]{2}_[0-9]+_.* were deleted)... 2023-03-20 07:11:30 destination > 2023-03-20 07:11:28 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):100 - Clearing storage area in destination completed for namespace null stream bounces bucketObject testAutomation 2023-03-20 07:11:30 destination > 2023-03-20 07:11:28 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):97 - Clearing storage area in destination started for namespace null stream campaigns bucketObject testAutomation pathFormat testAutomation/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_ 2023-03-20 07:11:30 destination > 2023-03-20 07:11:28 INFO i.a.i.d.s.S3StorageOperations(cleanUpBucketObject):250 - Storage bucket testAutomation has been cleaned-up (0 objects matching testAutomation/campaigns/[0-9]{4}_[0-9]{2}_[0-9]{2}_[0-9]+_.* were deleted)... 2023-03-20 07:11:30 destination > 2023-03-20 07:11:28 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):100 - Clearing storage area in destination completed for namespace null stream campaigns bucketObject testAutomation 2023-03-20 07:11:30 destination > 2023-03-20 07:11:28 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):97 - Clearing storage area in destination started for namespace null stream contacts bucketObject testAutomation pathFormat testAutomation/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_ 2023-03-20 07:11:30 destination > 2023-03-20 07:11:28 INFO i.a.i.d.s.S3StorageOperations(cleanUpBucketObject):250 - Storage bucket testAutomation has been cleaned-up (0 objects matching testAutomation/contacts/[0-9]{4}_[0-9]{2}_[0-9]{2}_[0-9]+_.* were deleted)... 2023-03-20 07:11:30 destination > 2023-03-20 07:11:28 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):100 - Clearing storage area in destination completed for namespace null stream contacts bucketObject testAutomation 2023-03-20 07:11:30 destination > 2023-03-20 07:11:28 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):97 - Clearing storage area in destination started for namespace null stream global_suppressions bucketObject testAutomation pathFormat testAutomation/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_ 2023-03-20 07:11:30 destination > 2023-03-20 07:11:28 INFO i.a.i.d.s.S3StorageOperations(cleanUpBucketObject):250 - Storage bucket testAutomation has been cleaned-up (0 objects matching testAutomation/global_suppressions/[0-9]{4}_[0-9]{2}_[0-9]{2}_[0-9]+_.* were deleted)... 2023-03-20 07:11:30 destination > 2023-03-20 07:11:28 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):100 - Clearing storage area in destination completed for namespace null stream global_suppressions bucketObject testAutomation 2023-03-20 07:11:30 destination > 2023-03-20 07:11:28 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):97 - Clearing storage area in destination started for namespace null stream invalid_emails bucketObject testAutomation pathFormat testAutomation/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_ 2023-03-20 07:11:30 destination > 2023-03-20 07:11:29 INFO i.a.i.d.s.S3StorageOperations(cleanUpBucketObject):250 - Storage bucket testAutomation has been cleaned-up (0 objects matching testAutomation/invalid_emails/[0-9]{4}_[0-9]{2}_[0-9]{2}_[0-9]+_.* were deleted)... 2023-03-20 07:11:30 destination > 2023-03-20 07:11:29 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):100 - Clearing storage area in destination completed for namespace null stream invalid_emails bucketObject testAutomation 2023-03-20 07:11:30 destination > 2023-03-20 07:11:29 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):97 - Clearing storage area in destination started for namespace null stream lists bucketObject testAutomation pathFormat testAutomation/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_ 2023-03-20 07:11:30 destination > 2023-03-20 07:11:29 INFO i.a.i.d.s.S3StorageOperations(cleanUpBucketObject):250 - Storage bucket testAutomation has been cleaned-up (0 objects matching testAutomation/lists/[0-9]{4}_[0-9]{2}_[0-9]{2}_[0-9]+_.* were deleted)... 2023-03-20 07:11:30 destination > 2023-03-20 07:11:29 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):100 - Clearing storage area in destination completed for namespace null stream lists bucketObject testAutomation 2023-03-20 07:11:30 destination > 2023-03-20 07:11:29 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):97 - Clearing storage area in destination started for namespace null stream messages bucketObject testAutomation pathFormat testAutomation/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_ 2023-03-20 07:11:30 destination > 2023-03-20 07:11:29 INFO i.a.i.d.s.S3StorageOperations(cleanUpBucketObject):250 - Storage bucket testAutomation has been cleaned-up (0 objects matching testAutomation/messages/[0-9]{4}_[0-9]{2}_[0-9]{2}_[0-9]+_.* were deleted)... 2023-03-20 07:11:30 destination > 2023-03-20 07:11:29 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):100 - Clearing storage area in destination completed for namespace null stream messages bucketObject testAutomation 2023-03-20 07:11:30 destination > 2023-03-20 07:11:29 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):97 - Clearing storage area in destination started for namespace null stream segments bucketObject testAutomation pathFormat testAutomation/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_ 2023-03-20 07:11:30 destination > 2023-03-20 07:11:29 INFO i.a.i.d.s.S3StorageOperations(cleanUpBucketObject):250 - Storage bucket testAutomation has been cleaned-up (0 objects matching testAutomation/segments/[0-9]{4}_[0-9]{2}_[0-9]{2}_[0-9]+_.* were deleted)... 2023-03-20 07:11:30 destination > 2023-03-20 07:11:29 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):100 - Clearing storage area in destination completed for namespace null stream segments bucketObject testAutomation 2023-03-20 07:11:30 destination > 2023-03-20 07:11:29 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):97 - Clearing storage area in destination started for namespace null stream single_sends bucketObject testAutomation pathFormat testAutomation/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_ 2023-03-20 07:11:30 destination > 2023-03-20 07:11:29 INFO i.a.i.d.s.S3StorageOperations(cleanUpBucketObject):250 - Storage bucket testAutomation has been cleaned-up (0 objects matching testAutomation/single_sends/[0-9]{4}_[0-9]{2}_[0-9]{2}_[0-9]+_.* were deleted)... 2023-03-20 07:11:30 destination > 2023-03-20 07:11:29 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):100 - Clearing storage area in destination completed for namespace null stream single_sends bucketObject testAutomation 2023-03-20 07:11:30 destination > 2023-03-20 07:11:29 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):97 - Clearing storage area in destination started for namespace null stream spam_reports bucketObject testAutomation pathFormat testAutomation/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_ 2023-03-20 07:11:30 destination > 2023-03-20 07:11:29 INFO i.a.i.d.s.S3StorageOperations(cleanUpBucketObject):250 - Storage bucket testAutomation has been cleaned-up (0 objects matching testAutomation/spam_reports/[0-9]{4}_[0-9]{2}_[0-9]{2}_[0-9]+_.* were deleted)... 2023-03-20 07:11:30 destination > 2023-03-20 07:11:29 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):100 - Clearing storage area in destination completed for namespace null stream spam_reports bucketObject testAutomation 2023-03-20 07:11:30 destination > 2023-03-20 07:11:29 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):97 - Clearing storage area in destination started for namespace null stream stats_automations bucketObject testAutomation pathFormat testAutomation/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_ 2023-03-20 07:11:30 destination > 2023-03-20 07:11:29 INFO i.a.i.d.s.S3StorageOperations(cleanUpBucketObject):250 - Storage bucket testAutomation has been cleaned-up (0 objects matching testAutomation/stats_automations/[0-9]{4}_[0-9]{2}_[0-9]{2}_[0-9]+_.* were deleted)... 2023-03-20 07:11:30 destination > 2023-03-20 07:11:29 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):100 - Clearing storage area in destination completed for namespace null stream stats_automations bucketObject testAutomation 2023-03-20 07:11:30 destination > 2023-03-20 07:11:29 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):97 - Clearing storage area in destination started for namespace null stream suppression_group_members bucketObject testAutomation pathFormat testAutomation/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_ 2023-03-20 07:11:30 destination > 2023-03-20 07:11:29 INFO i.a.i.d.s.S3StorageOperations(cleanUpBucketObject):250 - Storage bucket testAutomation has been cleaned-up (0 objects matching testAutomation/suppression_group_members/[0-9]{4}_[0-9]{2}_[0-9]{2}_[0-9]+_.* were deleted)... 2023-03-20 07:11:30 destination > 2023-03-20 07:11:29 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):100 - Clearing storage area in destination completed for namespace null stream suppression_group_members bucketObject testAutomation 2023-03-20 07:11:30 destination > 2023-03-20 07:11:29 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):97 - Clearing storage area in destination started for namespace null stream suppression_groups bucketObject testAutomation pathFormat testAutomation/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_ 2023-03-20 07:11:30 destination > 2023-03-20 07:11:29 INFO i.a.i.d.s.S3StorageOperations(cleanUpBucketObject):250 - Storage bucket testAutomation has been cleaned-up (0 objects matching testAutomation/suppression_groups/[0-9]{4}_[0-9]{2}_[0-9]{2}_[0-9]+_.* were deleted)... 2023-03-20 07:11:30 destination > 2023-03-20 07:11:29 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):100 - Clearing storage area in destination completed for namespace null stream suppression_groups bucketObject testAutomation 2023-03-20 07:11:30 destination > 2023-03-20 07:11:29 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):97 - Clearing storage area in destination started for namespace null stream templates bucketObject testAutomation pathFormat testAutomation/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_ 2023-03-20 07:11:30 destination > 2023-03-20 07:11:29 INFO i.a.i.d.s.S3StorageOperations(cleanUpBucketObject):250 - Storage bucket testAutomation has been cleaned-up (0 objects matching testAutomation/templates/[0-9]{4}_[0-9]{2}_[0-9]{2}_[0-9]+_.* were deleted)... 2023-03-20 07:11:30 destination > 2023-03-20 07:11:29 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):100 - Clearing storage area in destination completed for namespace null stream templates bucketObject testAutomation 2023-03-20 07:11:30 destination > 2023-03-20 07:11:29 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):104 - Preparing storage area in destination completed. 2023-03-20 07:11:10 INFO i.a.w.t.TemporalAttemptExecution(get):108 - Cloud storage job log path: /workspace/414/0/logs.log 2023-03-20 07:11:10 INFO i.a.w.t.TemporalAttemptExecution(get):111 - Executing worker wrapper. Airbyte version: 0.40.6 2023-03-20 07:11:10 INFO i.a.c.i.LineGobbler(voidCall):114 - 2023-03-20 07:11:10 INFO i.a.w.p.KubeProcessFactory(create):100 - Attempting to start pod = source-sendgrid-check-414-0-qmzmc for airbyte/source-sendgrid:0.2.14 with resources io.airbyte.config.ResourceRequirements@4ffb823e[cpuRequest=,cpuLimit=,memoryRequest=,memoryLimit=] 2023-03-20 07:11:10 INFO i.a.c.i.LineGobbler(voidCall):114 - ----- START CHECK ----- 2023-03-20 07:11:10 INFO i.a.w.p.KubeProcessFactory(create):103 - source-sendgrid-check-414-0-qmzmc stdoutLocalPort = 9018 2023-03-20 07:11:10 INFO i.a.c.i.LineGobbler(voidCall):114 - 2023-03-20 07:11:10 INFO i.a.w.p.KubeProcessFactory(create):106 - source-sendgrid-check-414-0-qmzmc stderrLocalPort = 9019 2023-03-20 07:11:10 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$9):589 - Creating stdout socket server... 2023-03-20 07:11:10 INFO i.a.w.p.KubePodProcess():520 - Creating pod source-sendgrid-check-414-0-qmzmc... 2023-03-20 07:11:10 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$10):607 - Creating stderr socket server... 2023-03-20 07:11:10 INFO i.a.w.p.KubePodProcess(waitForInitPodToRun):309 - Waiting for init container to be ready before copying files... 2023-03-20 07:11:10 INFO i.a.w.p.KubePodProcess(waitForInitPodToRun):313 - Init container present.. 2023-03-20 07:11:11 INFO i.a.w.p.KubePodProcess(waitForInitPodToRun):316 - Init container ready.. 2023-03-20 07:11:11 INFO i.a.w.p.KubePodProcess():551 - Copying files... 2023-03-20 07:11:11 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):258 - Uploading file: source_config.json 2023-03-20 07:11:11 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):266 - kubectl cp /tmp/2397cf71-600d-426f-9721-9df67d27b86a/source_config.json dev/source-sendgrid-check-414-0-qmzmc:/config/source_config.json -c init 2023-03-20 07:11:11 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):269 - Waiting for kubectl cp to complete 2023-03-20 07:11:11 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):283 - kubectl cp complete, closing process 2023-03-20 07:11:11 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):258 - Uploading file: FINISHED_UPLOADING 2023-03-20 07:11:11 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):266 - kubectl cp /tmp/d6f188b0-93d6-45f1-961b-dee59644396f/FINISHED_UPLOADING dev/source-sendgrid-check-414-0-qmzmc:/config/FINISHED_UPLOADING -c init 2023-03-20 07:11:11 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):269 - Waiting for kubectl cp to complete 2023-03-20 07:11:12 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):283 - kubectl cp complete, closing process 2023-03-20 07:11:12 INFO i.a.w.p.KubePodProcess():554 - Waiting until pod is ready... 2023-03-20 07:11:12 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$9):598 - Setting stdout... 2023-03-20 07:11:12 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$10):610 - Setting stderr... 2023-03-20 07:11:13 INFO i.a.w.p.KubePodProcess():570 - Reading pod IP... 2023-03-20 07:11:13 INFO i.a.w.p.KubePodProcess():572 - Pod IP: 12.0.4.96 2023-03-20 07:11:13 INFO i.a.w.p.KubePodProcess():579 - Using null stdin output stream... 2023-03-20 07:11:13 INFO i.a.w.i.DefaultAirbyteStreamFactory(internalLog):103 - Check succeeded 2023-03-20 07:11:13 INFO i.a.w.p.KubePodProcess(close):737 - (pod: dev / source-sendgrid-check-414-0-qmzmc) - Closed all resources for pod 2023-03-20 07:11:13 INFO i.a.w.t.TemporalAttemptExecution(get):132 - Stopping cancellation check scheduling... 2023-03-20 07:11:13 WARN i.a.c.i.LineGobbler(voidCall):119 - generic gobbler IOException: Socket closed. Typically happens when cancelling a job. 2023-03-20 07:11:13 INFO i.a.c.i.LineGobbler(voidCall):114 - 2023-03-20 07:11:13 INFO i.a.c.i.LineGobbler(voidCall):114 - ----- END CHECK ----- 2023-03-20 07:11:13 INFO i.a.c.i.LineGobbler(voidCall):114 - 2023-03-20 07:11:13 INFO i.a.w.t.TemporalAttemptExecution(get):108 - Cloud storage job log path: /workspace/414/0/logs.log 2023-03-20 07:11:13 INFO i.a.w.t.TemporalAttemptExecution(get):111 - Executing worker wrapper. Airbyte version: 0.40.6 2023-03-20 07:11:13 INFO i.a.c.i.LineGobbler(voidCall):114 - 2023-03-20 07:11:13 INFO i.a.w.p.KubeProcessFactory(create):100 - Attempting to start pod = destination-s3-check-414-0-jqesz for airbyte/destination-s3:0.3.15 with resources io.airbyte.config.ResourceRequirements@4ffb823e[cpuRequest=,cpuLimit=,memoryRequest=,memoryLimit=] 2023-03-20 07:11:13 INFO i.a.w.p.KubeProcessFactory(create):103 - destination-s3-check-414-0-jqesz stdoutLocalPort = 9022 2023-03-20 07:11:13 INFO i.a.c.i.LineGobbler(voidCall):114 - ----- START CHECK ----- 2023-03-20 07:11:13 INFO i.a.w.p.KubeProcessFactory(create):106 - destination-s3-check-414-0-jqesz stderrLocalPort = 9023 2023-03-20 07:11:13 INFO i.a.c.i.LineGobbler(voidCall):114 - 2023-03-20 07:11:13 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$9):589 - Creating stdout socket server... 2023-03-20 07:11:13 INFO i.a.w.p.KubePodProcess():520 - Creating pod destination-s3-check-414-0-jqesz... 2023-03-20 07:11:13 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$10):607 - Creating stderr socket server... 2023-03-20 07:11:13 INFO i.a.w.p.KubePodProcess(waitForInitPodToRun):309 - Waiting for init container to be ready before copying files... 2023-03-20 07:11:13 INFO i.a.w.p.KubePodProcess(waitForInitPodToRun):313 - Init container present.. 2023-03-20 07:11:15 INFO i.a.w.p.KubePodProcess(waitForInitPodToRun):316 - Init container ready.. 2023-03-20 07:11:15 INFO i.a.w.p.KubePodProcess():551 - Copying files... 2023-03-20 07:11:15 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):258 - Uploading file: source_config.json 2023-03-20 07:11:15 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):266 - kubectl cp /tmp/bcbb0f11-0b28-4187-a4e8-39735fc8b2db/source_config.json dev/destination-s3-check-414-0-jqesz:/config/source_config.json -c init 2023-03-20 07:11:15 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):269 - Waiting for kubectl cp to complete 2023-03-20 07:11:15 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):283 - kubectl cp complete, closing process 2023-03-20 07:11:15 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):258 - Uploading file: FINISHED_UPLOADING 2023-03-20 07:11:15 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):266 - kubectl cp /tmp/7d1907d5-83c5-4afd-9412-777c43981a64/FINISHED_UPLOADING dev/destination-s3-check-414-0-jqesz:/config/FINISHED_UPLOADING -c init 2023-03-20 07:11:15 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):269 - Waiting for kubectl cp to complete 2023-03-20 07:11:15 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):283 - kubectl cp complete, closing process 2023-03-20 07:11:15 INFO i.a.w.p.KubePodProcess():554 - Waiting until pod is ready... 2023-03-20 07:11:16 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$9):598 - Setting stdout... 2023-03-20 07:11:16 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$10):610 - Setting stderr... 2023-03-20 07:11:17 INFO i.a.w.p.KubePodProcess():570 - Reading pod IP... 2023-03-20 07:11:17 INFO i.a.w.p.KubePodProcess():572 - Pod IP: 12.0.3.51 2023-03-20 07:11:17 INFO i.a.w.p.KubePodProcess():579 - Using null stdin output stream... 2023-03-20 07:11:18 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2023-03-20 07:11:18 INFO i.a.i.b.IntegrationCliParser(parseOptions):118 - integration args: {check=null, config=source_config.json} 2023-03-20 07:11:18 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2023-03-20 07:11:18 INFO i.a.i.b.IntegrationRunner(runInternal):104 - Running integration: io.airbyte.integrations.destination.s3.S3Destination 2023-03-20 07:11:18 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2023-03-20 07:11:18 INFO i.a.i.b.IntegrationRunner(runInternal):105 - Command: CHECK 2023-03-20 07:11:18 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2023-03-20 07:11:18 INFO i.a.i.b.IntegrationRunner(runInternal):106 - Integration config: IntegrationConfig{command=CHECK, configPath='source_config.json', catalogPath='null', statePath='null'} 2023-03-20 07:11:18 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2023-03-20 07:11:18 WARN c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword airbyte_secret - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword 2023-03-20 07:11:18 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2023-03-20 07:11:18 WARN c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword examples - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword 2023-03-20 07:11:18 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2023-03-20 07:11:18 WARN c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword order - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword 2023-03-20 07:11:18 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2023-03-20 07:11:18 WARN c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword requires - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword 2023-03-20 07:11:18 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2023-03-20 07:11:18 INFO i.a.i.d.s.S3FormatConfigs(getS3FormatConfig):22 - S3 format config: {"flattening":"No flattening","compression":{"compression_type":"No Compression"},"format_type":"CSV"} 2023-03-20 07:11:18 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2023-03-20 07:11:18 INFO i.a.i.d.s.S3DestinationConfig(createS3Client):190 - Creating S3 client... 2023-03-20 07:11:19 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2023-03-20 07:11:19 INFO i.a.i.d.s.S3StorageOperations(createBucketObjectIfNotExists):102 - Storage Object redshift-dev-s3-bucket/testAutomation does not exist in bucket; creating... 2023-03-20 07:11:20 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2023-03-20 07:11:20 INFO i.a.i.d.s.S3StorageOperations(createBucketObjectIfNotExists):104 - Storage Object redshift-dev-s3-bucket/testAutomation has been created in bucket. 2023-03-20 07:11:20 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2023-03-20 07:11:20 INFO i.a.i.d.s.S3Destination(testIAMUserHasListObjectPermission):156 - Started testing if IAM user can call listObjects on the destination bucket 2023-03-20 07:11:20 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2023-03-20 07:11:20 INFO i.a.i.d.s.S3Destination(testIAMUserHasListObjectPermission):159 - Finished checking for listObjects permission 2023-03-20 07:11:20 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2023-03-20 07:11:20 INFO i.a.i.d.s.S3Destination(testSingleUpload):81 - Started testing if all required credentials assigned to user for single file uploading 2023-03-20 07:11:20 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2023-03-20 07:11:20 INFO i.a.i.d.s.S3Destination(testSingleUpload):91 - Finished checking for normal upload mode 2023-03-20 07:11:20 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2023-03-20 07:11:20 INFO i.a.i.d.s.S3Destination(testMultipartUpload):95 - Started testing if all required credentials assigned to user for multipart upload 2023-03-20 07:11:20 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2023-03-20 07:11:20 INFO a.m.s.StreamTransferManager(getMultiPartOutputStreams):329 - Initiated multipart upload to redshift-dev-s3-bucket/testAutomation/test_1679296280266 with full ID eFcPz6AG.72.Fcn6c4ivIfzOlSBda.eii86A_.d6Vd3MxG_b6dveN5eW8MsOvHZjycVH79WUz.nMdMIwVzcwC.jOrAIOHh0Vtkdj3R0bYB6gzzcRdT.3EjkXm4Dh4Xd5 2023-03-20 07:11:20 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2023-03-20 07:11:20 INFO a.m.s.MultiPartOutputStream(close):158 - Called close() on [MultipartOutputStream for parts 1 - 10000] 2023-03-20 07:11:20 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2023-03-20 07:11:20 INFO a.m.s.MultiPartOutputStream(close):158 - Called close() on [MultipartOutputStream for parts 1 - 10000] 2023-03-20 07:11:20 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2023-03-20 07:11:20 WARN a.m.s.MultiPartOutputStream(close):160 - [MultipartOutputStream for parts 1 - 10000] is already closed 2023-03-20 07:11:20 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2023-03-20 07:11:20 INFO a.m.s.StreamTransferManager(complete):367 - [Manager uploading to redshift-dev-s3-bucket/testAutomation/test_1679296280266 with id eFcPz6AG....Xm4Dh4Xd5]: Uploading leftover stream [Part number 1 containing 3.34 MB] 2023-03-20 07:11:20 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2023-03-20 07:11:20 INFO a.m.s.StreamTransferManager(uploadStreamPart):558 - [Manager uploading to redshift-dev-s3-bucket/testAutomation/test_1679296280266 with id eFcPz6AG....Xm4Dh4Xd5]: Finished uploading [Part number 1 containing 3.34 MB] 2023-03-20 07:11:20 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2023-03-20 07:11:20 INFO a.m.s.StreamTransferManager(complete):395 - [Manager uploading to redshift-dev-s3-bucket/testAutomation/test_1679296280266 with id eFcPz6AG....Xm4Dh4Xd5]: Completed 2023-03-20 07:11:20 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2023-03-20 07:11:20 INFO i.a.i.d.s.S3Destination(testMultipartUpload):119 - Finished verification for multipart upload mode 2023-03-20 07:11:20 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2023-03-20 07:11:20 INFO i.a.i.b.IntegrationRunner(runInternal):152 - Completed integration: io.airbyte.integrations.destination.s3.S3Destination 2023-03-20 07:11:21 INFO i.a.w.p.KubePodProcess(close):737 - (pod: dev / destination-s3-check-414-0-jqesz) - Closed all resources for pod 2023-03-20 07:11:21 INFO i.a.w.t.TemporalAttemptExecution(get):132 - Stopping cancellation check scheduling... 2023-03-20 07:11:21 INFO i.a.c.i.LineGobbler(voidCall):114 - 2023-03-20 07:11:21 INFO i.a.c.i.LineGobbler(voidCall):114 - ----- END CHECK ----- 2023-03-20 07:11:21 INFO i.a.c.i.LineGobbler(voidCall):114 - 2023-03-20 07:11:21 INFO i.a.w.t.TemporalAttemptExecution(get):108 - Cloud storage job log path: /workspace/414/0/logs.log 2023-03-20 07:11:21 INFO i.a.w.t.TemporalAttemptExecution(get):111 - Executing worker wrapper. Airbyte version: 0.40.6 2023-03-20 07:11:21 INFO i.a.c.f.EnvVariableFeatureFlags(getEnvOrDefault):49 - Using default value for environment variable LOG_CONNECTOR_MESSAGES: 'false' 2023-03-20 07:11:21 INFO i.a.c.EnvConfigs(getEnvOrDefault):1096 - Using default value for environment variable METRIC_CLIENT: '' 2023-03-20 07:11:21 WARN i.a.m.l.MetricClientFactory(initialize):60 - Metric client is already initialized to 2023-03-20 07:11:21 INFO i.a.c.f.EnvVariableFeatureFlags(getEnvOrDefault):49 - Using default value for environment variable LOG_CONNECTOR_MESSAGES: 'false' 2023-03-20 07:11:21 INFO i.a.w.g.DefaultReplicationWorker(run):122 - start sync worker. job id: 414 attempt id: 0 2023-03-20 07:11:21 INFO i.a.c.i.LineGobbler(voidCall):114 - 2023-03-20 07:11:21 INFO i.a.c.i.LineGobbler(voidCall):114 - ----- START REPLICATION ----- 2023-03-20 07:11:21 INFO i.a.c.i.LineGobbler(voidCall):114 - 2023-03-20 07:11:21 INFO i.a.w.g.DefaultReplicationWorker(run):135 - configured sync modes: {null.single_sends=full_refresh - overwrite, null.spam_reports=full_refresh - overwrite, null.messages=full_refresh - overwrite, null.segments=full_refresh - overwrite, null.suppression_groups=full_refresh - overwrite, null.invalid_emails=full_refresh - overwrite, null.suppression_group_members=full_refresh - overwrite, null.contacts=full_refresh - overwrite, null.bounces=full_refresh - overwrite, null.stats_automations=full_refresh - overwrite, null.campaigns=full_refresh - overwrite, null.lists=full_refresh - overwrite, null.global_suppressions=full_refresh - overwrite, null.templates=full_refresh - overwrite, null.blocks=full_refresh - overwrite} 2023-03-20 07:11:21 INFO i.a.w.i.DefaultAirbyteDestination(start):69 - Running destination... 2023-03-20 07:11:21 INFO i.a.w.p.KubeProcessFactory(create):100 - Attempting to start pod = destination-s3-write-414-0-lvfvs for airbyte/destination-s3:0.3.15 with resources io.airbyte.config.ResourceRequirements@2e00a6a4[cpuRequest=,cpuLimit=,memoryRequest=,memoryLimit=] 2023-03-20 07:11:21 INFO i.a.w.p.KubeProcessFactory(create):103 - destination-s3-write-414-0-lvfvs stdoutLocalPort = 9010 2023-03-20 07:11:21 INFO i.a.w.p.KubeProcessFactory(create):106 - destination-s3-write-414-0-lvfvs stderrLocalPort = 9011 2023-03-20 07:11:21 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$9):589 - Creating stdout socket server... 2023-03-20 07:11:21 INFO i.a.w.p.KubePodProcess():520 - Creating pod destination-s3-write-414-0-lvfvs... 2023-03-20 07:11:21 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$10):607 - Creating stderr socket server... 2023-03-20 07:11:21 INFO i.a.w.p.KubePodProcess(waitForInitPodToRun):309 - Waiting for init container to be ready before copying files... 2023-03-20 07:11:21 INFO i.a.w.p.KubePodProcess(waitForInitPodToRun):313 - Init container present.. 2023-03-20 07:11:23 INFO i.a.w.p.KubePodProcess(waitForInitPodToRun):316 - Init container ready.. 2023-03-20 07:11:23 INFO i.a.w.p.KubePodProcess():551 - Copying files... 2023-03-20 07:11:23 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):258 - Uploading file: destination_config.json 2023-03-20 07:11:23 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):266 - kubectl cp /tmp/e07e0bec-441b-43ba-8c72-9cba5a042b28/destination_config.json dev/destination-s3-write-414-0-lvfvs:/config/destination_config.json -c init 2023-03-20 07:11:23 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):269 - Waiting for kubectl cp to complete 2023-03-20 07:11:24 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):283 - kubectl cp complete, closing process 2023-03-20 07:11:24 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):258 - Uploading file: destination_catalog.json 2023-03-20 07:11:24 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):266 - kubectl cp /tmp/f74a1e39-1d97-42ff-a5f0-8a6f65d341e7/destination_catalog.json dev/destination-s3-write-414-0-lvfvs:/config/destination_catalog.json -c init 2023-03-20 07:11:24 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):269 - Waiting for kubectl cp to complete 2023-03-20 07:11:24 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):283 - kubectl cp complete, closing process 2023-03-20 07:11:24 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):258 - Uploading file: FINISHED_UPLOADING 2023-03-20 07:11:24 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):266 - kubectl cp /tmp/d1e2b12b-a215-4f4a-9a77-b79afb7fefa3/FINISHED_UPLOADING dev/destination-s3-write-414-0-lvfvs:/config/FINISHED_UPLOADING -c init 2023-03-20 07:11:24 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):269 - Waiting for kubectl cp to complete 2023-03-20 07:11:24 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):283 - kubectl cp complete, closing process 2023-03-20 07:11:24 INFO i.a.w.p.KubePodProcess():554 - Waiting until pod is ready... 2023-03-20 07:11:25 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$9):598 - Setting stdout... 2023-03-20 07:11:25 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$10):610 - Setting stderr... 2023-03-20 07:11:26 INFO i.a.w.p.KubePodProcess():570 - Reading pod IP... 2023-03-20 07:11:26 INFO i.a.w.p.KubePodProcess():572 - Pod IP: 12.0.4.69 2023-03-20 07:11:26 INFO i.a.w.p.KubePodProcess():575 - Creating stdin socket... 2023-03-20 07:11:26 INFO i.a.w.p.KubeProcessFactory(create):100 - Attempting to start pod = source-sendgrid-read-414-0-vgmug for airbyte/source-sendgrid:0.2.14 with resources io.airbyte.config.ResourceRequirements@79d11bfb[cpuRequest=,cpuLimit=,memoryRequest=,memoryLimit=] 2023-03-20 07:11:26 INFO i.a.w.p.KubeProcessFactory(create):103 - source-sendgrid-read-414-0-vgmug stdoutLocalPort = 9016 2023-03-20 07:11:26 INFO i.a.w.p.KubeProcessFactory(create):106 - source-sendgrid-read-414-0-vgmug stderrLocalPort = 9017 2023-03-20 07:11:26 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$9):589 - Creating stdout socket server... 2023-03-20 07:11:26 INFO i.a.w.p.KubePodProcess():520 - Creating pod source-sendgrid-read-414-0-vgmug... 2023-03-20 07:11:26 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$10):607 - Creating stderr socket server... 2023-03-20 07:11:26 INFO i.a.w.p.KubePodProcess(waitForInitPodToRun):309 - Waiting for init container to be ready before copying files... 2023-03-20 07:11:26 INFO i.a.w.p.KubePodProcess(waitForInitPodToRun):313 - Init container present.. 2023-03-20 07:11:27 INFO i.a.w.p.KubePodProcess(waitForInitPodToRun):316 - Init container ready.. 2023-03-20 07:11:27 INFO i.a.w.p.KubePodProcess():551 - Copying files... 2023-03-20 07:11:27 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):258 - Uploading file: source_config.json 2023-03-20 07:11:27 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):266 - kubectl cp /tmp/faad86db-d279-43a7-8e3e-71b0e3884d9a/source_config.json dev/source-sendgrid-read-414-0-vgmug:/config/source_config.json -c init 2023-03-20 07:11:27 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):269 - Waiting for kubectl cp to complete 2023-03-20 07:11:28 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):283 - kubectl cp complete, closing process 2023-03-20 07:11:28 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):258 - Uploading file: source_catalog.json 2023-03-20 07:11:28 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):266 - kubectl cp /tmp/0dfe05c5-b2d7-4ac6-a91c-e73cc1585b74/source_catalog.json dev/source-sendgrid-read-414-0-vgmug:/config/source_catalog.json -c init 2023-03-20 07:11:28 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):269 - Waiting for kubectl cp to complete 2023-03-20 07:11:29 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):283 - kubectl cp complete, closing process 2023-03-20 07:11:29 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):258 - Uploading file: FINISHED_UPLOADING 2023-03-20 07:11:29 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):266 - kubectl cp /tmp/525b5a6d-dfa9-4702-9549-2f659dc7079f/FINISHED_UPLOADING dev/source-sendgrid-read-414-0-vgmug:/config/FINISHED_UPLOADING -c init 2023-03-20 07:11:29 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):269 - Waiting for kubectl cp to complete 2023-03-20 07:11:29 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):283 - kubectl cp complete, closing process 2023-03-20 07:11:29 INFO i.a.w.p.KubePodProcess():554 - Waiting until pod is ready... 2023-03-20 07:11:30 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$9):598 - Setting stdout... 2023-03-20 07:11:30 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$10):610 - Setting stderr... 2023-03-20 07:11:30 INFO i.a.w.p.KubePodProcess():570 - Reading pod IP... 2023-03-20 07:11:30 INFO i.a.w.p.KubePodProcess():572 - Pod IP: 12.0.3.113 2023-03-20 07:11:30 INFO i.a.w.p.KubePodProcess():579 - Using null stdin output stream... 2023-03-20 07:11:30 INFO i.a.w.g.DefaultReplicationWorker(lambda$getDestinationOutputRunnable$7):435 - Destination output thread started. 2023-03-20 07:11:30 INFO i.a.w.g.DefaultReplicationWorker(run):177 - Waiting for source and destination threads to complete. 2023-03-20 07:11:30 INFO i.a.w.g.DefaultReplicationWorker(lambda$getReplicationRunnable$6):322 - Replication thread started. 2023-03-20 07:11:33 INFO i.a.w.g.DefaultReplicationWorker(lambda$getReplicationRunnable$6):355 - Source has no more messages, closing connection. 2023-03-20 07:11:34 INFO i.a.w.p.KubePodProcess(close):737 - (pod: dev / source-sendgrid-read-414-0-vgmug) - Closed all resources for pod 2023-03-20 07:11:35 INFO i.a.w.p.KubePodProcess(close):737 - (pod: dev / destination-s3-write-414-0-lvfvs) - Closed all resources for pod 2023-03-20 07:11:35 ERROR i.a.w.g.DefaultReplicationWorker(run):188 - Sync worker failed. java.util.concurrent.ExecutionException: io.airbyte.workers.general.DefaultReplicationWorker$SourceException: Source cannot be stopped! at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) ~[?:?] at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) ~[?:?] at io.airbyte.workers.general.DefaultReplicationWorker.run(DefaultReplicationWorker.java:181) ~[io.airbyte-airbyte-workers-0.40.6.jar:?] at io.airbyte.workers.general.DefaultReplicationWorker.run(DefaultReplicationWorker.java:68) ~[io.airbyte-airbyte-workers-0.40.6.jar:?] at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:159) ~[io.airbyte-airbyte-workers-0.40.6.jar:?] at java.lang.Thread.run(Thread.java:1589) [?:?] Suppressed: io.airbyte.workers.exception.WorkerException: Source process exit with code 1. This warning is normal if the job was cancelled. at io.airbyte.workers.internal.DefaultAirbyteSource.close(DefaultAirbyteSource.java:140) ~[io.airbyte-airbyte-workers-0.40.6.jar:?] at io.airbyte.workers.general.DefaultReplicationWorker.run(DefaultReplicationWorker.java:145) ~[io.airbyte-airbyte-workers-0.40.6.jar:?] at io.airbyte.workers.general.DefaultReplicationWorker.run(DefaultReplicationWorker.java:68) ~[io.airbyte-airbyte-workers-0.40.6.jar:?] at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:159) ~[io.airbyte-airbyte-workers-0.40.6.jar:?] at java.lang.Thread.run(Thread.java:1589) [?:?] Caused by: io.airbyte.workers.general.DefaultReplicationWorker$SourceException: Source cannot be stopped! at io.airbyte.workers.general.DefaultReplicationWorker.lambda$getReplicationRunnable$6(DefaultReplicationWorker.java:359) ~[io.airbyte-airbyte-workers-0.40.6.jar:?] at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804) ~[?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] ... 1 more Caused by: io.airbyte.workers.exception.WorkerException: Source process exit with code 1. This warning is normal if the job was cancelled. at io.airbyte.workers.internal.DefaultAirbyteSource.close(DefaultAirbyteSource.java:140) ~[io.airbyte-airbyte-workers-0.40.6.jar:?] at io.airbyte.workers.general.DefaultReplicationWorker.lambda$getReplicationRunnable$6(DefaultReplicationWorker.java:357) ~[io.airbyte-airbyte-workers-0.40.6.jar:?] at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804) ~[?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] ... 1 more 2023-03-20 07:11:35 INFO i.a.w.g.DefaultReplicationWorker(run):281 - Source did not output any state messages 2023-03-20 07:11:35 WARN i.a.w.g.DefaultReplicationWorker(run):292 - State capture: No state retained. 2023-03-20 07:11:35 INFO i.a.w.g.DefaultReplicationWorker(run):300 - sync summary: { "status" : "failed", "recordsSynced" : 14, "bytesSynced" : 4037, "startTime" : 1679296281742, "endTime" : 1679296295843, "totalStats" : { "recordsEmitted" : 14, "bytesEmitted" : 4037, "sourceStateMessagesEmitted" : 0, "destinationStateMessagesEmitted" : 0, "recordsCommitted" : 0, "meanSecondsBeforeSourceStateMessageEmitted" : 0, "maxSecondsBeforeSourceStateMessageEmitted" : 0, "maxSecondsBetweenStateMessageEmittedandCommitted" : 0, "meanSecondsBetweenStateMessageEmittedandCommitted" : 0 }, "streamStats" : [ { "streamName" : "global_suppressions", "stats" : { "recordsEmitted" : 3, "bytesEmitted" : 151 } }, { "streamName" : "campaigns", "stats" : { "recordsEmitted" : 5, "bytesEmitted" : 1043 } }, { "streamName" : "lists", "stats" : { "recordsEmitted" : 1, "bytesEmitted" : 187 } }, { "streamName" : "contacts", "stats" : { "recordsEmitted" : 5, "bytesEmitted" : 2656 } } ] } 2023-03-20 07:11:35 INFO i.a.w.g.DefaultReplicationWorker(run):301 - failures: [ { "failureOrigin" : "source", "failureType" : "system_error", "internalMessage" : "400 Client Error: Bad Request for url: https://api.sendgrid.com/v3/messages?query=last_event_time%20BETWEEN%20TIMESTAMP%20%22None%22%20AND%20TIMESTAMP%20%222023-03-20T07%3A11%3A33Z%22&limit=1000", "externalMessage" : "authorization required", "metadata" : { "attemptNumber" : 0, "jobId" : 414, "from_trace_message" : true, "connector_command" : "read" }, "stacktrace" : "Traceback (most recent call last):\n File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py\", line 115, in read\n yield from self._read_stream(\n File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py\", line 184, in _read_stream\n for record in record_iterator:\n File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py\", line 282, in _read_full_refresh\n for record in records:\n File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py\", line 421, in read_records\n response = self._send_request(request, request_kwargs)\n File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py\", line 339, in _send_request\n return backoff_handler(user_backoff_handler)(request, request_kwargs)\n File \"/usr/local/lib/python3.9/site-packages/backoff/_sync.py\", line 105, in retry\n ret = target(*args, **kwargs)\n File \"/usr/local/lib/python3.9/site-packages/backoff/_sync.py\", line 105, in retry\n ret = target(*args, **kwargs)\n File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py\", line 306, in _send\n raise exc\n File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py\", line 303, in _send\n response.raise_for_status()\n File \"/usr/local/lib/python3.9/site-packages/requests/models.py\", line 1021, in raise_for_status\n raise HTTPError(http_error_msg, response=self)\nrequests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://api.sendgrid.com/v3/messages?query=last_event_time%20BETWEEN%20TIMESTAMP%20%22None%22%20AND%20TIMESTAMP%20%222023-03-20T07%3A11%3A33Z%22&limit=1000\n", "timestamp" : 1679296293820 }, { "failureOrigin" : "source", "internalMessage" : "io.airbyte.workers.general.DefaultReplicationWorker$SourceException: Source cannot be stopped!", "externalMessage" : "Something went wrong within the source connector", "metadata" : { "attemptNumber" : 0, "jobId" : 414, "connector_command" : "read" }, "stacktrace" : "java.util.concurrent.CompletionException: io.airbyte.workers.general.DefaultReplicationWorker$SourceException: Source cannot be stopped!\n\tat java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315)\n\tat java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:320)\n\tat java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1807)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1589)\nCaused by: io.airbyte.workers.general.DefaultReplicationWorker$SourceException: Source cannot be stopped!\n\tat io.airbyte.workers.general.DefaultReplicationWorker.lambda$getReplicationRunnable$6(DefaultReplicationWorker.java:359)\n\tat java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)\n\t... 3 more\nCaused by: io.airbyte.workers.exception.WorkerException: Source process exit with code 1. This warning is normal if the job was cancelled.\n\tat io.airbyte.workers.internal.DefaultAirbyteSource.close(DefaultAirbyteSource.java:140)\n\tat io.airbyte.workers.general.DefaultReplicationWorker.lambda$getReplicationRunnable$6(DefaultReplicationWorker.java:357)\n\t... 4 more\n", "timestamp" : 1679296294114 } ] 2023-03-20 07:11:35 INFO i.a.w.t.TemporalAttemptExecution(get):132 - Stopping cancellation check scheduling... 2023-03-20 07:11:35 INFO i.a.w.t.s.ReplicationActivityImpl(lambda$replicate$3):166 - sync summary: io.airbyte.config.StandardSyncOutput@1cffd9f6[standardSyncSummary=io.airbyte.config.StandardSyncSummary@2bed5a74[status=failed,recordsSynced=14,bytesSynced=4037,startTime=1679296281742,endTime=1679296295843,totalStats=io.airbyte.config.SyncStats@7039f47[recordsEmitted=14,bytesEmitted=4037,sourceStateMessagesEmitted=0,destinationStateMessagesEmitted=0,recordsCommitted=0,meanSecondsBeforeSourceStateMessageEmitted=0,maxSecondsBeforeSourceStateMessageEmitted=0,maxSecondsBetweenStateMessageEmittedandCommitted=0,meanSecondsBetweenStateMessageEmittedandCommitted=0,additionalProperties={}],streamStats=[io.airbyte.config.StreamSyncStats@5105d91d[streamName=global_suppressions,stats=io.airbyte.config.SyncStats@33578aa2[recordsEmitted=3,bytesEmitted=151,sourceStateMessagesEmitted=,destinationStateMessagesEmitted=,recordsCommitted=,meanSecondsBeforeSourceStateMessageEmitted=,maxSecondsBeforeSourceStateMessageEmitted=,maxSecondsBetweenStateMessageEmittedandCommitted=,meanSecondsBetweenStateMessageEmittedandCommitted=,additionalProperties={}]], io.airbyte.config.StreamSyncStats@72a84846[streamName=campaigns,stats=io.airbyte.config.SyncStats@69ca08f3[recordsEmitted=5,bytesEmitted=1043,sourceStateMessagesEmitted=,destinationStateMessagesEmitted=,recordsCommitted=,meanSecondsBeforeSourceStateMessageEmitted=,maxSecondsBeforeSourceStateMessageEmitted=,maxSecondsBetweenStateMessageEmittedandCommitted=,meanSecondsBetweenStateMessageEmittedandCommitted=,additionalProperties={}]], io.airbyte.config.StreamSyncStats@1678f9e6[streamName=lists,stats=io.airbyte.config.SyncStats@19f56311[recordsEmitted=1,bytesEmitted=187,sourceStateMessagesEmitted=,destinationStateMessagesEmitted=,recordsCommitted=,meanSecondsBeforeSourceStateMessageEmitted=,maxSecondsBeforeSourceStateMessageEmitted=,maxSecondsBetweenStateMessageEmittedandCommitted=,meanSecondsBetweenStateMessageEmittedandCommitted=,additionalProperties={}]], io.airbyte.config.StreamSyncStats@29afe211[streamName=contacts,stats=io.airbyte.config.SyncStats@101bfa47[recordsEmitted=5,bytesEmitted=2656,sourceStateMessagesEmitted=,destinationStateMessagesEmitted=,recordsCommitted=,meanSecondsBeforeSourceStateMessageEmitted=,maxSecondsBeforeSourceStateMessageEmitted=,maxSecondsBetweenStateMessageEmittedandCommitted=,meanSecondsBetweenStateMessageEmittedandCommitted=,additionalProperties={}]]]],normalizationSummary=,state=,outputCatalog=io.airbyte.protocol.models.ConfiguredAirbyteCatalog@417d866b[streams=[io.airbyte.protocol.models.ConfiguredAirbyteStream@73a298be[stream=io.airbyte.protocol.models.AirbyteStream@254b53b1[name=blocks,jsonSchema={"type":"object","$schema":"http://json-schema.org/draft-07/schema#","properties":{"email":{"type":"string"},"reason":{"type":"string"},"status":{"type":"string"},"created":{"type":"integer"}}},supportedSyncModes=[full_refresh, incremental],sourceDefinedCursor=true,defaultCursorField=[created],sourceDefinedPrimaryKey=[[email]],namespace=,additionalProperties={}],syncMode=full_refresh,cursorField=[created],destinationSyncMode=overwrite,primaryKey=[[email]],additionalProperties={}], io.airbyte.protocol.models.ConfiguredAirbyteStream@6a4d4c9d[stream=io.airbyte.protocol.models.AirbyteStream@472ed161[name=bounces,jsonSchema={"type":"object","$schema":"http://json-schema.org/draft-07/schema#","properties":{"email":{"type":"string"},"reason":{"type":"string"},"status":{"type":"string"},"created":{"type":"integer"}}},supportedSyncModes=[full_refresh, incremental],sourceDefinedCursor=true,defaultCursorField=[created],sourceDefinedPrimaryKey=[[email]],namespace=,additionalProperties={}],syncMode=full_refresh,cursorField=[created],destinationSyncMode=overwrite,primaryKey=[[email]],additionalProperties={}], io.airbyte.protocol.models.ConfiguredAirbyteStream@20aed652[stream=io.airbyte.protocol.models.AirbyteStream@5f87cf2d[name=campaigns,jsonSchema={"type":"object","$schema":"http://json-schema.org/draft-07/schema#","properties":{"id":{"type":"string"},"name":{"type":"string"},"status":{"type":"string"},"is_abtest":{"type":"boolean"},"created_at":{"type":"string"},"updated_at":{"type":"string"}}},supportedSyncModes=[full_refresh],sourceDefinedCursor=,defaultCursorField=[],sourceDefinedPrimaryKey=[[id]],namespace=,additionalProperties={}],syncMode=full_refresh,cursorField=[],destinationSyncMode=overwrite,primaryKey=[[id]],additionalProperties={}], io.airbyte.protocol.models.ConfiguredAirbyteStream@5fae49e4[stream=io.airbyte.protocol.models.AirbyteStream@6e3a3ea9[name=contacts,jsonSchema={"type":"object","$schema":"http://json-schema.org/draft-07/schema#","properties":{"id":{"type":"string"},"city":{"type":"string"},"line":{"type":"string"},"email":{"type":"string"},"country":{"type":"string"},"facebook":{"type":"string"},"list_ids":{"type":["null","array"]},"whatsapp":{"type":"string"},"_metadata":{"type":"object","properties":{"self":{"type":"string"}}},"last_name":{"type":"string"},"created_at":{"type":"string"},"first_name":{"type":"string"},"updated_at":{"type":"string"},"postal_code":{"type":"string"},"unique_name":{"type":"string"},"phone_number":{"type":"string"},"custom_fields":{"type":"object"},"address_line_1":{"type":"string"},"address_line_2":{"type":"string"},"alternate_emails":{"type":["null","array"]},"state_province_region":{"type":"string"}}},supportedSyncModes=[full_refresh],sourceDefinedCursor=,defaultCursorField=[],sourceDefinedPrimaryKey=[[id]],namespace=,additionalProperties={}],syncMode=full_refresh,cursorField=[],destinationSyncMode=overwrite,primaryKey=[[id]],additionalProperties={}], io.airbyte.protocol.models.ConfiguredAirbyteStream@38f0697a[stream=io.airbyte.protocol.models.AirbyteStream@596f27ee[name=global_suppressions,jsonSchema={"type":"object","$schema":"http://json-schema.org/draft-07/schema#","properties":{"email":{"type":"string"},"created":{"type":"integer"}}},supportedSyncModes=[full_refresh, incremental],sourceDefinedCursor=true,defaultCursorField=[created],sourceDefinedPrimaryKey=[[email]],namespace=,additionalProperties={}],syncMode=full_refresh,cursorField=[created],destinationSyncMode=overwrite,primaryKey=[[email]],additionalProperties={}], io.airbyte.protocol.models.ConfiguredAirbyteStream@74a3b44e[stream=io.airbyte.protocol.models.AirbyteStream@4bd31305[name=invalid_emails,jsonSchema={"type":"object","$schema":"http://json-schema.org/draft-07/schema#","properties":{"email":{"type":"string"},"reason":{"type":"string"},"created":{"type":"integer"}}},supportedSyncModes=[full_refresh, incremental],sourceDefinedCursor=true,defaultCursorField=[created],sourceDefinedPrimaryKey=[[email]],namespace=,additionalProperties={}],syncMode=full_refresh,cursorField=[created],destinationSyncMode=overwrite,primaryKey=[[email]],additionalProperties={}], io.airbyte.protocol.models.ConfiguredAirbyteStream@705318bc[stream=io.airbyte.protocol.models.AirbyteStream@50a3110f[name=lists,jsonSchema={"type":"object","$schema":"http://json-schema.org/draft-07/schema#","properties":{"id":{"type":"string"},"name":{"type":"string"},"_metadata":{"type":"object","properties":{"self":{"type":"string"}}},"contact_count":{"type":"integer"}}},supportedSyncModes=[full_refresh],sourceDefinedCursor=,defaultCursorField=[],sourceDefinedPrimaryKey=[[id]],namespace=,additionalProperties={}],syncMode=full_refresh,cursorField=[],destinationSyncMode=overwrite,primaryKey=[[id]],additionalProperties={}], io.airbyte.protocol.models.ConfiguredAirbyteStream@394bce91[stream=io.airbyte.protocol.models.AirbyteStream@6ad3b986[name=messages,jsonSchema={"type":"object","$schema":"http://json-schema.org/draft-07/schema#","properties":{"msg_id":{"type":"string"},"status":{"type":"string"},"subject":{"type":"string"},"to_email":{"type":"string"},"from_email":{"type":"string"},"opens_count":{"type":"integer"},"clicks_count":{"type":"integer"},"last_event_time":{"type":"string"}}},supportedSyncModes=[full_refresh, incremental],sourceDefinedCursor=true,defaultCursorField=[last_event_time],sourceDefinedPrimaryKey=[[msg_id]],namespace=,additionalProperties={}],syncMode=full_refresh,cursorField=[last_event_time],destinationSyncMode=overwrite,primaryKey=[[msg_id]],additionalProperties={}], io.airbyte.protocol.models.ConfiguredAirbyteStream@2adca98[stream=io.airbyte.protocol.models.AirbyteStream@4a4e88ab[name=segments,jsonSchema={"type":"object","$schema":"http://json-schema.org/draft-07/schema#","properties":{"id":{"type":"string"},"name":{"type":"string"},"created_at":{"type":"string"},"updated_at":{"type":"string"},"contact_count":{"type":"integer"},"parent_list_id":{"type":["string","null"]},"sample_updated_at":{"type":"string"},"next_sample_update":{"type":"string"}}},supportedSyncModes=[full_refresh],sourceDefinedCursor=,defaultCursorField=[],sourceDefinedPrimaryKey=[[id]],namespace=,additionalProperties={}],syncMode=full_refresh,cursorField=[],destinationSyncMode=overwrite,primaryKey=[[id]],additionalProperties={}], io.airbyte.protocol.models.ConfiguredAirbyteStream@2d16e835[stream=io.airbyte.protocol.models.AirbyteStream@5a7e6b98[name=single_sends,jsonSchema={"type":"object","$schema":"http://json-schema.org/draft-07/schema#","properties":{"id":{"type":"string","format":"uuid"},"stats":{"type":"object","properties":{"opens":{"type":"integer"},"clicks":{"type":"integer"},"bounces":{"type":"integer"},"requests":{"type":"integer"},"delivered":{"type":"integer"},"bounce_drops":{"type":"integer"},"spam_reports":{"type":"integer"},"unique_opens":{"type":"integer"},"unsubscribes":{"type":"integer"},"unique_clicks":{"type":"integer"},"invalid_emails":{"type":"integer"},"spam_report_drops":{"type":"integer"}}},"ab_phase":{"type":"string"},"aggregation":{"type":"string"},"ab_variation":{"type":"string"}}},supportedSyncModes=[full_refresh],sourceDefinedCursor=,defaultCursorField=[],sourceDefinedPrimaryKey=[[id]],namespace=,additionalProperties={}],syncMode=full_refresh,cursorField=[],destinationSyncMode=overwrite,primaryKey=[[id]],additionalProperties={}], io.airbyte.protocol.models.ConfiguredAirbyteStream@36eb8345[stream=io.airbyte.protocol.models.AirbyteStream@85e4ee3[name=spam_reports,jsonSchema={"type":"object","$schema":"http://json-schema.org/draft-07/schema#","properties":{"ip":{"type":"string"},"email":{"type":"string"},"created":{"type":"integer"}}},supportedSyncModes=[full_refresh, incremental],sourceDefinedCursor=true,defaultCursorField=[created],sourceDefinedPrimaryKey=[[email]],namespace=,additionalProperties={}],syncMode=full_refresh,cursorField=[created],destinationSyncMode=overwrite,primaryKey=[[email]],additionalProperties={}], io.airbyte.protocol.models.ConfiguredAirbyteStream@1213589d[stream=io.airbyte.protocol.models.AirbyteStream@3b18d3ec[name=stats_automations,jsonSchema={"type":"object","$schema":"http://json-schema.org/draft-07/schema#","properties":{"id":{"type":"string"},"stats":{"type":"object","properties":{"opens":{"type":"integer"},"clicks":{"type":"integer"},"bounces":{"type":"integer"},"requests":{"type":"integer"},"delivered":{"type":"integer"},"bounce_drops":{"type":"integer"},"spam_reports":{"type":"integer"},"unique_opens":{"type":"integer"},"unsubscribes":{"type":"integer"},"unique_clicks":{"type":"integer"},"invalid_emails":{"type":"integer"},"spam_report_drops":{"type":"integer"}}},"step_id":{"type":"string"},"aggregation":{"type":"string"}}},supportedSyncModes=[full_refresh],sourceDefinedCursor=,defaultCursorField=[],sourceDefinedPrimaryKey=[[id]],namespace=,additionalProperties={}],syncMode=full_refresh,cursorField=[],destinationSyncMode=overwrite,primaryKey=[[id]],additionalProperties={}], io.airbyte.protocol.models.ConfiguredAirbyteStream@51038e61[stream=io.airbyte.protocol.models.AirbyteStream@2bd0ebf4[name=suppression_group_members,jsonSchema={"type":"object","$schema":"http://json-schema.org/draft-07/schema#","properties":{"email":{"type":"string"},"group_id":{"type":"integer"},"created_at":{"type":"integer"},"group_name":{"type":"string"}}},supportedSyncModes=[full_refresh],sourceDefinedCursor=,defaultCursorField=[],sourceDefinedPrimaryKey=[[group_id]],namespace=,additionalProperties={}],syncMode=full_refresh,cursorField=[],destinationSyncMode=overwrite,primaryKey=[[group_id]],additionalProperties={}], io.airbyte.protocol.models.ConfiguredAirbyteStream@238f89a3[stream=io.airbyte.protocol.models.AirbyteStream@5ef5a98d[name=suppression_groups,jsonSchema={"type":"object","$schema":"http://json-schema.org/draft-07/schema#","properties":{"id":{"type":"integer"},"name":{"type":"string"},"is_default":{"type":"boolean"},"description":{"type":"string"},"unsubscribes":{"type":"integer"}}},supportedSyncModes=[full_refresh],sourceDefinedCursor=,defaultCursorField=[],sourceDefinedPrimaryKey=[[id]],namespace=,additionalProperties={}],syncMode=full_refresh,cursorField=[],destinationSyncMode=overwrite,primaryKey=[[id]],additionalProperties={}], io.airbyte.protocol.models.ConfiguredAirbyteStream@134854b4[stream=io.airbyte.protocol.models.AirbyteStream@3f6b8262[name=templates,jsonSchema={"type":"object","$schema":"http://json-schema.org/draft-07/schema#","properties":{"id":{"type":["string"]},"name":{"type":["null","string"]},"versions":{"type":["array"]},"generation":{"type":["null","string"]},"updated_at":{"type":["null","string"]}}},supportedSyncModes=[full_refresh],sourceDefinedCursor=,defaultCursorField=[],sourceDefinedPrimaryKey=[[id]],namespace=,additionalProperties={}],syncMode=full_refresh,cursorField=[],destinationSyncMode=overwrite,primaryKey=[[id]],additionalProperties={}]],additionalProperties={}],failures=[io.airbyte.config.FailureReason@6076d6bd[failureOrigin=source,failureType=system_error,internalMessage=400 Client Error: Bad Request for url: https://api.sendgrid.com/v3/messages?query=last_event_time%20BETWEEN%20TIMESTAMP%20%22None%22%20AND%20TIMESTAMP%20%222023-03-20T07%3A11%3A33Z%22&limit=1000,externalMessage=authorization required,metadata=io.airbyte.config.Metadata@133cb2ff[additionalProperties={attemptNumber=0, jobId=414, from_trace_message=true, connector_command=read}],stacktrace=Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 115, in read yield from self._read_stream( File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 184, in _read_stream for record in record_iterator: File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 282, in _read_full_refresh for record in records: File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py", line 421, in read_records response = self._send_request(request, request_kwargs) File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py", line 339, in _send_request return backoff_handler(user_backoff_handler)(request, request_kwargs) File "/usr/local/lib/python3.9/site-packages/backoff/_sync.py", line 105, in retry ret = target(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/backoff/_sync.py", line 105, in retry ret = target(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py", line 306, in _send raise exc File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py", line 303, in _send response.raise_for_status() File "/usr/local/lib/python3.9/site-packages/requests/models.py", line 1021, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://api.sendgrid.com/v3/messages?query=last_event_time%20BETWEEN%20TIMESTAMP%20%22None%22%20AND%20TIMESTAMP%20%222023-03-20T07%3A11%3A33Z%22&limit=1000 ,retryable=,timestamp=1679296293820], io.airbyte.config.FailureReason@4200a9d8[failureOrigin=source,failureType=,internalMessage=io.airbyte.workers.general.DefaultReplicationWorker$SourceException: Source cannot be stopped!,externalMessage=Something went wrong within the source connector,metadata=io.airbyte.config.Metadata@211b5b68[additionalProperties={attemptNumber=0, jobId=414, connector_command=read}],stacktrace=java.util.concurrent.CompletionException: io.airbyte.workers.general.DefaultReplicationWorker$SourceException: Source cannot be stopped! at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315) at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:320) at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1807) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) at java.base/java.lang.Thread.run(Thread.java:1589) Caused by: io.airbyte.workers.general.DefaultReplicationWorker$SourceException: Source cannot be stopped! at io.airbyte.workers.general.DefaultReplicationWorker.lambda$getReplicationRunnable$6(DefaultReplicationWorker.java:359) at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804) ... 3 more Caused by: io.airbyte.workers.exception.WorkerException: Source process exit with code 1. This warning is normal if the job was cancelled. at io.airbyte.workers.internal.DefaultAirbyteSource.close(DefaultAirbyteSource.java:140) at io.airbyte.workers.general.DefaultReplicationWorker.lambda$getReplicationRunnable$6(DefaultReplicationWorker.java:357) ... 4 more ,retryable=,timestamp=1679296294114]]] 2023-03-20 07:11:35 INFO i.a.w.t.TemporalUtils(withBackgroundHeartbeat):312 - Stopping temporal heartbeating... 2023-03-20 07:11:35 INFO i.a.c.i.LineGobbler(voidCall):114 - 2023-03-20 07:11:35 INFO i.a.c.i.LineGobbler(voidCall):114 - ----- END REPLICATION ----- 2023-03-20 07:11:35 INFO i.a.c.i.LineGobbler(voidCall):114 - 2023-03-20 07:11:31 source > Read 0 records from blocks stream 2023-03-20 07:11:31 source > Finished syncing blocks 2023-03-20 07:11:31 source > SourceSendgrid runtimes: Syncing stream blocks 0:00:01.229770 2023-03-20 07:11:31 source > Syncing stream: bounces 2023-03-20 07:11:31 source > Read 0 records from bounces stream 2023-03-20 07:11:31 source > Finished syncing bounces 2023-03-20 07:11:31 source > SourceSendgrid runtimes: Syncing stream blocks 0:00:01.229770 Syncing stream bounces 0:00:00.150014 2023-03-20 07:11:31 source > Syncing stream: campaigns 2023-03-20 07:11:32 source > Read 5 records from campaigns stream 2023-03-20 07:11:32 source > Finished syncing campaigns 2023-03-20 07:11:32 source > SourceSendgrid runtimes: Syncing stream blocks 0:00:01.229770 Syncing stream bounces 0:00:00.150014 Syncing stream campaigns 0:00:00.344146 2023-03-20 07:11:32 source > Syncing stream: contacts 2023-03-20 07:11:32 source > Read 5 records from contacts stream 2023-03-20 07:11:32 source > Finished syncing contacts 2023-03-20 07:11:32 source > SourceSendgrid runtimes: Syncing stream blocks 0:00:01.229770 Syncing stream bounces 0:00:00.150014 Syncing stream campaigns 0:00:00.344146 Syncing stream contacts 0:00:00.663055 2023-03-20 07:11:32 source > Syncing stream: global_suppressions 2023-03-20 07:11:33 source > Read 3 records from global_suppressions stream 2023-03-20 07:11:33 source > Finished syncing global_suppressions 2023-03-20 07:11:33 source > SourceSendgrid runtimes: Syncing stream blocks 0:00:01.229770 Syncing stream bounces 0:00:00.150014 Syncing stream campaigns 0:00:00.344146 Syncing stream contacts 0:00:00.663055 Syncing stream global_suppressions 0:00:00.158298 2023-03-20 07:11:33 source > Syncing stream: invalid_emails 2023-03-20 07:11:33 source > Read 0 records from invalid_emails stream 2023-03-20 07:11:33 source > Finished syncing invalid_emails 2023-03-20 07:11:33 source > SourceSendgrid runtimes: Syncing stream blocks 0:00:01.229770 Syncing stream bounces 0:00:00.150014 Syncing stream campaigns 0:00:00.344146 Syncing stream contacts 0:00:00.663055 Syncing stream global_suppressions 0:00:00.158298 Syncing stream invalid_emails 0:00:00.175587 2023-03-20 07:11:33 source > Syncing stream: lists 2023-03-20 07:11:33 source > Read 1 records from lists stream 2023-03-20 07:11:33 source > Finished syncing lists 2023-03-20 07:11:33 source > SourceSendgrid runtimes: Syncing stream blocks 0:00:01.229770 Syncing stream bounces 0:00:00.150014 Syncing stream campaigns 0:00:00.344146 Syncing stream contacts 0:00:00.663055 Syncing stream global_suppressions 0:00:00.158298 Syncing stream invalid_emails 0:00:00.175587 Syncing stream lists 0:00:00.471232 2023-03-20 07:11:33 source > Syncing stream: messages 2023-03-20 07:11:33 source > {"errors":[{"message":"authorization required"}]} 2023-03-20 07:11:33 source > Encountered an exception while reading stream messages Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 115, in read yield from self._read_stream( File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 184, in _read_stream for record in record_iterator: File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 282, in _read_full_refresh for record in records: File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py", line 421, in read_records response = self._send_request(request, request_kwargs) File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py", line 339, in _send_request return backoff_handler(user_backoff_handler)(request, request_kwargs) File "/usr/local/lib/python3.9/site-packages/backoff/_sync.py", line 105, in retry ret = target(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/backoff/_sync.py", line 105, in retry ret = target(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py", line 306, in _send raise exc File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py", line 303, in _send response.raise_for_status() File "/usr/local/lib/python3.9/site-packages/requests/models.py", line 1021, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://api.sendgrid.com/v3/messages?query=last_event_time%20BETWEEN%20TIMESTAMP%20%22None%22%20AND%20TIMESTAMP%20%222023-03-20T07%3A11%3A33Z%22&limit=1000 2023-03-20 07:11:33 source > Finished syncing messages 2023-03-20 07:11:33 source > SourceSendgrid runtimes: Syncing stream blocks 0:00:01.229770 Syncing stream bounces 0:00:00.150014 Syncing stream campaigns 0:00:00.344146 Syncing stream contacts 0:00:00.663055 Syncing stream global_suppressions 0:00:00.158298 Syncing stream invalid_emails 0:00:00.175587 Syncing stream lists 0:00:00.471232 Syncing stream messages 0:00:00.097746 2023-03-20 07:11:33 source > 400 Client Error: Bad Request for url: https://api.sendgrid.com/v3/messages?query=last_event_time%20BETWEEN%20TIMESTAMP%20%22None%22%20AND%20TIMESTAMP%20%222023-03-20T07%3A11%3A33Z%22&limit=1000 Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 115, in read yield from self._read_stream( File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 184, in _read_stream for record in record_iterator: File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 282, in _read_full_refresh for record in records: File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py", line 421, in read_records response = self._send_request(request, request_kwargs) File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py", line 339, in _send_request return backoff_handler(user_backoff_handler)(request, request_kwargs) File "/usr/local/lib/python3.9/site-packages/backoff/_sync.py", line 105, in retry ret = target(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/backoff/_sync.py", line 105, in retry ret = target(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py", line 306, in _send raise exc File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py", line 303, in _send response.raise_for_status() File "/usr/local/lib/python3.9/site-packages/requests/models.py", line 1021, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://api.sendgrid.com/v3/messages?query=last_event_time%20BETWEEN%20TIMESTAMP%20%22None%22%20AND%20TIMESTAMP%20%222023-03-20T07%3A11%3A33Z%22&limit=1000 The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/airbyte/integration_code/main.py", line 13, in launch(source, sys.argv[1:]) File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py", line 123, in launch for message in source_entrypoint.run(parsed_args): File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py", line 114, in run for message in generator: File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 128, in read raise AirbyteTracedException.from_exception(e, message=display_message) from e airbyte_cdk.utils.traced_exception.AirbyteTracedException: 400 Client Error: Bad Request for url: https://api.sendgrid.com/v3/messages?query=last_event_time%20BETWEEN%20TIMESTAMP%20%22None%22%20AND%20TIMESTAMP%20%222023-03-20T07%3A11%3A33Z%22&limit=1000 2023-03-20 07:11:34 destination > 2023-03-20 07:11:34 INFO i.a.i.d.r.SerializedBufferingStrategy(lambda$addRecord$0):47 - Starting a new buffer for stream campaigns (current state: 0 bytes in 0 buffers) 2023-03-20 07:11:34 destination > 2023-03-20 07:11:34 INFO i.a.i.d.r.SerializedBufferingStrategy(lambda$addRecord$0):47 - Starting a new buffer for stream contacts (current state: 0 bytes in 1 buffers) 2023-03-20 07:11:34 destination > 2023-03-20 07:11:34 INFO i.a.i.d.r.SerializedBufferingStrategy(lambda$addRecord$0):47 - Starting a new buffer for stream global_suppressions (current state: 0 bytes in 2 buffers) 2023-03-20 07:11:34 destination > 2023-03-20 07:11:34 INFO i.a.i.d.r.SerializedBufferingStrategy(lambda$addRecord$0):47 - Starting a new buffer for stream lists (current state: 0 bytes in 3 buffers) 2023-03-20 07:11:34 destination > 2023-03-20 07:11:34 INFO i.a.i.b.FailureTrackingAirbyteMessageConsumer(close):62 - Airbyte message consumer: succeeded. 2023-03-20 07:11:34 destination > 2023-03-20 07:11:34 INFO i.a.i.d.b.BufferedStreamConsumer(close):171 - executing on success close procedure. 2023-03-20 07:11:34 destination > 2023-03-20 07:11:34 INFO i.a.i.d.r.SerializedBufferingStrategy(flushAll):100 - Flushing all 4 current buffers (0 bytes in total) 2023-03-20 07:11:34 destination > 2023-03-20 07:11:34 INFO i.a.i.d.r.SerializedBufferingStrategy(flushAll):102 - Flushing buffer of stream contacts (0 bytes) 2023-03-20 07:11:34 destination > 2023-03-20 07:11:34 INFO i.a.i.d.s.S3ConsumerFactory(lambda$flushBufferFunction$2):121 - Flushing buffer for stream contacts (0 bytes) to storage 2023-03-20 07:11:34 destination > 2023-03-20 07:11:34 INFO i.a.i.d.r.BaseSerializedBuffer(flush):138 - Finished writing data to f217f9d4-4744-4e16-b0f1-20b2fa3cf5f59325290635024080685.csv (3 KB) 2023-03-20 07:11:34 destination > 2023-03-20 07:11:34 INFO a.m.s.StreamTransferManager(getMultiPartOutputStreams):329 - Initiated multipart upload to redshift-dev-s3-bucket/testAutomation/contacts/2023_03_20_1679296287341_0.csv with full ID uqnKucwv43sv7Ofe8lzAzK6q3TSFNZ8Hx65PNAyNIdx.0EdJ7QyAzlP_Ba1PStEMjEz2Rq9aZE3oNAsDffh7BWtMRgsjIRgodZHI.XC__DN9m607sddWZ02D8nA0Clbn 2023-03-20 07:11:34 destination > 2023-03-20 07:11:34 INFO a.m.s.MultiPartOutputStream(close):158 - Called close() on [MultipartOutputStream for parts 1 - 10000] 2023-03-20 07:11:34 destination > 2023-03-20 07:11:34 INFO a.m.s.StreamTransferManager(complete):367 - [Manager uploading to redshift-dev-s3-bucket/testAutomation/contacts/2023_03_20_1679296287341_0.csv with id uqnKucwv4...D8nA0Clbn]: Uploading leftover stream [Part number 1 containing 0.00 MB] 2023-03-20 07:11:34 destination > 2023-03-20 07:11:34 INFO a.m.s.StreamTransferManager(uploadStreamPart):558 - [Manager uploading to redshift-dev-s3-bucket/testAutomation/contacts/2023_03_20_1679296287341_0.csv with id uqnKucwv4...D8nA0Clbn]: Finished uploading [Part number 1 containing 0.00 MB] 2023-03-20 07:11:34 destination > 2023-03-20 07:11:34 INFO a.m.s.StreamTransferManager(complete):395 - [Manager uploading to redshift-dev-s3-bucket/testAutomation/contacts/2023_03_20_1679296287341_0.csv with id uqnKucwv4...D8nA0Clbn]: Completed 2023-03-20 07:11:34 destination > 2023-03-20 07:11:34 INFO i.a.i.d.s.S3StorageOperations(loadDataIntoBucket):198 - Uploaded buffer file to storage: f217f9d4-4744-4e16-b0f1-20b2fa3cf5f59325290635024080685.csv -> testAutomation/contacts/2023_03_20_1679296287341_0.csv (filename: 2023_03_20_1679296287341_0.csv) 2023-03-20 07:11:34 destination > 2023-03-20 07:11:34 INFO i.a.i.d.r.FileBuffer(deleteFile):90 - Deleting tempFile data f217f9d4-4744-4e16-b0f1-20b2fa3cf5f59325290635024080685.csv 2023-03-20 07:11:34 destination > 2023-03-20 07:11:34 INFO i.a.i.d.r.SerializedBufferingStrategy(flushAll):102 - Flushing buffer of stream campaigns (0 bytes) 2023-03-20 07:11:34 destination > 2023-03-20 07:11:34 INFO i.a.i.d.s.S3ConsumerFactory(lambda$flushBufferFunction$2):121 - Flushing buffer for stream campaigns (0 bytes) to storage 2023-03-20 07:11:34 destination > 2023-03-20 07:11:34 INFO i.a.i.d.r.BaseSerializedBuffer(flush):138 - Finished writing data to 1c223890-0158-4876-b286-d3d35669f99a2279894964825728631.csv (1 KB) 2023-03-20 07:11:34 destination > 2023-03-20 07:11:34 INFO a.m.s.StreamTransferManager(getMultiPartOutputStreams):329 - Initiated multipart upload to redshift-dev-s3-bucket/testAutomation/campaigns/2023_03_20_1679296287341_0.csv with full ID TOzFK_MTE3oaYusQnRiXXM0aXZJJJWTjs7QL..Y5UxDfNpb7RxYU6S4HFCpJm_Q_lUo1dv797Kqn0eSw2.FKtfMrbG69O4KhTvHM8ilySZKPrrKLVM6getfmQD7.q7GS 2023-03-20 07:11:34 destination > 2023-03-20 07:11:34 INFO a.m.s.MultiPartOutputStream(close):158 - Called close() on [MultipartOutputStream for parts 1 - 10000] 2023-03-20 07:11:34 destination > 2023-03-20 07:11:34 INFO a.m.s.StreamTransferManager(complete):367 - [Manager uploading to redshift-dev-s3-bucket/testAutomation/campaigns/2023_03_20_1679296287341_0.csv with id TOzFK_MTE...mQD7.q7GS]: Uploading leftover stream [Part number 1 containing 0.00 MB] 2023-03-20 07:11:35 destination > 2023-03-20 07:11:35 INFO a.m.s.StreamTransferManager(uploadStreamPart):558 - [Manager uploading to redshift-dev-s3-bucket/testAutomation/campaigns/2023_03_20_1679296287341_0.csv with id TOzFK_MTE...mQD7.q7GS]: Finished uploading [Part number 1 containing 0.00 MB] 2023-03-20 07:11:35 destination > 2023-03-20 07:11:35 INFO a.m.s.StreamTransferManager(complete):395 - [Manager uploading to redshift-dev-s3-bucket/testAutomation/campaigns/2023_03_20_1679296287341_0.csv with id TOzFK_MTE...mQD7.q7GS]: Completed 2023-03-20 07:11:35 destination > 2023-03-20 07:11:35 INFO i.a.i.d.s.S3StorageOperations(loadDataIntoBucket):198 - Uploaded buffer file to storage: 1c223890-0158-4876-b286-d3d35669f99a2279894964825728631.csv -> testAutomation/campaigns/2023_03_20_1679296287341_0.csv (filename: 2023_03_20_1679296287341_0.csv) 2023-03-20 07:11:35 destination > 2023-03-20 07:11:35 INFO i.a.i.d.r.FileBuffer(deleteFile):90 - Deleting tempFile data 1c223890-0158-4876-b286-d3d35669f99a2279894964825728631.csv 2023-03-20 07:11:35 destination > 2023-03-20 07:11:35 INFO i.a.i.d.r.SerializedBufferingStrategy(flushAll):102 - Flushing buffer of stream lists (0 bytes) 2023-03-20 07:11:35 destination > 2023-03-20 07:11:35 INFO i.a.i.d.s.S3ConsumerFactory(lambda$flushBufferFunction$2):121 - Flushing buffer for stream lists (0 bytes) to storage 2023-03-20 07:11:35 destination > 2023-03-20 07:11:35 INFO i.a.i.d.r.BaseSerializedBuffer(flush):138 - Finished writing data to c5024c2b-eb72-4141-a242-a55bca03fa3813027913685977670061.csv (316 bytes) 2023-03-20 07:11:35 destination > 2023-03-20 07:11:35 INFO a.m.s.StreamTransferManager(getMultiPartOutputStreams):329 - Initiated multipart upload to redshift-dev-s3-bucket/testAutomation/lists/2023_03_20_1679296287341_0.csv with full ID 1FhKtRhq7tjfjn087EYWH1mYNtjSmz1c9fUa7RZ4ErxVVOmJu6SXwCnQtkXBO8ZIXcVM9VDMFzWIPi.BLWaHWshhvs1QI2xPQhgqQ3OyGN.6PTEPELZhwgr4BOaH3Y7G 2023-03-20 07:11:35 destination > 2023-03-20 07:11:35 INFO a.m.s.MultiPartOutputStream(close):158 - Called close() on [MultipartOutputStream for parts 1 - 10000] 2023-03-20 07:11:35 destination > 2023-03-20 07:11:35 INFO a.m.s.StreamTransferManager(complete):367 - [Manager uploading to redshift-dev-s3-bucket/testAutomation/lists/2023_03_20_1679296287341_0.csv with id 1FhKtRhq7...4BOaH3Y7G]: Uploading leftover stream [Part number 1 containing 0.00 MB] 2023-03-20 07:11:35 destination > 2023-03-20 07:11:35 INFO a.m.s.StreamTransferManager(uploadStreamPart):558 - [Manager uploading to redshift-dev-s3-bucket/testAutomation/lists/2023_03_20_1679296287341_0.csv with id 1FhKtRhq7...4BOaH3Y7G]: Finished uploading [Part number 1 containing 0.00 MB] 2023-03-20 07:11:35 destination > 2023-03-20 07:11:35 INFO a.m.s.StreamTransferManager(complete):395 - [Manager uploading to redshift-dev-s3-bucket/testAutomation/lists/2023_03_20_1679296287341_0.csv with id 1FhKtRhq7...4BOaH3Y7G]: Completed 2023-03-20 07:11:35 destination > 2023-03-20 07:11:35 INFO i.a.i.d.s.S3StorageOperations(loadDataIntoBucket):198 - Uploaded buffer file to storage: c5024c2b-eb72-4141-a242-a55bca03fa3813027913685977670061.csv -> testAutomation/lists/2023_03_20_1679296287341_0.csv (filename: 2023_03_20_1679296287341_0.csv) 2023-03-20 07:11:35 destination > 2023-03-20 07:11:35 INFO i.a.i.d.r.FileBuffer(deleteFile):90 - Deleting tempFile data c5024c2b-eb72-4141-a242-a55bca03fa3813027913685977670061.csv 2023-03-20 07:11:35 destination > 2023-03-20 07:11:35 INFO i.a.i.d.r.SerializedBufferingStrategy(flushAll):102 - Flushing buffer of stream global_suppressions (0 bytes) 2023-03-20 07:11:35 destination > 2023-03-20 07:11:35 INFO i.a.i.d.s.S3ConsumerFactory(lambda$flushBufferFunction$2):121 - Flushing buffer for stream global_suppressions (0 bytes) to storage 2023-03-20 07:11:35 destination > 2023-03-20 07:11:35 INFO i.a.i.d.r.BaseSerializedBuffer(flush):138 - Finished writing data to 339692c2-22f1-467e-8891-2b0c315434657992719876889681729.csv (396 bytes) 2023-03-20 07:11:35 destination > 2023-03-20 07:11:35 INFO a.m.s.StreamTransferManager(getMultiPartOutputStreams):329 - Initiated multipart upload to redshift-dev-s3-bucket/testAutomation/global_suppressions/2023_03_20_1679296287341_0.csv with full ID IlChDiT3EiRJebvQlx7YNQ7D5mBOegu2qw62Fag64dpqFDF44idNUuWRThDpUK84IlwLrYtf.vx8kjxEQM1p.IY1dsGlxOc7kEp8DJg461y8QPA55qYxVinNRAqaeXmM 2023-03-20 07:11:35 destination > 2023-03-20 07:11:35 INFO a.m.s.MultiPartOutputStream(close):158 - Called close() on [MultipartOutputStream for parts 1 - 10000] 2023-03-20 07:11:35 destination > 2023-03-20 07:11:35 INFO a.m.s.StreamTransferManager(complete):367 - [Manager uploading to redshift-dev-s3-bucket/testAutomation/global_suppressions/2023_03_20_1679296287341_0.csv with id IlChDiT3E...NRAqaeXmM]: Uploading leftover stream [Part number 1 containing 0.00 MB] 2023-03-20 07:11:35 destination > 2023-03-20 07:11:35 INFO a.m.s.StreamTransferManager(uploadStreamPart):558 - [Manager uploading to redshift-dev-s3-bucket/testAutomation/global_suppressions/2023_03_20_1679296287341_0.csv with id IlChDiT3E...NRAqaeXmM]: Finished uploading [Part number 1 containing 0.00 MB] 2023-03-20 07:11:35 destination > 2023-03-20 07:11:35 INFO a.m.s.StreamTransferManager(complete):395 - [Manager uploading to redshift-dev-s3-bucket/testAutomation/global_suppressions/2023_03_20_1679296287341_0.csv with id IlChDiT3E...NRAqaeXmM]: Completed 2023-03-20 07:11:35 destination > 2023-03-20 07:11:35 INFO i.a.i.d.s.S3StorageOperations(loadDataIntoBucket):198 - Uploaded buffer file to storage: 339692c2-22f1-467e-8891-2b0c315434657992719876889681729.csv -> testAutomation/global_suppressions/2023_03_20_1679296287341_0.csv (filename: 2023_03_20_1679296287341_0.csv) 2023-03-20 07:11:35 destination > 2023-03-20 07:11:35 INFO i.a.i.d.r.FileBuffer(deleteFile):90 - Deleting tempFile data 339692c2-22f1-467e-8891-2b0c315434657992719876889681729.csv 2023-03-20 07:11:35 destination > 2023-03-20 07:11:35 INFO i.a.i.d.r.SerializedBufferingStrategy(close):122 - Closing buffer for stream contacts 2023-03-20 07:11:35 destination > 2023-03-20 07:11:35 INFO i.a.i.d.r.SerializedBufferingStrategy(close):122 - Closing buffer for stream campaigns 2023-03-20 07:11:35 destination > 2023-03-20 07:11:35 INFO i.a.i.d.r.SerializedBufferingStrategy(close):122 - Closing buffer for stream lists 2023-03-20 07:11:35 destination > 2023-03-20 07:11:35 INFO i.a.i.d.r.SerializedBufferingStrategy(close):122 - Closing buffer for stream global_suppressions 2023-03-20 07:11:35 destination > 2023-03-20 07:11:35 INFO i.a.i.b.IntegrationRunner(runInternal):152 - Completed integration: io.airbyte.integrations.destination.s3.S3Destination 2023-03-20 07:11:36 INFO i.a.v.j.JsonSchemaValidator(test):71 - JSON schema validation failed. errors: $.compression_codec: is missing but it is required, $.format_type: does not have a value in the enumeration [Avro] 2023-03-20 07:11:58 destination > 2023-03-20 07:11:55 INFO i.a.i.b.IntegrationCliParser(parseOptions):118 - integration args: {catalog=destination_catalog.json, write=null, config=destination_config.json} 2023-03-20 07:11:58 source > Starting syncing SourceSendgrid 2023-03-20 07:11:58 destination > 2023-03-20 07:11:55 INFO i.a.i.b.IntegrationRunner(runInternal):104 - Running integration: io.airbyte.integrations.destination.s3.S3Destination 2023-03-20 07:11:58 source > Syncing stream: blocks 2023-03-20 07:11:58 destination > 2023-03-20 07:11:55 INFO i.a.i.b.IntegrationRunner(runInternal):105 - Command: WRITE 2023-03-20 07:11:58 destination > 2023-03-20 07:11:55 INFO i.a.i.b.IntegrationRunner(runInternal):106 - Integration config: IntegrationConfig{command=WRITE, configPath='destination_config.json', catalogPath='destination_catalog.json', statePath='null'} 2023-03-20 07:11:58 destination > 2023-03-20 07:11:55 WARN c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword airbyte_secret - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword 2023-03-20 07:11:58 source > Read 0 records from blocks stream 2023-03-20 07:11:58 destination > 2023-03-20 07:11:55 WARN c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword examples - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword 2023-03-20 07:11:58 destination > 2023-03-20 07:11:55 WARN c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword order - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword 2023-03-20 07:11:58 destination > 2023-03-20 07:11:55 WARN c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword requires - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword 2023-03-20 07:11:58 destination > 2023-03-20 07:11:55 INFO i.a.i.d.s.S3FormatConfigs(getS3FormatConfig):22 - S3 format config: {"flattening":"No flattening","compression":{"compression_type":"No Compression"},"format_type":"CSV"} 2023-03-20 07:11:36 INFO i.a.w.t.TemporalAttemptExecution(get):108 - Cloud storage job log path: /workspace/414/1/logs.log 2023-03-20 07:11:36 INFO i.a.w.t.TemporalAttemptExecution(get):111 - Executing worker wrapper. Airbyte version: 0.40.6 2023-03-20 07:11:36 INFO i.a.c.i.LineGobbler(voidCall):114 - 2023-03-20 07:11:36 INFO i.a.w.p.KubeProcessFactory(create):100 - Attempting to start pod = source-sendgrid-check-414-1-ufueb for airbyte/source-sendgrid:0.2.14 with resources io.airbyte.config.ResourceRequirements@4ffb823e[cpuRequest=,cpuLimit=,memoryRequest=,memoryLimit=] 2023-03-20 07:11:36 INFO i.a.w.p.KubeProcessFactory(create):103 - source-sendgrid-check-414-1-ufueb stdoutLocalPort = 9020 2023-03-20 07:11:36 INFO i.a.w.p.KubeProcessFactory(create):106 - source-sendgrid-check-414-1-ufueb stderrLocalPort = 9021 2023-03-20 07:11:36 INFO i.a.c.i.LineGobbler(voidCall):114 - ----- START CHECK ----- 2023-03-20 07:11:36 INFO i.a.c.i.LineGobbler(voidCall):114 - 2023-03-20 07:11:36 INFO i.a.w.p.KubePodProcess():520 - Creating pod source-sendgrid-check-414-1-ufueb... 2023-03-20 07:11:36 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$9):589 - Creating stdout socket server... 2023-03-20 07:11:36 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$10):607 - Creating stderr socket server... 2023-03-20 07:11:36 INFO i.a.w.p.KubePodProcess(waitForInitPodToRun):309 - Waiting for init container to be ready before copying files... 2023-03-20 07:11:36 INFO i.a.w.p.KubePodProcess(waitForInitPodToRun):313 - Init container present.. 2023-03-20 07:11:38 INFO i.a.w.p.KubePodProcess(waitForInitPodToRun):316 - Init container ready.. 2023-03-20 07:11:38 INFO i.a.w.p.KubePodProcess():551 - Copying files... 2023-03-20 07:11:38 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):258 - Uploading file: source_config.json 2023-03-20 07:11:38 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):266 - kubectl cp /tmp/1263013a-dd0e-4ab8-ad9c-120543a55483/source_config.json dev/source-sendgrid-check-414-1-ufueb:/config/source_config.json -c init 2023-03-20 07:11:38 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):269 - Waiting for kubectl cp to complete 2023-03-20 07:11:38 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):283 - kubectl cp complete, closing process 2023-03-20 07:11:38 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):258 - Uploading file: FINISHED_UPLOADING 2023-03-20 07:11:38 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):266 - kubectl cp /tmp/4497b607-b8e7-4f48-9190-de6fb08158a5/FINISHED_UPLOADING dev/source-sendgrid-check-414-1-ufueb:/config/FINISHED_UPLOADING -c init 2023-03-20 07:11:38 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):269 - Waiting for kubectl cp to complete 2023-03-20 07:11:39 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):283 - kubectl cp complete, closing process 2023-03-20 07:11:39 INFO i.a.w.p.KubePodProcess():554 - Waiting until pod is ready... 2023-03-20 07:11:39 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$9):598 - Setting stdout... 2023-03-20 07:11:39 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$10):610 - Setting stderr... 2023-03-20 07:11:40 INFO i.a.w.p.KubePodProcess():570 - Reading pod IP... 2023-03-20 07:11:40 INFO i.a.w.p.KubePodProcess():572 - Pod IP: 12.0.3.52 2023-03-20 07:11:40 INFO i.a.w.p.KubePodProcess():579 - Using null stdin output stream... 2023-03-20 07:11:41 INFO i.a.w.i.DefaultAirbyteStreamFactory(internalLog):103 - Check succeeded 2023-03-20 07:11:42 INFO i.a.w.p.KubePodProcess(close):737 - (pod: dev / source-sendgrid-check-414-1-ufueb) - Closed all resources for pod 2023-03-20 07:11:42 INFO i.a.w.t.TemporalAttemptExecution(get):132 - Stopping cancellation check scheduling... 2023-03-20 07:11:42 INFO i.a.c.i.LineGobbler(voidCall):114 - 2023-03-20 07:11:42 INFO i.a.c.i.LineGobbler(voidCall):114 - ----- END CHECK ----- 2023-03-20 07:11:42 INFO i.a.c.i.LineGobbler(voidCall):114 - 2023-03-20 07:11:42 INFO i.a.w.t.TemporalAttemptExecution(get):108 - Cloud storage job log path: /workspace/414/1/logs.log 2023-03-20 07:11:42 INFO i.a.w.t.TemporalAttemptExecution(get):111 - Executing worker wrapper. Airbyte version: 0.40.6 2023-03-20 07:11:42 INFO i.a.c.i.LineGobbler(voidCall):114 - 2023-03-20 07:11:42 INFO i.a.w.p.KubeProcessFactory(create):100 - Attempting to start pod = destination-s3-check-414-1-ystvb for airbyte/destination-s3:0.3.15 with resources io.airbyte.config.ResourceRequirements@4ffb823e[cpuRequest=,cpuLimit=,memoryRequest=,memoryLimit=] 2023-03-20 07:11:42 INFO i.a.c.i.LineGobbler(voidCall):114 - ----- START CHECK ----- 2023-03-20 07:11:42 INFO i.a.w.p.KubeProcessFactory(create):103 - destination-s3-check-414-1-ystvb stdoutLocalPort = 9028 2023-03-20 07:11:42 INFO i.a.c.i.LineGobbler(voidCall):114 - 2023-03-20 07:11:42 INFO i.a.w.p.KubeProcessFactory(create):106 - destination-s3-check-414-1-ystvb stderrLocalPort = 9029 2023-03-20 07:11:42 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$9):589 - Creating stdout socket server... 2023-03-20 07:11:42 INFO i.a.w.p.KubePodProcess():520 - Creating pod destination-s3-check-414-1-ystvb... 2023-03-20 07:11:42 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$10):607 - Creating stderr socket server... 2023-03-20 07:11:42 INFO i.a.w.p.KubePodProcess(waitForInitPodToRun):309 - Waiting for init container to be ready before copying files... 2023-03-20 07:11:42 INFO i.a.w.p.KubePodProcess(waitForInitPodToRun):313 - Init container present.. 2023-03-20 07:11:44 INFO i.a.w.p.KubePodProcess(waitForInitPodToRun):316 - Init container ready.. 2023-03-20 07:11:44 INFO i.a.w.p.KubePodProcess():551 - Copying files... 2023-03-20 07:11:44 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):258 - Uploading file: source_config.json 2023-03-20 07:11:44 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):266 - kubectl cp /tmp/f7e09500-6f85-4231-b26e-2cfbe52aa582/source_config.json dev/destination-s3-check-414-1-ystvb:/config/source_config.json -c init 2023-03-20 07:11:44 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):269 - Waiting for kubectl cp to complete 2023-03-20 07:11:44 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):283 - kubectl cp complete, closing process 2023-03-20 07:11:44 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):258 - Uploading file: FINISHED_UPLOADING 2023-03-20 07:11:44 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):266 - kubectl cp /tmp/c4bf9545-1682-4f6d-9de8-046a13d4f24c/FINISHED_UPLOADING dev/destination-s3-check-414-1-ystvb:/config/FINISHED_UPLOADING -c init 2023-03-20 07:11:44 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):269 - Waiting for kubectl cp to complete 2023-03-20 07:11:44 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):283 - kubectl cp complete, closing process 2023-03-20 07:11:44 INFO i.a.w.p.KubePodProcess():554 - Waiting until pod is ready... 2023-03-20 07:11:45 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$9):598 - Setting stdout... 2023-03-20 07:11:45 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$10):610 - Setting stderr... 2023-03-20 07:11:46 INFO i.a.w.p.KubePodProcess():570 - Reading pod IP... 2023-03-20 07:11:46 INFO i.a.w.p.KubePodProcess():572 - Pod IP: 12.0.4.68 2023-03-20 07:11:46 INFO i.a.w.p.KubePodProcess():579 - Using null stdin output stream... 2023-03-20 07:11:47 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2023-03-20 07:11:47 INFO i.a.i.b.IntegrationCliParser(parseOptions):118 - integration args: {check=null, config=source_config.json} 2023-03-20 07:11:47 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2023-03-20 07:11:47 INFO i.a.i.b.IntegrationRunner(runInternal):104 - Running integration: io.airbyte.integrations.destination.s3.S3Destination 2023-03-20 07:11:47 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2023-03-20 07:11:47 INFO i.a.i.b.IntegrationRunner(runInternal):105 - Command: CHECK 2023-03-20 07:11:47 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2023-03-20 07:11:47 INFO i.a.i.b.IntegrationRunner(runInternal):106 - Integration config: IntegrationConfig{command=CHECK, configPath='source_config.json', catalogPath='null', statePath='null'} 2023-03-20 07:11:47 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2023-03-20 07:11:47 WARN c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword airbyte_secret - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword 2023-03-20 07:11:47 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2023-03-20 07:11:47 WARN c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword examples - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword 2023-03-20 07:11:47 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2023-03-20 07:11:47 WARN c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword order - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword 2023-03-20 07:11:47 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2023-03-20 07:11:47 WARN c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword requires - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword 2023-03-20 07:11:47 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2023-03-20 07:11:47 INFO i.a.i.d.s.S3FormatConfigs(getS3FormatConfig):22 - S3 format config: {"flattening":"No flattening","compression":{"compression_type":"No Compression"},"format_type":"CSV"} 2023-03-20 07:11:47 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2023-03-20 07:11:47 INFO i.a.i.d.s.S3DestinationConfig(createS3Client):190 - Creating S3 client... 2023-03-20 07:11:48 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2023-03-20 07:11:48 INFO i.a.i.d.s.S3StorageOperations(createBucketObjectIfNotExists):102 - Storage Object redshift-dev-s3-bucket/testAutomation does not exist in bucket; creating... 2023-03-20 07:11:48 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2023-03-20 07:11:48 INFO i.a.i.d.s.S3StorageOperations(createBucketObjectIfNotExists):104 - Storage Object redshift-dev-s3-bucket/testAutomation has been created in bucket. 2023-03-20 07:11:48 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2023-03-20 07:11:48 INFO i.a.i.d.s.S3Destination(testIAMUserHasListObjectPermission):156 - Started testing if IAM user can call listObjects on the destination bucket 2023-03-20 07:11:48 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2023-03-20 07:11:48 INFO i.a.i.d.s.S3Destination(testIAMUserHasListObjectPermission):159 - Finished checking for listObjects permission 2023-03-20 07:11:48 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2023-03-20 07:11:48 INFO i.a.i.d.s.S3Destination(testSingleUpload):81 - Started testing if all required credentials assigned to user for single file uploading 2023-03-20 07:11:48 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2023-03-20 07:11:48 INFO i.a.i.d.s.S3Destination(testSingleUpload):91 - Finished checking for normal upload mode 2023-03-20 07:11:48 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2023-03-20 07:11:48 INFO i.a.i.d.s.S3Destination(testMultipartUpload):95 - Started testing if all required credentials assigned to user for multipart upload 2023-03-20 07:11:48 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2023-03-20 07:11:48 INFO a.m.s.StreamTransferManager(getMultiPartOutputStreams):329 - Initiated multipart upload to redshift-dev-s3-bucket/testAutomation/test_1679296308881 with full ID 33Ofu2Y0VSbKLaqMPAtyYclaoWzO5axcDzq_3tX6BeA0j.3UWqjr99_Ln5RrWj6u_mWSUJfO7N7oV6m14Ygq6Kq0ZzRLL.dPGJ_ivUWYH5Rhy.q9shx7xrp6GqHPd7MS 2023-03-20 07:11:48 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2023-03-20 07:11:48 INFO a.m.s.MultiPartOutputStream(close):158 - Called close() on [MultipartOutputStream for parts 1 - 10000] 2023-03-20 07:11:49 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2023-03-20 07:11:48 INFO a.m.s.MultiPartOutputStream(close):158 - Called close() on [MultipartOutputStream for parts 1 - 10000] 2023-03-20 07:11:49 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2023-03-20 07:11:49 WARN a.m.s.MultiPartOutputStream(close):160 - [MultipartOutputStream for parts 1 - 10000] is already closed 2023-03-20 07:11:49 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2023-03-20 07:11:49 INFO a.m.s.StreamTransferManager(complete):367 - [Manager uploading to redshift-dev-s3-bucket/testAutomation/test_1679296308881 with id 33Ofu2Y0V...6GqHPd7MS]: Uploading leftover stream [Part number 1 containing 3.34 MB] 2023-03-20 07:11:49 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2023-03-20 07:11:49 INFO a.m.s.StreamTransferManager(uploadStreamPart):558 - [Manager uploading to redshift-dev-s3-bucket/testAutomation/test_1679296308881 with id 33Ofu2Y0V...6GqHPd7MS]: Finished uploading [Part number 1 containing 3.34 MB] 2023-03-20 07:11:49 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2023-03-20 07:11:49 INFO a.m.s.StreamTransferManager(complete):395 - [Manager uploading to redshift-dev-s3-bucket/testAutomation/test_1679296308881 with id 33Ofu2Y0V...6GqHPd7MS]: Completed 2023-03-20 07:11:49 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2023-03-20 07:11:49 INFO i.a.i.d.s.S3Destination(testMultipartUpload):119 - Finished verification for multipart upload mode 2023-03-20 07:11:49 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2023-03-20 07:11:49 INFO i.a.i.b.IntegrationRunner(runInternal):152 - Completed integration: io.airbyte.integrations.destination.s3.S3Destination 2023-03-20 07:11:49 INFO i.a.w.p.KubePodProcess(close):737 - (pod: dev / destination-s3-check-414-1-ystvb) - Closed all resources for pod 2023-03-20 07:11:49 INFO i.a.w.t.TemporalAttemptExecution(get):132 - Stopping cancellation check scheduling... 2023-03-20 07:11:49 INFO i.a.c.i.LineGobbler(voidCall):114 - 2023-03-20 07:11:49 INFO i.a.c.i.LineGobbler(voidCall):114 - ----- END CHECK ----- 2023-03-20 07:11:49 INFO i.a.c.i.LineGobbler(voidCall):114 - 2023-03-20 07:11:50 INFO i.a.w.t.TemporalAttemptExecution(get):108 - Cloud storage job log path: /workspace/414/1/logs.log 2023-03-20 07:11:50 INFO i.a.w.t.TemporalAttemptExecution(get):111 - Executing worker wrapper. Airbyte version: 0.40.6 2023-03-20 07:11:50 INFO i.a.c.f.EnvVariableFeatureFlags(getEnvOrDefault):49 - Using default value for environment variable LOG_CONNECTOR_MESSAGES: 'false' 2023-03-20 07:11:50 INFO i.a.c.EnvConfigs(getEnvOrDefault):1096 - Using default value for environment variable METRIC_CLIENT: '' 2023-03-20 07:11:50 WARN i.a.m.l.MetricClientFactory(initialize):60 - Metric client is already initialized to 2023-03-20 07:11:50 INFO i.a.c.f.EnvVariableFeatureFlags(getEnvOrDefault):49 - Using default value for environment variable LOG_CONNECTOR_MESSAGES: 'false' 2023-03-20 07:11:50 INFO i.a.w.g.DefaultReplicationWorker(run):122 - start sync worker. job id: 414 attempt id: 1 2023-03-20 07:11:50 INFO i.a.c.i.LineGobbler(voidCall):114 - 2023-03-20 07:11:50 INFO i.a.w.g.DefaultReplicationWorker(run):135 - configured sync modes: {null.single_sends=full_refresh - overwrite, null.spam_reports=full_refresh - overwrite, null.messages=full_refresh - overwrite, null.segments=full_refresh - overwrite, null.suppression_groups=full_refresh - overwrite, null.invalid_emails=full_refresh - overwrite, null.suppression_group_members=full_refresh - overwrite, null.contacts=full_refresh - overwrite, null.bounces=full_refresh - overwrite, null.stats_automations=full_refresh - overwrite, null.campaigns=full_refresh - overwrite, null.lists=full_refresh - overwrite, null.global_suppressions=full_refresh - overwrite, null.templates=full_refresh - overwrite, null.blocks=full_refresh - overwrite} 2023-03-20 07:11:50 INFO i.a.w.i.DefaultAirbyteDestination(start):69 - Running destination... 2023-03-20 07:11:50 INFO i.a.w.p.KubeProcessFactory(create):100 - Attempting to start pod = destination-s3-write-414-1-evwkr for airbyte/destination-s3:0.3.15 with resources io.airbyte.config.ResourceRequirements@27699cc7[cpuRequest=,cpuLimit=,memoryRequest=,memoryLimit=] 2023-03-20 07:11:50 INFO i.a.w.p.KubeProcessFactory(create):103 - destination-s3-write-414-1-evwkr stdoutLocalPort = 9030 2023-03-20 07:11:50 INFO i.a.w.p.KubeProcessFactory(create):106 - destination-s3-write-414-1-evwkr stderrLocalPort = 9031 2023-03-20 07:11:50 INFO i.a.c.i.LineGobbler(voidCall):114 - ----- START REPLICATION ----- 2023-03-20 07:11:50 INFO i.a.c.i.LineGobbler(voidCall):114 - 2023-03-20 07:11:50 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$9):589 - Creating stdout socket server... 2023-03-20 07:11:50 INFO i.a.w.p.KubePodProcess():520 - Creating pod destination-s3-write-414-1-evwkr... 2023-03-20 07:11:50 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$10):607 - Creating stderr socket server... 2023-03-20 07:11:50 INFO i.a.w.p.KubePodProcess(waitForInitPodToRun):309 - Waiting for init container to be ready before copying files... 2023-03-20 07:11:50 INFO i.a.w.p.KubePodProcess(waitForInitPodToRun):313 - Init container present.. 2023-03-20 07:11:52 INFO i.a.w.p.KubePodProcess(waitForInitPodToRun):316 - Init container ready.. 2023-03-20 07:11:52 INFO i.a.w.p.KubePodProcess():551 - Copying files... 2023-03-20 07:11:52 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):258 - Uploading file: destination_config.json 2023-03-20 07:11:52 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):266 - kubectl cp /tmp/65b89dcc-97c1-45ea-976c-7ac82d6a5261/destination_config.json dev/destination-s3-write-414-1-evwkr:/config/destination_config.json -c init 2023-03-20 07:11:52 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):269 - Waiting for kubectl cp to complete 2023-03-20 07:11:52 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):283 - kubectl cp complete, closing process 2023-03-20 07:11:52 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):258 - Uploading file: destination_catalog.json 2023-03-20 07:11:52 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):266 - kubectl cp /tmp/84ccfb38-bdea-482f-b9b4-a619f526fbb9/destination_catalog.json dev/destination-s3-write-414-1-evwkr:/config/destination_catalog.json -c init 2023-03-20 07:11:52 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):269 - Waiting for kubectl cp to complete 2023-03-20 07:11:52 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):283 - kubectl cp complete, closing process 2023-03-20 07:11:52 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):258 - Uploading file: FINISHED_UPLOADING 2023-03-20 07:11:52 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):266 - kubectl cp /tmp/5ad4c221-729d-4008-b245-f552f7bbd889/FINISHED_UPLOADING dev/destination-s3-write-414-1-evwkr:/config/FINISHED_UPLOADING -c init 2023-03-20 07:11:52 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):269 - Waiting for kubectl cp to complete 2023-03-20 07:11:53 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):283 - kubectl cp complete, closing process 2023-03-20 07:11:53 INFO i.a.w.p.KubePodProcess():554 - Waiting until pod is ready... 2023-03-20 07:11:53 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$9):598 - Setting stdout... 2023-03-20 07:11:53 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$10):610 - Setting stderr... 2023-03-20 07:11:54 INFO i.a.w.p.KubePodProcess():570 - Reading pod IP... 2023-03-20 07:11:54 INFO i.a.w.p.KubePodProcess():572 - Pod IP: 12.0.3.114 2023-03-20 07:11:54 INFO i.a.w.p.KubePodProcess():575 - Creating stdin socket... 2023-03-20 07:11:54 INFO i.a.w.p.KubeProcessFactory(create):100 - Attempting to start pod = source-sendgrid-read-414-1-nvibe for airbyte/source-sendgrid:0.2.14 with resources io.airbyte.config.ResourceRequirements@7f92ad3b[cpuRequest=,cpuLimit=,memoryRequest=,memoryLimit=] 2023-03-20 07:11:54 INFO i.a.w.p.KubeProcessFactory(create):103 - source-sendgrid-read-414-1-nvibe stdoutLocalPort = 9024 2023-03-20 07:11:54 INFO i.a.w.p.KubeProcessFactory(create):106 - source-sendgrid-read-414-1-nvibe stderrLocalPort = 9025 2023-03-20 07:11:54 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$9):589 - Creating stdout socket server... 2023-03-20 07:11:54 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$10):607 - Creating stderr socket server... 2023-03-20 07:11:54 INFO i.a.w.p.KubePodProcess():520 - Creating pod source-sendgrid-read-414-1-nvibe... 2023-03-20 07:11:54 INFO i.a.w.p.KubePodProcess(waitForInitPodToRun):309 - Waiting for init container to be ready before copying files... 2023-03-20 07:11:54 INFO i.a.w.p.KubePodProcess(waitForInitPodToRun):313 - Init container present.. 2023-03-20 07:11:56 INFO i.a.w.p.KubePodProcess(waitForInitPodToRun):316 - Init container ready.. 2023-03-20 07:11:56 INFO i.a.w.p.KubePodProcess():551 - Copying files... 2023-03-20 07:11:56 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):258 - Uploading file: source_config.json 2023-03-20 07:11:56 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):266 - kubectl cp /tmp/e0d55ac5-f8f5-49c9-aaa8-5898f949823e/source_config.json dev/source-sendgrid-read-414-1-nvibe:/config/source_config.json -c init 2023-03-20 07:11:56 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):269 - Waiting for kubectl cp to complete 2023-03-20 07:11:56 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):283 - kubectl cp complete, closing process 2023-03-20 07:11:56 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):258 - Uploading file: source_catalog.json 2023-03-20 07:11:56 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):266 - kubectl cp /tmp/da3088e5-5dd1-4866-b6ea-82182e2e3cf1/source_catalog.json dev/source-sendgrid-read-414-1-nvibe:/config/source_catalog.json -c init 2023-03-20 07:11:56 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):269 - Waiting for kubectl cp to complete 2023-03-20 07:11:57 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):283 - kubectl cp complete, closing process 2023-03-20 07:11:57 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):258 - Uploading file: FINISHED_UPLOADING 2023-03-20 07:11:57 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):266 - kubectl cp /tmp/6c92d7c1-e84c-41f0-bae8-ce20e7499b48/FINISHED_UPLOADING dev/source-sendgrid-read-414-1-nvibe:/config/FINISHED_UPLOADING -c init 2023-03-20 07:11:57 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):269 - Waiting for kubectl cp to complete 2023-03-20 07:11:57 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):283 - kubectl cp complete, closing process 2023-03-20 07:11:57 INFO i.a.w.p.KubePodProcess():554 - Waiting until pod is ready... 2023-03-20 07:11:57 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$9):598 - Setting stdout... 2023-03-20 07:11:57 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$10):610 - Setting stderr... 2023-03-20 07:11:58 INFO i.a.w.p.KubePodProcess():570 - Reading pod IP... 2023-03-20 07:11:58 INFO i.a.w.p.KubePodProcess():572 - Pod IP: 12.0.4.70 2023-03-20 07:11:58 INFO i.a.w.p.KubePodProcess():579 - Using null stdin output stream... 2023-03-20 07:11:58 INFO i.a.w.g.DefaultReplicationWorker(run):177 - Waiting for source and destination threads to complete. 2023-03-20 07:11:58 INFO i.a.w.g.DefaultReplicationWorker(lambda$getDestinationOutputRunnable$7):435 - Destination output thread started. 2023-03-20 07:11:58 INFO i.a.w.g.DefaultReplicationWorker(lambda$getReplicationRunnable$6):322 - Replication thread started. 2023-03-20 07:12:02 INFO i.a.w.g.DefaultReplicationWorker(lambda$getReplicationRunnable$6):355 - Source has no more messages, closing connection. 2023-03-20 07:12:03 INFO i.a.w.p.KubePodProcess(close):737 - (pod: dev / source-sendgrid-read-414-1-nvibe) - Closed all resources for pod 2023-03-20 07:12:05 INFO i.a.w.p.KubePodProcess(close):737 - (pod: dev / destination-s3-write-414-1-evwkr) - Closed all resources for pod 2023-03-20 07:12:05 ERROR i.a.w.g.DefaultReplicationWorker(run):188 - Sync worker failed. java.util.concurrent.ExecutionException: io.airbyte.workers.general.DefaultReplicationWorker$SourceException: Source cannot be stopped! at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) ~[?:?] at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) ~[?:?] at io.airbyte.workers.general.DefaultReplicationWorker.run(DefaultReplicationWorker.java:181) ~[io.airbyte-airbyte-workers-0.40.6.jar:?] at io.airbyte.workers.general.DefaultReplicationWorker.run(DefaultReplicationWorker.java:68) ~[io.airbyte-airbyte-workers-0.40.6.jar:?] at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:159) ~[io.airbyte-airbyte-workers-0.40.6.jar:?] at java.lang.Thread.run(Thread.java:1589) [?:?] Suppressed: io.airbyte.workers.exception.WorkerException: Source process exit with code 1. This warning is normal if the job was cancelled. at io.airbyte.workers.internal.DefaultAirbyteSource.close(DefaultAirbyteSource.java:140) ~[io.airbyte-airbyte-workers-0.40.6.jar:?] at io.airbyte.workers.general.DefaultReplicationWorker.run(DefaultReplicationWorker.java:145) ~[io.airbyte-airbyte-workers-0.40.6.jar:?] at io.airbyte.workers.general.DefaultReplicationWorker.run(DefaultReplicationWorker.java:68) ~[io.airbyte-airbyte-workers-0.40.6.jar:?] at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:159) ~[io.airbyte-airbyte-workers-0.40.6.jar:?] at java.lang.Thread.run(Thread.java:1589) [?:?] Caused by: io.airbyte.workers.general.DefaultReplicationWorker$SourceException: Source cannot be stopped! at io.airbyte.workers.general.DefaultReplicationWorker.lambda$getReplicationRunnable$6(DefaultReplicationWorker.java:359) ~[io.airbyte-airbyte-workers-0.40.6.jar:?] at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804) ~[?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] ... 1 more Caused by: io.airbyte.workers.exception.WorkerException: Source process exit with code 1. This warning is normal if the job was cancelled. at io.airbyte.workers.internal.DefaultAirbyteSource.close(DefaultAirbyteSource.java:140) ~[io.airbyte-airbyte-workers-0.40.6.jar:?] at io.airbyte.workers.general.DefaultReplicationWorker.lambda$getReplicationRunnable$6(DefaultReplicationWorker.java:357) ~[io.airbyte-airbyte-workers-0.40.6.jar:?] at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804) ~[?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] ... 1 more 2023-03-20 07:12:05 INFO i.a.w.g.DefaultReplicationWorker(run):281 - Source did not output any state messages 2023-03-20 07:12:05 WARN i.a.w.g.DefaultReplicationWorker(run):292 - State capture: No state retained. 2023-03-20 07:12:05 INFO i.a.w.g.DefaultReplicationWorker(run):300 - sync summary: { "status" : "failed", "recordsSynced" : 14, "bytesSynced" : 4037, "startTime" : 1679296310323, "endTime" : 1679296325436, "totalStats" : { "recordsEmitted" : 14, "bytesEmitted" : 4037, "sourceStateMessagesEmitted" : 0, "destinationStateMessagesEmitted" : 0, "recordsCommitted" : 0, "meanSecondsBeforeSourceStateMessageEmitted" : 0, "maxSecondsBeforeSourceStateMessageEmitted" : 0, "maxSecondsBetweenStateMessageEmittedandCommitted" : 0, "meanSecondsBetweenStateMessageEmittedandCommitted" : 0 }, "streamStats" : [ { "streamName" : "global_suppressions", "stats" : { "recordsEmitted" : 3, "bytesEmitted" : 151 } }, { "streamName" : "campaigns", "stats" : { "recordsEmitted" : 5, "bytesEmitted" : 1043 } }, { "streamName" : "lists", "stats" : { "recordsEmitted" : 1, "bytesEmitted" : 187 } }, { "streamName" : "contacts", "stats" : { "recordsEmitted" : 5, "bytesEmitted" : 2656 } } ] } 2023-03-20 07:12:05 INFO i.a.w.g.DefaultReplicationWorker(run):301 - failures: [ { "failureOrigin" : "source", "failureType" : "system_error", "internalMessage" : "400 Client Error: Bad Request for url: https://api.sendgrid.com/v3/messages?query=last_event_time%20BETWEEN%20TIMESTAMP%20%22None%22%20AND%20TIMESTAMP%20%222023-03-20T07%3A12%3A02Z%22&limit=1000", "externalMessage" : "authorization required", "metadata" : { "attemptNumber" : 1, "jobId" : 414, "from_trace_message" : true, "connector_command" : "read" }, "stacktrace" : "Traceback (most recent call last):\n File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py\", line 115, in read\n yield from self._read_stream(\n File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py\", line 184, in _read_stream\n for record in record_iterator:\n File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py\", line 282, in _read_full_refresh\n for record in records:\n File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py\", line 421, in read_records\n response = self._send_request(request, request_kwargs)\n File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py\", line 339, in _send_request\n return backoff_handler(user_backoff_handler)(request, request_kwargs)\n File \"/usr/local/lib/python3.9/site-packages/backoff/_sync.py\", line 105, in retry\n ret = target(*args, **kwargs)\n File \"/usr/local/lib/python3.9/site-packages/backoff/_sync.py\", line 105, in retry\n ret = target(*args, **kwargs)\n File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py\", line 306, in _send\n raise exc\n File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py\", line 303, in _send\n response.raise_for_status()\n File \"/usr/local/lib/python3.9/site-packages/requests/models.py\", line 1021, in raise_for_status\n raise HTTPError(http_error_msg, response=self)\nrequests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://api.sendgrid.com/v3/messages?query=last_event_time%20BETWEEN%20TIMESTAMP%20%22None%22%20AND%20TIMESTAMP%20%222023-03-20T07%3A12%3A02Z%22&limit=1000\n", "timestamp" : 1679296322661 }, { "failureOrigin" : "source", "internalMessage" : "io.airbyte.workers.general.DefaultReplicationWorker$SourceException: Source cannot be stopped!", "externalMessage" : "Something went wrong within the source connector", "metadata" : { "attemptNumber" : 1, "jobId" : 414, "connector_command" : "read" }, "stacktrace" : "java.util.concurrent.CompletionException: io.airbyte.workers.general.DefaultReplicationWorker$SourceException: Source cannot be stopped!\n\tat java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315)\n\tat java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:320)\n\tat java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1807)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1589)\nCaused by: io.airbyte.workers.general.DefaultReplicationWorker$SourceException: Source cannot be stopped!\n\tat io.airbyte.workers.general.DefaultReplicationWorker.lambda$getReplicationRunnable$6(DefaultReplicationWorker.java:359)\n\tat java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)\n\t... 3 more\nCaused by: io.airbyte.workers.exception.WorkerException: Source process exit with code 1. This warning is normal if the job was cancelled.\n\tat io.airbyte.workers.internal.DefaultAirbyteSource.close(DefaultAirbyteSource.java:140)\n\tat io.airbyte.workers.general.DefaultReplicationWorker.lambda$getReplicationRunnable$6(DefaultReplicationWorker.java:357)\n\t... 4 more\n", "timestamp" : 1679296323032 } ] 2023-03-20 07:12:05 INFO i.a.c.i.LineGobbler(voidCall):114 - 2023-03-20 07:12:05 INFO i.a.c.i.LineGobbler(voidCall):114 - ----- END REPLICATION ----- 2023-03-20 07:12:05 INFO i.a.c.i.LineGobbler(voidCall):114 - 2023-03-20 07:12:05 INFO i.a.w.t.TemporalAttemptExecution(get):132 - Stopping cancellation check scheduling... 2023-03-20 07:12:05 INFO i.a.w.t.s.ReplicationActivityImpl(lambda$replicate$3):166 - sync summary: io.airbyte.config.StandardSyncOutput@45c7458[standardSyncSummary=io.airbyte.config.StandardSyncSummary@4f5ba287[status=failed,recordsSynced=14,bytesSynced=4037,startTime=1679296310323,endTime=1679296325436,totalStats=io.airbyte.config.SyncStats@7e11a0c6[recordsEmitted=14,bytesEmitted=4037,sourceStateMessagesEmitted=0,destinationStateMessagesEmitted=0,recordsCommitted=0,meanSecondsBeforeSourceStateMessageEmitted=0,maxSecondsBeforeSourceStateMessageEmitted=0,maxSecondsBetweenStateMessageEmittedandCommitted=0,meanSecondsBetweenStateMessageEmittedandCommitted=0,additionalProperties={}],streamStats=[io.airbyte.config.StreamSyncStats@2fd46143[streamName=global_suppressions,stats=io.airbyte.config.SyncStats@48cd3e55[recordsEmitted=3,bytesEmitted=151,sourceStateMessagesEmitted=,destinationStateMessagesEmitted=,recordsCommitted=,meanSecondsBeforeSourceStateMessageEmitted=,maxSecondsBeforeSourceStateMessageEmitted=,maxSecondsBetweenStateMessageEmittedandCommitted=,meanSecondsBetweenStateMessageEmittedandCommitted=,additionalProperties={}]], io.airbyte.config.StreamSyncStats@5a90e251[streamName=campaigns,stats=io.airbyte.config.SyncStats@29f46e55[recordsEmitted=5,bytesEmitted=1043,sourceStateMessagesEmitted=,destinationStateMessagesEmitted=,recordsCommitted=,meanSecondsBeforeSourceStateMessageEmitted=,maxSecondsBeforeSourceStateMessageEmitted=,maxSecondsBetweenStateMessageEmittedandCommitted=,meanSecondsBetweenStateMessageEmittedandCommitted=,additionalProperties={}]], io.airbyte.config.StreamSyncStats@2526cc51[streamName=lists,stats=io.airbyte.config.SyncStats@4386136[recordsEmitted=1,bytesEmitted=187,sourceStateMessagesEmitted=,destinationStateMessagesEmitted=,recordsCommitted=,meanSecondsBeforeSourceStateMessageEmitted=,maxSecondsBeforeSourceStateMessageEmitted=,maxSecondsBetweenStateMessageEmittedandCommitted=,meanSecondsBetweenStateMessageEmittedandCommitted=,additionalProperties={}]], io.airbyte.config.StreamSyncStats@5967898a[streamName=contacts,stats=io.airbyte.config.SyncStats@536bd235[recordsEmitted=5,bytesEmitted=2656,sourceStateMessagesEmitted=,destinationStateMessagesEmitted=,recordsCommitted=,meanSecondsBeforeSourceStateMessageEmitted=,maxSecondsBeforeSourceStateMessageEmitted=,maxSecondsBetweenStateMessageEmittedandCommitted=,meanSecondsBetweenStateMessageEmittedandCommitted=,additionalProperties={}]]]],normalizationSummary=,state=,outputCatalog=io.airbyte.protocol.models.ConfiguredAirbyteCatalog@403cd84d[streams=[io.airbyte.protocol.models.ConfiguredAirbyteStream@7ca30ad[stream=io.airbyte.protocol.models.AirbyteStream@6204d207[name=blocks,jsonSchema={"type":"object","$schema":"http://json-schema.org/draft-07/schema#","properties":{"email":{"type":"string"},"reason":{"type":"string"},"status":{"type":"string"},"created":{"type":"integer"}}},supportedSyncModes=[full_refresh, incremental],sourceDefinedCursor=true,defaultCursorField=[created],sourceDefinedPrimaryKey=[[email]],namespace=,additionalProperties={}],syncMode=full_refresh,cursorField=[created],destinationSyncMode=overwrite,primaryKey=[[email]],additionalProperties={}], io.airbyte.protocol.models.ConfiguredAirbyteStream@6ff35e08[stream=io.airbyte.protocol.models.AirbyteStream@49ab1d0b[name=bounces,jsonSchema={"type":"object","$schema":"http://json-schema.org/draft-07/schema#","properties":{"email":{"type":"string"},"reason":{"type":"string"},"status":{"type":"string"},"created":{"type":"integer"}}},supportedSyncModes=[full_refresh, incremental],sourceDefinedCursor=true,defaultCursorField=[created],sourceDefinedPrimaryKey=[[email]],namespace=,additionalProperties={}],syncMode=full_refresh,cursorField=[created],destinationSyncMode=overwrite,primaryKey=[[email]],additionalProperties={}], io.airbyte.protocol.models.ConfiguredAirbyteStream@1f3213cb[stream=io.airbyte.protocol.models.AirbyteStream@5be26ec0[name=campaigns,jsonSchema={"type":"object","$schema":"http://json-schema.org/draft-07/schema#","properties":{"id":{"type":"string"},"name":{"type":"string"},"status":{"type":"string"},"is_abtest":{"type":"boolean"},"created_at":{"type":"string"},"updated_at":{"type":"string"}}},supportedSyncModes=[full_refresh],sourceDefinedCursor=,defaultCursorField=[],sourceDefinedPrimaryKey=[[id]],namespace=,additionalProperties={}],syncMode=full_refresh,cursorField=[],destinationSyncMode=overwrite,primaryKey=[[id]],additionalProperties={}], io.airbyte.protocol.models.ConfiguredAirbyteStream@2e1478aa[stream=io.airbyte.protocol.models.AirbyteStream@3fc66b26[name=contacts,jsonSchema={"type":"object","$schema":"http://json-schema.org/draft-07/schema#","properties":{"id":{"type":"string"},"city":{"type":"string"},"line":{"type":"string"},"email":{"type":"string"},"country":{"type":"string"},"facebook":{"type":"string"},"list_ids":{"type":["null","array"]},"whatsapp":{"type":"string"},"_metadata":{"type":"object","properties":{"self":{"type":"string"}}},"last_name":{"type":"string"},"created_at":{"type":"string"},"first_name":{"type":"string"},"updated_at":{"type":"string"},"postal_code":{"type":"string"},"unique_name":{"type":"string"},"phone_number":{"type":"string"},"custom_fields":{"type":"object"},"address_line_1":{"type":"string"},"address_line_2":{"type":"string"},"alternate_emails":{"type":["null","array"]},"state_province_region":{"type":"string"}}},supportedSyncModes=[full_refresh],sourceDefinedCursor=,defaultCursorField=[],sourceDefinedPrimaryKey=[[id]],namespace=,additionalProperties={}],syncMode=full_refresh,cursorField=[],destinationSyncMode=overwrite,primaryKey=[[id]],additionalProperties={}], io.airbyte.protocol.models.ConfiguredAirbyteStream@30e58b5e[stream=io.airbyte.protocol.models.AirbyteStream@783967ec[name=global_suppressions,jsonSchema={"type":"object","$schema":"http://json-schema.org/draft-07/schema#","properties":{"email":{"type":"string"},"created":{"type":"integer"}}},supportedSyncModes=[full_refresh, incremental],sourceDefinedCursor=true,defaultCursorField=[created],sourceDefinedPrimaryKey=[[email]],namespace=,additionalProperties={}],syncMode=full_refresh,cursorField=[created],destinationSyncMode=overwrite,primaryKey=[[email]],additionalProperties={}], io.airbyte.protocol.models.ConfiguredAirbyteStream@75e58169[stream=io.airbyte.protocol.models.AirbyteStream@79765ba8[name=invalid_emails,jsonSchema={"type":"object","$schema":"http://json-schema.org/draft-07/schema#","properties":{"email":{"type":"string"},"reason":{"type":"string"},"created":{"type":"integer"}}},supportedSyncModes=[full_refresh, incremental],sourceDefinedCursor=true,defaultCursorField=[created],sourceDefinedPrimaryKey=[[email]],namespace=,additionalProperties={}],syncMode=full_refresh,cursorField=[created],destinationSyncMode=overwrite,primaryKey=[[email]],additionalProperties={}], io.airbyte.protocol.models.ConfiguredAirbyteStream@655580a3[stream=io.airbyte.protocol.models.AirbyteStream@56608de2[name=lists,jsonSchema={"type":"object","$schema":"http://json-schema.org/draft-07/schema#","properties":{"id":{"type":"string"},"name":{"type":"string"},"_metadata":{"type":"object","properties":{"self":{"type":"string"}}},"contact_count":{"type":"integer"}}},supportedSyncModes=[full_refresh],sourceDefinedCursor=,defaultCursorField=[],sourceDefinedPrimaryKey=[[id]],namespace=,additionalProperties={}],syncMode=full_refresh,cursorField=[],destinationSyncMode=overwrite,primaryKey=[[id]],additionalProperties={}], io.airbyte.protocol.models.ConfiguredAirbyteStream@f57a08e[stream=io.airbyte.protocol.models.AirbyteStream@44b701d7[name=messages,jsonSchema={"type":"object","$schema":"http://json-schema.org/draft-07/schema#","properties":{"msg_id":{"type":"string"},"status":{"type":"string"},"subject":{"type":"string"},"to_email":{"type":"string"},"from_email":{"type":"string"},"opens_count":{"type":"integer"},"clicks_count":{"type":"integer"},"last_event_time":{"type":"string"}}},supportedSyncModes=[full_refresh, incremental],sourceDefinedCursor=true,defaultCursorField=[last_event_time],sourceDefinedPrimaryKey=[[msg_id]],namespace=,additionalProperties={}],syncMode=full_refresh,cursorField=[last_event_time],destinationSyncMode=overwrite,primaryKey=[[msg_id]],additionalProperties={}], io.airbyte.protocol.models.ConfiguredAirbyteStream@d2ed17a[stream=io.airbyte.protocol.models.AirbyteStream@5f7342a0[name=segments,jsonSchema={"type":"object","$schema":"http://json-schema.org/draft-07/schema#","properties":{"id":{"type":"string"},"name":{"type":"string"},"created_at":{"type":"string"},"updated_at":{"type":"string"},"contact_count":{"type":"integer"},"parent_list_id":{"type":["string","null"]},"sample_updated_at":{"type":"string"},"next_sample_update":{"type":"string"}}},supportedSyncModes=[full_refresh],sourceDefinedCursor=,defaultCursorField=[],sourceDefinedPrimaryKey=[[id]],namespace=,additionalProperties={}],syncMode=full_refresh,cursorField=[],destinationSyncMode=overwrite,primaryKey=[[id]],additionalProperties={}], io.airbyte.protocol.models.ConfiguredAirbyteStream@6d12ca10[stream=io.airbyte.protocol.models.AirbyteStream@11d7c7dc[name=single_sends,jsonSchema={"type":"object","$schema":"http://json-schema.org/draft-07/schema#","properties":{"id":{"type":"string","format":"uuid"},"stats":{"type":"object","properties":{"opens":{"type":"integer"},"clicks":{"type":"integer"},"bounces":{"type":"integer"},"requests":{"type":"integer"},"delivered":{"type":"integer"},"bounce_drops":{"type":"integer"},"spam_reports":{"type":"integer"},"unique_opens":{"type":"integer"},"unsubscribes":{"type":"integer"},"unique_clicks":{"type":"integer"},"invalid_emails":{"type":"integer"},"spam_report_drops":{"type":"integer"}}},"ab_phase":{"type":"string"},"aggregation":{"type":"string"},"ab_variation":{"type":"string"}}},supportedSyncModes=[full_refresh],sourceDefinedCursor=,defaultCursorField=[],sourceDefinedPrimaryKey=[[id]],namespace=,additionalProperties={}],syncMode=full_refresh,cursorField=[],destinationSyncMode=overwrite,primaryKey=[[id]],additionalProperties={}], io.airbyte.protocol.models.ConfiguredAirbyteStream@6a09619d[stream=io.airbyte.protocol.models.AirbyteStream@2faa483e[name=spam_reports,jsonSchema={"type":"object","$schema":"http://json-schema.org/draft-07/schema#","properties":{"ip":{"type":"string"},"email":{"type":"string"},"created":{"type":"integer"}}},supportedSyncModes=[full_refresh, incremental],sourceDefinedCursor=true,defaultCursorField=[created],sourceDefinedPrimaryKey=[[email]],namespace=,additionalProperties={}],syncMode=full_refresh,cursorField=[created],destinationSyncMode=overwrite,primaryKey=[[email]],additionalProperties={}], io.airbyte.protocol.models.ConfiguredAirbyteStream@54135591[stream=io.airbyte.protocol.models.AirbyteStream@7b5591e8[name=stats_automations,jsonSchema={"type":"object","$schema":"http://json-schema.org/draft-07/schema#","properties":{"id":{"type":"string"},"stats":{"type":"object","properties":{"opens":{"type":"integer"},"clicks":{"type":"integer"},"bounces":{"type":"integer"},"requests":{"type":"integer"},"delivered":{"type":"integer"},"bounce_drops":{"type":"integer"},"spam_reports":{"type":"integer"},"unique_opens":{"type":"integer"},"unsubscribes":{"type":"integer"},"unique_clicks":{"type":"integer"},"invalid_emails":{"type":"integer"},"spam_report_drops":{"type":"integer"}}},"step_id":{"type":"string"},"aggregation":{"type":"string"}}},supportedSyncModes=[full_refresh],sourceDefinedCursor=,defaultCursorField=[],sourceDefinedPrimaryKey=[[id]],namespace=,additionalProperties={}],syncMode=full_refresh,cursorField=[],destinationSyncMode=overwrite,primaryKey=[[id]],additionalProperties={}], io.airbyte.protocol.models.ConfiguredAirbyteStream@5a710296[stream=io.airbyte.protocol.models.AirbyteStream@27674a5e[name=suppression_group_members,jsonSchema={"type":"object","$schema":"http://json-schema.org/draft-07/schema#","properties":{"email":{"type":"string"},"group_id":{"type":"integer"},"created_at":{"type":"integer"},"group_name":{"type":"string"}}},supportedSyncModes=[full_refresh],sourceDefinedCursor=,defaultCursorField=[],sourceDefinedPrimaryKey=[[group_id]],namespace=,additionalProperties={}],syncMode=full_refresh,cursorField=[],destinationSyncMode=overwrite,primaryKey=[[group_id]],additionalProperties={}], io.airbyte.protocol.models.ConfiguredAirbyteStream@69162cfe[stream=io.airbyte.protocol.models.AirbyteStream@3e9b3ae5[name=suppression_groups,jsonSchema={"type":"object","$schema":"http://json-schema.org/draft-07/schema#","properties":{"id":{"type":"integer"},"name":{"type":"string"},"is_default":{"type":"boolean"},"description":{"type":"string"},"unsubscribes":{"type":"integer"}}},supportedSyncModes=[full_refresh],sourceDefinedCursor=,defaultCursorField=[],sourceDefinedPrimaryKey=[[id]],namespace=,additionalProperties={}],syncMode=full_refresh,cursorField=[],destinationSyncMode=overwrite,primaryKey=[[id]],additionalProperties={}], io.airbyte.protocol.models.ConfiguredAirbyteStream@6cacfa12[stream=io.airbyte.protocol.models.AirbyteStream@71846863[name=templates,jsonSchema={"type":"object","$schema":"http://json-schema.org/draft-07/schema#","properties":{"id":{"type":["string"]},"name":{"type":["null","string"]},"versions":{"type":["array"]},"generation":{"type":["null","string"]},"updated_at":{"type":["null","string"]}}},supportedSyncModes=[full_refresh],sourceDefinedCursor=,defaultCursorField=[],sourceDefinedPrimaryKey=[[id]],namespace=,additionalProperties={}],syncMode=full_refresh,cursorField=[],destinationSyncMode=overwrite,primaryKey=[[id]],additionalProperties={}]],additionalProperties={}],failures=[io.airbyte.config.FailureReason@29addb71[failureOrigin=source,failureType=system_error,internalMessage=400 Client Error: Bad Request for url: https://api.sendgrid.com/v3/messages?query=last_event_time%20BETWEEN%20TIMESTAMP%20%22None%22%20AND%20TIMESTAMP%20%222023-03-20T07%3A12%3A02Z%22&limit=1000,externalMessage=authorization required,metadata=io.airbyte.config.Metadata@4e8690b3[additionalProperties={attemptNumber=1, jobId=414, from_trace_message=true, connector_command=read}],stacktrace=Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 115, in read yield from self._read_stream( File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 184, in _read_stream for record in record_iterator: File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 282, in _read_full_refresh for record in records: File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py", line 421, in read_records response = self._send_request(request, request_kwargs) File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py", line 339, in _send_request return backoff_handler(user_backoff_handler)(request, request_kwargs) File "/usr/local/lib/python3.9/site-packages/backoff/_sync.py", line 105, in retry ret = target(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/backoff/_sync.py", line 105, in retry ret = target(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py", line 306, in _send raise exc File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py", line 303, in _send response.raise_for_status() File "/usr/local/lib/python3.9/site-packages/requests/models.py", line 1021, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://api.sendgrid.com/v3/messages?query=last_event_time%20BETWEEN%20TIMESTAMP%20%22None%22%20AND%20TIMESTAMP%20%222023-03-20T07%3A12%3A02Z%22&limit=1000 ,retryable=,timestamp=1679296322661], io.airbyte.config.FailureReason@45719f1b[failureOrigin=source,failureType=,internalMessage=io.airbyte.workers.general.DefaultReplicationWorker$SourceException: Source cannot be stopped!,externalMessage=Something went wrong within the source connector,metadata=io.airbyte.config.Metadata@176420a6[additionalProperties={attemptNumber=1, jobId=414, connector_command=read}],stacktrace=java.util.concurrent.CompletionException: io.airbyte.workers.general.DefaultReplicationWorker$SourceException: Source cannot be stopped! at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315) at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:320) at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1807) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) at java.base/java.lang.Thread.run(Thread.java:1589) Caused by: io.airbyte.workers.general.DefaultReplicationWorker$SourceException: Source cannot be stopped! at io.airbyte.workers.general.DefaultReplicationWorker.lambda$getReplicationRunnable$6(DefaultReplicationWorker.java:359) at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804) ... 3 more Caused by: io.airbyte.workers.exception.WorkerException: Source process exit with code 1. This warning is normal if the job was cancelled. at io.airbyte.workers.internal.DefaultAirbyteSource.close(DefaultAirbyteSource.java:140) at io.airbyte.workers.general.DefaultReplicationWorker.lambda$getReplicationRunnable$6(DefaultReplicationWorker.java:357) ... 4 more ,retryable=,timestamp=1679296323032]]] 2023-03-20 07:12:05 INFO i.a.w.t.TemporalUtils(withBackgroundHeartbeat):312 - Stopping temporal heartbeating... 2023-03-20 07:11:58 destination > 2023-03-20 07:11:55 INFO i.a.i.d.s.S3DestinationConfig(createS3Client):190 - Creating S3 client... 2023-03-20 07:11:58 destination > 2023-03-20 07:11:56 INFO i.a.i.d.s.SerializedBufferFactory(getCreateFunction):49 - S3 format config: S3CsvFormatConfig{flattening=NO, compression=NO_COMPRESSION} 2023-03-20 07:11:58 destination > 2023-03-20 07:11:56 INFO i.a.i.d.s.S3ConsumerFactory(lambda$toWriteConfig$0):83 - Write config: WriteConfig{streamName=blocks, namespace=null, outputBucketPath=testAutomation, pathFormat=testAutomation/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_, fullOutputPath=testAutomation/blocks/2023_03_20_1679296315680_, syncMode=overwrite} 2023-03-20 07:11:58 destination > 2023-03-20 07:11:56 INFO i.a.i.d.s.S3ConsumerFactory(lambda$toWriteConfig$0):83 - Write config: WriteConfig{streamName=bounces, namespace=null, outputBucketPath=testAutomation, pathFormat=testAutomation/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_, fullOutputPath=testAutomation/bounces/2023_03_20_1679296315680_, syncMode=overwrite} 2023-03-20 07:11:58 destination > 2023-03-20 07:11:56 INFO i.a.i.d.s.S3ConsumerFactory(lambda$toWriteConfig$0):83 - Write config: WriteConfig{streamName=campaigns, namespace=null, outputBucketPath=testAutomation, pathFormat=testAutomation/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_, fullOutputPath=testAutomation/campaigns/2023_03_20_1679296315680_, syncMode=overwrite} 2023-03-20 07:11:58 destination > 2023-03-20 07:11:56 INFO i.a.i.d.s.S3ConsumerFactory(lambda$toWriteConfig$0):83 - Write config: WriteConfig{streamName=contacts, namespace=null, outputBucketPath=testAutomation, pathFormat=testAutomation/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_, fullOutputPath=testAutomation/contacts/2023_03_20_1679296315680_, syncMode=overwrite} 2023-03-20 07:11:58 destination > 2023-03-20 07:11:56 INFO i.a.i.d.s.S3ConsumerFactory(lambda$toWriteConfig$0):83 - Write config: WriteConfig{streamName=global_suppressions, namespace=null, outputBucketPath=testAutomation, pathFormat=testAutomation/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_, fullOutputPath=testAutomation/global_suppressions/2023_03_20_1679296315680_, syncMode=overwrite} 2023-03-20 07:11:58 destination > 2023-03-20 07:11:56 INFO i.a.i.d.s.S3ConsumerFactory(lambda$toWriteConfig$0):83 - Write config: WriteConfig{streamName=invalid_emails, namespace=null, outputBucketPath=testAutomation, pathFormat=testAutomation/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_, fullOutputPath=testAutomation/invalid_emails/2023_03_20_1679296315680_, syncMode=overwrite} 2023-03-20 07:11:58 destination > 2023-03-20 07:11:56 INFO i.a.i.d.s.S3ConsumerFactory(lambda$toWriteConfig$0):83 - Write config: WriteConfig{streamName=lists, namespace=null, outputBucketPath=testAutomation, pathFormat=testAutomation/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_, fullOutputPath=testAutomation/lists/2023_03_20_1679296315680_, syncMode=overwrite} 2023-03-20 07:11:58 destination > 2023-03-20 07:11:56 INFO i.a.i.d.s.S3ConsumerFactory(lambda$toWriteConfig$0):83 - Write config: WriteConfig{streamName=messages, namespace=null, outputBucketPath=testAutomation, pathFormat=testAutomation/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_, fullOutputPath=testAutomation/messages/2023_03_20_1679296315680_, syncMode=overwrite} 2023-03-20 07:11:58 destination > 2023-03-20 07:11:56 INFO i.a.i.d.s.S3ConsumerFactory(lambda$toWriteConfig$0):83 - Write config: WriteConfig{streamName=segments, namespace=null, outputBucketPath=testAutomation, pathFormat=testAutomation/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_, fullOutputPath=testAutomation/segments/2023_03_20_1679296315680_, syncMode=overwrite} 2023-03-20 07:11:58 destination > 2023-03-20 07:11:56 INFO i.a.i.d.s.S3ConsumerFactory(lambda$toWriteConfig$0):83 - Write config: WriteConfig{streamName=single_sends, namespace=null, outputBucketPath=testAutomation, pathFormat=testAutomation/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_, fullOutputPath=testAutomation/single_sends/2023_03_20_1679296315680_, syncMode=overwrite} 2023-03-20 07:11:58 destination > 2023-03-20 07:11:56 INFO i.a.i.d.s.S3ConsumerFactory(lambda$toWriteConfig$0):83 - Write config: WriteConfig{streamName=spam_reports, namespace=null, outputBucketPath=testAutomation, pathFormat=testAutomation/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_, fullOutputPath=testAutomation/spam_reports/2023_03_20_1679296315680_, syncMode=overwrite} 2023-03-20 07:11:58 destination > 2023-03-20 07:11:56 INFO i.a.i.d.s.S3ConsumerFactory(lambda$toWriteConfig$0):83 - Write config: WriteConfig{streamName=stats_automations, namespace=null, outputBucketPath=testAutomation, pathFormat=testAutomation/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_, fullOutputPath=testAutomation/stats_automations/2023_03_20_1679296315680_, syncMode=overwrite} 2023-03-20 07:11:58 destination > 2023-03-20 07:11:56 INFO i.a.i.d.s.S3ConsumerFactory(lambda$toWriteConfig$0):83 - Write config: WriteConfig{streamName=suppression_group_members, namespace=null, outputBucketPath=testAutomation, pathFormat=testAutomation/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_, fullOutputPath=testAutomation/suppression_group_members/2023_03_20_1679296315680_, syncMode=overwrite} 2023-03-20 07:11:58 destination > 2023-03-20 07:11:56 INFO i.a.i.d.s.S3ConsumerFactory(lambda$toWriteConfig$0):83 - Write config: WriteConfig{streamName=suppression_groups, namespace=null, outputBucketPath=testAutomation, pathFormat=testAutomation/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_, fullOutputPath=testAutomation/suppression_groups/2023_03_20_1679296315680_, syncMode=overwrite} 2023-03-20 07:11:58 destination > 2023-03-20 07:11:56 INFO i.a.i.d.s.S3ConsumerFactory(lambda$toWriteConfig$0):83 - Write config: WriteConfig{streamName=templates, namespace=null, outputBucketPath=testAutomation, pathFormat=testAutomation/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_, fullOutputPath=testAutomation/templates/2023_03_20_1679296315680_, syncMode=overwrite} 2023-03-20 07:11:58 destination > 2023-03-20 07:11:56 INFO i.a.i.d.b.BufferedStreamConsumer(startTracked):116 - class io.airbyte.integrations.destination.buffered_stream_consumer.BufferedStreamConsumer started. 2023-03-20 07:11:58 destination > 2023-03-20 07:11:56 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):90 - Preparing bucket in destination started for 15 streams 2023-03-20 07:11:58 destination > 2023-03-20 07:11:56 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):97 - Clearing storage area in destination started for namespace null stream blocks bucketObject testAutomation pathFormat testAutomation/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_ 2023-03-20 07:11:58 destination > 2023-03-20 07:11:56 INFO i.a.i.d.s.S3StorageOperations(cleanUpBucketObject):250 - Storage bucket testAutomation has been cleaned-up (0 objects matching testAutomation/blocks/[0-9]{4}_[0-9]{2}_[0-9]{2}_[0-9]+_.* were deleted)... 2023-03-20 07:11:58 destination > 2023-03-20 07:11:56 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):100 - Clearing storage area in destination completed for namespace null stream blocks bucketObject testAutomation 2023-03-20 07:11:58 source > Finished syncing blocks 2023-03-20 07:11:58 destination > 2023-03-20 07:11:56 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):97 - Clearing storage area in destination started for namespace null stream bounces bucketObject testAutomation pathFormat testAutomation/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_ 2023-03-20 07:11:58 destination > 2023-03-20 07:11:57 INFO i.a.i.d.s.S3StorageOperations(cleanUpBucketObject):250 - Storage bucket testAutomation has been cleaned-up (0 objects matching testAutomation/bounces/[0-9]{4}_[0-9]{2}_[0-9]{2}_[0-9]+_.* were deleted)... 2023-03-20 07:11:58 destination > 2023-03-20 07:11:57 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):100 - Clearing storage area in destination completed for namespace null stream bounces bucketObject testAutomation 2023-03-20 07:11:58 destination > 2023-03-20 07:11:57 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):97 - Clearing storage area in destination started for namespace null stream campaigns bucketObject testAutomation pathFormat testAutomation/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_ 2023-03-20 07:11:58 destination > 2023-03-20 07:11:57 INFO i.a.i.d.s.S3StorageOperations(cleanUpObjects):301 - Deleting objects testAutomation/campaigns/2023_03_20_1679296287341_0.csv 2023-03-20 07:11:58 source > SourceSendgrid runtimes: Syncing stream blocks 0:00:00.191188 2023-03-20 07:11:58 destination > 2023-03-20 07:11:57 INFO i.a.i.d.s.S3StorageOperations(cleanUpBucketObject):250 - Storage bucket testAutomation has been cleaned-up (1 objects matching testAutomation/campaigns/[0-9]{4}_[0-9]{2}_[0-9]{2}_[0-9]+_.* were deleted)... 2023-03-20 07:11:58 destination > 2023-03-20 07:11:57 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):100 - Clearing storage area in destination completed for namespace null stream campaigns bucketObject testAutomation 2023-03-20 07:11:58 destination > 2023-03-20 07:11:57 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):97 - Clearing storage area in destination started for namespace null stream contacts bucketObject testAutomation pathFormat testAutomation/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_ 2023-03-20 07:11:58 destination > 2023-03-20 07:11:57 INFO i.a.i.d.s.S3StorageOperations(cleanUpObjects):301 - Deleting objects testAutomation/contacts/2023_03_20_1679296287341_0.csv 2023-03-20 07:11:58 source > Syncing stream: bounces 2023-03-20 07:11:58 destination > 2023-03-20 07:11:57 INFO i.a.i.d.s.S3StorageOperations(cleanUpBucketObject):250 - Storage bucket testAutomation has been cleaned-up (1 objects matching testAutomation/contacts/[0-9]{4}_[0-9]{2}_[0-9]{2}_[0-9]+_.* were deleted)... 2023-03-20 07:11:58 destination > 2023-03-20 07:11:57 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):100 - Clearing storage area in destination completed for namespace null stream contacts bucketObject testAutomation 2023-03-20 07:11:58 destination > 2023-03-20 07:11:57 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):97 - Clearing storage area in destination started for namespace null stream global_suppressions bucketObject testAutomation pathFormat testAutomation/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_ 2023-03-20 07:11:58 destination > 2023-03-20 07:11:57 INFO i.a.i.d.s.S3StorageOperations(cleanUpObjects):301 - Deleting objects testAutomation/global_suppressions/2023_03_20_1679296287341_0.csv 2023-03-20 07:11:58 destination > 2023-03-20 07:11:57 INFO i.a.i.d.s.S3StorageOperations(cleanUpBucketObject):250 - Storage bucket testAutomation has been cleaned-up (1 objects matching testAutomation/global_suppressions/[0-9]{4}_[0-9]{2}_[0-9]{2}_[0-9]+_.* were deleted)... 2023-03-20 07:11:58 destination > 2023-03-20 07:11:57 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):100 - Clearing storage area in destination completed for namespace null stream global_suppressions bucketObject testAutomation 2023-03-20 07:11:58 destination > 2023-03-20 07:11:57 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):97 - Clearing storage area in destination started for namespace null stream invalid_emails bucketObject testAutomation pathFormat testAutomation/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_ 2023-03-20 07:11:58 destination > 2023-03-20 07:11:57 INFO i.a.i.d.s.S3StorageOperations(cleanUpBucketObject):250 - Storage bucket testAutomation has been cleaned-up (0 objects matching testAutomation/invalid_emails/[0-9]{4}_[0-9]{2}_[0-9]{2}_[0-9]+_.* were deleted)... 2023-03-20 07:11:58 destination > 2023-03-20 07:11:57 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):100 - Clearing storage area in destination completed for namespace null stream invalid_emails bucketObject testAutomation 2023-03-20 07:11:58 destination > 2023-03-20 07:11:57 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):97 - Clearing storage area in destination started for namespace null stream lists bucketObject testAutomation pathFormat testAutomation/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_ 2023-03-20 07:11:58 destination > 2023-03-20 07:11:57 INFO i.a.i.d.s.S3StorageOperations(cleanUpObjects):301 - Deleting objects testAutomation/lists/2023_03_20_1679296287341_0.csv 2023-03-20 07:11:58 destination > 2023-03-20 07:11:57 INFO i.a.i.d.s.S3StorageOperations(cleanUpBucketObject):250 - Storage bucket testAutomation has been cleaned-up (1 objects matching testAutomation/lists/[0-9]{4}_[0-9]{2}_[0-9]{2}_[0-9]+_.* were deleted)... 2023-03-20 07:11:58 destination > 2023-03-20 07:11:57 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):100 - Clearing storage area in destination completed for namespace null stream lists bucketObject testAutomation 2023-03-20 07:11:58 destination > 2023-03-20 07:11:57 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):97 - Clearing storage area in destination started for namespace null stream messages bucketObject testAutomation pathFormat testAutomation/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_ 2023-03-20 07:11:58 destination > 2023-03-20 07:11:58 INFO i.a.i.d.s.S3StorageOperations(cleanUpBucketObject):250 - Storage bucket testAutomation has been cleaned-up (0 objects matching testAutomation/messages/[0-9]{4}_[0-9]{2}_[0-9]{2}_[0-9]+_.* were deleted)... 2023-03-20 07:11:58 destination > 2023-03-20 07:11:58 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):100 - Clearing storage area in destination completed for namespace null stream messages bucketObject testAutomation 2023-03-20 07:11:58 destination > 2023-03-20 07:11:58 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):97 - Clearing storage area in destination started for namespace null stream segments bucketObject testAutomation pathFormat testAutomation/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_ 2023-03-20 07:11:58 destination > 2023-03-20 07:11:58 INFO i.a.i.d.s.S3StorageOperations(cleanUpBucketObject):250 - Storage bucket testAutomation has been cleaned-up (0 objects matching testAutomation/segments/[0-9]{4}_[0-9]{2}_[0-9]{2}_[0-9]+_.* were deleted)... 2023-03-20 07:11:58 destination > 2023-03-20 07:11:58 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):100 - Clearing storage area in destination completed for namespace null stream segments bucketObject testAutomation 2023-03-20 07:11:58 destination > 2023-03-20 07:11:58 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):97 - Clearing storage area in destination started for namespace null stream single_sends bucketObject testAutomation pathFormat testAutomation/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_ 2023-03-20 07:11:58 destination > 2023-03-20 07:11:58 INFO i.a.i.d.s.S3StorageOperations(cleanUpBucketObject):250 - Storage bucket testAutomation has been cleaned-up (0 objects matching testAutomation/single_sends/[0-9]{4}_[0-9]{2}_[0-9]{2}_[0-9]+_.* were deleted)... 2023-03-20 07:11:58 destination > 2023-03-20 07:11:58 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):100 - Clearing storage area in destination completed for namespace null stream single_sends bucketObject testAutomation 2023-03-20 07:11:58 destination > 2023-03-20 07:11:58 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):97 - Clearing storage area in destination started for namespace null stream spam_reports bucketObject testAutomation pathFormat testAutomation/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_ 2023-03-20 07:11:58 destination > 2023-03-20 07:11:58 INFO i.a.i.d.s.S3StorageOperations(cleanUpBucketObject):250 - Storage bucket testAutomation has been cleaned-up (0 objects matching testAutomation/spam_reports/[0-9]{4}_[0-9]{2}_[0-9]{2}_[0-9]+_.* were deleted)... 2023-03-20 07:11:58 destination > 2023-03-20 07:11:58 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):100 - Clearing storage area in destination completed for namespace null stream spam_reports bucketObject testAutomation 2023-03-20 07:11:58 destination > 2023-03-20 07:11:58 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):97 - Clearing storage area in destination started for namespace null stream stats_automations bucketObject testAutomation pathFormat testAutomation/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_ 2023-03-20 07:11:58 destination > 2023-03-20 07:11:58 INFO i.a.i.d.s.S3StorageOperations(cleanUpBucketObject):250 - Storage bucket testAutomation has been cleaned-up (0 objects matching testAutomation/stats_automations/[0-9]{4}_[0-9]{2}_[0-9]{2}_[0-9]+_.* were deleted)... 2023-03-20 07:11:58 destination > 2023-03-20 07:11:58 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):100 - Clearing storage area in destination completed for namespace null stream stats_automations bucketObject testAutomation 2023-03-20 07:11:58 destination > 2023-03-20 07:11:58 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):97 - Clearing storage area in destination started for namespace null stream suppression_group_members bucketObject testAutomation pathFormat testAutomation/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_ 2023-03-20 07:11:58 destination > 2023-03-20 07:11:58 INFO i.a.i.d.s.S3StorageOperations(cleanUpBucketObject):250 - Storage bucket testAutomation has been cleaned-up (0 objects matching testAutomation/suppression_group_members/[0-9]{4}_[0-9]{2}_[0-9]{2}_[0-9]+_.* were deleted)... 2023-03-20 07:11:58 destination > 2023-03-20 07:11:58 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):100 - Clearing storage area in destination completed for namespace null stream suppression_group_members bucketObject testAutomation 2023-03-20 07:11:58 destination > 2023-03-20 07:11:58 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):97 - Clearing storage area in destination started for namespace null stream suppression_groups bucketObject testAutomation pathFormat testAutomation/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_ 2023-03-20 07:11:58 destination > 2023-03-20 07:11:58 INFO i.a.i.d.s.S3StorageOperations(cleanUpBucketObject):250 - Storage bucket testAutomation has been cleaned-up (0 objects matching testAutomation/suppression_groups/[0-9]{4}_[0-9]{2}_[0-9]{2}_[0-9]+_.* were deleted)... 2023-03-20 07:11:58 destination > 2023-03-20 07:11:58 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):100 - Clearing storage area in destination completed for namespace null stream suppression_groups bucketObject testAutomation 2023-03-20 07:11:58 destination > 2023-03-20 07:11:58 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):97 - Clearing storage area in destination started for namespace null stream templates bucketObject testAutomation pathFormat testAutomation/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_ 2023-03-20 07:11:58 destination > 2023-03-20 07:11:58 INFO i.a.i.d.s.S3StorageOperations(cleanUpBucketObject):250 - Storage bucket testAutomation has been cleaned-up (0 objects matching testAutomation/templates/[0-9]{4}_[0-9]{2}_[0-9]{2}_[0-9]+_.* were deleted)... 2023-03-20 07:11:58 destination > 2023-03-20 07:11:58 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):100 - Clearing storage area in destination completed for namespace null stream templates bucketObject testAutomation 2023-03-20 07:11:58 destination > 2023-03-20 07:11:58 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):104 - Preparing storage area in destination completed. 2023-03-20 07:11:58 source > Read 0 records from bounces stream 2023-03-20 07:11:58 source > Finished syncing bounces 2023-03-20 07:11:58 source > SourceSendgrid runtimes: Syncing stream blocks 0:00:00.191188 Syncing stream bounces 0:00:00.143466 2023-03-20 07:11:58 source > Syncing stream: campaigns 2023-03-20 07:11:59 source > Read 5 records from campaigns stream 2023-03-20 07:11:59 source > Finished syncing campaigns 2023-03-20 07:11:59 source > SourceSendgrid runtimes: Syncing stream blocks 0:00:00.191188 Syncing stream bounces 0:00:00.143466 Syncing stream campaigns 0:00:00.287596 2023-03-20 07:11:59 source > Syncing stream: contacts 2023-03-20 07:11:59 source > Read 5 records from contacts stream 2023-03-20 07:11:59 source > Finished syncing contacts 2023-03-20 07:11:59 source > SourceSendgrid runtimes: Syncing stream blocks 0:00:00.191188 Syncing stream bounces 0:00:00.143466 Syncing stream campaigns 0:00:00.287596 Syncing stream contacts 0:00:00.603141 2023-03-20 07:11:59 source > Syncing stream: global_suppressions 2023-03-20 07:12:00 source > Read 3 records from global_suppressions stream 2023-03-20 07:12:00 source > Finished syncing global_suppressions 2023-03-20 07:12:00 source > SourceSendgrid runtimes: Syncing stream blocks 0:00:00.191188 Syncing stream bounces 0:00:00.143466 Syncing stream campaigns 0:00:00.287596 Syncing stream contacts 0:00:00.603141 Syncing stream global_suppressions 0:00:01.215598 2023-03-20 07:12:00 source > Syncing stream: invalid_emails 2023-03-20 07:12:02 source > Read 0 records from invalid_emails stream 2023-03-20 07:12:02 source > Finished syncing invalid_emails 2023-03-20 07:12:02 source > SourceSendgrid runtimes: Syncing stream blocks 0:00:00.191188 Syncing stream bounces 0:00:00.143466 Syncing stream campaigns 0:00:00.287596 Syncing stream contacts 0:00:00.603141 Syncing stream global_suppressions 0:00:01.215598 Syncing stream invalid_emails 0:00:01.203192 2023-03-20 07:12:02 source > Syncing stream: lists 2023-03-20 07:12:02 source > Read 1 records from lists stream 2023-03-20 07:12:02 source > Finished syncing lists 2023-03-20 07:12:02 source > SourceSendgrid runtimes: Syncing stream blocks 0:00:00.191188 Syncing stream bounces 0:00:00.143466 Syncing stream campaigns 0:00:00.287596 Syncing stream contacts 0:00:00.603141 Syncing stream global_suppressions 0:00:01.215598 Syncing stream invalid_emails 0:00:01.203192 Syncing stream lists 0:00:00.505035 2023-03-20 07:12:02 source > Syncing stream: messages 2023-03-20 07:12:02 source > {"errors":[{"message":"authorization required"}]} 2023-03-20 07:12:02 source > Encountered an exception while reading stream messages Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 115, in read yield from self._read_stream( File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 184, in _read_stream for record in record_iterator: File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 282, in _read_full_refresh for record in records: File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py", line 421, in read_records response = self._send_request(request, request_kwargs) File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py", line 339, in _send_request return backoff_handler(user_backoff_handler)(request, request_kwargs) File "/usr/local/lib/python3.9/site-packages/backoff/_sync.py", line 105, in retry ret = target(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/backoff/_sync.py", line 105, in retry ret = target(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py", line 306, in _send raise exc File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py", line 303, in _send response.raise_for_status() File "/usr/local/lib/python3.9/site-packages/requests/models.py", line 1021, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://api.sendgrid.com/v3/messages?query=last_event_time%20BETWEEN%20TIMESTAMP%20%22None%22%20AND%20TIMESTAMP%20%222023-03-20T07%3A12%3A02Z%22&limit=1000 2023-03-20 07:12:02 source > Finished syncing messages 2023-03-20 07:12:02 source > SourceSendgrid runtimes: Syncing stream blocks 0:00:00.191188 Syncing stream bounces 0:00:00.143466 Syncing stream campaigns 0:00:00.287596 Syncing stream contacts 0:00:00.603141 Syncing stream global_suppressions 0:00:01.215598 Syncing stream invalid_emails 0:00:01.203192 Syncing stream lists 0:00:00.505035 Syncing stream messages 0:00:00.100866 2023-03-20 07:12:02 source > 400 Client Error: Bad Request for url: https://api.sendgrid.com/v3/messages?query=last_event_time%20BETWEEN%20TIMESTAMP%20%22None%22%20AND%20TIMESTAMP%20%222023-03-20T07%3A12%3A02Z%22&limit=1000 Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 115, in read yield from self._read_stream( File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 184, in _read_stream for record in record_iterator: File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 282, in _read_full_refresh for record in records: File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py", line 421, in read_records response = self._send_request(request, request_kwargs) File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py", line 339, in _send_request return backoff_handler(user_backoff_handler)(request, request_kwargs) File "/usr/local/lib/python3.9/site-packages/backoff/_sync.py", line 105, in retry ret = target(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/backoff/_sync.py", line 105, in retry ret = target(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py", line 306, in _send raise exc File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py", line 303, in _send response.raise_for_status() File "/usr/local/lib/python3.9/site-packages/requests/models.py", line 1021, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://api.sendgrid.com/v3/messages?query=last_event_time%20BETWEEN%20TIMESTAMP%20%22None%22%20AND%20TIMESTAMP%20%222023-03-20T07%3A12%3A02Z%22&limit=1000 The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/airbyte/integration_code/main.py", line 13, in launch(source, sys.argv[1:]) File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py", line 123, in launch for message in source_entrypoint.run(parsed_args): File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py", line 114, in run for message in generator: File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 128, in read raise AirbyteTracedException.from_exception(e, message=display_message) from e airbyte_cdk.utils.traced_exception.AirbyteTracedException: 400 Client Error: Bad Request for url: https://api.sendgrid.com/v3/messages?query=last_event_time%20BETWEEN%20TIMESTAMP%20%22None%22%20AND%20TIMESTAMP%20%222023-03-20T07%3A12%3A02Z%22&limit=1000 2023-03-20 07:12:03 destination > 2023-03-20 07:12:03 INFO i.a.i.d.r.SerializedBufferingStrategy(lambda$addRecord$0):47 - Starting a new buffer for stream campaigns (current state: 0 bytes in 0 buffers) 2023-03-20 07:12:03 destination > 2023-03-20 07:12:03 INFO i.a.i.d.r.SerializedBufferingStrategy(lambda$addRecord$0):47 - Starting a new buffer for stream contacts (current state: 0 bytes in 1 buffers) 2023-03-20 07:12:03 destination > 2023-03-20 07:12:03 INFO i.a.i.d.r.SerializedBufferingStrategy(lambda$addRecord$0):47 - Starting a new buffer for stream global_suppressions (current state: 0 bytes in 2 buffers) 2023-03-20 07:12:03 destination > 2023-03-20 07:12:03 INFO i.a.i.d.r.SerializedBufferingStrategy(lambda$addRecord$0):47 - Starting a new buffer for stream lists (current state: 0 bytes in 3 buffers) 2023-03-20 07:12:03 destination > 2023-03-20 07:12:03 INFO i.a.i.b.FailureTrackingAirbyteMessageConsumer(close):62 - Airbyte message consumer: succeeded. 2023-03-20 07:12:03 destination > 2023-03-20 07:12:03 INFO i.a.i.d.b.BufferedStreamConsumer(close):171 - executing on success close procedure. 2023-03-20 07:12:03 destination > 2023-03-20 07:12:03 INFO i.a.i.d.r.SerializedBufferingStrategy(flushAll):100 - Flushing all 4 current buffers (0 bytes in total) 2023-03-20 07:12:03 destination > 2023-03-20 07:12:03 INFO i.a.i.d.r.SerializedBufferingStrategy(flushAll):102 - Flushing buffer of stream contacts (0 bytes) 2023-03-20 07:12:03 destination > 2023-03-20 07:12:03 INFO i.a.i.d.s.S3ConsumerFactory(lambda$flushBufferFunction$2):121 - Flushing buffer for stream contacts (0 bytes) to storage 2023-03-20 07:12:03 destination > 2023-03-20 07:12:03 INFO i.a.i.d.r.BaseSerializedBuffer(flush):138 - Finished writing data to 3ce4f44e-da3e-47aa-b44e-4a017dfc24e510794916427391552126.csv (3 KB) 2023-03-20 07:12:03 destination > 2023-03-20 07:12:03 INFO a.m.s.StreamTransferManager(getMultiPartOutputStreams):329 - Initiated multipart upload to redshift-dev-s3-bucket/testAutomation/contacts/2023_03_20_1679296315680_0.csv with full ID mW72iTmCNbcIW43xwTjmASuumzFhUV.lFtP4dXifOPxhQKj5T4J6LmjTL1ToR3qC3JFt.407XEBKvIS0nzaF0_tMHIK6eetJHwaTy6Wu24F3h9WckomCaI_AcbSmg8xC 2023-03-20 07:12:03 destination > 2023-03-20 07:12:03 INFO a.m.s.MultiPartOutputStream(close):158 - Called close() on [MultipartOutputStream for parts 1 - 10000] 2023-03-20 07:12:03 destination > 2023-03-20 07:12:03 INFO a.m.s.StreamTransferManager(complete):367 - [Manager uploading to redshift-dev-s3-bucket/testAutomation/contacts/2023_03_20_1679296315680_0.csv with id mW72iTmCN...AcbSmg8xC]: Uploading leftover stream [Part number 1 containing 0.00 MB] 2023-03-20 07:12:03 destination > 2023-03-20 07:12:03 INFO a.m.s.StreamTransferManager(uploadStreamPart):558 - [Manager uploading to redshift-dev-s3-bucket/testAutomation/contacts/2023_03_20_1679296315680_0.csv with id mW72iTmCN...AcbSmg8xC]: Finished uploading [Part number 1 containing 0.00 MB] 2023-03-20 07:12:03 destination > 2023-03-20 07:12:03 INFO a.m.s.StreamTransferManager(complete):395 - [Manager uploading to redshift-dev-s3-bucket/testAutomation/contacts/2023_03_20_1679296315680_0.csv with id mW72iTmCN...AcbSmg8xC]: Completed 2023-03-20 07:12:03 destination > 2023-03-20 07:12:03 INFO i.a.i.d.s.S3StorageOperations(loadDataIntoBucket):198 - Uploaded buffer file to storage: 3ce4f44e-da3e-47aa-b44e-4a017dfc24e510794916427391552126.csv -> testAutomation/contacts/2023_03_20_1679296315680_0.csv (filename: 2023_03_20_1679296315680_0.csv) 2023-03-20 07:12:03 destination > 2023-03-20 07:12:03 INFO i.a.i.d.r.FileBuffer(deleteFile):90 - Deleting tempFile data 3ce4f44e-da3e-47aa-b44e-4a017dfc24e510794916427391552126.csv 2023-03-20 07:12:03 destination > 2023-03-20 07:12:03 INFO i.a.i.d.r.SerializedBufferingStrategy(flushAll):102 - Flushing buffer of stream campaigns (0 bytes) 2023-03-20 07:12:03 destination > 2023-03-20 07:12:03 INFO i.a.i.d.s.S3ConsumerFactory(lambda$flushBufferFunction$2):121 - Flushing buffer for stream campaigns (0 bytes) to storage 2023-03-20 07:12:03 destination > 2023-03-20 07:12:03 INFO i.a.i.d.r.BaseSerializedBuffer(flush):138 - Finished writing data to 4cc1ebdc-3fc9-480b-a9e9-d1a27b86aec012881871462414609178.csv (1 KB) 2023-03-20 07:12:03 destination > 2023-03-20 07:12:03 INFO a.m.s.StreamTransferManager(getMultiPartOutputStreams):329 - Initiated multipart upload to redshift-dev-s3-bucket/testAutomation/campaigns/2023_03_20_1679296315680_0.csv with full ID WO_LIoMEIj9XRSrGqGdtkn9gB6Ob6VxPFox5hwpyUiwC1i.OYWUnKPzyXYB6fnP7meS..NFUX47y47GOFQ_KQfQMZ6sDKjd9RZX4DG2SLJXJyclpGt1mT6X9R7F5c5Ix 2023-03-20 07:12:03 destination > 2023-03-20 07:12:03 INFO a.m.s.MultiPartOutputStream(close):158 - Called close() on [MultipartOutputStream for parts 1 - 10000] 2023-03-20 07:12:03 destination > 2023-03-20 07:12:03 INFO a.m.s.StreamTransferManager(complete):367 - [Manager uploading to redshift-dev-s3-bucket/testAutomation/campaigns/2023_03_20_1679296315680_0.csv with id WO_LIoMEI...9R7F5c5Ix]: Uploading leftover stream [Part number 1 containing 0.00 MB] 2023-03-20 07:12:03 destination > 2023-03-20 07:12:03 INFO a.m.s.StreamTransferManager(uploadStreamPart):558 - [Manager uploading to redshift-dev-s3-bucket/testAutomation/campaigns/2023_03_20_1679296315680_0.csv with id WO_LIoMEI...9R7F5c5Ix]: Finished uploading [Part number 1 containing 0.00 MB] 2023-03-20 07:12:03 destination > 2023-03-20 07:12:03 INFO a.m.s.StreamTransferManager(complete):395 - [Manager uploading to redshift-dev-s3-bucket/testAutomation/campaigns/2023_03_20_1679296315680_0.csv with id WO_LIoMEI...9R7F5c5Ix]: Completed 2023-03-20 07:12:03 destination > 2023-03-20 07:12:03 INFO i.a.i.d.s.S3StorageOperations(loadDataIntoBucket):198 - Uploaded buffer file to storage: 4cc1ebdc-3fc9-480b-a9e9-d1a27b86aec012881871462414609178.csv -> testAutomation/campaigns/2023_03_20_1679296315680_0.csv (filename: 2023_03_20_1679296315680_0.csv) 2023-03-20 07:12:03 destination > 2023-03-20 07:12:03 INFO i.a.i.d.r.FileBuffer(deleteFile):90 - Deleting tempFile data 4cc1ebdc-3fc9-480b-a9e9-d1a27b86aec012881871462414609178.csv 2023-03-20 07:12:03 destination > 2023-03-20 07:12:03 INFO i.a.i.d.r.SerializedBufferingStrategy(flushAll):102 - Flushing buffer of stream lists (0 bytes) 2023-03-20 07:12:03 destination > 2023-03-20 07:12:03 INFO i.a.i.d.s.S3ConsumerFactory(lambda$flushBufferFunction$2):121 - Flushing buffer for stream lists (0 bytes) to storage 2023-03-20 07:12:03 destination > 2023-03-20 07:12:03 INFO i.a.i.d.r.BaseSerializedBuffer(flush):138 - Finished writing data to fbb09a35-d07e-400b-af2e-577e8c2f6e673415644558923790229.csv (316 bytes) 2023-03-20 07:12:04 destination > 2023-03-20 07:12:04 INFO a.m.s.StreamTransferManager(getMultiPartOutputStreams):329 - Initiated multipart upload to redshift-dev-s3-bucket/testAutomation/lists/2023_03_20_1679296315680_0.csv with full ID vOnncgFgtT2421bXhYPgDojAqNEWy2GJNVrzyToiT6HPwCk7cZPhiDQM2NtTo9IBtbK5zocqetakuK7FRDFD9VMeZXz0NRYRSJyceYvlErSQvd8t7jD3sF79UyDaX8QJ 2023-03-20 07:12:04 destination > 2023-03-20 07:12:04 INFO a.m.s.MultiPartOutputStream(close):158 - Called close() on [MultipartOutputStream for parts 1 - 10000] 2023-03-20 07:12:04 destination > 2023-03-20 07:12:04 INFO a.m.s.StreamTransferManager(complete):367 - [Manager uploading to redshift-dev-s3-bucket/testAutomation/lists/2023_03_20_1679296315680_0.csv with id vOnncgFgt...9UyDaX8QJ]: Uploading leftover stream [Part number 1 containing 0.00 MB] 2023-03-20 07:12:04 destination > 2023-03-20 07:12:04 INFO a.m.s.StreamTransferManager(uploadStreamPart):558 - [Manager uploading to redshift-dev-s3-bucket/testAutomation/lists/2023_03_20_1679296315680_0.csv with id vOnncgFgt...9UyDaX8QJ]: Finished uploading [Part number 1 containing 0.00 MB] 2023-03-20 07:12:04 destination > 2023-03-20 07:12:04 INFO a.m.s.StreamTransferManager(complete):395 - [Manager uploading to redshift-dev-s3-bucket/testAutomation/lists/2023_03_20_1679296315680_0.csv with id vOnncgFgt...9UyDaX8QJ]: Completed 2023-03-20 07:12:04 destination > 2023-03-20 07:12:04 INFO i.a.i.d.s.S3StorageOperations(loadDataIntoBucket):198 - Uploaded buffer file to storage: fbb09a35-d07e-400b-af2e-577e8c2f6e673415644558923790229.csv -> testAutomation/lists/2023_03_20_1679296315680_0.csv (filename: 2023_03_20_1679296315680_0.csv) 2023-03-20 07:12:04 destination > 2023-03-20 07:12:04 INFO i.a.i.d.r.FileBuffer(deleteFile):90 - Deleting tempFile data fbb09a35-d07e-400b-af2e-577e8c2f6e673415644558923790229.csv 2023-03-20 07:12:04 destination > 2023-03-20 07:12:04 INFO i.a.i.d.r.SerializedBufferingStrategy(flushAll):102 - Flushing buffer of stream global_suppressions (0 bytes) 2023-03-20 07:12:04 destination > 2023-03-20 07:12:04 INFO i.a.i.d.s.S3ConsumerFactory(lambda$flushBufferFunction$2):121 - Flushing buffer for stream global_suppressions (0 bytes) to storage 2023-03-20 07:12:04 destination > 2023-03-20 07:12:04 INFO i.a.i.d.r.BaseSerializedBuffer(flush):138 - Finished writing data to 870925c1-09b4-4a7b-8b3b-d507158a7da54276454361031220463.csv (396 bytes) 2023-03-20 07:12:04 destination > 2023-03-20 07:12:04 INFO a.m.s.StreamTransferManager(getMultiPartOutputStreams):329 - Initiated multipart upload to redshift-dev-s3-bucket/testAutomation/global_suppressions/2023_03_20_1679296315680_0.csv with full ID AZzjXkIjF06bqwR2ezTpbc_9.F_sXTLodGVzvLV7Rd5EGj3IcrCGcHWXgDaOB11g.9bFYb_DKhEMm1baTph20aI0YGg6exLPHRPhIS7gCbMgQTh1bi7pHzf8uXDOIqR8 2023-03-20 07:12:04 destination > 2023-03-20 07:12:04 INFO a.m.s.MultiPartOutputStream(close):158 - Called close() on [MultipartOutputStream for parts 1 - 10000] 2023-03-20 07:12:04 destination > 2023-03-20 07:12:04 INFO a.m.s.StreamTransferManager(complete):367 - [Manager uploading to redshift-dev-s3-bucket/testAutomation/global_suppressions/2023_03_20_1679296315680_0.csv with id AZzjXkIjF...8uXDOIqR8]: Uploading leftover stream [Part number 1 containing 0.00 MB] 2023-03-20 07:12:04 destination > 2023-03-20 07:12:04 INFO a.m.s.StreamTransferManager(uploadStreamPart):558 - [Manager uploading to redshift-dev-s3-bucket/testAutomation/global_suppressions/2023_03_20_1679296315680_0.csv with id AZzjXkIjF...8uXDOIqR8]: Finished uploading [Part number 1 containing 0.00 MB] 2023-03-20 07:12:04 destination > 2023-03-20 07:12:04 INFO a.m.s.StreamTransferManager(complete):395 - [Manager uploading to redshift-dev-s3-bucket/testAutomation/global_suppressions/2023_03_20_1679296315680_0.csv with id AZzjXkIjF...8uXDOIqR8]: Completed 2023-03-20 07:12:04 destination > 2023-03-20 07:12:04 INFO i.a.i.d.s.S3StorageOperations(loadDataIntoBucket):198 - Uploaded buffer file to storage: 870925c1-09b4-4a7b-8b3b-d507158a7da54276454361031220463.csv -> testAutomation/global_suppressions/2023_03_20_1679296315680_0.csv (filename: 2023_03_20_1679296315680_0.csv) 2023-03-20 07:12:04 destination > 2023-03-20 07:12:04 INFO i.a.i.d.r.FileBuffer(deleteFile):90 - Deleting tempFile data 870925c1-09b4-4a7b-8b3b-d507158a7da54276454361031220463.csv 2023-03-20 07:12:04 destination > 2023-03-20 07:12:04 INFO i.a.i.d.r.SerializedBufferingStrategy(close):122 - Closing buffer for stream contacts 2023-03-20 07:12:04 destination > 2023-03-20 07:12:04 INFO i.a.i.d.r.SerializedBufferingStrategy(close):122 - Closing buffer for stream campaigns 2023-03-20 07:12:04 destination > 2023-03-20 07:12:04 INFO i.a.i.d.r.SerializedBufferingStrategy(close):122 - Closing buffer for stream lists 2023-03-20 07:12:04 destination > 2023-03-20 07:12:04 INFO i.a.i.d.r.SerializedBufferingStrategy(close):122 - Closing buffer for stream global_suppressions 2023-03-20 07:12:04 destination > 2023-03-20 07:12:04 INFO i.a.i.b.IntegrationRunner(runInternal):152 - Completed integration: io.airbyte.integrations.destination.s3.S3Destination 2023-03-20 07:12:05 INFO i.a.v.j.JsonSchemaValidator(test):71 - JSON schema validation failed. errors: $.compression_codec: is missing but it is required, $.format_type: does not have a value in the enumeration [Avro] 2023-03-20 07:12:28 destination > 2023-03-20 07:12:23 INFO i.a.i.b.IntegrationCliParser(parseOptions):118 - integration args: {catalog=destination_catalog.json, write=null, config=destination_config.json} 2023-03-20 07:12:28 source > Starting syncing SourceSendgrid 2023-03-20 07:12:28 destination > 2023-03-20 07:12:23 INFO i.a.i.b.IntegrationRunner(runInternal):104 - Running integration: io.airbyte.integrations.destination.s3.S3Destination 2023-03-20 07:12:28 destination > 2023-03-20 07:12:23 INFO i.a.i.b.IntegrationRunner(runInternal):105 - Command: WRITE 2023-03-20 07:12:28 destination > 2023-03-20 07:12:23 INFO i.a.i.b.IntegrationRunner(runInternal):106 - Integration config: IntegrationConfig{command=WRITE, configPath='destination_config.json', catalogPath='destination_catalog.json', statePath='null'} 2023-03-20 07:12:28 source > Syncing stream: blocks 2023-03-20 07:12:28 destination > 2023-03-20 07:12:23 WARN c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword airbyte_secret - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword 2023-03-20 07:12:28 destination > 2023-03-20 07:12:23 WARN c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword examples - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword 2023-03-20 07:12:28 destination > 2023-03-20 07:12:23 WARN c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword order - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword 2023-03-20 07:12:28 destination > 2023-03-20 07:12:23 WARN c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword requires - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword 2023-03-20 07:12:28 source > Read 0 records from blocks stream 2023-03-20 07:12:28 destination > 2023-03-20 07:12:23 INFO i.a.i.d.s.S3FormatConfigs(getS3FormatConfig):22 - S3 format config: {"flattening":"No flattening","compression":{"compression_type":"No Compression"},"format_type":"CSV"} 2023-03-20 07:12:28 destination > 2023-03-20 07:12:23 INFO i.a.i.d.s.S3DestinationConfig(createS3Client):190 - Creating S3 client... 2023-03-20 07:12:28 destination > 2023-03-20 07:12:24 INFO i.a.i.d.s.SerializedBufferFactory(getCreateFunction):49 - S3 format config: S3CsvFormatConfig{flattening=NO, compression=NO_COMPRESSION} 2023-03-20 07:12:28 source > Finished syncing blocks 2023-03-20 07:12:28 destination > 2023-03-20 07:12:24 INFO i.a.i.d.s.S3ConsumerFactory(lambda$toWriteConfig$0):83 - Write config: WriteConfig{streamName=blocks, namespace=null, outputBucketPath=testAutomation, pathFormat=testAutomation/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_, fullOutputPath=testAutomation/blocks/2023_03_20_1679296343675_, syncMode=overwrite} 2023-03-20 07:12:28 destination > 2023-03-20 07:12:24 INFO i.a.i.d.s.S3ConsumerFactory(lambda$toWriteConfig$0):83 - Write config: WriteConfig{streamName=bounces, namespace=null, outputBucketPath=testAutomation, pathFormat=testAutomation/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_, fullOutputPath=testAutomation/bounces/2023_03_20_1679296343675_, syncMode=overwrite} 2023-03-20 07:12:28 source > SourceSendgrid runtimes: Syncing stream blocks 0:00:00.150198 2023-03-20 07:12:28 destination > 2023-03-20 07:12:24 INFO i.a.i.d.s.S3ConsumerFactory(lambda$toWriteConfig$0):83 - Write config: WriteConfig{streamName=campaigns, namespace=null, outputBucketPath=testAutomation, pathFormat=testAutomation/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_, fullOutputPath=testAutomation/campaigns/2023_03_20_1679296343675_, syncMode=overwrite} 2023-03-20 07:12:28 destination > 2023-03-20 07:12:24 INFO i.a.i.d.s.S3ConsumerFactory(lambda$toWriteConfig$0):83 - Write config: WriteConfig{streamName=contacts, namespace=null, outputBucketPath=testAutomation, pathFormat=testAutomation/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_, fullOutputPath=testAutomation/contacts/2023_03_20_1679296343675_, syncMode=overwrite} 2023-03-20 07:12:28 source > Syncing stream: bounces 2023-03-20 07:12:28 destination > 2023-03-20 07:12:24 INFO i.a.i.d.s.S3ConsumerFactory(lambda$toWriteConfig$0):83 - Write config: WriteConfig{streamName=global_suppressions, namespace=null, outputBucketPath=testAutomation, pathFormat=testAutomation/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_, fullOutputPath=testAutomation/global_suppressions/2023_03_20_1679296343675_, syncMode=overwrite} 2023-03-20 07:12:28 destination > 2023-03-20 07:12:24 INFO i.a.i.d.s.S3ConsumerFactory(lambda$toWriteConfig$0):83 - Write config: WriteConfig{streamName=invalid_emails, namespace=null, outputBucketPath=testAutomation, pathFormat=testAutomation/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_, fullOutputPath=testAutomation/invalid_emails/2023_03_20_1679296343675_, syncMode=overwrite} 2023-03-20 07:12:28 destination > 2023-03-20 07:12:24 INFO i.a.i.d.s.S3ConsumerFactory(lambda$toWriteConfig$0):83 - Write config: WriteConfig{streamName=lists, namespace=null, outputBucketPath=testAutomation, pathFormat=testAutomation/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_, fullOutputPath=testAutomation/lists/2023_03_20_1679296343675_, syncMode=overwrite} 2023-03-20 07:12:28 destination > 2023-03-20 07:12:24 INFO i.a.i.d.s.S3ConsumerFactory(lambda$toWriteConfig$0):83 - Write config: WriteConfig{streamName=messages, namespace=null, outputBucketPath=testAutomation, pathFormat=testAutomation/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_, fullOutputPath=testAutomation/messages/2023_03_20_1679296343675_, syncMode=overwrite} 2023-03-20 07:12:28 destination > 2023-03-20 07:12:24 INFO i.a.i.d.s.S3ConsumerFactory(lambda$toWriteConfig$0):83 - Write config: WriteConfig{streamName=segments, namespace=null, outputBucketPath=testAutomation, pathFormat=testAutomation/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_, fullOutputPath=testAutomation/segments/2023_03_20_1679296343675_, syncMode=overwrite} 2023-03-20 07:12:28 destination > 2023-03-20 07:12:24 INFO i.a.i.d.s.S3ConsumerFactory(lambda$toWriteConfig$0):83 - Write config: WriteConfig{streamName=single_sends, namespace=null, outputBucketPath=testAutomation, pathFormat=testAutomation/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_, fullOutputPath=testAutomation/single_sends/2023_03_20_1679296343675_, syncMode=overwrite} 2023-03-20 07:12:28 destination > 2023-03-20 07:12:24 INFO i.a.i.d.s.S3ConsumerFactory(lambda$toWriteConfig$0):83 - Write config: WriteConfig{streamName=spam_reports, namespace=null, outputBucketPath=testAutomation, pathFormat=testAutomation/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_, fullOutputPath=testAutomation/spam_reports/2023_03_20_1679296343675_, syncMode=overwrite} 2023-03-20 07:12:28 destination > 2023-03-20 07:12:24 INFO i.a.i.d.s.S3ConsumerFactory(lambda$toWriteConfig$0):83 - Write config: WriteConfig{streamName=stats_automations, namespace=null, outputBucketPath=testAutomation, pathFormat=testAutomation/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_, fullOutputPath=testAutomation/stats_automations/2023_03_20_1679296343675_, syncMode=overwrite} 2023-03-20 07:12:28 destination > 2023-03-20 07:12:24 INFO i.a.i.d.s.S3ConsumerFactory(lambda$toWriteConfig$0):83 - Write config: WriteConfig{streamName=suppression_group_members, namespace=null, outputBucketPath=testAutomation, pathFormat=testAutomation/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_, fullOutputPath=testAutomation/suppression_group_members/2023_03_20_1679296343675_, syncMode=overwrite} 2023-03-20 07:12:28 destination > 2023-03-20 07:12:24 INFO i.a.i.d.s.S3ConsumerFactory(lambda$toWriteConfig$0):83 - Write config: WriteConfig{streamName=suppression_groups, namespace=null, outputBucketPath=testAutomation, pathFormat=testAutomation/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_, fullOutputPath=testAutomation/suppression_groups/2023_03_20_1679296343675_, syncMode=overwrite} 2023-03-20 07:12:28 destination > 2023-03-20 07:12:24 INFO i.a.i.d.s.S3ConsumerFactory(lambda$toWriteConfig$0):83 - Write config: WriteConfig{streamName=templates, namespace=null, outputBucketPath=testAutomation, pathFormat=testAutomation/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_, fullOutputPath=testAutomation/templates/2023_03_20_1679296343675_, syncMode=overwrite} 2023-03-20 07:12:28 destination > 2023-03-20 07:12:24 INFO i.a.i.d.b.BufferedStreamConsumer(startTracked):116 - class io.airbyte.integrations.destination.buffered_stream_consumer.BufferedStreamConsumer started. 2023-03-20 07:12:28 destination > 2023-03-20 07:12:24 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):90 - Preparing bucket in destination started for 15 streams 2023-03-20 07:12:28 destination > 2023-03-20 07:12:24 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):97 - Clearing storage area in destination started for namespace null stream blocks bucketObject testAutomation pathFormat testAutomation/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_ 2023-03-20 07:12:28 destination > 2023-03-20 07:12:24 INFO i.a.i.d.s.S3StorageOperations(cleanUpBucketObject):250 - Storage bucket testAutomation has been cleaned-up (0 objects matching testAutomation/blocks/[0-9]{4}_[0-9]{2}_[0-9]{2}_[0-9]+_.* were deleted)... 2023-03-20 07:12:28 destination > 2023-03-20 07:12:24 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):100 - Clearing storage area in destination completed for namespace null stream blocks bucketObject testAutomation 2023-03-20 07:12:28 destination > 2023-03-20 07:12:24 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):97 - Clearing storage area in destination started for namespace null stream bounces bucketObject testAutomation pathFormat testAutomation/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_ 2023-03-20 07:12:28 destination > 2023-03-20 07:12:25 INFO i.a.i.d.s.S3StorageOperations(cleanUpBucketObject):250 - Storage bucket testAutomation has been cleaned-up (0 objects matching testAutomation/bounces/[0-9]{4}_[0-9]{2}_[0-9]{2}_[0-9]+_.* were deleted)... 2023-03-20 07:12:28 destination > 2023-03-20 07:12:25 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):100 - Clearing storage area in destination completed for namespace null stream bounces bucketObject testAutomation 2023-03-20 07:12:28 destination > 2023-03-20 07:12:25 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):97 - Clearing storage area in destination started for namespace null stream campaigns bucketObject testAutomation pathFormat testAutomation/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_ 2023-03-20 07:12:28 destination > 2023-03-20 07:12:25 INFO i.a.i.d.s.S3StorageOperations(cleanUpObjects):301 - Deleting objects testAutomation/campaigns/2023_03_20_1679296315680_0.csv 2023-03-20 07:12:28 destination > 2023-03-20 07:12:25 INFO i.a.i.d.s.S3StorageOperations(cleanUpBucketObject):250 - Storage bucket testAutomation has been cleaned-up (1 objects matching testAutomation/campaigns/[0-9]{4}_[0-9]{2}_[0-9]{2}_[0-9]+_.* were deleted)... 2023-03-20 07:12:28 destination > 2023-03-20 07:12:25 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):100 - Clearing storage area in destination completed for namespace null stream campaigns bucketObject testAutomation 2023-03-20 07:12:28 destination > 2023-03-20 07:12:25 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):97 - Clearing storage area in destination started for namespace null stream contacts bucketObject testAutomation pathFormat testAutomation/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_ 2023-03-20 07:12:28 destination > 2023-03-20 07:12:25 INFO i.a.i.d.s.S3StorageOperations(cleanUpObjects):301 - Deleting objects testAutomation/contacts/2023_03_20_1679296315680_0.csv 2023-03-20 07:12:28 destination > 2023-03-20 07:12:25 INFO i.a.i.d.s.S3StorageOperations(cleanUpBucketObject):250 - Storage bucket testAutomation has been cleaned-up (1 objects matching testAutomation/contacts/[0-9]{4}_[0-9]{2}_[0-9]{2}_[0-9]+_.* were deleted)... 2023-03-20 07:12:28 destination > 2023-03-20 07:12:25 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):100 - Clearing storage area in destination completed for namespace null stream contacts bucketObject testAutomation 2023-03-20 07:12:28 destination > 2023-03-20 07:12:25 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):97 - Clearing storage area in destination started for namespace null stream global_suppressions bucketObject testAutomation pathFormat testAutomation/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_ 2023-03-20 07:12:28 destination > 2023-03-20 07:12:25 INFO i.a.i.d.s.S3StorageOperations(cleanUpObjects):301 - Deleting objects testAutomation/global_suppressions/2023_03_20_1679296315680_0.csv 2023-03-20 07:12:28 destination > 2023-03-20 07:12:25 INFO i.a.i.d.s.S3StorageOperations(cleanUpBucketObject):250 - Storage bucket testAutomation has been cleaned-up (1 objects matching testAutomation/global_suppressions/[0-9]{4}_[0-9]{2}_[0-9]{2}_[0-9]+_.* were deleted)... 2023-03-20 07:12:28 destination > 2023-03-20 07:12:25 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):100 - Clearing storage area in destination completed for namespace null stream global_suppressions bucketObject testAutomation 2023-03-20 07:12:28 destination > 2023-03-20 07:12:25 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):97 - Clearing storage area in destination started for namespace null stream invalid_emails bucketObject testAutomation pathFormat testAutomation/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_ 2023-03-20 07:12:28 destination > 2023-03-20 07:12:25 INFO i.a.i.d.s.S3StorageOperations(cleanUpBucketObject):250 - Storage bucket testAutomation has been cleaned-up (0 objects matching testAutomation/invalid_emails/[0-9]{4}_[0-9]{2}_[0-9]{2}_[0-9]+_.* were deleted)... 2023-03-20 07:12:28 destination > 2023-03-20 07:12:25 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):100 - Clearing storage area in destination completed for namespace null stream invalid_emails bucketObject testAutomation 2023-03-20 07:12:28 destination > 2023-03-20 07:12:25 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):97 - Clearing storage area in destination started for namespace null stream lists bucketObject testAutomation pathFormat testAutomation/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_ 2023-03-20 07:12:28 destination > 2023-03-20 07:12:25 INFO i.a.i.d.s.S3StorageOperations(cleanUpObjects):301 - Deleting objects testAutomation/lists/2023_03_20_1679296315680_0.csv 2023-03-20 07:12:28 destination > 2023-03-20 07:12:25 INFO i.a.i.d.s.S3StorageOperations(cleanUpBucketObject):250 - Storage bucket testAutomation has been cleaned-up (1 objects matching testAutomation/lists/[0-9]{4}_[0-9]{2}_[0-9]{2}_[0-9]+_.* were deleted)... 2023-03-20 07:12:28 destination > 2023-03-20 07:12:25 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):100 - Clearing storage area in destination completed for namespace null stream lists bucketObject testAutomation 2023-03-20 07:12:28 destination > 2023-03-20 07:12:25 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):97 - Clearing storage area in destination started for namespace null stream messages bucketObject testAutomation pathFormat testAutomation/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_ 2023-03-20 07:12:28 destination > 2023-03-20 07:12:26 INFO i.a.i.d.s.S3StorageOperations(cleanUpBucketObject):250 - Storage bucket testAutomation has been cleaned-up (0 objects matching testAutomation/messages/[0-9]{4}_[0-9]{2}_[0-9]{2}_[0-9]+_.* were deleted)... 2023-03-20 07:12:28 destination > 2023-03-20 07:12:26 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):100 - Clearing storage area in destination completed for namespace null stream messages bucketObject testAutomation 2023-03-20 07:12:28 destination > 2023-03-20 07:12:26 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):97 - Clearing storage area in destination started for namespace null stream segments bucketObject testAutomation pathFormat testAutomation/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_ 2023-03-20 07:12:28 destination > 2023-03-20 07:12:26 INFO i.a.i.d.s.S3StorageOperations(cleanUpBucketObject):250 - Storage bucket testAutomation has been cleaned-up (0 objects matching testAutomation/segments/[0-9]{4}_[0-9]{2}_[0-9]{2}_[0-9]+_.* were deleted)... 2023-03-20 07:12:28 destination > 2023-03-20 07:12:26 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):100 - Clearing storage area in destination completed for namespace null stream segments bucketObject testAutomation 2023-03-20 07:12:28 destination > 2023-03-20 07:12:26 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):97 - Clearing storage area in destination started for namespace null stream single_sends bucketObject testAutomation pathFormat testAutomation/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_ 2023-03-20 07:12:28 destination > 2023-03-20 07:12:26 INFO i.a.i.d.s.S3StorageOperations(cleanUpBucketObject):250 - Storage bucket testAutomation has been cleaned-up (0 objects matching testAutomation/single_sends/[0-9]{4}_[0-9]{2}_[0-9]{2}_[0-9]+_.* were deleted)... 2023-03-20 07:12:28 destination > 2023-03-20 07:12:26 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):100 - Clearing storage area in destination completed for namespace null stream single_sends bucketObject testAutomation 2023-03-20 07:12:28 destination > 2023-03-20 07:12:26 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):97 - Clearing storage area in destination started for namespace null stream spam_reports bucketObject testAutomation pathFormat testAutomation/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_ 2023-03-20 07:12:28 destination > 2023-03-20 07:12:26 INFO i.a.i.d.s.S3StorageOperations(cleanUpBucketObject):250 - Storage bucket testAutomation has been cleaned-up (0 objects matching testAutomation/spam_reports/[0-9]{4}_[0-9]{2}_[0-9]{2}_[0-9]+_.* were deleted)... 2023-03-20 07:12:28 destination > 2023-03-20 07:12:26 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):100 - Clearing storage area in destination completed for namespace null stream spam_reports bucketObject testAutomation 2023-03-20 07:12:28 destination > 2023-03-20 07:12:26 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):97 - Clearing storage area in destination started for namespace null stream stats_automations bucketObject testAutomation pathFormat testAutomation/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_ 2023-03-20 07:12:28 destination > 2023-03-20 07:12:26 INFO i.a.i.d.s.S3StorageOperations(cleanUpBucketObject):250 - Storage bucket testAutomation has been cleaned-up (0 objects matching testAutomation/stats_automations/[0-9]{4}_[0-9]{2}_[0-9]{2}_[0-9]+_.* were deleted)... 2023-03-20 07:12:28 destination > 2023-03-20 07:12:26 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):100 - Clearing storage area in destination completed for namespace null stream stats_automations bucketObject testAutomation 2023-03-20 07:12:28 destination > 2023-03-20 07:12:26 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):97 - Clearing storage area in destination started for namespace null stream suppression_group_members bucketObject testAutomation pathFormat testAutomation/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_ 2023-03-20 07:12:28 destination > 2023-03-20 07:12:26 INFO i.a.i.d.s.S3StorageOperations(cleanUpBucketObject):250 - Storage bucket testAutomation has been cleaned-up (0 objects matching testAutomation/suppression_group_members/[0-9]{4}_[0-9]{2}_[0-9]{2}_[0-9]+_.* were deleted)... 2023-03-20 07:12:28 destination > 2023-03-20 07:12:26 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):100 - Clearing storage area in destination completed for namespace null stream suppression_group_members bucketObject testAutomation 2023-03-20 07:12:28 destination > 2023-03-20 07:12:26 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):97 - Clearing storage area in destination started for namespace null stream suppression_groups bucketObject testAutomation pathFormat testAutomation/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_ 2023-03-20 07:12:28 destination > 2023-03-20 07:12:26 INFO i.a.i.d.s.S3StorageOperations(cleanUpBucketObject):250 - Storage bucket testAutomation has been cleaned-up (0 objects matching testAutomation/suppression_groups/[0-9]{4}_[0-9]{2}_[0-9]{2}_[0-9]+_.* were deleted)... 2023-03-20 07:12:28 destination > 2023-03-20 07:12:26 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):100 - Clearing storage area in destination completed for namespace null stream suppression_groups bucketObject testAutomation 2023-03-20 07:12:28 destination > 2023-03-20 07:12:26 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):97 - Clearing storage area in destination started for namespace null stream templates bucketObject testAutomation pathFormat testAutomation/${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_ 2023-03-20 07:12:28 destination > 2023-03-20 07:12:26 INFO i.a.i.d.s.S3StorageOperations(cleanUpBucketObject):250 - Storage bucket testAutomation has been cleaned-up (0 objects matching testAutomation/templates/[0-9]{4}_[0-9]{2}_[0-9]{2}_[0-9]+_.* were deleted)... 2023-03-20 07:12:28 destination > 2023-03-20 07:12:26 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):100 - Clearing storage area in destination completed for namespace null stream templates bucketObject testAutomation 2023-03-20 07:12:28 destination > 2023-03-20 07:12:26 INFO i.a.i.d.s.S3ConsumerFactory(lambda$onStartFunction$1):104 - Preparing storage area in destination completed. 2023-03-20 07:12:05 INFO i.a.w.t.TemporalAttemptExecution(get):108 - Cloud storage job log path: /workspace/414/2/logs.log 2023-03-20 07:12:05 INFO i.a.w.t.TemporalAttemptExecution(get):111 - Executing worker wrapper. Airbyte version: 0.40.6 2023-03-20 07:12:05 INFO i.a.c.i.LineGobbler(voidCall):114 - 2023-03-20 07:12:05 INFO i.a.w.p.KubeProcessFactory(create):100 - Attempting to start pod = source-sendgrid-check-414-2-ihmlv for airbyte/source-sendgrid:0.2.14 with resources io.airbyte.config.ResourceRequirements@4ffb823e[cpuRequest=,cpuLimit=,memoryRequest=,memoryLimit=] 2023-03-20 07:12:05 INFO i.a.c.i.LineGobbler(voidCall):114 - ----- START CHECK ----- 2023-03-20 07:12:05 INFO i.a.c.i.LineGobbler(voidCall):114 - 2023-03-20 07:12:05 INFO i.a.w.p.KubeProcessFactory(create):103 - source-sendgrid-check-414-2-ihmlv stdoutLocalPort = 9026 2023-03-20 07:12:05 INFO i.a.w.p.KubeProcessFactory(create):106 - source-sendgrid-check-414-2-ihmlv stderrLocalPort = 9027 2023-03-20 07:12:05 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$9):589 - Creating stdout socket server... 2023-03-20 07:12:05 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$10):607 - Creating stderr socket server... 2023-03-20 07:12:05 INFO i.a.w.p.KubePodProcess():520 - Creating pod source-sendgrid-check-414-2-ihmlv... 2023-03-20 07:12:06 INFO i.a.w.p.KubePodProcess(waitForInitPodToRun):309 - Waiting for init container to be ready before copying files... 2023-03-20 07:12:06 INFO i.a.w.p.KubePodProcess(waitForInitPodToRun):313 - Init container present.. 2023-03-20 07:12:07 INFO i.a.w.p.KubePodProcess(waitForInitPodToRun):316 - Init container ready.. 2023-03-20 07:12:07 INFO i.a.w.p.KubePodProcess():551 - Copying files... 2023-03-20 07:12:07 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):258 - Uploading file: source_config.json 2023-03-20 07:12:07 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):266 - kubectl cp /tmp/7c9407fa-ebcf-4d91-a0cd-750f2c378594/source_config.json dev/source-sendgrid-check-414-2-ihmlv:/config/source_config.json -c init 2023-03-20 07:12:07 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):269 - Waiting for kubectl cp to complete 2023-03-20 07:12:07 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):283 - kubectl cp complete, closing process 2023-03-20 07:12:07 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):258 - Uploading file: FINISHED_UPLOADING 2023-03-20 07:12:07 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):266 - kubectl cp /tmp/4c96c329-4c6e-42fb-b946-4c26573851c2/FINISHED_UPLOADING dev/source-sendgrid-check-414-2-ihmlv:/config/FINISHED_UPLOADING -c init 2023-03-20 07:12:07 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):269 - Waiting for kubectl cp to complete 2023-03-20 07:12:07 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):283 - kubectl cp complete, closing process 2023-03-20 07:12:07 INFO i.a.w.p.KubePodProcess():554 - Waiting until pod is ready... 2023-03-20 07:12:08 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$9):598 - Setting stdout... 2023-03-20 07:12:08 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$10):610 - Setting stderr... 2023-03-20 07:12:09 INFO i.a.w.p.KubePodProcess():570 - Reading pod IP... 2023-03-20 07:12:09 INFO i.a.w.p.KubePodProcess():572 - Pod IP: 12.0.4.71 2023-03-20 07:12:09 INFO i.a.w.p.KubePodProcess():579 - Using null stdin output stream... 2023-03-20 07:12:09 INFO i.a.w.i.DefaultAirbyteStreamFactory(internalLog):103 - Check succeeded 2023-03-20 07:12:10 INFO i.a.w.p.KubePodProcess(close):737 - (pod: dev / source-sendgrid-check-414-2-ihmlv) - Closed all resources for pod 2023-03-20 07:12:10 INFO i.a.w.t.TemporalAttemptExecution(get):132 - Stopping cancellation check scheduling... 2023-03-20 07:12:10 INFO i.a.c.i.LineGobbler(voidCall):114 - 2023-03-20 07:12:10 INFO i.a.c.i.LineGobbler(voidCall):114 - ----- END CHECK ----- 2023-03-20 07:12:10 INFO i.a.c.i.LineGobbler(voidCall):114 - 2023-03-20 07:12:10 INFO i.a.w.t.TemporalAttemptExecution(get):108 - Cloud storage job log path: /workspace/414/2/logs.log 2023-03-20 07:12:10 INFO i.a.w.t.TemporalAttemptExecution(get):111 - Executing worker wrapper. Airbyte version: 0.40.6 2023-03-20 07:12:10 INFO i.a.w.p.KubeProcessFactory(create):100 - Attempting to start pod = destination-s3-check-414-2-vdgpl for airbyte/destination-s3:0.3.15 with resources io.airbyte.config.ResourceRequirements@4ffb823e[cpuRequest=,cpuLimit=,memoryRequest=,memoryLimit=] 2023-03-20 07:12:10 INFO i.a.c.i.LineGobbler(voidCall):114 - 2023-03-20 07:12:10 INFO i.a.w.p.KubeProcessFactory(create):103 - destination-s3-check-414-2-vdgpl stdoutLocalPort = 9038 2023-03-20 07:12:10 INFO i.a.c.i.LineGobbler(voidCall):114 - ----- START CHECK ----- 2023-03-20 07:12:10 INFO i.a.w.p.KubeProcessFactory(create):106 - destination-s3-check-414-2-vdgpl stderrLocalPort = 9039 2023-03-20 07:12:10 INFO i.a.c.i.LineGobbler(voidCall):114 - 2023-03-20 07:12:10 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$9):589 - Creating stdout socket server... 2023-03-20 07:12:10 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$10):607 - Creating stderr socket server... 2023-03-20 07:12:10 INFO i.a.w.p.KubePodProcess():520 - Creating pod destination-s3-check-414-2-vdgpl... 2023-03-20 07:12:10 INFO i.a.w.p.KubePodProcess(waitForInitPodToRun):309 - Waiting for init container to be ready before copying files... 2023-03-20 07:12:10 INFO i.a.w.p.KubePodProcess(waitForInitPodToRun):313 - Init container present.. 2023-03-20 07:12:11 INFO i.a.w.p.KubePodProcess(waitForInitPodToRun):316 - Init container ready.. 2023-03-20 07:12:11 INFO i.a.w.p.KubePodProcess():551 - Copying files... 2023-03-20 07:12:11 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):258 - Uploading file: source_config.json 2023-03-20 07:12:11 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):266 - kubectl cp /tmp/190d9590-fcb6-4ad3-8bcc-a3051f1cd9c0/source_config.json dev/destination-s3-check-414-2-vdgpl:/config/source_config.json -c init 2023-03-20 07:12:11 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):269 - Waiting for kubectl cp to complete 2023-03-20 07:12:12 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):283 - kubectl cp complete, closing process 2023-03-20 07:12:12 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):258 - Uploading file: FINISHED_UPLOADING 2023-03-20 07:12:12 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):266 - kubectl cp /tmp/b4b104f7-1440-4743-a8eb-0f92e7d27079/FINISHED_UPLOADING dev/destination-s3-check-414-2-vdgpl:/config/FINISHED_UPLOADING -c init 2023-03-20 07:12:12 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):269 - Waiting for kubectl cp to complete 2023-03-20 07:12:12 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):283 - kubectl cp complete, closing process 2023-03-20 07:12:12 INFO i.a.w.p.KubePodProcess():554 - Waiting until pod is ready... 2023-03-20 07:12:13 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$9):598 - Setting stdout... 2023-03-20 07:12:13 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$10):610 - Setting stderr... 2023-03-20 07:12:14 INFO i.a.w.p.KubePodProcess():570 - Reading pod IP... 2023-03-20 07:12:14 INFO i.a.w.p.KubePodProcess():572 - Pod IP: 12.0.3.51 2023-03-20 07:12:14 INFO i.a.w.p.KubePodProcess():579 - Using null stdin output stream... 2023-03-20 07:12:14 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2023-03-20 07:12:14 INFO i.a.i.b.IntegrationCliParser(parseOptions):118 - integration args: {check=null, config=source_config.json} 2023-03-20 07:12:14 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2023-03-20 07:12:14 INFO i.a.i.b.IntegrationRunner(runInternal):104 - Running integration: io.airbyte.integrations.destination.s3.S3Destination 2023-03-20 07:12:14 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2023-03-20 07:12:14 INFO i.a.i.b.IntegrationRunner(runInternal):105 - Command: CHECK 2023-03-20 07:12:14 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2023-03-20 07:12:14 INFO i.a.i.b.IntegrationRunner(runInternal):106 - Integration config: IntegrationConfig{command=CHECK, configPath='source_config.json', catalogPath='null', statePath='null'} 2023-03-20 07:12:15 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2023-03-20 07:12:15 WARN c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword airbyte_secret - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword 2023-03-20 07:12:15 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2023-03-20 07:12:15 WARN c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword examples - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword 2023-03-20 07:12:15 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2023-03-20 07:12:15 WARN c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword order - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword 2023-03-20 07:12:15 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2023-03-20 07:12:15 WARN c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword requires - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword 2023-03-20 07:12:15 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2023-03-20 07:12:15 INFO i.a.i.d.s.S3FormatConfigs(getS3FormatConfig):22 - S3 format config: {"flattening":"No flattening","compression":{"compression_type":"No Compression"},"format_type":"CSV"} 2023-03-20 07:12:15 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2023-03-20 07:12:15 INFO i.a.i.d.s.S3DestinationConfig(createS3Client):190 - Creating S3 client... 2023-03-20 07:12:16 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2023-03-20 07:12:16 INFO i.a.i.d.s.S3StorageOperations(createBucketObjectIfNotExists):102 - Storage Object redshift-dev-s3-bucket/testAutomation does not exist in bucket; creating... 2023-03-20 07:12:16 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2023-03-20 07:12:16 INFO i.a.i.d.s.S3StorageOperations(createBucketObjectIfNotExists):104 - Storage Object redshift-dev-s3-bucket/testAutomation has been created in bucket. 2023-03-20 07:12:16 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2023-03-20 07:12:16 INFO i.a.i.d.s.S3Destination(testIAMUserHasListObjectPermission):156 - Started testing if IAM user can call listObjects on the destination bucket 2023-03-20 07:12:16 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2023-03-20 07:12:16 INFO i.a.i.d.s.S3Destination(testIAMUserHasListObjectPermission):159 - Finished checking for listObjects permission 2023-03-20 07:12:16 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2023-03-20 07:12:16 INFO i.a.i.d.s.S3Destination(testSingleUpload):81 - Started testing if all required credentials assigned to user for single file uploading 2023-03-20 07:12:16 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2023-03-20 07:12:16 INFO i.a.i.d.s.S3Destination(testSingleUpload):91 - Finished checking for normal upload mode 2023-03-20 07:12:16 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2023-03-20 07:12:16 INFO i.a.i.d.s.S3Destination(testMultipartUpload):95 - Started testing if all required credentials assigned to user for multipart upload 2023-03-20 07:12:16 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2023-03-20 07:12:16 INFO a.m.s.StreamTransferManager(getMultiPartOutputStreams):329 - Initiated multipart upload to redshift-dev-s3-bucket/testAutomation/test_1679296336424 with full ID RfeC0aFDZ8hD9sXV0sXph0H9hPzMdAbUiQrokWVll_7aiwDrqP.NUOk5kCPSu0xGp74T73TAaJY7NXbnc.QplzCO5jCgBizS1GbDMn6vjUD2S41VVNu0E55UazBt0F8x 2023-03-20 07:12:16 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2023-03-20 07:12:16 INFO a.m.s.MultiPartOutputStream(close):158 - Called close() on [MultipartOutputStream for parts 1 - 10000] 2023-03-20 07:12:16 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2023-03-20 07:12:16 INFO a.m.s.MultiPartOutputStream(close):158 - Called close() on [MultipartOutputStream for parts 1 - 10000] 2023-03-20 07:12:16 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2023-03-20 07:12:16 WARN a.m.s.MultiPartOutputStream(close):160 - [MultipartOutputStream for parts 1 - 10000] is already closed 2023-03-20 07:12:16 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2023-03-20 07:12:16 INFO a.m.s.StreamTransferManager(complete):367 - [Manager uploading to redshift-dev-s3-bucket/testAutomation/test_1679296336424 with id RfeC0aFDZ...UazBt0F8x]: Uploading leftover stream [Part number 1 containing 3.34 MB] 2023-03-20 07:12:16 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2023-03-20 07:12:16 INFO a.m.s.StreamTransferManager(uploadStreamPart):558 - [Manager uploading to redshift-dev-s3-bucket/testAutomation/test_1679296336424 with id RfeC0aFDZ...UazBt0F8x]: Finished uploading [Part number 1 containing 3.34 MB] 2023-03-20 07:12:16 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2023-03-20 07:12:16 INFO a.m.s.StreamTransferManager(complete):395 - [Manager uploading to redshift-dev-s3-bucket/testAutomation/test_1679296336424 with id RfeC0aFDZ...UazBt0F8x]: Completed 2023-03-20 07:12:16 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2023-03-20 07:12:16 INFO i.a.i.d.s.S3Destination(testMultipartUpload):119 - Finished verification for multipart upload mode 2023-03-20 07:12:16 INFO i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):62 - 2023-03-20 07:12:16 INFO i.a.i.b.IntegrationRunner(runInternal):152 - Completed integration: io.airbyte.integrations.destination.s3.S3Destination 2023-03-20 07:12:17 INFO i.a.w.p.KubePodProcess(close):737 - (pod: dev / destination-s3-check-414-2-vdgpl) - Closed all resources for pod 2023-03-20 07:12:17 INFO i.a.w.t.TemporalAttemptExecution(get):132 - Stopping cancellation check scheduling... 2023-03-20 07:12:17 INFO i.a.c.i.LineGobbler(voidCall):114 - 2023-03-20 07:12:17 INFO i.a.c.i.LineGobbler(voidCall):114 - ----- END CHECK ----- 2023-03-20 07:12:17 INFO i.a.c.i.LineGobbler(voidCall):114 - 2023-03-20 07:12:17 INFO i.a.w.t.TemporalAttemptExecution(get):108 - Cloud storage job log path: /workspace/414/2/logs.log 2023-03-20 07:12:17 INFO i.a.w.t.TemporalAttemptExecution(get):111 - Executing worker wrapper. Airbyte version: 0.40.6 2023-03-20 07:12:17 INFO i.a.c.f.EnvVariableFeatureFlags(getEnvOrDefault):49 - Using default value for environment variable LOG_CONNECTOR_MESSAGES: 'false' 2023-03-20 07:12:17 INFO i.a.c.EnvConfigs(getEnvOrDefault):1096 - Using default value for environment variable METRIC_CLIENT: '' 2023-03-20 07:12:17 WARN i.a.m.l.MetricClientFactory(initialize):60 - Metric client is already initialized to 2023-03-20 07:12:17 INFO i.a.c.f.EnvVariableFeatureFlags(getEnvOrDefault):49 - Using default value for environment variable LOG_CONNECTOR_MESSAGES: 'false' 2023-03-20 07:12:17 INFO i.a.w.g.DefaultReplicationWorker(run):122 - start sync worker. job id: 414 attempt id: 2 2023-03-20 07:12:17 INFO i.a.w.g.DefaultReplicationWorker(run):135 - configured sync modes: {null.single_sends=full_refresh - overwrite, null.spam_reports=full_refresh - overwrite, null.messages=full_refresh - overwrite, null.segments=full_refresh - overwrite, null.suppression_groups=full_refresh - overwrite, null.invalid_emails=full_refresh - overwrite, null.suppression_group_members=full_refresh - overwrite, null.contacts=full_refresh - overwrite, null.bounces=full_refresh - overwrite, null.stats_automations=full_refresh - overwrite, null.campaigns=full_refresh - overwrite, null.lists=full_refresh - overwrite, null.global_suppressions=full_refresh - overwrite, null.templates=full_refresh - overwrite, null.blocks=full_refresh - overwrite} 2023-03-20 07:12:17 INFO i.a.w.i.DefaultAirbyteDestination(start):69 - Running destination... 2023-03-20 07:12:17 INFO i.a.w.p.KubeProcessFactory(create):100 - Attempting to start pod = destination-s3-write-414-2-xnfsj for airbyte/destination-s3:0.3.15 with resources io.airbyte.config.ResourceRequirements@3dce13e9[cpuRequest=,cpuLimit=,memoryRequest=,memoryLimit=] 2023-03-20 07:12:17 INFO i.a.w.p.KubeProcessFactory(create):103 - destination-s3-write-414-2-xnfsj stdoutLocalPort = 9032 2023-03-20 07:12:17 INFO i.a.w.p.KubeProcessFactory(create):106 - destination-s3-write-414-2-xnfsj stderrLocalPort = 9033 2023-03-20 07:12:17 INFO i.a.c.i.LineGobbler(voidCall):114 - 2023-03-20 07:12:17 INFO i.a.c.i.LineGobbler(voidCall):114 - ----- START REPLICATION ----- 2023-03-20 07:12:17 INFO i.a.c.i.LineGobbler(voidCall):114 - 2023-03-20 07:12:17 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$9):589 - Creating stdout socket server... 2023-03-20 07:12:17 INFO i.a.w.p.KubePodProcess():520 - Creating pod destination-s3-write-414-2-xnfsj... 2023-03-20 07:12:17 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$10):607 - Creating stderr socket server... 2023-03-20 07:12:17 INFO i.a.w.p.KubePodProcess(waitForInitPodToRun):309 - Waiting for init container to be ready before copying files... 2023-03-20 07:12:17 INFO i.a.w.p.KubePodProcess(waitForInitPodToRun):313 - Init container present.. 2023-03-20 07:12:19 INFO i.a.w.p.KubePodProcess(waitForInitPodToRun):316 - Init container ready.. 2023-03-20 07:12:19 INFO i.a.w.p.KubePodProcess():551 - Copying files... 2023-03-20 07:12:19 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):258 - Uploading file: destination_config.json 2023-03-20 07:12:19 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):266 - kubectl cp /tmp/217c6573-a041-47c6-9ddd-9be49f097c30/destination_config.json dev/destination-s3-write-414-2-xnfsj:/config/destination_config.json -c init 2023-03-20 07:12:19 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):269 - Waiting for kubectl cp to complete 2023-03-20 07:12:19 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):283 - kubectl cp complete, closing process 2023-03-20 07:12:19 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):258 - Uploading file: destination_catalog.json 2023-03-20 07:12:19 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):266 - kubectl cp /tmp/39f0b00c-e7ed-4d18-8b90-295616a3a7c8/destination_catalog.json dev/destination-s3-write-414-2-xnfsj:/config/destination_catalog.json -c init 2023-03-20 07:12:19 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):269 - Waiting for kubectl cp to complete 2023-03-20 07:12:19 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):283 - kubectl cp complete, closing process 2023-03-20 07:12:19 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):258 - Uploading file: FINISHED_UPLOADING 2023-03-20 07:12:19 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):266 - kubectl cp /tmp/e539e378-6ffb-4b85-baf3-5f1c9a24e3e8/FINISHED_UPLOADING dev/destination-s3-write-414-2-xnfsj:/config/FINISHED_UPLOADING -c init 2023-03-20 07:12:19 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):269 - Waiting for kubectl cp to complete 2023-03-20 07:12:20 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):283 - kubectl cp complete, closing process 2023-03-20 07:12:20 INFO i.a.w.p.KubePodProcess():554 - Waiting until pod is ready... 2023-03-20 07:12:21 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$9):598 - Setting stdout... 2023-03-20 07:12:21 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$10):610 - Setting stderr... 2023-03-20 07:12:22 INFO i.a.w.p.KubePodProcess():570 - Reading pod IP... 2023-03-20 07:12:22 INFO i.a.w.p.KubePodProcess():572 - Pod IP: 12.0.4.69 2023-03-20 07:12:22 INFO i.a.w.p.KubePodProcess():575 - Creating stdin socket... 2023-03-20 07:12:22 INFO i.a.w.p.KubeProcessFactory(create):100 - Attempting to start pod = source-sendgrid-read-414-2-qowgo for airbyte/source-sendgrid:0.2.14 with resources io.airbyte.config.ResourceRequirements@2b0fd970[cpuRequest=,cpuLimit=,memoryRequest=,memoryLimit=] 2023-03-20 07:12:22 INFO i.a.w.p.KubeProcessFactory(create):103 - source-sendgrid-read-414-2-qowgo stdoutLocalPort = 9036 2023-03-20 07:12:22 INFO i.a.w.p.KubeProcessFactory(create):106 - source-sendgrid-read-414-2-qowgo stderrLocalPort = 9037 2023-03-20 07:12:22 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$9):589 - Creating stdout socket server... 2023-03-20 07:12:22 INFO i.a.w.p.KubePodProcess():520 - Creating pod source-sendgrid-read-414-2-qowgo... 2023-03-20 07:12:22 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$10):607 - Creating stderr socket server... 2023-03-20 07:12:22 INFO i.a.w.p.KubePodProcess(waitForInitPodToRun):309 - Waiting for init container to be ready before copying files... 2023-03-20 07:12:23 INFO i.a.w.p.KubePodProcess(waitForInitPodToRun):313 - Init container present.. 2023-03-20 07:12:24 INFO i.a.w.p.KubePodProcess(waitForInitPodToRun):316 - Init container ready.. 2023-03-20 07:12:25 INFO i.a.w.p.KubePodProcess():551 - Copying files... 2023-03-20 07:12:25 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):258 - Uploading file: source_config.json 2023-03-20 07:12:25 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):266 - kubectl cp /tmp/5bba07ae-fe6d-466b-84e4-405da57e665e/source_config.json dev/source-sendgrid-read-414-2-qowgo:/config/source_config.json -c init 2023-03-20 07:12:25 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):269 - Waiting for kubectl cp to complete 2023-03-20 07:12:25 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):283 - kubectl cp complete, closing process 2023-03-20 07:12:25 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):258 - Uploading file: source_catalog.json 2023-03-20 07:12:25 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):266 - kubectl cp /tmp/21e10daf-7342-42c3-a813-b0173576f858/source_catalog.json dev/source-sendgrid-read-414-2-qowgo:/config/source_catalog.json -c init 2023-03-20 07:12:25 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):269 - Waiting for kubectl cp to complete 2023-03-20 07:12:26 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):283 - kubectl cp complete, closing process 2023-03-20 07:12:26 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):258 - Uploading file: FINISHED_UPLOADING 2023-03-20 07:12:26 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):266 - kubectl cp /tmp/7d576106-a16f-401c-9fbe-e5e4cc50c7f8/FINISHED_UPLOADING dev/source-sendgrid-read-414-2-qowgo:/config/FINISHED_UPLOADING -c init 2023-03-20 07:12:26 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):269 - Waiting for kubectl cp to complete 2023-03-20 07:12:26 INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):283 - kubectl cp complete, closing process 2023-03-20 07:12:26 INFO i.a.w.p.KubePodProcess():554 - Waiting until pod is ready... 2023-03-20 07:12:27 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$9):598 - Setting stdout... 2023-03-20 07:12:27 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$10):610 - Setting stderr... 2023-03-20 07:12:28 INFO i.a.w.p.KubePodProcess():570 - Reading pod IP... 2023-03-20 07:12:28 INFO i.a.w.p.KubePodProcess():572 - Pod IP: 12.0.3.113 2023-03-20 07:12:28 INFO i.a.w.p.KubePodProcess():579 - Using null stdin output stream... 2023-03-20 07:12:28 INFO i.a.w.g.DefaultReplicationWorker(run):177 - Waiting for source and destination threads to complete. 2023-03-20 07:12:28 INFO i.a.w.g.DefaultReplicationWorker(lambda$getDestinationOutputRunnable$7):435 - Destination output thread started. 2023-03-20 07:12:28 INFO i.a.w.g.DefaultReplicationWorker(lambda$getReplicationRunnable$6):322 - Replication thread started. 2023-03-20 07:12:31 INFO i.a.w.g.DefaultReplicationWorker(lambda$getReplicationRunnable$6):355 - Source has no more messages, closing connection. 2023-03-20 07:12:32 INFO i.a.w.p.KubePodProcess(close):737 - (pod: dev / source-sendgrid-read-414-2-qowgo) - Closed all resources for pod 2023-03-20 07:12:34 INFO i.a.w.p.KubePodProcess(close):737 - (pod: dev / destination-s3-write-414-2-xnfsj) - Closed all resources for pod 2023-03-20 07:12:34 ERROR i.a.w.g.DefaultReplicationWorker(run):188 - Sync worker failed. java.util.concurrent.ExecutionException: io.airbyte.workers.general.DefaultReplicationWorker$SourceException: Source cannot be stopped! at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) ~[?:?] at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) ~[?:?] at io.airbyte.workers.general.DefaultReplicationWorker.run(DefaultReplicationWorker.java:181) ~[io.airbyte-airbyte-workers-0.40.6.jar:?] at io.airbyte.workers.general.DefaultReplicationWorker.run(DefaultReplicationWorker.java:68) ~[io.airbyte-airbyte-workers-0.40.6.jar:?] at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:159) ~[io.airbyte-airbyte-workers-0.40.6.jar:?] at java.lang.Thread.run(Thread.java:1589) [?:?] Suppressed: io.airbyte.workers.exception.WorkerException: Source process exit with code 1. This warning is normal if the job was cancelled. at io.airbyte.workers.internal.DefaultAirbyteSource.close(DefaultAirbyteSource.java:140) ~[io.airbyte-airbyte-workers-0.40.6.jar:?] at io.airbyte.workers.general.DefaultReplicationWorker.run(DefaultReplicationWorker.java:145) ~[io.airbyte-airbyte-workers-0.40.6.jar:?] at io.airbyte.workers.general.DefaultReplicationWorker.run(DefaultReplicationWorker.java:68) ~[io.airbyte-airbyte-workers-0.40.6.jar:?] at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:159) ~[io.airbyte-airbyte-workers-0.40.6.jar:?] at java.lang.Thread.run(Thread.java:1589) [?:?] Caused by: io.airbyte.workers.general.DefaultReplicationWorker$SourceException: Source cannot be stopped! at io.airbyte.workers.general.DefaultReplicationWorker.lambda$getReplicationRunnable$6(DefaultReplicationWorker.java:359) ~[io.airbyte-airbyte-workers-0.40.6.jar:?] at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804) ~[?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] ... 1 more Caused by: io.airbyte.workers.exception.WorkerException: Source process exit with code 1. This warning is normal if the job was cancelled. at io.airbyte.workers.internal.DefaultAirbyteSource.close(DefaultAirbyteSource.java:140) ~[io.airbyte-airbyte-workers-0.40.6.jar:?] at io.airbyte.workers.general.DefaultReplicationWorker.lambda$getReplicationRunnable$6(DefaultReplicationWorker.java:357) ~[io.airbyte-airbyte-workers-0.40.6.jar:?] at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804) ~[?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] ... 1 more 2023-03-20 07:12:34 INFO i.a.w.g.DefaultReplicationWorker(run):281 - Source did not output any state messages 2023-03-20 07:12:34 WARN i.a.w.g.DefaultReplicationWorker(run):292 - State capture: No state retained. 2023-03-20 07:12:34 INFO i.a.w.g.DefaultReplicationWorker(run):300 - sync summary: { "status" : "failed", "recordsSynced" : 14, "bytesSynced" : 4037, "startTime" : 1679296337785, "endTime" : 1679296354282, "totalStats" : { "recordsEmitted" : 14, "bytesEmitted" : 4037, "sourceStateMessagesEmitted" : 0, "destinationStateMessagesEmitted" : 0, "recordsCommitted" : 0, "meanSecondsBeforeSourceStateMessageEmitted" : 0, "maxSecondsBeforeSourceStateMessageEmitted" : 0, "maxSecondsBetweenStateMessageEmittedandCommitted" : 0, "meanSecondsBetweenStateMessageEmittedandCommitted" : 0 }, "streamStats" : [ { "streamName" : "global_suppressions", "stats" : { "recordsEmitted" : 3, "bytesEmitted" : 151 } }, { "streamName" : "campaigns", "stats" : { "recordsEmitted" : 5, "bytesEmitted" : 1043 } }, { "streamName" : "lists", "stats" : { "recordsEmitted" : 1, "bytesEmitted" : 187 } }, { "streamName" : "contacts", "stats" : { "recordsEmitted" : 5, "bytesEmitted" : 2656 } } ] } 2023-03-20 07:12:34 INFO i.a.w.g.DefaultReplicationWorker(run):301 - failures: [ { "failureOrigin" : "source", "failureType" : "system_error", "internalMessage" : "400 Client Error: Bad Request for url: https://api.sendgrid.com/v3/messages?query=last_event_time%20BETWEEN%20TIMESTAMP%20%22None%22%20AND%20TIMESTAMP%20%222023-03-20T07%3A12%3A31Z%22&limit=1000", "externalMessage" : "authorization required", "metadata" : { "attemptNumber" : 2, "jobId" : 414, "from_trace_message" : true, "connector_command" : "read" }, "stacktrace" : "Traceback (most recent call last):\n File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py\", line 115, in read\n yield from self._read_stream(\n File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py\", line 184, in _read_stream\n for record in record_iterator:\n File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py\", line 282, in _read_full_refresh\n for record in records:\n File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py\", line 421, in read_records\n response = self._send_request(request, request_kwargs)\n File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py\", line 339, in _send_request\n return backoff_handler(user_backoff_handler)(request, request_kwargs)\n File \"/usr/local/lib/python3.9/site-packages/backoff/_sync.py\", line 105, in retry\n ret = target(*args, **kwargs)\n File \"/usr/local/lib/python3.9/site-packages/backoff/_sync.py\", line 105, in retry\n ret = target(*args, **kwargs)\n File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py\", line 306, in _send\n raise exc\n File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py\", line 303, in _send\n response.raise_for_status()\n File \"/usr/local/lib/python3.9/site-packages/requests/models.py\", line 1021, in raise_for_status\n raise HTTPError(http_error_msg, response=self)\nrequests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://api.sendgrid.com/v3/messages?query=last_event_time%20BETWEEN%20TIMESTAMP%20%22None%22%20AND%20TIMESTAMP%20%222023-03-20T07%3A12%3A31Z%22&limit=1000\n", "timestamp" : 1679296351266 }, { "failureOrigin" : "source", "internalMessage" : "io.airbyte.workers.general.DefaultReplicationWorker$SourceException: Source cannot be stopped!", "externalMessage" : "Something went wrong within the source connector", "metadata" : { "attemptNumber" : 2, "jobId" : 414, "connector_command" : "read" }, "stacktrace" : "java.util.concurrent.CompletionException: io.airbyte.workers.general.DefaultReplicationWorker$SourceException: Source cannot be stopped!\n\tat java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315)\n\tat java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:320)\n\tat java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1807)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1589)\nCaused by: io.airbyte.workers.general.DefaultReplicationWorker$SourceException: Source cannot be stopped!\n\tat io.airbyte.workers.general.DefaultReplicationWorker.lambda$getReplicationRunnable$6(DefaultReplicationWorker.java:359)\n\tat java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)\n\t... 3 more\nCaused by: io.airbyte.workers.exception.WorkerException: Source process exit with code 1. This warning is normal if the job was cancelled.\n\tat io.airbyte.workers.internal.DefaultAirbyteSource.close(DefaultAirbyteSource.java:140)\n\tat io.airbyte.workers.general.DefaultReplicationWorker.lambda$getReplicationRunnable$6(DefaultReplicationWorker.java:357)\n\t... 4 more\n", "timestamp" : 1679296352560 } ] 2023-03-20 07:12:34 INFO i.a.w.t.TemporalAttemptExecution(get):132 - Stopping cancellation check scheduling... 2023-03-20 07:12:34 INFO i.a.w.t.s.ReplicationActivityImpl(lambda$replicate$3):166 - sync summary: io.airbyte.config.StandardSyncOutput@53807b5e[standardSyncSummary=io.airbyte.config.StandardSyncSummary@73e43cac[status=failed,recordsSynced=14,bytesSynced=4037,startTime=1679296337785,endTime=1679296354282,totalStats=io.airbyte.config.SyncStats@629eb42[recordsEmitted=14,bytesEmitted=4037,sourceStateMessagesEmitted=0,destinationStateMessagesEmitted=0,recordsCommitted=0,meanSecondsBeforeSourceStateMessageEmitted=0,maxSecondsBeforeSourceStateMessageEmitted=0,maxSecondsBetweenStateMessageEmittedandCommitted=0,meanSecondsBetweenStateMessageEmittedandCommitted=0,additionalProperties={}],streamStats=[io.airbyte.config.StreamSyncStats@235751d9[streamName=global_suppressions,stats=io.airbyte.config.SyncStats@73dd8466[recordsEmitted=3,bytesEmitted=151,sourceStateMessagesEmitted=,destinationStateMessagesEmitted=,recordsCommitted=,meanSecondsBeforeSourceStateMessageEmitted=,maxSecondsBeforeSourceStateMessageEmitted=,maxSecondsBetweenStateMessageEmittedandCommitted=,meanSecondsBetweenStateMessageEmittedandCommitted=,additionalProperties={}]], io.airbyte.config.StreamSyncStats@210ec7ed[streamName=campaigns,stats=io.airbyte.config.SyncStats@68344b75[recordsEmitted=5,bytesEmitted=1043,sourceStateMessagesEmitted=,destinationStateMessagesEmitted=,recordsCommitted=,meanSecondsBeforeSourceStateMessageEmitted=,maxSecondsBeforeSourceStateMessageEmitted=,maxSecondsBetweenStateMessageEmittedandCommitted=,meanSecondsBetweenStateMessageEmittedandCommitted=,additionalProperties={}]], io.airbyte.config.StreamSyncStats@71740633[streamName=lists,stats=io.airbyte.config.SyncStats@6e9552cf[recordsEmitted=1,bytesEmitted=187,sourceStateMessagesEmitted=,destinationStateMessagesEmitted=,recordsCommitted=,meanSecondsBeforeSourceStateMessageEmitted=,maxSecondsBeforeSourceStateMessageEmitted=,maxSecondsBetweenStateMessageEmittedandCommitted=,meanSecondsBetweenStateMessageEmittedandCommitted=,additionalProperties={}]], io.airbyte.config.StreamSyncStats@39f3d15f[streamName=contacts,stats=io.airbyte.config.SyncStats@73564af7[recordsEmitted=5,bytesEmitted=2656,sourceStateMessagesEmitted=,destinationStateMessagesEmitted=,recordsCommitted=,meanSecondsBeforeSourceStateMessageEmitted=,maxSecondsBeforeSourceStateMessageEmitted=,maxSecondsBetweenStateMessageEmittedandCommitted=,meanSecondsBetweenStateMessageEmittedandCommitted=,additionalProperties={}]]]],normalizationSummary=,state=,outputCatalog=io.airbyte.protocol.models.ConfiguredAirbyteCatalog@22429f30[streams=[io.airbyte.protocol.models.ConfiguredAirbyteStream@6605b29d[stream=io.airbyte.protocol.models.AirbyteStream@415bee2b[name=blocks,jsonSchema={"type":"object","$schema":"http://json-schema.org/draft-07/schema#","properties":{"email":{"type":"string"},"reason":{"type":"string"},"status":{"type":"string"},"created":{"type":"integer"}}},supportedSyncModes=[full_refresh, incremental],sourceDefinedCursor=true,defaultCursorField=[created],sourceDefinedPrimaryKey=[[email]],namespace=,additionalProperties={}],syncMode=full_refresh,cursorField=[created],destinationSyncMode=overwrite,primaryKey=[[email]],additionalProperties={}], io.airbyte.protocol.models.ConfiguredAirbyteStream@9b1505[stream=io.airbyte.protocol.models.AirbyteStream@3616a139[name=bounces,jsonSchema={"type":"object","$schema":"http://json-schema.org/draft-07/schema#","properties":{"email":{"type":"string"},"reason":{"type":"string"},"status":{"type":"string"},"created":{"type":"integer"}}},supportedSyncModes=[full_refresh, incremental],sourceDefinedCursor=true,defaultCursorField=[created],sourceDefinedPrimaryKey=[[email]],namespace=,additionalProperties={}],syncMode=full_refresh,cursorField=[created],destinationSyncMode=overwrite,primaryKey=[[email]],additionalProperties={}], io.airbyte.protocol.models.ConfiguredAirbyteStream@7dcc7c3c[stream=io.airbyte.protocol.models.AirbyteStream@6378ff18[name=campaigns,jsonSchema={"type":"object","$schema":"http://json-schema.org/draft-07/schema#","properties":{"id":{"type":"string"},"name":{"type":"string"},"status":{"type":"string"},"is_abtest":{"type":"boolean"},"created_at":{"type":"string"},"updated_at":{"type":"string"}}},supportedSyncModes=[full_refresh],sourceDefinedCursor=,defaultCursorField=[],sourceDefinedPrimaryKey=[[id]],namespace=,additionalProperties={}],syncMode=full_refresh,cursorField=[],destinationSyncMode=overwrite,primaryKey=[[id]],additionalProperties={}], io.airbyte.protocol.models.ConfiguredAirbyteStream@3b13fd4f[stream=io.airbyte.protocol.models.AirbyteStream@380f8c7d[name=contacts,jsonSchema={"type":"object","$schema":"http://json-schema.org/draft-07/schema#","properties":{"id":{"type":"string"},"city":{"type":"string"},"line":{"type":"string"},"email":{"type":"string"},"country":{"type":"string"},"facebook":{"type":"string"},"list_ids":{"type":["null","array"]},"whatsapp":{"type":"string"},"_metadata":{"type":"object","properties":{"self":{"type":"string"}}},"last_name":{"type":"string"},"created_at":{"type":"string"},"first_name":{"type":"string"},"updated_at":{"type":"string"},"postal_code":{"type":"string"},"unique_name":{"type":"string"},"phone_number":{"type":"string"},"custom_fields":{"type":"object"},"address_line_1":{"type":"string"},"address_line_2":{"type":"string"},"alternate_emails":{"type":["null","array"]},"state_province_region":{"type":"string"}}},supportedSyncModes=[full_refresh],sourceDefinedCursor=,defaultCursorField=[],sourceDefinedPrimaryKey=[[id]],namespace=,additionalProperties={}],syncMode=full_refresh,cursorField=[],destinationSyncMode=overwrite,primaryKey=[[id]],additionalProperties={}], io.airbyte.protocol.models.ConfiguredAirbyteStream@26bc3adc[stream=io.airbyte.protocol.models.AirbyteStream@298912c[name=global_suppressions,jsonSchema={"type":"object","$schema":"http://json-schema.org/draft-07/schema#","properties":{"email":{"type":"string"},"created":{"type":"integer"}}},supportedSyncModes=[full_refresh, incremental],sourceDefinedCursor=true,defaultCursorField=[created],sourceDefinedPrimaryKey=[[email]],namespace=,additionalProperties={}],syncMode=full_refresh,cursorField=[created],destinationSyncMode=overwrite,primaryKey=[[email]],additionalProperties={}], io.airbyte.protocol.models.ConfiguredAirbyteStream@26c5edb5[stream=io.airbyte.protocol.models.AirbyteStream@62edf174[name=invalid_emails,jsonSchema={"type":"object","$schema":"http://json-schema.org/draft-07/schema#","properties":{"email":{"type":"string"},"reason":{"type":"string"},"created":{"type":"integer"}}},supportedSyncModes=[full_refresh, incremental],sourceDefinedCursor=true,defaultCursorField=[created],sourceDefinedPrimaryKey=[[email]],namespace=,additionalProperties={}],syncMode=full_refresh,cursorField=[created],destinationSyncMode=overwrite,primaryKey=[[email]],additionalProperties={}], io.airbyte.protocol.models.ConfiguredAirbyteStream@25c05d2f[stream=io.airbyte.protocol.models.AirbyteStream@6317a94a[name=lists,jsonSchema={"type":"object","$schema":"http://json-schema.org/draft-07/schema#","properties":{"id":{"type":"string"},"name":{"type":"string"},"_metadata":{"type":"object","properties":{"self":{"type":"string"}}},"contact_count":{"type":"integer"}}},supportedSyncModes=[full_refresh],sourceDefinedCursor=,defaultCursorField=[],sourceDefinedPrimaryKey=[[id]],namespace=,additionalProperties={}],syncMode=full_refresh,cursorField=[],destinationSyncMode=overwrite,primaryKey=[[id]],additionalProperties={}], io.airbyte.protocol.models.ConfiguredAirbyteStream@6a3648d8[stream=io.airbyte.protocol.models.AirbyteStream@67dd62bb[name=messages,jsonSchema={"type":"object","$schema":"http://json-schema.org/draft-07/schema#","properties":{"msg_id":{"type":"string"},"status":{"type":"string"},"subject":{"type":"string"},"to_email":{"type":"string"},"from_email":{"type":"string"},"opens_count":{"type":"integer"},"clicks_count":{"type":"integer"},"last_event_time":{"type":"string"}}},supportedSyncModes=[full_refresh, incremental],sourceDefinedCursor=true,defaultCursorField=[last_event_time],sourceDefinedPrimaryKey=[[msg_id]],namespace=,additionalProperties={}],syncMode=full_refresh,cursorField=[last_event_time],destinationSyncMode=overwrite,primaryKey=[[msg_id]],additionalProperties={}], io.airbyte.protocol.models.ConfiguredAirbyteStream@4053624a[stream=io.airbyte.protocol.models.AirbyteStream@1ed0def3[name=segments,jsonSchema={"type":"object","$schema":"http://json-schema.org/draft-07/schema#","properties":{"id":{"type":"string"},"name":{"type":"string"},"created_at":{"type":"string"},"updated_at":{"type":"string"},"contact_count":{"type":"integer"},"parent_list_id":{"type":["string","null"]},"sample_updated_at":{"type":"string"},"next_sample_update":{"type":"string"}}},supportedSyncModes=[full_refresh],sourceDefinedCursor=,defaultCursorField=[],sourceDefinedPrimaryKey=[[id]],namespace=,additionalProperties={}],syncMode=full_refresh,cursorField=[],destinationSyncMode=overwrite,primaryKey=[[id]],additionalProperties={}], io.airbyte.protocol.models.ConfiguredAirbyteStream@46f82579[stream=io.airbyte.protocol.models.AirbyteStream@4abc5fa7[name=single_sends,jsonSchema={"type":"object","$schema":"http://json-schema.org/draft-07/schema#","properties":{"id":{"type":"string","format":"uuid"},"stats":{"type":"object","properties":{"opens":{"type":"integer"},"clicks":{"type":"integer"},"bounces":{"type":"integer"},"requests":{"type":"integer"},"delivered":{"type":"integer"},"bounce_drops":{"type":"integer"},"spam_reports":{"type":"integer"},"unique_opens":{"type":"integer"},"unsubscribes":{"type":"integer"},"unique_clicks":{"type":"integer"},"invalid_emails":{"type":"integer"},"spam_report_drops":{"type":"integer"}}},"ab_phase":{"type":"string"},"aggregation":{"type":"string"},"ab_variation":{"type":"string"}}},supportedSyncModes=[full_refresh],sourceDefinedCursor=,defaultCursorField=[],sourceDefinedPrimaryKey=[[id]],namespace=,additionalProperties={}],syncMode=full_refresh,cursorField=[],destinationSyncMode=overwrite,primaryKey=[[id]],additionalProperties={}], io.airbyte.protocol.models.ConfiguredAirbyteStream@11a63588[stream=io.airbyte.protocol.models.AirbyteStream@9194609[name=spam_reports,jsonSchema={"type":"object","$schema":"http://json-schema.org/draft-07/schema#","properties":{"ip":{"type":"string"},"email":{"type":"string"},"created":{"type":"integer"}}},supportedSyncModes=[full_refresh, incremental],sourceDefinedCursor=true,defaultCursorField=[created],sourceDefinedPrimaryKey=[[email]],namespace=,additionalProperties={}],syncMode=full_refresh,cursorField=[created],destinationSyncMode=overwrite,primaryKey=[[email]],additionalProperties={}], io.airbyte.protocol.models.ConfiguredAirbyteStream@e4d79be[stream=io.airbyte.protocol.models.AirbyteStream@26244eb7[name=stats_automations,jsonSchema={"type":"object","$schema":"http://json-schema.org/draft-07/schema#","properties":{"id":{"type":"string"},"stats":{"type":"object","properties":{"opens":{"type":"integer"},"clicks":{"type":"integer"},"bounces":{"type":"integer"},"requests":{"type":"integer"},"delivered":{"type":"integer"},"bounce_drops":{"type":"integer"},"spam_reports":{"type":"integer"},"unique_opens":{"type":"integer"},"unsubscribes":{"type":"integer"},"unique_clicks":{"type":"integer"},"invalid_emails":{"type":"integer"},"spam_report_drops":{"type":"integer"}}},"step_id":{"type":"string"},"aggregation":{"type":"string"}}},supportedSyncModes=[full_refresh],sourceDefinedCursor=,defaultCursorField=[],sourceDefinedPrimaryKey=[[id]],namespace=,additionalProperties={}],syncMode=full_refresh,cursorField=[],destinationSyncMode=overwrite,primaryKey=[[id]],additionalProperties={}], io.airbyte.protocol.models.ConfiguredAirbyteStream@68e258e[stream=io.airbyte.protocol.models.AirbyteStream@45640258[name=suppression_group_members,jsonSchema={"type":"object","$schema":"http://json-schema.org/draft-07/schema#","properties":{"email":{"type":"string"},"group_id":{"type":"integer"},"created_at":{"type":"integer"},"group_name":{"type":"string"}}},supportedSyncModes=[full_refresh],sourceDefinedCursor=,defaultCursorField=[],sourceDefinedPrimaryKey=[[group_id]],namespace=,additionalProperties={}],syncMode=full_refresh,cursorField=[],destinationSyncMode=overwrite,primaryKey=[[group_id]],additionalProperties={}], io.airbyte.protocol.models.ConfiguredAirbyteStream@200113c3[stream=io.airbyte.protocol.models.AirbyteStream@24d4b082[name=suppression_groups,jsonSchema={"type":"object","$schema":"http://json-schema.org/draft-07/schema#","properties":{"id":{"type":"integer"},"name":{"type":"string"},"is_default":{"type":"boolean"},"description":{"type":"string"},"unsubscribes":{"type":"integer"}}},supportedSyncModes=[full_refresh],sourceDefinedCursor=,defaultCursorField=[],sourceDefinedPrimaryKey=[[id]],namespace=,additionalProperties={}],syncMode=full_refresh,cursorField=[],destinationSyncMode=overwrite,primaryKey=[[id]],additionalProperties={}], io.airbyte.protocol.models.ConfiguredAirbyteStream@530153c3[stream=io.airbyte.protocol.models.AirbyteStream@3692fd39[name=templates,jsonSchema={"type":"object","$schema":"http://json-schema.org/draft-07/schema#","properties":{"id":{"type":["string"]},"name":{"type":["null","string"]},"versions":{"type":["array"]},"generation":{"type":["null","string"]},"updated_at":{"type":["null","string"]}}},supportedSyncModes=[full_refresh],sourceDefinedCursor=,defaultCursorField=[],sourceDefinedPrimaryKey=[[id]],namespace=,additionalProperties={}],syncMode=full_refresh,cursorField=[],destinationSyncMode=overwrite,primaryKey=[[id]],additionalProperties={}]],additionalProperties={}],failures=[io.airbyte.config.FailureReason@1e256f23[failureOrigin=source,failureType=system_error,internalMessage=400 Client Error: Bad Request for url: https://api.sendgrid.com/v3/messages?query=last_event_time%20BETWEEN%20TIMESTAMP%20%22None%22%20AND%20TIMESTAMP%20%222023-03-20T07%3A12%3A31Z%22&limit=1000,externalMessage=authorization required,metadata=io.airbyte.config.Metadata@1f749cc5[additionalProperties={attemptNumber=2, jobId=414, from_trace_message=true, connector_command=read}],stacktrace=Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 115, in read yield from self._read_stream( File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 184, in _read_stream for record in record_iterator: File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 282, in _read_full_refresh for record in records: File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py", line 421, in read_records response = self._send_request(request, request_kwargs) File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py", line 339, in _send_request return backoff_handler(user_backoff_handler)(request, request_kwargs) File "/usr/local/lib/python3.9/site-packages/backoff/_sync.py", line 105, in retry ret = target(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/backoff/_sync.py", line 105, in retry ret = target(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py", line 306, in _send raise exc File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py", line 303, in _send response.raise_for_status() File "/usr/local/lib/python3.9/site-packages/requests/models.py", line 1021, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://api.sendgrid.com/v3/messages?query=last_event_time%20BETWEEN%20TIMESTAMP%20%22None%22%20AND%20TIMESTAMP%20%222023-03-20T07%3A12%3A31Z%22&limit=1000 ,retryable=,timestamp=1679296351266], io.airbyte.config.FailureReason@46324ba3[failureOrigin=source,failureType=,internalMessage=io.airbyte.workers.general.DefaultReplicationWorker$SourceException: Source cannot be stopped!,externalMessage=Something went wrong within the source connector,metadata=io.airbyte.config.Metadata@67e81d69[additionalProperties={attemptNumber=2, jobId=414, connector_command=read}],stacktrace=java.util.concurrent.CompletionException: io.airbyte.workers.general.DefaultReplicationWorker$SourceException: Source cannot be stopped! at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315) at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:320) at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1807) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) at java.base/java.lang.Thread.run(Thread.java:1589) Caused by: io.airbyte.workers.general.DefaultReplicationWorker$SourceException: Source cannot be stopped! at io.airbyte.workers.general.DefaultReplicationWorker.lambda$getReplicationRunnable$6(DefaultReplicationWorker.java:359) at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804) ... 3 more Caused by: io.airbyte.workers.exception.WorkerException: Source process exit with code 1. This warning is normal if the job was cancelled. at io.airbyte.workers.internal.DefaultAirbyteSource.close(DefaultAirbyteSource.java:140) at io.airbyte.workers.general.DefaultReplicationWorker.lambda$getReplicationRunnable$6(DefaultReplicationWorker.java:357) ... 4 more ,retryable=,timestamp=1679296352560]]] 2023-03-20 07:12:34 INFO i.a.w.t.TemporalUtils(withBackgroundHeartbeat):312 - Stopping temporal heartbeating... 2023-03-20 07:12:34 INFO i.a.c.i.LineGobbler(voidCall):114 - 2023-03-20 07:12:34 INFO i.a.c.i.LineGobbler(voidCall):114 - ----- END REPLICATION ----- 2023-03-20 07:12:34 INFO i.a.c.i.LineGobbler(voidCall):114 - 2023-03-20 07:12:29 source > Read 0 records from bounces stream 2023-03-20 07:12:29 source > Finished syncing bounces 2023-03-20 07:12:29 source > SourceSendgrid runtimes: Syncing stream blocks 0:00:00.150198 Syncing stream bounces 0:00:01.188906 2023-03-20 07:12:29 source > Syncing stream: campaigns 2023-03-20 07:12:29 source > Read 5 records from campaigns stream 2023-03-20 07:12:29 source > Finished syncing campaigns 2023-03-20 07:12:29 source > SourceSendgrid runtimes: Syncing stream blocks 0:00:00.150198 Syncing stream bounces 0:00:01.188906 Syncing stream campaigns 0:00:00.259714 2023-03-20 07:12:29 source > Syncing stream: contacts 2023-03-20 07:12:30 source > Read 5 records from contacts stream 2023-03-20 07:12:30 source > Finished syncing contacts 2023-03-20 07:12:30 source > SourceSendgrid runtimes: Syncing stream blocks 0:00:00.150198 Syncing stream bounces 0:00:01.188906 Syncing stream campaigns 0:00:00.259714 Syncing stream contacts 0:00:00.660341 2023-03-20 07:12:30 source > Syncing stream: global_suppressions 2023-03-20 07:12:30 source > Read 3 records from global_suppressions stream 2023-03-20 07:12:30 source > Finished syncing global_suppressions 2023-03-20 07:12:30 source > SourceSendgrid runtimes: Syncing stream blocks 0:00:00.150198 Syncing stream bounces 0:00:01.188906 Syncing stream campaigns 0:00:00.259714 Syncing stream contacts 0:00:00.660341 Syncing stream global_suppressions 0:00:00.140370 2023-03-20 07:12:30 source > Syncing stream: invalid_emails 2023-03-20 07:12:30 source > Read 0 records from invalid_emails stream 2023-03-20 07:12:30 source > Finished syncing invalid_emails 2023-03-20 07:12:30 source > SourceSendgrid runtimes: Syncing stream blocks 0:00:00.150198 Syncing stream bounces 0:00:01.188906 Syncing stream campaigns 0:00:00.259714 Syncing stream contacts 0:00:00.660341 Syncing stream global_suppressions 0:00:00.140370 Syncing stream invalid_emails 0:00:00.206035 2023-03-20 07:12:30 source > Syncing stream: lists 2023-03-20 07:12:31 source > Read 1 records from lists stream 2023-03-20 07:12:31 source > Finished syncing lists 2023-03-20 07:12:31 source > SourceSendgrid runtimes: Syncing stream blocks 0:00:00.150198 Syncing stream bounces 0:00:01.188906 Syncing stream campaigns 0:00:00.259714 Syncing stream contacts 0:00:00.660341 Syncing stream global_suppressions 0:00:00.140370 Syncing stream invalid_emails 0:00:00.206035 Syncing stream lists 0:00:00.501347 2023-03-20 07:12:31 source > Syncing stream: messages 2023-03-20 07:12:31 source > {"errors":[{"message":"authorization required"}]} 2023-03-20 07:12:31 source > Encountered an exception while reading stream messages Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 115, in read yield from self._read_stream( File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 184, in _read_stream for record in record_iterator: File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 282, in _read_full_refresh for record in records: File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py", line 421, in read_records response = self._send_request(request, request_kwargs) File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py", line 339, in _send_request return backoff_handler(user_backoff_handler)(request, request_kwargs) File "/usr/local/lib/python3.9/site-packages/backoff/_sync.py", line 105, in retry ret = target(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/backoff/_sync.py", line 105, in retry ret = target(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py", line 306, in _send raise exc File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py", line 303, in _send response.raise_for_status() File "/usr/local/lib/python3.9/site-packages/requests/models.py", line 1021, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://api.sendgrid.com/v3/messages?query=last_event_time%20BETWEEN%20TIMESTAMP%20%22None%22%20AND%20TIMESTAMP%20%222023-03-20T07%3A12%3A31Z%22&limit=1000 2023-03-20 07:12:31 source > Finished syncing messages 2023-03-20 07:12:31 source > SourceSendgrid runtimes: Syncing stream blocks 0:00:00.150198 Syncing stream bounces 0:00:01.188906 Syncing stream campaigns 0:00:00.259714 Syncing stream contacts 0:00:00.660341 Syncing stream global_suppressions 0:00:00.140370 Syncing stream invalid_emails 0:00:00.206035 Syncing stream lists 0:00:00.501347 Syncing stream messages 0:00:00.177079 2023-03-20 07:12:31 source > 400 Client Error: Bad Request for url: https://api.sendgrid.com/v3/messages?query=last_event_time%20BETWEEN%20TIMESTAMP%20%22None%22%20AND%20TIMESTAMP%20%222023-03-20T07%3A12%3A31Z%22&limit=1000 Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 115, in read yield from self._read_stream( File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 184, in _read_stream for record in record_iterator: File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 282, in _read_full_refresh for record in records: File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py", line 421, in read_records response = self._send_request(request, request_kwargs) File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py", line 339, in _send_request return backoff_handler(user_backoff_handler)(request, request_kwargs) File "/usr/local/lib/python3.9/site-packages/backoff/_sync.py", line 105, in retry ret = target(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/backoff/_sync.py", line 105, in retry ret = target(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py", line 306, in _send raise exc File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py", line 303, in _send response.raise_for_status() File "/usr/local/lib/python3.9/site-packages/requests/models.py", line 1021, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://api.sendgrid.com/v3/messages?query=last_event_time%20BETWEEN%20TIMESTAMP%20%22None%22%20AND%20TIMESTAMP%20%222023-03-20T07%3A12%3A31Z%22&limit=1000 The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/airbyte/integration_code/main.py", line 13, in launch(source, sys.argv[1:]) File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py", line 123, in launch for message in source_entrypoint.run(parsed_args): File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py", line 114, in run for message in generator: File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 128, in read raise AirbyteTracedException.from_exception(e, message=display_message) from e airbyte_cdk.utils.traced_exception.AirbyteTracedException: 400 Client Error: Bad Request for url: https://api.sendgrid.com/v3/messages?query=last_event_time%20BETWEEN%20TIMESTAMP%20%22None%22%20AND%20TIMESTAMP%20%222023-03-20T07%3A12%3A31Z%22&limit=1000 2023-03-20 07:12:32 destination > 2023-03-20 07:12:32 INFO i.a.i.d.r.SerializedBufferingStrategy(lambda$addRecord$0):47 - Starting a new buffer for stream campaigns (current state: 0 bytes in 0 buffers) 2023-03-20 07:12:32 destination > 2023-03-20 07:12:32 INFO i.a.i.d.r.SerializedBufferingStrategy(lambda$addRecord$0):47 - Starting a new buffer for stream contacts (current state: 0 bytes in 1 buffers) 2023-03-20 07:12:32 destination > 2023-03-20 07:12:32 INFO i.a.i.d.r.SerializedBufferingStrategy(lambda$addRecord$0):47 - Starting a new buffer for stream global_suppressions (current state: 0 bytes in 2 buffers) 2023-03-20 07:12:32 destination > 2023-03-20 07:12:32 INFO i.a.i.d.r.SerializedBufferingStrategy(lambda$addRecord$0):47 - Starting a new buffer for stream lists (current state: 0 bytes in 3 buffers) 2023-03-20 07:12:33 destination > 2023-03-20 07:12:33 INFO i.a.i.b.FailureTrackingAirbyteMessageConsumer(close):62 - Airbyte message consumer: succeeded. 2023-03-20 07:12:33 destination > 2023-03-20 07:12:33 INFO i.a.i.d.b.BufferedStreamConsumer(close):171 - executing on success close procedure. 2023-03-20 07:12:33 destination > 2023-03-20 07:12:33 INFO i.a.i.d.r.SerializedBufferingStrategy(flushAll):100 - Flushing all 4 current buffers (0 bytes in total) 2023-03-20 07:12:33 destination > 2023-03-20 07:12:33 INFO i.a.i.d.r.SerializedBufferingStrategy(flushAll):102 - Flushing buffer of stream contacts (0 bytes) 2023-03-20 07:12:33 destination > 2023-03-20 07:12:33 INFO i.a.i.d.s.S3ConsumerFactory(lambda$flushBufferFunction$2):121 - Flushing buffer for stream contacts (0 bytes) to storage 2023-03-20 07:12:33 destination > 2023-03-20 07:12:33 INFO i.a.i.d.r.BaseSerializedBuffer(flush):138 - Finished writing data to daa9ebe7-cb41-46ee-8feb-bb282becbf6913801347943086757091.csv (3 KB) 2023-03-20 07:12:33 destination > 2023-03-20 07:12:33 INFO a.m.s.StreamTransferManager(getMultiPartOutputStreams):329 - Initiated multipart upload to redshift-dev-s3-bucket/testAutomation/contacts/2023_03_20_1679296343675_0.csv with full ID PkOkZKcEB7XnyeuO42DPkpIB8mlL7.TgCWb8PWigZVWG1JSynpVziFR7xJaNJOAitaUT_IT.VGqTzcn6hROxLwTu7sk11YHbO1cYR4BA1H9AIVDy2u2wKws7f5rkFVbS 2023-03-20 07:12:33 destination > 2023-03-20 07:12:33 INFO a.m.s.MultiPartOutputStream(close):158 - Called close() on [MultipartOutputStream for parts 1 - 10000] 2023-03-20 07:12:33 destination > 2023-03-20 07:12:33 INFO a.m.s.StreamTransferManager(complete):367 - [Manager uploading to redshift-dev-s3-bucket/testAutomation/contacts/2023_03_20_1679296343675_0.csv with id PkOkZKcEB...7f5rkFVbS]: Uploading leftover stream [Part number 1 containing 0.00 MB] 2023-03-20 07:12:33 destination > 2023-03-20 07:12:33 INFO a.m.s.StreamTransferManager(uploadStreamPart):558 - [Manager uploading to redshift-dev-s3-bucket/testAutomation/contacts/2023_03_20_1679296343675_0.csv with id PkOkZKcEB...7f5rkFVbS]: Finished uploading [Part number 1 containing 0.00 MB] 2023-03-20 07:12:33 destination > 2023-03-20 07:12:33 INFO a.m.s.StreamTransferManager(complete):395 - [Manager uploading to redshift-dev-s3-bucket/testAutomation/contacts/2023_03_20_1679296343675_0.csv with id PkOkZKcEB...7f5rkFVbS]: Completed 2023-03-20 07:12:33 destination > 2023-03-20 07:12:33 INFO i.a.i.d.s.S3StorageOperations(loadDataIntoBucket):198 - Uploaded buffer file to storage: daa9ebe7-cb41-46ee-8feb-bb282becbf6913801347943086757091.csv -> testAutomation/contacts/2023_03_20_1679296343675_0.csv (filename: 2023_03_20_1679296343675_0.csv) 2023-03-20 07:12:33 destination > 2023-03-20 07:12:33 INFO i.a.i.d.r.FileBuffer(deleteFile):90 - Deleting tempFile data daa9ebe7-cb41-46ee-8feb-bb282becbf6913801347943086757091.csv 2023-03-20 07:12:33 destination > 2023-03-20 07:12:33 INFO i.a.i.d.r.SerializedBufferingStrategy(flushAll):102 - Flushing buffer of stream campaigns (0 bytes) 2023-03-20 07:12:33 destination > 2023-03-20 07:12:33 INFO i.a.i.d.s.S3ConsumerFactory(lambda$flushBufferFunction$2):121 - Flushing buffer for stream campaigns (0 bytes) to storage 2023-03-20 07:12:33 destination > 2023-03-20 07:12:33 INFO i.a.i.d.r.BaseSerializedBuffer(flush):138 - Finished writing data to d6088785-eec1-4887-88fa-c2c2d248248b4958229896563294427.csv (1 KB) 2023-03-20 07:12:33 destination > 2023-03-20 07:12:33 INFO a.m.s.StreamTransferManager(getMultiPartOutputStreams):329 - Initiated multipart upload to redshift-dev-s3-bucket/testAutomation/campaigns/2023_03_20_1679296343675_0.csv with full ID dERlmbc9XFIR_gDYcLr19FJkUMbWTsc6X7EyzqB44wmtC6yz.fZyjlQkkUAeeW9.Heq7iOFZKXUt.yGIhtx4VA1OjLIqbAUk4iOmX9mL0Jk1WDyUqJGescTV9OTkBI8K 2023-03-20 07:12:33 destination > 2023-03-20 07:12:33 INFO a.m.s.MultiPartOutputStream(close):158 - Called close() on [MultipartOutputStream for parts 1 - 10000] 2023-03-20 07:12:33 destination > 2023-03-20 07:12:33 INFO a.m.s.StreamTransferManager(complete):367 - [Manager uploading to redshift-dev-s3-bucket/testAutomation/campaigns/2023_03_20_1679296343675_0.csv with id dERlmbc9X...V9OTkBI8K]: Uploading leftover stream [Part number 1 containing 0.00 MB] 2023-03-20 07:12:33 destination > 2023-03-20 07:12:33 INFO a.m.s.StreamTransferManager(uploadStreamPart):558 - [Manager uploading to redshift-dev-s3-bucket/testAutomation/campaigns/2023_03_20_1679296343675_0.csv with id dERlmbc9X...V9OTkBI8K]: Finished uploading [Part number 1 containing 0.00 MB] 2023-03-20 07:12:33 destination > 2023-03-20 07:12:33 INFO a.m.s.StreamTransferManager(complete):395 - [Manager uploading to redshift-dev-s3-bucket/testAutomation/campaigns/2023_03_20_1679296343675_0.csv with id dERlmbc9X...V9OTkBI8K]: Completed 2023-03-20 07:12:33 destination > 2023-03-20 07:12:33 INFO i.a.i.d.s.S3StorageOperations(loadDataIntoBucket):198 - Uploaded buffer file to storage: d6088785-eec1-4887-88fa-c2c2d248248b4958229896563294427.csv -> testAutomation/campaigns/2023_03_20_1679296343675_0.csv (filename: 2023_03_20_1679296343675_0.csv) 2023-03-20 07:12:33 destination > 2023-03-20 07:12:33 INFO i.a.i.d.r.FileBuffer(deleteFile):90 - Deleting tempFile data d6088785-eec1-4887-88fa-c2c2d248248b4958229896563294427.csv 2023-03-20 07:12:33 destination > 2023-03-20 07:12:33 INFO i.a.i.d.r.SerializedBufferingStrategy(flushAll):102 - Flushing buffer of stream lists (0 bytes) 2023-03-20 07:12:33 destination > 2023-03-20 07:12:33 INFO i.a.i.d.s.S3ConsumerFactory(lambda$flushBufferFunction$2):121 - Flushing buffer for stream lists (0 bytes) to storage 2023-03-20 07:12:33 destination > 2023-03-20 07:12:33 INFO i.a.i.d.r.BaseSerializedBuffer(flush):138 - Finished writing data to bbd52405-5fae-4a2d-8210-4b76e821f54213937298451763153422.csv (316 bytes) 2023-03-20 07:12:33 destination > 2023-03-20 07:12:33 INFO a.m.s.StreamTransferManager(getMultiPartOutputStreams):329 - Initiated multipart upload to redshift-dev-s3-bucket/testAutomation/lists/2023_03_20_1679296343675_0.csv with full ID ThdgugAfglL_.sF8ZyaaVG4IYUQkMN5CzbefeBSUaYMTF3Mjmx3u7bawBZBZ4PHqDhthLbStuWvSlGwNdHP7AHIQMgwpY7dZClCRtcf19Vu_yLYSFLTOiPBB3z0dOxBP 2023-03-20 07:12:33 destination > 2023-03-20 07:12:33 INFO a.m.s.MultiPartOutputStream(close):158 - Called close() on [MultipartOutputStream for parts 1 - 10000] 2023-03-20 07:12:33 destination > 2023-03-20 07:12:33 INFO a.m.s.StreamTransferManager(complete):367 - [Manager uploading to redshift-dev-s3-bucket/testAutomation/lists/2023_03_20_1679296343675_0.csv with id ThdgugAfg...B3z0dOxBP]: Uploading leftover stream [Part number 1 containing 0.00 MB] 2023-03-20 07:12:33 destination > 2023-03-20 07:12:33 INFO a.m.s.StreamTransferManager(uploadStreamPart):558 - [Manager uploading to redshift-dev-s3-bucket/testAutomation/lists/2023_03_20_1679296343675_0.csv with id ThdgugAfg...B3z0dOxBP]: Finished uploading [Part number 1 containing 0.00 MB] 2023-03-20 07:12:33 destination > 2023-03-20 07:12:33 INFO a.m.s.StreamTransferManager(complete):395 - [Manager uploading to redshift-dev-s3-bucket/testAutomation/lists/2023_03_20_1679296343675_0.csv with id ThdgugAfg...B3z0dOxBP]: Completed 2023-03-20 07:12:33 destination > 2023-03-20 07:12:33 INFO i.a.i.d.s.S3StorageOperations(loadDataIntoBucket):198 - Uploaded buffer file to storage: bbd52405-5fae-4a2d-8210-4b76e821f54213937298451763153422.csv -> testAutomation/lists/2023_03_20_1679296343675_0.csv (filename: 2023_03_20_1679296343675_0.csv) 2023-03-20 07:12:33 destination > 2023-03-20 07:12:33 INFO i.a.i.d.r.FileBuffer(deleteFile):90 - Deleting tempFile data bbd52405-5fae-4a2d-8210-4b76e821f54213937298451763153422.csv 2023-03-20 07:12:33 destination > 2023-03-20 07:12:33 INFO i.a.i.d.r.SerializedBufferingStrategy(flushAll):102 - Flushing buffer of stream global_suppressions (0 bytes) 2023-03-20 07:12:33 destination > 2023-03-20 07:12:33 INFO i.a.i.d.s.S3ConsumerFactory(lambda$flushBufferFunction$2):121 - Flushing buffer for stream global_suppressions (0 bytes) to storage 2023-03-20 07:12:33 destination > 2023-03-20 07:12:33 INFO i.a.i.d.r.BaseSerializedBuffer(flush):138 - Finished writing data to afcf4467-7887-4150-bea0-a96d611ec2b516975764265561466577.csv (396 bytes) 2023-03-20 07:12:33 destination > 2023-03-20 07:12:33 INFO a.m.s.StreamTransferManager(getMultiPartOutputStreams):329 - Initiated multipart upload to redshift-dev-s3-bucket/testAutomation/global_suppressions/2023_03_20_1679296343675_0.csv with full ID XzYKgVDggAAcEuMzpPE0HbTpZVHPvglGdhTcWHUHO_EzmJAtCynoUMc5CZdwUtMUysuP_EX.E8hdgAp1tmB8VZ8WciOqspmOE7G4z8iru7twwEyvx4sJjJT1G1BXcNt4 2023-03-20 07:12:33 destination > 2023-03-20 07:12:33 INFO a.m.s.MultiPartOutputStream(close):158 - Called close() on [MultipartOutputStream for parts 1 - 10000] 2023-03-20 07:12:33 destination > 2023-03-20 07:12:33 INFO a.m.s.StreamTransferManager(complete):367 - [Manager uploading to redshift-dev-s3-bucket/testAutomation/global_suppressions/2023_03_20_1679296343675_0.csv with id XzYKgVDgg...1G1BXcNt4]: Uploading leftover stream [Part number 1 containing 0.00 MB] 2023-03-20 07:12:33 destination > 2023-03-20 07:12:33 INFO a.m.s.StreamTransferManager(uploadStreamPart):558 - [Manager uploading to redshift-dev-s3-bucket/testAutomation/global_suppressions/2023_03_20_1679296343675_0.csv with id XzYKgVDgg...1G1BXcNt4]: Finished uploading [Part number 1 containing 0.00 MB] 2023-03-20 07:12:33 destination > 2023-03-20 07:12:33 INFO a.m.s.StreamTransferManager(complete):395 - [Manager uploading to redshift-dev-s3-bucket/testAutomation/global_suppressions/2023_03_20_1679296343675_0.csv with id XzYKgVDgg...1G1BXcNt4]: Completed 2023-03-20 07:12:33 destination > 2023-03-20 07:12:33 INFO i.a.i.d.s.S3StorageOperations(loadDataIntoBucket):198 - Uploaded buffer file to storage: afcf4467-7887-4150-bea0-a96d611ec2b516975764265561466577.csv -> testAutomation/global_suppressions/2023_03_20_1679296343675_0.csv (filename: 2023_03_20_1679296343675_0.csv) 2023-03-20 07:12:33 destination > 2023-03-20 07:12:33 INFO i.a.i.d.r.FileBuffer(deleteFile):90 - Deleting tempFile data afcf4467-7887-4150-bea0-a96d611ec2b516975764265561466577.csv 2023-03-20 07:12:33 destination > 2023-03-20 07:12:33 INFO i.a.i.d.r.SerializedBufferingStrategy(close):122 - Closing buffer for stream contacts 2023-03-20 07:12:33 destination > 2023-03-20 07:12:33 INFO i.a.i.d.r.SerializedBufferingStrategy(close):122 - Closing buffer for stream campaigns 2023-03-20 07:12:33 destination > 2023-03-20 07:12:33 INFO i.a.i.d.r.SerializedBufferingStrategy(close):122 - Closing buffer for stream lists 2023-03-20 07:12:33 destination > 2023-03-20 07:12:33 INFO i.a.i.d.r.SerializedBufferingStrategy(close):122 - Closing buffer for stream global_suppressions 2023-03-20 07:12:33 destination > 2023-03-20 07:12:33 INFO i.a.i.b.IntegrationRunner(runInternal):152 - Completed integration: io.airbyte.integrations.destination.s3.S3Destination