Summary
After an unexpected stoppage due to a Kubernetes cluster incident, all connections between MySQL (source) and PostgreSQL (destination) in Airbyte have stopped working. The error message ‘io.temporal.serviceclient.CheckedExceptionWrapper: io.airbyte.workers.exception.WorkerException: Running the launcher replication-orchestrator failed’ is encountered upon restarting Airbyte.
Question
I am currently experiencing an issue with Airbyte (version 0.50.39) running on Kubernetes, where all connections have stopped working following an unexpected stoppage due to a Kubernetes cluster incident. Previously, the connection between MySQL (source) and PostgreSQL (destination) was functioning correctly. The Airbyte instance was operating normally, syncing data from MySQL to PostgreSQL. However, following an incident with our Kubernetes cluster, which caused Airbyte to stop, all connections have ceased to function upon restarting Airbyte.
The primary error text is:
message='io.temporal.serviceclient.CheckedExceptionWrapper: io.airbyte.workers.exception.WorkerException: Running the launcher replication-orchestrator failed', type='java.lang.RuntimeException', nonRetryable=false
And here is the log
2024-01-05 18:11:11 DEBUG i.a.c.h.LogClientSingleton(setJobMdc):198 - Setting kube job mdc
2024-01-05 18:10:57 platform > Executing worker wrapper. Airbyte version: 0.50.38
2024-01-05 18:10:57 platform > Attempt 0 to save workflow id for cancellation
2024-01-05 18:10:57 platform >
2024-01-05 18:10:57 platform > ----- START CHECK -----
2024-01-05 18:10:57 platform >
2024-01-05 18:10:57 platform > Using default value for environment variable APPLY_FIELD_SELECTION: 'false'
2024-01-05 18:10:57 platform > Using default value for environment variable FIELD_SELECTION_WORKSPACES: ''
2024-01-05 18:10:57 platform > Using default value for environment variable SIDECAR_KUBE_CPU_LIMIT: '2.0'
2024-01-05 18:10:57 platform > Using default value for environment variable SOCAT_KUBE_CPU_LIMIT: '2.0'
2024-01-05 18:10:57 platform > Using default value for environment variable SIDECAR_KUBE_CPU_REQUEST: '0.1'
2024-01-05 18:10:57 platform > Using default value for environment variable SOCAT_KUBE_CPU_REQUEST: '0.1'
2024-01-05 18:10:57 platform > Using default value for environment variable LAUNCHDARKLY_KEY: ''
2024-01-05 18:10:57 platform > Using default value for environment variable FEATURE_FLAG_CLIENT: ''
2024-01-05 18:10:57 platform > Using default value for environment variable OTEL_COLLECTOR_ENDPOINT: ''
2024-01-05 18:10:57 platform > Attempting to start pod = source-mysql-check-398-0-crpiy for airbyte/source-mysql:3.3.1 with resources ConnectorResourceRequirements[main=io.airbyte.config.ResourceRequirements@425939fe[cpuRequest=,cpuLimit=,memoryRequest=,memoryLimit=,additionalProperties={}], heartbeat=io.airbyte.config.ResourceRequirements@5391cf05[cpuRequest=0.1,cpuLimit=2.0,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}], stdErr=io.airbyte.config.ResourceRequirements@315db069[cpuRequest=0.25,cpuLimit=2,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}], stdIn=io.airbyte.config.ResourceRequirements@69d373ca[cpuRequest=0.1,cpuLimit=2.0,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}], stdOut=io.airbyte.config.ResourceRequirements@69d373ca[cpuRequest=0.1,cpuLimit=2.0,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}]] and allowedHosts io.airbyte.config.AllowedHosts@624e31c7[hosts=[185.17.30.32, *.datadoghq.com, *.datadoghq.eu, *.sentry.io],additionalProperties={}]
2024-01-05 18:10:57 platform > workerPorts remaining capacity: %s
2024-01-05 18:10:57 platform > source-mysql-check-398-0-crpiy stdoutLocalPort = 9024
2024-01-05 18:10:57 platform > workerPorts remaining capacity: %s
2024-01-05 18:10:57 platform > source-mysql-check-398-0-crpiy stderrLocalPort = 9025
2024-01-05 18:10:57 platform > Using default value for environment variable SYNC_JOB_INIT_RETRY_TIMEOUT_MINUTES: '5'
2024-01-05 18:10:57 platform > Creating stdout socket server...
2024-01-05 18:10:57 platform > Creating stderr socket server...
2024-01-05 18:10:57 platform > Creating pod source-mysql-check-398-0-crpiy...
2024-01-05 18:10:57 platform > Trying to configure client from Kubernetes config...
2024-01-05 18:10:57 platform > Did not find Kubernetes config at: [/root/.kube/config]. Ignoring.
2024-01-05 18:10:57 platform > Trying to configure client from service account...
2024-01-05 18:10:57 platform > Found service account host and port: 10.43.0.1:443
2024-01-05 18:10:57 platform > Found service account ca cert at: [/var/run/secrets/kubernetes.io/serviceaccount/ca.crt}].
2024-01-05 18:10:57 platform > Found service account token at: [/var/run/secrets/kubernetes.io/serviceaccount/token].
2024-01-05 18:10:57 platform > Trying to configure client namespace from Kubernetes service account namespace path...
2024-01-05 18:10:57 platform > Found service account namespace at: [/var/run/secrets/kubernetes.io/serviceaccount/namespace].
2024-01-05 18:10:57 platform > Ready to run resync and reflector for v1/namespaces/airbyte/pods with resync 0
2024-01-05 18:10:57 platform > Resync skipped due to 0 full resync period for v1/namespaces/airbyte/pods
2024-01-05 18:10:58 platform > Waiting for init container to be ready before copying files...
2024-01-05 18:10:58 platform > Ready to run resync and reflector for v1/namespaces/airbyte/pods with resync 0
2024-01-05 18:10:58 platform > Resync skipped due to 0 full resync period for v1/namespaces/airbyte/pods
2024-01-05 18:10:59 platform > Init container ready..
2024-01-05 18:10:59 platform > Copying files...
2024-01-05 18:10:59 platform > Uploading file: source_config.json
2024-01-05 18:10:59 platform > kubectl cp /tmp/38e53c44-8086-42f2-9443-b999e890daf0/source_config.json airbyte/source-mysql-check-398-0-crpiy:/config/source_config.json -c init
2024-01-05 18:10:59 platform > Waiting for kubectl cp to complete
2024-01-05 18:10:59 platform > kubectl cp complete, closing process
2024-01-05 18:10:59 platform > Uploading file: FINISHED_UPLOADING
2024-01-05 18:10:59 platform > kubectl cp /tmp/50f0aec7-4c3a-4698-95b3-8bc57271df08/FINISHED_UPLOADING airbyte/source-mysql-check-398-0-crpiy:/config/FINISHED_UPLOADING -c init
2024-01-05 18:10:59 platform > Waiting for kubectl cp to complete
2024-01-05 18:10:59 platform > kubectl cp complete, closing process
2024-01-05 18:10:59 platform > Waiting until pod is ready...
2024-01-05 18:10:59 platform > Ready to run resync and reflector for v1/namespaces/airbyte/pods with resync 0
2024-01-05 18:10:59 platform > Resync skipped due to 0 full resync period for v1/namespaces/airbyte/pods
2024-01-05 18:11:00 platform > Setting stdout...
2024-01-05 18:11:00 platform > Setting stderr...
2024-01-05 18:11:01 platform > Reading pod IP...
2024-01-05 18:11:01 platform > Pod IP: 10.42.0.191
2024-01-05 18:11:01 platform > Using null stdin output stream...
2024-01-05 18:11:01 platform > Reading messages from protocol version 0.2.0
2024-01-05 18:11:01 platform > WARN StatusConsoleListener The use of package scanning to locate plugins is deprecated and will be removed in a future release
2024-01-05 18:11:01 platform > WARN StatusConsoleListener The use of package scanning to locate plugins is deprecated and will be removed in a future release
2024-01-05 18:11:01 platform > WARN StatusConsoleListener The use of package scanning to locate plugins is deprecated and will be removed in a future release
2024-01-05 18:11:01 platform > WARN StatusConsoleListener The use of package scanning to locate plugins is deprecated and will be removed in a future release
2024-01-05 18:11:01 platform > 2024-01-05 18:11:01 INFO i.a.i.s.m.MySqlSource(main):583 - starting source: class io.airbyte.integrations.source.mysql.MySqlSource
2024-01-05 18:11:01 platform > 2024-01-05 18:11:01 INFO i.a.c.i.b.IntegrationCliParser(parseOptions):126 - integration args: {check=null, config=source_config.json}
2024-01-05 18:11:01 platform > 2024-01-05 18:11:01 INFO i.a.c.i.b.IntegrationRunner(runInternal):132 - Running integration: io.airbyte.cdk.integrations.base.ssh.SshWrappedSource
2024-01-05 18:11:01 platform > 2024-01-05 18:11:01 INFO i.a.c.i.b.IntegrationRunner(runInternal):133 - Command: CHECK
2024-01-05 18:11:01 platform > 2024-01-05 18:11:01 INFO i.a.c.i.b.IntegrationRunner(runInternal):134 - Integration config: IntegrationConfig{command=CHECK, configPath='source_config.json', catalogPath='null', statePath='null'}
2024-01-05 18:11:01 platform > 2024-01-05 18:11:01 WARN c.n.s.JsonMetaSchema(newValidator):278 - Unknown keyword order - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2024-01-05 18:11:01 platform > 2024-01-05 18:11:01 WARN c.n.s.JsonMetaSchema(newValidator):278 - Unknown keyword airbyte_secret - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2024-01-05 18:11:01 platform > 2024-01-05 18:11:01 WARN c.n.s.JsonMetaSchema(newValidator):278 - Unknown keyword always_show - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2024-01-05 18:11:01 platform > 2024-01-05 18:11:01 WARN c.n.s.JsonMetaSchema(newValidator):278 - Unknown keyword display_type - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2024-01-05 18:11:02 platform > 2024-01-05 18:11:02 INFO i.a.c.i.b.s.SshTunnel(getInstance):204 - Starting connection with method: NO_TUNNEL
2024-01-05 18:11:02 platform > 2024-01-05 18:11:02 INFO c.z.h.HikariDataSource(<init>):80 - HikariPool-1 - Starting...
2024-01-05 18:11:02 platform > 2024-01-05 18:11:02 INFO c.z.h.HikariDataSource(<init>):82 - HikariPool-1 - Start completed.
2024-01-05 18:11:02 platform > 2024-01-05 18:11:02 INFO i.a.c.i.s.j.AbstractJdbcSource(lambda$getCheckOperations$1):140 - Attempting to get metadata from the database to see if we can connect.
2024-01-05 18:11:02 platform > 2024-01-05 18:11:02 INFO c.z.h.HikariDataSource(close):350 - HikariPool-1 - Shutdown initiated...
2024-01-05 18:11:02 platform > 2024-01-05 18:11:02 INFO c.z.h.HikariDataSource(close):352 - HikariPool-1 - Shutdown completed.
2024-01-05 18:11:03 platform > 2024-01-05 18:11:02 INFO i.a.c.i.b.IntegrationRunner(runInternal):231 - Completed integration: io.airbyte.cdk.integrations.base.ssh.SshWrappedSource
2024-01-05 18:11:03 platform > 2024-01-05 18:11:02 INFO i.a.i.s.m.MySqlSource(main):585 - completed source: class io.airbyte.integrations.source.mysql.MySqlSource
2024-01-05 18:11:03 platform > Gently closing process source-mysql-check-398-0-crpiy
2024-01-05 18:11:03 platform > Stopping watcher for v1/namespaces/airbyte/pods at v4768837
2024-01-05 18:11:03 platform > Force closing the watch io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@7c2fa383
2024-01-05 18:11:03 platform > Closing websocket io.fabric8.kubernetes.client.okhttp.OkHttpWebSocketImpl@422f697f
2024-01-05 18:11:03 platform > (pod: airbyte / source-mysql-check-398-0-crpiy) - Closed all resources for pod
2024-01-05 18:11:03 platform > Check connection job received output: io.airbyte.config.StandardCheckConnectionOutput@5eb4bc13[status=succeeded,message=<null>,additionalProperties={}]
2024-01-05 18:11:03 platform >
2024-01-05 18:11:03 platform > ----- END CHECK -----
2024-01-05 18:11:03 platform >
2024-01-05 18:11:03 platform > Cloud storage job log path: /workspace/398/0/logs.log
2024-01-05 18:11:03 platform > Executing worker wrapper. Airbyte version: 0.50.38
2024-01-05 18:11:03 platform > Attempt 0 to save workflow id for cancellation
2024-01-05 18:11:03 platform > Using default value for environment variable APPLY_FIELD_SELECTION: 'false'
2024-01-05 18:11:03 platform > Using default value for environment variable FIELD_SELECTION_WORKSPACES: ''
2024-01-05 18:11:03 platform > Using default value for environment variable SIDECAR_KUBE_CPU_LIMIT: '2.0'
2024-01-05 18:11:03 platform > Using default value for environment variable SOCAT_KUBE_CPU_LIMIT: '2.0'
2024-01-05 18:11:03 platform > Using default value for environment variable SIDECAR_KUBE_CPU_REQUEST: '0.1'
2024-01-05 18:11:03 platform > Using default value for environment variable SOCAT_KUBE_CPU_REQUEST: '0.1'
2024-01-05 18:11:03 platform > Using default value for environment variable LAUNCHDARKLY_KEY: ''
2024-01-05 18:11:03 platform > Using default value for environment variable FEATURE_FLAG_CLIENT: ''
2024-01-05 18:11:03 platform > Using default value for environment variable OTEL_COLLECTOR_ENDPOINT: ''
2024-01-05 18:11:03 platform > Attempting to start pod = destination-postgres-check-398-0-iwwfn for airbyte/destination-postgres:0.4.0 with resources ConnectorResourceRequirements[main=io.airbyte.config.ResourceRequirements@3341c27c[cpuRequest=,cpuLimit=,memoryRequest=,memoryLimit=,additionalProperties={}], heartbeat=io.airbyte.config.ResourceRequirements@5391cf05[cpuRequest=0.1,cpuLimit=2.0,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}], stdErr=io.airbyte.config.ResourceRequirements@315db069[cpuRequest=0.25,cpuLimit=2,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}], stdIn=io.airbyte.config.ResourceRequirements@69d373ca[cpuRequest=0.1,cpuLimit=2.0,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}], stdOut=io.airbyte.config.ResourceRequirements@69d373ca[cpuRequest=0.1,cpuLimit=2.0,memoryRequest=25Mi,memoryLimit=50Mi,additionalProperties={}]] and allowedHosts null
2024-01-05 18:11:03 platform > workerPorts remaining capacity: %s
2024-01-05 18:11:03 platform > destination-postgres-check-398-0-iwwfn stdoutLocalPort = 9026
2024-01-05 18:11:03 platform > workerPorts remaining capacity: %s
2024-01-05 18:11:03 platform > destination-postgres-check-398-0-iwwfn stderrLocalPort = 9027
2024-01-05 18:11:03 platform >
2024-01-05 18:11:03 platform > ----- START CHECK -----
2024-01-05 18:11:03 platform >
2024-01-05 18:11:03 platform > Creating stdout socket server...
2024-01-05 18:11:03 platform > Creating pod destination-postgres-check-398-0-iwwfn...
2024-01-05 18:11:03 platform > Creating stderr socket server...
2024-01-05 18:11:03 platform > Ready to run resync and reflector for v1/namespaces/airbyte/pods with resync 0
2024-01-05 18:11:03 platform > Resync skipped due to 0 full resync period for v1/namespaces/airbyte/pods
2024-01-05 18:11:03 platform > Waiting for init container to be ready before copying files...
2024-01-05 18:11:03 platform > Ready to run resync and reflector for v1/namespaces/airbyte/pods with resync 0
2024-01-05 18:11:03 platform > Resync skipped due to 0 full resync period for v1/namespaces/airbyte/pods
2024-01-05 18:11:04 platform > Init container ready..
2024-01-05 18:11:04 platform > Copying files...
2024-01-05 18:11:04 platform > Uploading file: source_config.json
2024-01-05 18:11:04 platform > kubectl cp /tmp/5904b584-5fdd-4e92-aaf9-ff9ced785903/source_config.json airbyte/destination-postgres-check-398-0-iwwfn:/config/source_config.json -c init
2024-01-05 18:11:04 platform > Waiting for kubectl cp to complete
2024-01-05 18:11:04 platform > kubectl cp complete, closing process
2024-01-05 18:11:04 platform > Uploading file: FINISHED_UPLOADING
2024-01-05 18:11:04 platform > kubectl cp /tmp/df2d49ce-a288-4165-bcd5-3be2e6cbe0e3/FINISHED_UPLOADING airbyte/destination-postgres-check-398-0-iwwfn:/config/FINISHED_UPLOADING -c init
2024-01-05 18:11:04 platform > Waiting for kubectl cp to complete
2024-01-05 18:11:04 platform > kubectl cp complete, closing process
2024-01-05 18:11:04 platform > Waiting until pod is ready...
2024-01-05 18:11:04 platform > Ready to run resync and reflector for v1/namespaces/airbyte/pods with resync 0
2024-01-05 18:11:04 platform > Resync skipped due to 0 full resync period for v1/namespaces/airbyte/pods
2024-01-05 18:11:05 platform > Setting stdout...
2024-01-05 18:11:05 platform > Setting stderr...
2024-01-05 18:11:06 platform > Reading pod IP...
2024-01-05 18:11:06 platform > Pod IP: 10.42.0.192
2024-01-05 18:11:06 platform > Using null stdin output stream...
2024-01-05 18:11:06 platform > Reading messages from protocol version 0.2.0
2024-01-05 18:11:06 platform > INFO i.a.i.d.p.PostgresDestination(main):98 starting destination: class io.airbyte.integrations.destination.postgres.PostgresDestination
2024-01-05 18:11:06 platform > INFO i.a.i.b.IntegrationCliParser(parseOptions):126 integration args: {check=null, config=source_config.json}
2024-01-05 18:11:06 platform > INFO i.a.i.b.IntegrationRunner(runInternal):106 Running integration: io.airbyte.integrations.base.ssh.SshWrappedDestination
2024-01-05 18:11:06 platform > INFO i.a.i.b.IntegrationRunner(runInternal):107 Command: CHECK
2024-01-05 18:11:06 platform > INFO i.a.i.b.IntegrationRunner(runInternal):108 Integration config: IntegrationConfig{command=CHECK, configPath='source_config.json', catalogPath='null', statePath='null'}
2024-01-05 18:11:07 platform > WARN c.n.s.JsonMetaSchema(newValidator):278 Unknown keyword order - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2024-01-05 18:11:07 platform > WARN c.n.s.JsonMetaSchema(newValidator):278 Unknown keyword airbyte_secret - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2024-01-05 18:11:07 platform > INFO i.a.i.b.s.SshTunnel(getInstance):204 Starting connection with method: NO_TUNNEL
2024-01-05 18:11:07 platform > INFO c.z.h.HikariDataSource(<init>):80 HikariPool-1 - Starting...
2024-01-05 18:11:07 platform > INFO c.z.h.HikariDataSource(<init>):82 HikariPool-1 - Start completed.
2024-01-05 18:11:08 platform > INFO c.z.h.HikariDataSource(close):350 HikariPool-1 - Shutdown initiated...
2024-01-05 18:11:08 platform > INFO c.z.h.HikariDataSource(close):352 HikariPool-1 - Shutdown completed.
2024-01-05 18:11:08 platform > INFO i.a.i.b.IntegrationRunner(runInternal):197 Completed integration: io.airbyte.integrations.base.ssh.SshWrappedDestination
2024-01-05 18:11:08 platform > INFO i.a.i.d.p.PostgresDestination(main):100 completed destination: class io.airbyte.integrations.destination.postgres.PostgresDestination
2024-01-05 18:11:08 platform > Gently closing process destination-postgres-check-398-0-iwwfn
2024-01-05 18:11:09 platform > Stopping watcher for v1/namespaces/airbyte/pods at v4768868
2024-01-05 18:11:09 platform > Force closing the watch io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@6ca795d
2024-01-05 18:11:09 platform > Closing websocket io.fabric8.kubernetes.client.okhttp.OkHttpWebSocketImpl@4f072074
2024-01-05 18:11:09 platform > (pod: airbyte / destination-postgres-check-398-0-iwwfn) - Closed all resources for pod
2024-01-05 18:11:09 platform > Check connection job received output: io.airbyte.config.StandardCheckConnectionOutput@7d9b3390[status=succeeded,message=<null>,additionalProperties={}]
2024-01-05 18:11:09 platform >
2024-01-05 18:11:09 platform > ----- END CHECK -----
2024-01-05 18:11:09 platform >
2024-01-05 18:11:10 platform > Cloud storage job log path: /workspace/398/0/logs.log
2024-01-05 18:11:10 platform > Executing worker wrapper. Airbyte version: 0.50.38
2024-01-05 18:11:10 platform > Attempt 0 to save workflow id for cancellation
2024-01-05 18:11:10 platform > Creating an interceptor chain that will apply interceptors in the following order: [software.amazon.awssdk.core.internal.interceptor.HttpChecksumValidationInterceptor@1dae6416, software.amazon.awssdk.awscore.interceptor.HelpfulUnknownHostExceptionInterceptor@731f4662, software.amazon.awssdk.awscore.eventstream.EventStreamInitialRequestInterceptor@3decccf3, software.amazon.awssdk.awscore.interceptor.TraceIdExecutionInterceptor@609b9f4e, software.amazon.awssdk.services.s3.internal.handlers.EnableChunkedEncodingInterceptor@e7bc99f, software.amazon.awssdk.services.s3.internal.handlers.DisableDoubleUrlEncodingInterceptor@7deb264, software.amazon.awssdk.services.s3.internal.handlers.EnableTrailingChecksumInterceptor@5c8e7d38, software.amazon.awssdk.services.s3.internal.handlers.CreateMultipartUploadRequestInterceptor@6053f808, software.amazon.awssdk.services.s3.internal.handlers.GetObjectInterceptor@30b4e2ac, software.amazon.awssdk.services.s3.internal.handlers.AsyncChecksumValidationInterceptor@428b3b4c, software.amazon.awssdk.services.s3.internal.handlers.EndpointAddressInterceptor@6dc3e998, software.amazon.awssdk.services.s3.internal.handlers.ExceptionTranslationInterceptor@122fcf90, software.amazon.awssdk.services.s3.internal.handlers.GetBucketPolicyInterceptor@519eefb, software.amazon.awssdk.services.s3.internal.handlers.PutObjectInterceptor@1e53f44f, software.amazon.awssdk.services.s3.internal.handlers.SyncChecksumValidationInterceptor@6cae4134, software.amazon.awssdk.services.s3.internal.handlers.DecodeUrlEncodedResponseInterceptor@91407e, software.amazon.awssdk.services.s3.internal.handlers.CreateBucketInterceptor@4ab508a1]
2024-01-05 18:11:10 platform > Interceptor 'software.amazon.awssdk.services.s3.internal.handlers.EnableTrailingChecksumInterceptor@5c8e7d38' modified the message with its modifyHttpRequest method.
2024-01-05 18:11:10 platform > Interceptor 'software.amazon.awssdk.services.s3.internal.handlers.EndpointAddressInterceptor@6dc3e998' modified the message with its modifyHttpRequest method.
2024-01-05 18:11:10 platform > Sending Request: DefaultSdkHttpFullRequest(httpMethod=GET, protocol=http, host=airbyte-minio-svc, port=9000, encodedPath=/state-storage//state/airbyte/orchestrator-repl-job-398-attempt-0/FAILED, headers=[amz-sdk-invocation-id, User-Agent, x-amz-te], queryParameters=[])
2024-01-05 18:11:10 platform > AWS4 Canonical Request: GET
/state-storage//state/airbyte/orchestrator-repl-job-398-attempt-0/FAILED
amz-sdk-invocation-id:f564b05a-07cb-514d-cfcf-1f8143b74791
amz-sdk-request:attempt=1; max=4
host:airbyte-minio-svc:9000
x-amz-content-sha256:UNSIGNED-PAYLOAD
x-amz-date:20240105T181110Z
x-amz-te:append-md5
amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date;x-amz-te
UNSIGNED-PAYLOAD
2024-01-05 18:11:10 platform > AWS4 String to sign: AWS4-HMAC-SHA256
20240105T181110Z
20240105/us-east-1/s3/aws4_request
71a4b427bb790cb5e8d6441d7e155a35d70efe7c2a9564d5415329defb92c6ab
2024-01-05 18:11:10 platform > Received failed response: 404, Request ID: 17A785F4C54A7CB2, Extended Request ID: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8
2024-01-05 18:11:10 platform > Received failed response: 404, Request ID: 17A785F4C54A7CB2, Extended Request ID: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8
2024-01-05 18:11:11 platform > Retry State: RetryManager(completeFailureBackoffPolicy=BackoffPolicy(minInterval=PT10S, maxInterval=PT30M, base=3), partialFailureBackoffPolicy=null, successiveCompleteFailureLimit=5, totalCompleteFailureLimit=10, successivePartialFailureLimit=1000, totalPartialFailureLimit=10, successiveCompleteFailures=1, totalCompleteFailures=1, successivePartialFailures=0, totalPartialFailures=0)
Backoff before next attempt: 10 seconds```
Unfortunately, I can't understand the reason of failure from the log. I have already tried to upgrade the version of Airbyte. If I create some new connections the error is the same
<br>
---
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. [Click here](https://airbytehq.slack.com/archives/C021JANJ6TY/p1704479542968579) if you want to access the original thread.
[Join the conversation on Slack](https://slack.airbyte.com)
<sub>
["airbyte", "kubernetes", "mysql", "postgresql", "connection-failure", "error", "io.temporal.serviceclient.checkedexceptionwrapper", "io.airbyte.workers.exception.workerexception", "replication-orchestrator"]
</sub>