Source connection throws unrelated error: The Access Key Id you provided does not exist in our records. (Service: S3, Status Code: 403

Platform, Deploy & Infra Issues
, ,
1
  • Is this your first time deploying Airbyte?: No
  • OS Version / Instance: Ubuntu
  • Memory / Disk: you can use something like 4Gb / 1 Tb
  • Deployment: K8s
  • Airbyte Version: * 0.39.29-alpha
  • Source name/version: latest
  • Destination name/version:
  • Step: The issue is happening creating a new source?
  • Description: Setting up a new mysql connecter source throws an error about an S3 access key id, wtf?

Why should setting up a new mysql connector source care about S3 having the incorrect access key?!?

Regardless of the fact you are trying to collect metrics for the installation imho a new source setup should not fail based on an incorrect access token for S3:

Internal Server Error: The Access Key Id you provided does not exist in our records. (Service: S3, Status Code: 403, Request ID: 16FD5E7331C6F3BA, Extended Request ID: null)

2

Hey could you share the complete logs of scheduler/server?

3

server:

kubectl logs -n airbyte airbyte-server-5c88698df6-x72bp
2022-06-30 16:43:28 INFO i.a.c.EnvConfigs(getEnvOrDefault):967 - Using default value for environment variable CONFIG_DATABASE_USER: 'airbyte'
2022-06-30 16:43:28 INFO i.a.c.EnvConfigs(getEnvOrDefault):967 - Using default value for environment variable CONFIG_DATABASE_PASSWORD: '*****'
2022-06-30 16:43:28 INFO i.a.c.EnvConfigs(getEnvOrDefault):967 - Using default value for environment variable CONFIG_DATABASE_URL: 'jdbc:postgresql://airbyte-postgresql:5432/db-airbyte'
2022-06-30 16:43:28 INFO c.z.h.HikariDataSource(<init>):80 - HikariPool-1 - Starting...
2022-06-30 16:43:28 INFO c.z.h.HikariDataSource(<init>):82 - HikariPool-1 - Start completed.
2022-06-30 16:43:28 INFO c.z.h.HikariDataSource(<init>):80 - HikariPool-2 - Starting...
2022-06-30 16:43:28 INFO c.z.h.HikariDataSource(<init>):82 - HikariPool-2 - Start completed.
2022-06-30 16:43:29 WARN c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword existingJavaType - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2022-06-30 16:43:29 INFO i.a.s.ServerApp(getServer):163 - Checking databases..
2022-06-30 16:43:30 INFO i.a.s.ServerApp(assertDatabasesReady):139 - Checking configs database flyway migration version..
2022-06-30 16:43:30 INFO i.a.c.EnvConfigs(getEnvOrDefault):967 - Using default value for environment variable CONFIGS_DATABASE_INITIALIZATION_TIMEOUT_MS: '60000'
2022-06-30 16:43:30 WARN i.a.d.c.DatabaseAvailabilityCheck(check):37 - Waiting for database to become available...
2022-06-30 16:43:30 INFO i.a.d.c.DatabaseAvailabilityCheck(lambda$isDatabaseConnected$1):74 - Testing airbyte configs database connection...
Cannot end publish with com.van.logging.aws.S3PublishHelper@596e7d53 due to error: Cannot end publishing: Cannot publish to S3: The Access Key Id you provided does not exist in our records. (Service: Amazon S3; Status Code: 403; Error Code: InvalidAccessKeyId; Request ID: 16FD73A99ACA5CB5; S3 Extended Request ID: d0820355-cb3b-4572-b35e-9f6432a1dbc5; Proxy: null)
2022-06-30 16:43:30 INFO i.a.d.c.DatabaseAvailabilityCheck(check):56 - Database available.
2022-06-30 16:43:30 INFO o.f.c.i.l.s.Slf4jLog(info):49 - Flyway Community Edition 7.14.0 by Redgate
2022-06-30 16:43:30 INFO o.f.c.i.l.s.Slf4jLog(info):49 - Database: jdbc:postgresql://airbyte-postgresql:5432/db-airbyte (PostgreSQL 11.14)
2022-06-30 16:43:30 INFO o.f.c.i.l.s.Slf4jLog(info):49 - Flyway Community Edition 7.14.0 by Redgate
2022-06-30 16:43:30 INFO i.a.d.c.DatabaseMigrationCheck(check):46 - Current database migration version 0.39.17.001.
2022-06-30 16:43:30 INFO i.a.d.c.DatabaseMigrationCheck(check):47 - Minimum Flyway version required 0.35.15.001.
2022-06-30 16:43:30 INFO i.a.d.c.DatabaseMigrationCheck(check):62 - Verified that database has been migrated to the required minimum version 60000.
2022-06-30 16:43:30 INFO i.a.s.ServerApp(assertDatabasesReady):145 - Checking jobs database flyway migration version..
2022-06-30 16:43:30 INFO i.a.c.EnvConfigs(getEnvOrDefault):967 - Using default value for environment variable JOBS_DATABASE_INITIALIZATION_TIMEOUT_MS: '60000'
2022-06-30 16:43:30 WARN i.a.d.c.DatabaseAvailabilityCheck(check):37 - Waiting for database to become available...
2022-06-30 16:43:30 INFO i.a.d.c.DatabaseAvailabilityCheck(lambda$isDatabaseConnected$1):74 - Testing airbyte jobs database connection...
2022-06-30 16:43:30 INFO i.a.d.c.DatabaseAvailabilityCheck(check):56 - Database available.
2022-06-30 16:43:30 INFO o.f.c.i.l.s.Slf4jLog(info):49 - Flyway Community Edition 7.14.0 by Redgate
2022-06-30 16:43:30 INFO o.f.c.i.l.s.Slf4jLog(info):49 - Database: jdbc:postgresql://airbyte-postgresql:5432/db-airbyte (PostgreSQL 11.14)
2022-06-30 16:43:30 INFO o.f.c.i.l.s.Slf4jLog(info):49 - Flyway Community Edition 7.14.0 by Redgate
2022-06-30 16:43:30 INFO i.a.d.c.DatabaseMigrationCheck(check):46 - Current database migration version 0.35.62.001.
2022-06-30 16:43:30 INFO i.a.d.c.DatabaseMigrationCheck(check):47 - Minimum Flyway version required 0.29.15.001.
2022-06-30 16:43:30 INFO i.a.d.c.DatabaseMigrationCheck(check):62 - Verified that database has been migrated to the required minimum version 60000.
2022-06-30 16:43:30 INFO i.a.s.ServerApp(getServer):166 - Creating Staged Resource folder...
2022-06-30 16:43:30 INFO i.a.s.ServerApp(getServer):169 - Creating config repository...
2022-06-30 16:43:30 INFO i.a.c.EnvConfigs(getEnvOrDefault):967 - Using default value for environment variable SECRET_PERSISTENCE: 'TESTING_CONFIG_DB_TABLE'
2022-06-30 16:43:30 INFO i.a.c.EnvConfigs(getEnvOrDefault):967 - Using default value for environment variable SECRET_PERSISTENCE: 'TESTING_CONFIG_DB_TABLE'
2022-06-30 16:43:30 INFO i.a.c.EnvConfigs(getEnvOrDefault):967 - Using default value for environment variable SECRET_PERSISTENCE: 'TESTING_CONFIG_DB_TABLE'
2022-06-30 16:43:30 INFO i.a.s.ServerApp(getServer):185 - Creating jobs persistence...
2022-06-30 16:43:30 INFO i.a.c.EnvConfigs(getEnvOrDefault):967 - Using default value for environment variable DEPLOYMENT_MODE: 'OSS'
2022-06-30 16:43:30 INFO i.a.c.EnvConfigs(getEnvOrDefault):967 - Using default value for environment variable TEMPORAL_HISTORY_RETENTION_IN_DAYS: '30'
2022-06-30 16:43:30 INFO i.a.c.EnvConfigs(getEnvOrDefault):967 - Using default value for environment variable ACTIVITY_MAX_ATTEMPT: '5'
2022-06-30 16:43:30 INFO i.a.c.EnvConfigs(getEnvOrDefault):967 - Using default value for environment variable ACTIVITY_INITIAL_DELAY_BETWEEN_ATTEMPTS_SECONDS: '30'
2022-06-30 16:43:30 INFO i.a.c.EnvConfigs(getEnvOrDefault):967 - Using default value for environment variable ACTIVITY_MAX_DELAY_BETWEEN_ATTEMPTS_SECONDS: '600'
2022-06-30 16:43:30 INFO i.a.c.EnvConfigs(getEnvOrDefault):967 - Using default value for environment variable TEMPORAL_CLOUD_ENABLED: 'false'
2022-06-30 16:43:30 INFO i.a.w.t.TemporalUtils(getTemporalClientWhenConnected):232 - Waiting for temporal server...
2022-06-30 16:43:30 WARN i.a.w.t.TemporalUtils(getTemporalClientWhenConnected):243 - Waiting for namespace default to be initialized in temporal...
2022-06-30 16:43:33 INFO i.t.s.WorkflowServiceStubsImpl(<init>):188 - Created GRPC client for channel: ManagedChannelOrphanWrapper{delegate=ManagedChannelImpl{logId=1, target=airbyte-temporal:7233}}
2022-06-30 16:43:38 INFO i.a.w.t.TemporalUtils(getTemporalClientWhenConnected):260 - Temporal namespace default initialized!
2022-06-30 16:43:38 INFO i.a.c.EnvConfigs(getEnvOrDefault):967 - Using default value for environment variable TEMPORAL_CLOUD_ENABLED: 'false'
2022-06-30 16:43:38 INFO i.a.s.ServerApp(migrateExistingConnectionsToTemporalScheduler):250 - Migration to temporal scheduler has already been performed
2022-06-30 16:43:38 INFO i.a.s.ServerApp(getServer):221 - Starting server...
2022-06-30 16:43:38 INFO o.e.j.u.l.Log(initialized):169 - Logging initialized @11137ms to org.eclipse.jetty.util.log.Slf4jLog
2022-06-30 16:43:38 INFO o.e.j.s.Server(doStart):360 - jetty-9.4.31.v20200723; built: 2020-07-23T17:57:36.812Z; git: 450ba27947e13e66baa8cd1ce7e85a4461cacc1d; jvm 17.0.1+12-39
Jun 30, 2022 4:43:38 PM org.glassfish.jersey.server.wadl.WadlFeature configure
WARNING: JAXBContext implementation could not be found. WADL feature is disabled.
2022-06-30 16:43:39 INFO o.h.v.i.u.Version(<clinit>):21 - HV000001: Hibernate Validator 6.1.2.Final
2022-06-30 16:43:39 INFO o.e.j.s.h.ContextHandler(doStart):860 - Started o.e.j.s.ServletContextHandler@dbcf7c3{/,null,AVAILABLE}
2022-06-30 16:43:39 INFO o.e.j.s.AbstractConnector(doStart):331 - Started ServerConnector@42cc420b{HTTP/1.1, (http/1.1)}{0.0.0.0:8001}
2022-06-30 16:43:39 INFO o.e.j.s.Server(doStart):400 - Started @12025ms
2022-06-30 16:43:39 INFO i.a.s.ServerApp(start):129 -
    ___    _      __          __
   /   |  (_)____/ /_  __  __/ /____
  / /| | / / ___/ __ \/ / / / __/ _ \
 / ___ |/ / /  / /_/ / /_/ / /_/  __/
/_/  |_/_/_/  /_.___/\__, /\__/\___/
                    /____/
--------------------------------------
 Now ready at http://localhost:8000/
--------------------------------------
Version: 0.39.29-alpha
4

There is no scheduler in this version of airbyte?

5

Where do you see that AWS s3 error? It could be because of the s3 logs configured for the deployment can you check if that is the case?

6

I am getting the same error when trying to to do fresh install on AWS EKS with default values in Helm chart and add Slack connector. Error pops up both in webapp UI and server logs.

Screen Shot 2022-07-06 at 4.49.54 pm
Screen Shot 2022-07-06 at 4.49.54 pm1920×1055 132 KB 

2022-07-06 06:26:58 ERROR i.a.s.e.UncaughtExceptionMapper(toResponse):22 - Uncaught exception
software.amazon.awssdk.services.s3.model.S3Exception: The Access Key Id you provided does not exist in our records. (Service: S3, Status Code: 403, Request ID: 16FF2980479F1C21, Extended Request ID: null)
	at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleErrorResponse(AwsXmlPredicatedResponseHandler.java:156) ~[aws-xml-protocol-2.16.84.jar:?]
	at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleResponse(AwsXmlPredicatedResponseHandler.java:106) ~[aws-xml-protocol-2.16.84.jar:?]
	at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:84) ~[aws-xml-protocol-2.16.84.jar:?]
	at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:42) ~[aws-xml-protocol-2.16.84.jar:?]
	at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler$Crc32ValidationResponseHandler.handle(AwsSyncClientHandler.java:94) ~[aws-core-2.16.84.jar:?]
	at software.amazon.awssdk.core.internal.handler.BaseClientHandler.lambda$successTransformationResponseHandler$6(BaseClientHandler.java:270) ~[sdk-core-2.16.84.jar:?]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:40) ~[sdk-core-2.16.84.jar:?]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:30) ~[sdk-core-2.16.84.jar:?]
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[sdk-core-2.16.84.jar:?]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:73) ~[sdk-core-2.16.84.jar:?]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:42) ~[sdk-core-2.16.84.jar:?]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:78) ~[sdk-core-2.16.84.jar:?]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:40) ~[sdk-core-2.16.84.jar:?]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:50) ~[sdk-core-2.16.84.jar:?]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:36) ~[sdk-core-2.16.84.jar:?]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:64) ~[sdk-core-2.16.84.jar:?]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:34) ~[sdk-core-2.16.84.jar:?]
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[sdk-core-2.16.84.jar:?]
	at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:56) ~[sdk-core-2.16.84.jar:?]
	at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:36) ~[sdk-core-2.16.84.jar:?]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:80) ~[sdk-core-2.16.84.jar:?]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:60) ~[sdk-core-2.16.84.jar:?]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:42) ~[sdk-core-2.16.84.jar:?]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:48) ~[sdk-core-2.16.84.jar:?]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:31) ~[sdk-core-2.16.84.jar:?]
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[sdk-core-2.16.84.jar:?]
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[sdk-core-2.16.84.jar:?]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37) ~[sdk-core-2.16.84.jar:?]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26) ~[sdk-core-2.16.84.jar:?]
	at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:193) ~[sdk-core-2.16.84.jar:?]
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:135) ~[sdk-core-2.16.84.jar:?]
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:161) ~[sdk-core-2.16.84.jar:?]
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:114) ~[sdk-core-2.16.84.jar:?]
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:169) ~[sdk-core-2.16.84.jar:?]
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:95) ~[sdk-core-2.16.84.jar:?]
	at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45) ~[sdk-core-2.16.84.jar:?]
	at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:55) ~[aws-core-2.16.84.jar:?]
	at software.amazon.awssdk.services.s3.DefaultS3Client.listObjectsV2(DefaultS3Client.java:6032) ~[s3-2.16.84.jar:?]
	at software.amazon.awssdk.services.s3.paginators.ListObjectsV2Iterable$ListObjectsV2ResponseFetcher.nextPage(ListObjectsV2Iterable.java:153) ~[s3-2.16.84.jar:?]
	at software.amazon.awssdk.services.s3.paginators.ListObjectsV2Iterable$ListObjectsV2ResponseFetcher.nextPage(ListObjectsV2Iterable.java:144) ~[s3-2.16.84.jar:?]
	at software.amazon.awssdk.core.pagination.sync.PaginatedResponsesIterator.next(PaginatedResponsesIterator.java:58) ~[sdk-core-2.16.84.jar:?]
	at io.airbyte.config.helpers.S3Logs.getAscendingObjectKeys(S3Logs.java:160) ~[io.airbyte.airbyte-config-config-models-0.39.34-alpha.jar:?]
	at io.airbyte.config.helpers.S3Logs.tailCloudLog(S3Logs.java:104) ~[io.airbyte.airbyte-config-config-models-0.39.34-alpha.jar:?]
	at io.airbyte.config.helpers.LogClientSingleton.getJobLogFile(LogClientSingleton.java:113) ~[io.airbyte.airbyte-config-config-models-0.39.34-alpha.jar:?]
	at io.airbyte.server.converters.JobConverter.getLogRead(JobConverter.java:205) ~[io.airbyte-airbyte-server-0.39.34-alpha.jar:?]
	at io.airbyte.server.converters.JobConverter.getSynchronousJobRead(JobConverter.java:225) ~[io.airbyte-airbyte-server-0.39.34-alpha.jar:?]
	at io.airbyte.server.converters.JobConverter.getSynchronousJobRead(JobConverter.java:212) ~[io.airbyte-airbyte-server-0.39.34-alpha.jar:?]
	at io.airbyte.server.handlers.SchedulerHandler.reportConnectionStatus(SchedulerHandler.java:329) ~[io.airbyte-airbyte-server-0.39.34-alpha.jar:?]
	at io.airbyte.server.handlers.SchedulerHandler.checkSourceConnectionFromSourceCreate(SchedulerHandler.java:147) ~[io.airbyte-airbyte-server-0.39.34-alpha.jar:?]
	at io.airbyte.server.apis.ConfigurationApi.lambda$executeSourceCheckConnection$77(ConfigurationApi.java:742) ~[io.airbyte-airbyte-server-0.39.34-alpha.jar:?]
	at io.airbyte.server.apis.ConfigurationApi.execute(ConfigurationApi.java:868) ~[io.airbyte-airbyte-server-0.39.34-alpha.jar:?]
	at io.airbyte.server.apis.ConfigurationApi.executeSourceCheckConnection(ConfigurationApi.java:742) ~[io.airbyte-airbyte-server-0.39.34-alpha.jar:?]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[?:?]
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
	at java.lang.reflect.Method.invoke(Method.java:568) ~[?:?]
	at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52) ~[jersey-server-2.31.jar:?]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124) ~[jersey-server-2.31.jar:?]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167) ~[jersey-server-2.31.jar:?]
	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:219) ~[jersey-server-2.31.jar:?]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79) ~[jersey-server-2.31.jar:?]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:469) ~[jersey-server-2.31.jar:?]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:391) ~[jersey-server-2.31.jar:?]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:80) ~[jersey-server-2.31.jar:?]
	at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:253) ~[jersey-server-2.31.jar:?]
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) ~[jersey-common-2.31.jar:?]
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) ~[jersey-common-2.31.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[jersey-common-2.31.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[jersey-common-2.31.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:244) ~[jersey-common-2.31.jar:?]
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265) ~[jersey-common-2.31.jar:?]
	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:232) ~[jersey-server-2.31.jar:?]
	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680) ~[jersey-server-2.31.jar:?]
	at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394) ~[jersey-container-servlet-core-2.31.jar:?]
	at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346) ~[jersey-container-servlet-core-2.31.jar:?]
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366) ~[jersey-container-servlet-core-2.31.jar:?]
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319) ~[jersey-container-servlet-core-2.31.jar:?]
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205) ~[jersey-container-servlet-core-2.31.jar:?]
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:763) ~[jetty-servlet-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:569) ~[jetty-servlet-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1377) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:507) ~[jetty-servlet-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1292) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.server.Server.handle(Server.java:501) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:556) [jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375) [jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273) [jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) [jetty-io-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) [jetty-io-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) [jetty-io-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336) [jetty-util-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313) [jetty-util-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171) [jetty-util-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129) [jetty-util-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375) [jetty-util-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806) [jetty-util-9.4.31.v20200723.jar:9.4.31.v20200723]
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938) [jetty-util-9.4.31.v20200723.jar:9.4.31.v20200723]
	at java.lang.Thread.run(Thread.java:833) [?:?]
2022-07-06 06:26:58 ERROR i.a.s.RequestLogger(filter):110 - REQ 10.17.29.106 POST 500 /api/v1/scheduler/sources/check_connection - {"connectionConfiguration":"REDACTED","sourceDefinitionId":"c2281cee-86f9-4a86-bb48-d23286b4c7bd"}
2022-07-06 06:27:00 INFO i.a.s.RequestLogger(filter):112 - REQ 10.17.29.106 GET 200 /api/v1/health
2022-07-06 06:27:01 INFO i.a.s.RequestLogger(filter):112 - REQ 10.17.16.217 GET 200 /api/v1/health
2022-07-06 06:27:04 INFO i.a.s.RequestLogger(filter):112 - REQ 10.17.29.106 GET 200 /api/v1/health
2022-07-06 06:27:07 INFO i.a.s.RequestLogger(filter):112 - REQ 10.17.16.217 GET 200 /api/v1/health
Cannot end publish with com.van.logging.aws.S3PublishHelper@1f3e5a94 due to error: Cannot end publishing: Cannot publish to S3: The Access Key Id you provided does not exist in our records. (Service: Amazon S3; Status Code: 403; Error Code: InvalidAccessKeyId; Request ID: 16FF298342022D82; S3 Extended Request ID: 5204669b-5790-42ea-bf9f-6d1ad657198e; Proxy: null)

Version: 0.39.34-alpha

7

Here is server and worker env, if that helps:

Server

AIRBYTE_MINIO_PORT=tcp://172.20.9.18:9000
AIRBYTE_MINIO_PORT_9000_TCP=tcp://172.20.9.18:9000
AIRBYTE_MINIO_PORT_9000_TCP_ADDR=172.20.9.18
AIRBYTE_MINIO_PORT_9000_TCP_PORT=9000
AIRBYTE_MINIO_PORT_9000_TCP_PROTO=tcp
AIRBYTE_MINIO_PORT_9001_TCP=tcp://172.20.9.18:9001
AIRBYTE_MINIO_PORT_9001_TCP_ADDR=172.20.9.18
AIRBYTE_MINIO_PORT_9001_TCP_PORT=9001
AIRBYTE_MINIO_PORT_9001_TCP_PROTO=tcp
AIRBYTE_MINIO_SERVICE_HOST=172.20.9.18
AIRBYTE_MINIO_SERVICE_PORT=9000
AIRBYTE_MINIO_SERVICE_PORT_MINIO_API=9000
AIRBYTE_MINIO_SERVICE_PORT_MINIO_CONSOLE=9001
AIRBYTE_POSTGRESQL_PORT=tcp://172.20.4.165:5432
AIRBYTE_POSTGRESQL_PORT_5432_TCP=tcp://172.20.4.165:5432
AIRBYTE_POSTGRESQL_PORT_5432_TCP_ADDR=172.20.4.165
AIRBYTE_POSTGRESQL_PORT_5432_TCP_PORT=5432
AIRBYTE_POSTGRESQL_PORT_5432_TCP_PROTO=tcp
AIRBYTE_POSTGRESQL_SERVICE_HOST=172.20.4.165
AIRBYTE_POSTGRESQL_SERVICE_PORT=5432
AIRBYTE_POSTGRESQL_SERVICE_PORT_TCP_POSTGRESQL=5432
AIRBYTE_SERVER_PORT=tcp://172.20.94.173:8001
AIRBYTE_SERVER_PORT_8001_TCP=tcp://172.20.94.173:8001
AIRBYTE_SERVER_PORT_8001_TCP_ADDR=172.20.94.173
AIRBYTE_SERVER_PORT_8001_TCP_PORT=8001
AIRBYTE_SERVER_PORT_8001_TCP_PROTO=tcp
AIRBYTE_SERVER_SERVICE_HOST=172.20.94.173
AIRBYTE_SERVER_SERVICE_PORT=8001
AIRBYTE_SERVER_SERVICE_PORT_HTTP=8001
AIRBYTE_TEMPORAL_PORT=tcp://172.20.236.185:7233
AIRBYTE_TEMPORAL_PORT_7233_TCP=tcp://172.20.236.185:7233
AIRBYTE_TEMPORAL_PORT_7233_TCP_ADDR=172.20.236.185
AIRBYTE_TEMPORAL_PORT_7233_TCP_PORT=7233
AIRBYTE_TEMPORAL_PORT_7233_TCP_PROTO=tcp
AIRBYTE_TEMPORAL_SERVICE_HOST=172.20.236.185
AIRBYTE_TEMPORAL_SERVICE_PORT=7233
AIRBYTE_VERSION=0.39.34-alpha
AIRBYTE_WEBAPP_PORT=tcp://172.20.190.55:80
AIRBYTE_WEBAPP_PORT_80_TCP=tcp://172.20.190.55:80
AIRBYTE_WEBAPP_PORT_80_TCP_ADDR=172.20.190.55
AIRBYTE_WEBAPP_PORT_80_TCP_PORT=80
AIRBYTE_WEBAPP_PORT_80_TCP_PROTO=tcp
AIRBYTE_WEBAPP_SERVICE_HOST=172.20.190.55
AIRBYTE_WEBAPP_SERVICE_PORT=80
AIRBYTE_WEBAPP_SERVICE_PORT_HTTP=80
APPLICATION=airbyte-server
AWS_ACCESS_KEY_ID=minio
AWS_SECRET_ACCESS_KEY=minio123
CONFIGS_DATABASE_MINIMUM_FLYWAY_MIGRATION_VERSION=0.35.15.001
CONFIG_ROOT=/configs
DATABASE_PASSWORD=airbyte
DATABASE_URL=jdbc:postgresql://airbyte-postgresql:5432/db-airbyte
DATABASE_USER=airbyte
GCS_LOG_BUCKET=
GOOGLE_APPLICATION_CREDENTIALS=
HOME=/root
HOSTNAME=airbyte-server-749744fdb-tp794
JAVA_HOME=/usr/local/openjdk-17
JAVA_VERSION=17.0.1
JOBS_DATABASE_MINIMUM_FLYWAY_MIGRATION_VERSION=0.29.15.001
JOB_MAIN_CONTAINER_CPU_LIMIT=
JOB_MAIN_CONTAINER_CPU_REQUEST=
JOB_MAIN_CONTAINER_MEMORY_LIMIT=
JOB_MAIN_CONTAINER_MEMORY_REQUEST=
KUBERNETES_PORT=tcp://172.20.0.1:443
KUBERNETES_PORT_443_TCP=tcp://172.20.0.1:443
KUBERNETES_PORT_443_TCP_ADDR=172.20.0.1
KUBERNETES_PORT_443_TCP_PORT=443
KUBERNETES_PORT_443_TCP_PROTO=tcp
KUBERNETES_SERVICE_HOST=172.20.0.1
KUBERNETES_SERVICE_PORT=443
KUBERNETES_SERVICE_PORT_HTTPS=443
LANG=C.UTF-8
LOG_LEVEL=INFO
PATH=/usr/local/openjdk-17/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
S3_LOG_BUCKET=airbyte-dev-logs
S3_LOG_BUCKET_REGION=
S3_MINIO_ENDPOINT=http://airbyte-minio:9000
S3_PATH_STYLE_ACCESS=true
TEMPORAL_HOST=airbyte-temporal:7233
TERM=xterm
TRACKING_STRATEGY=segment
VERSION=0.39.34-alpha
WEBAPP_URL=http://airbyte-webapp:80
WORKER_ENVIRONMENT=kubernetes
WORKSPACE_ROOT=/workspace

Worker:

ACTIVITY_INITIAL_DELAY_BETWEEN_ATTEMPTS_SECONDS=
ACTIVITY_MAX_ATTEMPT=
ACTIVITY_MAX_DELAY_BETWEEN_ATTEMPTS_SECONDS=
AIRBYTE_MINIO_PORT=tcp://172.20.9.18:9000
AIRBYTE_MINIO_PORT_9000_TCP=tcp://172.20.9.18:9000
AIRBYTE_MINIO_PORT_9000_TCP_ADDR=172.20.9.18
AIRBYTE_MINIO_PORT_9000_TCP_PORT=9000
AIRBYTE_MINIO_PORT_9000_TCP_PROTO=tcp
AIRBYTE_MINIO_PORT_9001_TCP=tcp://172.20.9.18:9001
AIRBYTE_MINIO_PORT_9001_TCP_ADDR=172.20.9.18
AIRBYTE_MINIO_PORT_9001_TCP_PORT=9001
AIRBYTE_MINIO_PORT_9001_TCP_PROTO=tcp
AIRBYTE_MINIO_SERVICE_HOST=172.20.9.18
AIRBYTE_MINIO_SERVICE_PORT=9000
AIRBYTE_MINIO_SERVICE_PORT_MINIO_API=9000
AIRBYTE_MINIO_SERVICE_PORT_MINIO_CONSOLE=9001
AIRBYTE_POSTGRESQL_PORT=tcp://172.20.4.165:5432
AIRBYTE_POSTGRESQL_PORT_5432_TCP=tcp://172.20.4.165:5432
AIRBYTE_POSTGRESQL_PORT_5432_TCP_ADDR=172.20.4.165
AIRBYTE_POSTGRESQL_PORT_5432_TCP_PORT=5432
AIRBYTE_POSTGRESQL_PORT_5432_TCP_PROTO=tcp
AIRBYTE_POSTGRESQL_SERVICE_HOST=172.20.4.165
AIRBYTE_POSTGRESQL_SERVICE_PORT=5432
AIRBYTE_POSTGRESQL_SERVICE_PORT_TCP_POSTGRESQL=5432
AIRBYTE_SERVER_PORT=tcp://172.20.94.173:8001
AIRBYTE_SERVER_PORT_8001_TCP=tcp://172.20.94.173:8001
AIRBYTE_SERVER_PORT_8001_TCP_ADDR=172.20.94.173
AIRBYTE_SERVER_PORT_8001_TCP_PORT=8001
AIRBYTE_SERVER_PORT_8001_TCP_PROTO=tcp
AIRBYTE_SERVER_SERVICE_HOST=172.20.94.173
AIRBYTE_SERVER_SERVICE_PORT=8001
AIRBYTE_SERVER_SERVICE_PORT_HTTP=8001
AIRBYTE_TEMPORAL_PORT=tcp://172.20.236.185:7233
AIRBYTE_TEMPORAL_PORT_7233_TCP=tcp://172.20.236.185:7233
AIRBYTE_TEMPORAL_PORT_7233_TCP_ADDR=172.20.236.185
AIRBYTE_TEMPORAL_PORT_7233_TCP_PORT=7233
AIRBYTE_TEMPORAL_PORT_7233_TCP_PROTO=tcp
AIRBYTE_TEMPORAL_SERVICE_HOST=172.20.236.185
AIRBYTE_TEMPORAL_SERVICE_PORT=7233
AIRBYTE_VERSION=0.39.34-alpha
AIRBYTE_WEBAPP_PORT=tcp://172.20.190.55:80
AIRBYTE_WEBAPP_PORT_80_TCP=tcp://172.20.190.55:80
AIRBYTE_WEBAPP_PORT_80_TCP_ADDR=172.20.190.55
AIRBYTE_WEBAPP_PORT_80_TCP_PORT=80
AIRBYTE_WEBAPP_PORT_80_TCP_PROTO=tcp
AIRBYTE_WEBAPP_SERVICE_HOST=172.20.190.55
AIRBYTE_WEBAPP_SERVICE_PORT=80
AIRBYTE_WEBAPP_SERVICE_PORT_HTTP=80
APPLICATION=airbyte-workers
AWS_ACCESS_KEY_ID=minio
AWS_SECRET_ACCESS_KEY=minio123
CONFIGS_DATABASE_MINIMUM_FLYWAY_MIGRATION_VERSION=0.35.15.001
CONFIG_ROOT=/configs
DATABASE_HOST=airbyte-postgresql
DATABASE_PASSWORD=airbyte
DATABASE_PORT=5432
DATABASE_URL=jdbc:postgresql://airbyte-postgresql:5432/db-airbyte
DATABASE_USER=airbyte
GCS_LOG_BUCKET=
GOOGLE_APPLICATION_CREDENTIALS=
HOME=/root
HOSTNAME=airbyte-worker-5779b74dcc-6766x
INTERNAL_API_HOST=airbyte-server:8001
JAVA_HOME=/usr/local/openjdk-17
JAVA_VERSION=17.0.1
JOBS_DATABASE_MINIMUM_FLYWAY_MIGRATION_VERSION=0.29.15.001
JOB_KUBE_NAMESPACE=airbyte
JOB_MAIN_CONTAINER_CPU_LIMIT=
JOB_MAIN_CONTAINER_CPU_REQUEST=
JOB_MAIN_CONTAINER_MEMORY_LIMIT=
JOB_MAIN_CONTAINER_MEMORY_REQUEST=
KUBERNETES_PORT=tcp://172.20.0.1:443
KUBERNETES_PORT_443_TCP=tcp://172.20.0.1:443
KUBERNETES_PORT_443_TCP_ADDR=172.20.0.1
KUBERNETES_PORT_443_TCP_PORT=443
KUBERNETES_PORT_443_TCP_PROTO=tcp
KUBERNETES_SERVICE_HOST=172.20.0.1
KUBERNETES_SERVICE_PORT=443
KUBERNETES_SERVICE_PORT_HTTPS=443
LANG=C.UTF-8
LOCAL_ROOT=/tmp/airbyte_local
LOG_LEVEL=INFO
METRIC_CLIENT=
OTEL_COLLECTOR_ENDPOINT=
PATH=/usr/local/openjdk-17/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
S3_LOG_BUCKET=airbyte-dev-logs
S3_LOG_BUCKET_REGION=
S3_MINIO_ENDPOINT=http://airbyte-minio:9000
S3_PATH_STYLE_ACCESS=true
STATE_STORAGE_MINIO_ACCESS_KEY=minio
STATE_STORAGE_MINIO_BUCKET_NAME=airbyte-state-storage
STATE_STORAGE_MINIO_ENDPOINT=http://airbyte-minio:9000
STATE_STORAGE_MINIO_SECRET_ACCESS_KEY=minio123
TEMPORAL_HOST=airbyte-temporal:7233
TEMPORAL_WORKER_PORTS=9001,9002,9003,9004,9005,9006,9007,9008,9009,9010,9011,9012,9013,9014,9015,9016,9017,9018,9019,9020,9021,9022,9023,9024,9025,9026,9027,9028,9029,9030,9031,9032,9033,9034,9035,9036,9037,9038,9039,9040
TERM=xterm
TRACKING_STRATEGY=segment
VERSION=0.39.34-alpha
WEBAPP_URL=http://airbyte-webapp:80
WORKER_ENVIRONMENT=kubernetes
WORKFLOW_FAILURE_RESTART_DELAY_SECONDS=
WORKSPACE_DOCKER_MOUNT=workspace
WORKSPACE_ROOT=/workspace
8

Hey could you try deploying with kustomize as that can help us debug this in much simpler way

9

It looks like it’s actually helm issue. Putting Minio auth in 9.0.0-compatible way helped us Helm deployment: The Access Key Id you provided does not exist · Issue #8805 · airbytehq/airbyte · GitHub

11

A new test message with 20 chars.