Invalid Access Key for Source Connecter s3

100mitra · June 30, 2022, 12:28pm

Is this your first time deploying Airbyte?: Yes
OS Version / Instance: MacOS
Memory / Disk: 16 GB / 512 GB
Deployment: K8s
Airbyte Version: 0.39-28-alpha
Source name/version: S3
Destination name/version: Snowflake
Step: The issue happened during creating and checking connection for source
Description:

Error Message :
Even after providing proper access key and secret key for S3 source, I am keep getting this message. This happened only while deploying airbyte with helm-charts, but with docker-compose it does not show any error and provide proper syncing.

I tried to run the request api to http://localhost:8001/api/v1/sources/check_connection with corresponding sourceId and mentioned below is the response I received.

{
  "message": "Internal Server Error: The Access Key Id you provided does not exist in our records. (Service: S3, Status Code: 403, Request ID: 16FD64796935167A, Extended Request ID: null)",
  "exceptionClassName": "software.amazon.awssdk.services.s3.model.S3Exception",
  "exceptionStack": [
    "software.amazon.awssdk.services.s3.model.S3Exception: The Access Key Id you provided does not exist in our records. (Service: S3, Status Code: 403, Request ID: 16FD64796935167A, Extended Request ID: null)",
    "\tat software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleErrorResponse(AwsXmlPredicatedResponseHandler.java:156)",
    "\tat software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleResponse(AwsXmlPredicatedResponseHandler.java:106)",
    "\tat software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:84)",
    "\tat software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:42)",
    "\tat software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler$Crc32ValidationResponseHandler.handle(AwsSyncClientHandler.java:94)",
    "\tat software.amazon.awssdk.core.internal.handler.BaseClientHandler.lambda$successTransformationResponseHandler$6(BaseClientHandler.java:270)",
    "\tat software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:40)",
    "\tat software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:30)",
    "\tat software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)",
    "\tat software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:73)",
    "\tat software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:42)",
    "\tat software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:78)",
    "\tat software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:40)",
    "\tat software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:50)",
    "\tat software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:36)",
    "\tat software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:64)",
    "\tat software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:34)",
    "\tat software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)",
    "\tat software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:56)",
    "\tat software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:36)",
    "\tat software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:80)",
    "\tat software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:60)",
    "\tat software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:42)",
    "\tat software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:48)",
    "\tat software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:31)",
    "\tat software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)",
    "\tat software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)",
    "\tat software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37)",
    "\tat software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26)",
    "\tat software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:193)",
    "\tat software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:135)",
    "\tat software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:161)",
    "\tat software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:114)",
    "\tat software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:169)",
    "\tat software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:95)",
    "\tat software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45)",
    "\tat software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:55)",
    "\tat software.amazon.awssdk.services.s3.DefaultS3Client.listObjectsV2(DefaultS3Client.java:6032)",
    "\tat software.amazon.awssdk.services.s3.paginators.ListObjectsV2Iterable$ListObjectsV2ResponseFetcher.nextPage(ListObjectsV2Iterable.java:153)",
    "\tat software.amazon.awssdk.services.s3.paginators.ListObjectsV2Iterable$ListObjectsV2ResponseFetcher.nextPage(ListObjectsV2Iterable.java:144)",
    "\tat software.amazon.awssdk.core.pagination.sync.PaginatedResponsesIterator.next(PaginatedResponsesIterator.java:58)",
    "\tat io.airbyte.config.helpers.S3Logs.getAscendingObjectKeys(S3Logs.java:160)",
    "\tat io.airbyte.config.helpers.S3Logs.tailCloudLog(S3Logs.java:104)",
    "\tat io.airbyte.config.helpers.LogClientSingleton.getJobLogFile(LogClientSingleton.java:113)",
    "\tat io.airbyte.server.converters.JobConverter.getLogRead(JobConverter.java:205)",
    "\tat io.airbyte.server.converters.JobConverter.getSynchronousJobRead(JobConverter.java:225)",
    "\tat io.airbyte.server.converters.JobConverter.getSynchronousJobRead(JobConverter.java:212)",
    "\tat io.airbyte.server.handlers.SchedulerHandler.reportConnectionStatus(SchedulerHandler.java:328)",
    "\tat io.airbyte.server.handlers.SchedulerHandler.checkSourceConnectionFromSourceId(SchedulerHandler.java:130)",
    "\tat io.airbyte.server.apis.ConfigurationApi.lambda$checkConnectionToSource$34(ConfigurationApi.java:477)",
    "\tat io.airbyte.server.apis.ConfigurationApi.execute(ConfigurationApi.java:865)",
    "\tat io.airbyte.server.apis.ConfigurationApi.checkConnectionToSource(ConfigurationApi.java:477)",
    "\tat java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)",
    "\tat java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)",
    "\tat java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)",
    "\tat java.base/java.lang.reflect.Method.invoke(Method.java:568)",
    "\tat org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)",
    "\tat org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124)",
    "\tat org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167)",
    "\tat org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:219)",
    "\tat org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79)",
    "\tat org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:469)",
    "\tat org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:391)",
    "\tat org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:80)",
    "\tat org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:253)",
    "\tat org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)",
    "\tat org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)",
    "\tat org.glassfish.jersey.internal.Errors.process(Errors.java:292)",
    "\tat org.glassfish.jersey.internal.Errors.process(Errors.java:274)",
    "\tat org.glassfish.jersey.internal.Errors.process(Errors.java:244)",
    "\tat org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)",
    "\tat org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:232)",
    "\tat org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680)",
    "\tat org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)",
    "\tat org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)",
    "\tat org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366)",
    "\tat org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319)",
    "\tat org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)",
    "\tat org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:763)",
    "\tat org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:569)",
    "\tat org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)",
    "\tat org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1377)",
    "\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)",
    "\tat org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:507)",
    "\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)",
    "\tat org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1292)",
    "\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)",
    "\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)",
    "\tat org.eclipse.jetty.server.Server.handle(Server.java:501)",
    "\tat org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)",
    "\tat org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:556)",
    "\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)",
    "\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273)",
    "\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)",
    "\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)",
    "\tat org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)",
    "\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)",
    "\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)",
    "\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)",
    "\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)",
    "\tat org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375)",
    "\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)",
    "\tat org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)",
    "\tat java.base/java.lang.Thread.run(Thread.java:833)"
  ],
  "rootCauseExceptionStack": []
}

Could someone guide me where I am loosing the context ?
Thanks in advance.

Ufou · June 30, 2022, 4:36pm

I have the same issue, also deployed via helm chart on k8s, will try reverting to an earlier version

alafanechere · July 1, 2022, 12:26pm

Hey, I think this is probably related to the value mapping of some logging-related env variables in the Helm Chart.
Did you try to set logging to S3, Minio or GCS? I’m not able to exactly identify the root cause at the moment but could your please share the output of a printenv command run on the airbyte-server and the rendered Helm chart for the airbyte-server ? (Please obfuscate any secret). It will help identify the problem. I’d also suggest trying to deploy with Kustomize (in the ./kube folder) to check if the error also happens with this deployment method.

100mitra · July 5, 2022, 7:02am

Thanks @alafanechere for your suggestion, I will try that and post it soon.

100mitra · July 7, 2022, 11:39am

Hey @alafanechere , as our previous discussion , here are the env var that I received from svc/airbyte-server.

kubectl exec -n airbyte svc/airbyte-server – env

PATH=/usr/local/openjdk-17/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=airbyte-server-79796d8d5c-55grd
S3_LOG_BUCKET=airbyte-dev-logs
AWS_ACCESS_KEY_ID=minio
S3_MINIO_ENDPOINT=http://airbyte-minio:9000
GCS_LOG_BUCKET=
TRACKING_STRATEGY=segment
LOG_LEVEL=INFO
JOB_MAIN_CONTAINER_CPU_LIMIT=
JOB_MAIN_CONTAINER_MEMORY_LIMIT=
S3_LOG_BUCKET_REGION=
DATABASE_PASSWORD=airbyte
DATABASE_URL=jdbc:postgresql://airbyte-postgresql:5432/db-airbyte
WEBAPP_URL=http://airbyte-webapp:80
JOB_MAIN_CONTAINER_MEMORY_REQUEST=
JOB_MAIN_CONTAINER_CPU_REQUEST=
CONFIGS_DATABASE_MINIMUM_FLYWAY_MIGRATION_VERSION=0.35.15.001
AIRBYTE_VERSION=0.39.28-alpha
DATABASE_USER=airbyte
WORKER_ENVIRONMENT=kubernetes
WORKSPACE_ROOT=/workspace
GOOGLE_APPLICATION_CREDENTIALS=
JOBS_DATABASE_MINIMUM_FLYWAY_MIGRATION_VERSION=0.29.15.001
CONFIG_ROOT=/configs
TEMPORAL_HOST=airbyte-temporal:7233
AWS_SECRET_ACCESS_KEY=minio123
S3_PATH_STYLE_ACCESS=true
AIRBYTE_MINIO_PORT=tcp://10.110.31.65:9000
AIRBYTE_MINIO_PORT_9001_TCP=tcp://10.110.31.65:9001
AIRBYTE_MINIO_PORT_9001_TCP_PORT=9001
AIRBYTE_POSTGRESQL_SERVICE_HOST=10.99.33.114
AIRBYTE_POSTGRESQL_PORT=tcp://10.99.33.114:5432
AIRBYTE_SERVER_SERVICE_PORT=8001
AIRBYTE_SERVER_PORT=tcp://10.102.20.131:8001
AIRBYTE_MINIO_SERVICE_PORT_MINIO_CONSOLE=9001
AIRBYTE_POSTGRESQL_PORT_5432_TCP_ADDR=10.99.33.114
KUBERNETES_SERVICE_PORT_HTTPS=443
KUBERNETES_PORT_443_TCP=tcp://10.96.0.1:443
KUBERNETES_PORT_443_TCP_PROTO=tcp
KUBERNETES_PORT_443_TCP_PORT=443
AIRBYTE_MINIO_PORT_9000_TCP=tcp://10.110.31.65:9000
AIRBYTE_POSTGRESQL_PORT_5432_TCP_PORT=5432
AIRBYTE_TEMPORAL_SERVICE_HOST=10.108.140.23
AIRBYTE_WEBAPP_PORT_80_TCP_PORT=80
AIRBYTE_WEBAPP_PORT_80_TCP_ADDR=10.105.216.5
KUBERNETES_SERVICE_PORT=443
KUBERNETES_PORT_443_TCP_ADDR=10.96.0.1
AIRBYTE_MINIO_PORT_9000_TCP_ADDR=10.110.31.65
AIRBYTE_WEBAPP_SERVICE_HOST=10.105.216.5
AIRBYTE_WEBAPP_PORT_80_TCP=tcp://10.105.216.5:80
AIRBYTE_POSTGRESQL_PORT_5432_TCP_PROTO=tcp
AIRBYTE_WEBAPP_SERVICE_PORT_HTTP=80
AIRBYTE_WEBAPP_PORT=tcp://10.105.216.5:80
KUBERNETES_SERVICE_HOST=10.96.0.1
KUBERNETES_PORT=tcp://10.96.0.1:443
AIRBYTE_SERVER_PORT_8001_TCP_PORT=8001
AIRBYTE_SERVER_PORT_8001_TCP_ADDR=10.102.20.131
AIRBYTE_MINIO_PORT_9001_TCP_ADDR=10.110.31.65
AIRBYTE_TEMPORAL_PORT_7233_TCP_PORT=7233
AIRBYTE_MINIO_SERVICE_PORT=9000
AIRBYTE_MINIO_SERVICE_PORT_MINIO_API=9000
AIRBYTE_POSTGRESQL_SERVICE_PORT=5432
AIRBYTE_SERVER_PORT_8001_TCP_PROTO=tcp
AIRBYTE_POSTGRESQL_SERVICE_PORT_TCP_POSTGRESQL=5432
AIRBYTE_POSTGRESQL_PORT_5432_TCP=tcp://10.99.33.114:5432
AIRBYTE_TEMPORAL_SERVICE_PORT=7233
AIRBYTE_SERVER_SERVICE_HOST=10.102.20.131
AIRBYTE_SERVER_SERVICE_PORT_HTTP=8001
AIRBYTE_SERVER_PORT_8001_TCP=tcp://10.102.20.131:8001
AIRBYTE_TEMPORAL_PORT_7233_TCP_ADDR=10.108.140.23
AIRBYTE_MINIO_PORT_9001_TCP_PROTO=tcp
AIRBYTE_WEBAPP_PORT_80_TCP_PROTO=tcp
AIRBYTE_TEMPORAL_PORT_7233_TCP=tcp://10.108.140.23:7233
AIRBYTE_WEBAPP_SERVICE_PORT=80
AIRBYTE_TEMPORAL_PORT=tcp://10.108.140.23:7233
AIRBYTE_TEMPORAL_PORT_7233_TCP_PROTO=tcp
AIRBYTE_MINIO_SERVICE_HOST=10.110.31.65
AIRBYTE_MINIO_PORT_9000_TCP_PROTO=tcp
AIRBYTE_MINIO_PORT_9000_TCP_PORT=9000
JAVA_HOME=/usr/local/openjdk-17
LANG=C.UTF-8
JAVA_VERSION=17.0.1
APPLICATION=airbyte-server
VERSION=0.39.28-alpha
HOME=/root

alafanechere · July 7, 2022, 1:26pm

Does your error persist if you remove the following env var from the server pod?
AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY

100mitra · July 12, 2022, 8:10am

Hi @alafanechere , just FYI, I tried to deploy airbyte with kustomize with kubectl apply -k kube/overlays/stable and again setup source, destination and connection and it worked.

It is not working with helm and I will also post the result when I remove the AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY .

I have a hunch that there are certain env variables that are conflicting while using helm, and try to find what is different.
Meanwhile @Ufou , you can also try using Kustomize if you are still facing the issue, may be that could help you.

ENV var for svc/airbyte-server-svc that is obatined while running things with Kustomize are :

PATH=/usr/local/openjdk-17/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=airbyte-server-76c7c9cd6c-5wkkr
S3_MINIO_ENDPOINT=http://airbyte-minio-svc:9000
CONFIGS_DATABASE_MINIMUM_FLYWAY_MIGRATION_VERSION=0.35.15.001
JOBS_DATABASE_MINIMUM_FLYWAY_MIGRATION_VERSION=0.29.15.001
AIRBYTE_VERSION=0.39.28-alpha
TRACKING_STRATEGY=segment
TEMPORAL_HOST=airbyte-temporal-svc:7233
AWS_ACCESS_KEY_ID=minio
S3_LOG_BUCKET=airbyte-dev-logs
S3_PATH_STYLE_ACCESS=true
GCS_LOG_BUCKET=
DATABASE_URL=jdbc:postgresql://airbyte-db-svc:5432/airbyte
WORKER_ENVIRONMENT=kubernetes
JOB_MAIN_CONTAINER_CPU_REQUEST=
JOB_MAIN_CONTAINER_MEMORY_LIMIT=
JOB_MAIN_CONTAINER_CPU_LIMIT=
S3_LOG_BUCKET_REGION=
GOOGLE_APPLICATION_CREDENTIALS=
CONFIG_ROOT=/configs
DATABASE_PASSWORD=docker
WORKSPACE_ROOT=/workspace
LOG_LEVEL=INFO
DATABASE_USER=docker
WEBAPP_URL=airbyte-webapp-svc:80
JOB_MAIN_CONTAINER_MEMORY_REQUEST=
AWS_SECRET_ACCESS_KEY=minio123
AIRBYTE_MINIO_SVC_SERVICE_HOST=10.108.150.15
KUBERNETES_SERVICE_HOST=10.96.0.1
AIRBYTE_WEBAPP_SVC_PORT_80_TCP=tcp://10.111.21.2:80
AIRBYTE_TEMPORAL_SVC_PORT_7233_TCP_PORT=7233
AIRBYTE_MINIO_SVC_PORT_9000_TCP_PROTO=tcp
AIRBYTE_DB_SVC_PORT_5432_TCP_PROTO=tcp
AIRBYTE_TEMPORAL_SVC_PORT_7233_TCP=tcp://10.109.251.23:7233
AIRBYTE_MINIO_SVC_PORT=tcp://10.108.150.15:9000
AIRBYTE_MINIO_SVC_PORT_9000_TCP=tcp://10.108.150.15:9000
KUBERNETES_PORT_443_TCP=tcp://10.96.0.1:443
AIRBYTE_SERVER_SVC_SERVICE_PORT=8001
AIRBYTE_SERVER_SVC_PORT=tcp://10.104.22.76:8001
AIRBYTE_DB_SVC_SERVICE_PORT=5432
AIRBYTE_WEBAPP_SVC_PORT=tcp://10.111.21.2:80
AIRBYTE_SERVER_SVC_SERVICE_HOST=10.104.22.76
AIRBYTE_SERVER_SVC_PORT_8001_TCP_ADDR=10.104.22.76
AIRBYTE_TEMPORAL_SVC_PORT=tcp://10.109.251.23:7233
AIRBYTE_TEMPORAL_SVC_PORT_7233_TCP_ADDR=10.109.251.23
KUBERNETES_PORT=tcp://10.96.0.1:443
KUBERNETES_PORT_443_TCP_PROTO=tcp
AIRBYTE_DB_SVC_PORT_5432_TCP=tcp://10.103.218.151:5432
AIRBYTE_DB_SVC_PORT_5432_TCP_PORT=5432
AIRBYTE_WEBAPP_SVC_PORT_80_TCP_ADDR=10.111.21.2
AIRBYTE_TEMPORAL_SVC_SERVICE_PORT=7233
KUBERNETES_SERVICE_PORT_HTTPS=443
AIRBYTE_WEBAPP_SVC_PORT_80_TCP_PROTO=tcp
AIRBYTE_WEBAPP_SVC_PORT_80_TCP_PORT=80
AIRBYTE_SERVER_SVC_PORT_8001_TCP=tcp://10.104.22.76:8001
AIRBYTE_TEMPORAL_SVC_PORT_7233_TCP_PROTO=tcp
AIRBYTE_MINIO_SVC_SERVICE_PORT=9000
AIRBYTE_MINIO_SVC_PORT_9000_TCP_PORT=9000
AIRBYTE_MINIO_SVC_PORT_9000_TCP_ADDR=10.108.150.15
KUBERNETES_PORT_443_TCP_ADDR=10.96.0.1
AIRBYTE_DB_SVC_PORT=tcp://10.103.218.151:5432
AIRBYTE_DB_SVC_PORT_5432_TCP_ADDR=10.103.218.151
AIRBYTE_WEBAPP_SVC_SERVICE_HOST=10.111.21.2
AIRBYTE_SERVER_SVC_PORT_8001_TCP_PROTO=tcp
KUBERNETES_SERVICE_PORT=443
KUBERNETES_PORT_443_TCP_PORT=443
AIRBYTE_DB_SVC_SERVICE_HOST=10.103.218.151
AIRBYTE_WEBAPP_SVC_SERVICE_PORT=80
AIRBYTE_SERVER_SVC_PORT_8001_TCP_PORT=8001
AIRBYTE_TEMPORAL_SVC_SERVICE_HOST=10.109.251.23
JAVA_HOME=/usr/local/openjdk-17
LANG=C.UTF-8
JAVA_VERSION=17.0.1
APPLICATION=airbyte-server
VERSION=0.39.28-alpha
HOME=/root

alafanechere · July 12, 2022, 3:02pm

Thank you for the feedback @100mitra. It would be interesting to compare the env var set by helm to those set by kustomize. FYI we plan on working on better maintaining this helm chart in the future to avoid this kind of discrepancies.

josephbrownskilljar · February 14, 2023, 7:37pm

I’m getting the same error when trying to set up an S3 source.

Without providing an AWS_SECRET_ACCESS_KEY:

The connection tests failed.

ClientError('An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied')

When providing both AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY:


The connection tests failed.

ClientError('An error occurred (InvalidAccessKeyId) when calling the ListObjectsV2 operation: The AWS Access Key Id you provided does not exist in our records.')

These are fresh access keys that are confirmed to work at present.

Topic		Replies	Views
Source S3: Error - The AWS Access Key Id you provided does not exist in our records Connector Questions & Issues getting-started , source-s3	4	3739	August 25, 2022
Destination set up - s3 public bucket access denied Connector Questions & Issues destination-s3 , getting-started , connectors	7	1188	February 2, 2023
AWS EKS S3 destination Auth Error Connector Questions & Issues destination-s3 , connectors	3	368	March 17, 2023
Cannot use EC2 IAM Role to create S3 source Connector Questions & Issues getting-started , source-s3	5	902	July 14, 2022
Oops something went wrong - after creating a new connection - Duplicate key Connector Questions & Issues connectors	3	951	January 26, 2023

Invalid Access Key for Source Connecter s3

Related Topics