- Is this your first time deploying Airbyte?: Yes
- OS Version / Instance: Windows
- Memory / Disk: 16GB / 500 GB
- Deployment: Docker
- Airbyte Version: 0.40.2
- Source name/version: Amazon S3/0.1.18
- Destination name/version: NA
- Step: Adding New Source
- Description:
While setting up Amazon S3 as a source on the Airbyte UI, I am encountering the following error:
ClientError('An error occurred (InvalidAccessKeyId) when calling the ListObjectsV2 operation: The AWS Access Key Id you provided does not exist in our records.')
Full log:
2022-08-24 21:59:42 INFO i.a.w.t.TemporalAttemptExecution(get):105 - Docker volume job log path: /tmp/workspace/d8427963-4141-43b4-aee7-b40241234b63/0/logs.log
2022-08-24 21:59:42 INFO i.a.w.t.TemporalAttemptExecution(get):110 - Executing worker wrapper. Airbyte version: 0.40.2
2022-08-24 21:59:42 INFO i.a.c.i.LineGobbler(voidCall):83 - Checking if airbyte/source-s3:0.1.18 exists...
2022-08-24 21:59:42 INFO i.a.c.i.LineGobbler(voidCall):83 - airbyte/source-s3:0.1.18 was found locally.
2022-08-24 21:59:42 INFO i.a.w.p.DockerProcessFactory(create):108 - Creating docker job ID: d8427963-4141-43b4-aee7-b40241234b63
2022-08-24 21:59:42 INFO i.a.w.p.DockerProcessFactory(create):163 - Preparing command: docker run --rm --init -i -w /data/d8427963-4141-43b4-aee7-b40241234b63/0 --log-driver none --name source-s3-check-d8427963-4141-43b4-aee7-b40241234b63-0-qtyyk --network host -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -e DEPLOYMENT_MODE=OSS -e USE_STREAM_CAPABLE_STATE=true -e WORKER_ENVIRONMENT=DOCKER -e AIRBYTE_ROLE= -e WORKER_JOB_ATTEMPT=0 -e WORKER_CONNECTOR_IMAGE=airbyte/source-s3:0.1.18 -e AIRBYTE_VERSION=0.40.2 -e WORKER_JOB_ID=d8427963-4141-43b4-aee7-b40241234b63 airbyte/source-s3:0.1.18 check --config source_config.json
2022-08-24 21:59:44 INFO i.a.w.i.DefaultAirbyteStreamFactory(internalLog):99 - initialised stream with format:
2022-08-24 21:59:44 INFO i.a.w.i.DefaultAirbyteStreamFactory(internalLog):99 - initialised stream with format: {'additional_reader_options': '{}', 'newlines_in_values': False, 'advanced_options': '{}', 'infer_datatypes': True, 'double_quote': True, 'quote_char': '"', 'block_size': 10000, 'delimiter': ',', 'filetype': 'csv', 'encoding': 'utf8'}
2022-08-24 21:59:44 INFO i.a.w.i.DefaultAirbyteStreamFactory(internalLog):99 - Iterating S3 bucket 'amp-dev-data-pipeline-dag-stack-poc'
2022-08-24 21:59:44 ERROR i.a.w.i.DefaultAirbyteStreamFactory(internalLog):95 - Traceback (most recent call last):
File "/airbyte/integration_code/source_s3/source_files_abstract/source.py", line 62, in check_connection
for file_info in self.stream_class(**config).filepath_iterator():
File "/airbyte/integration_code/source_s3/stream.py", line 78, in filepath_iterator
yield from self._list_bucket(accept_key=lambda k: not k.endswith("/"))
File "/airbyte/integration_code/source_s3/stream.py", line 50, in _list_bucket
response = client.list_objects_v2(**kwargs)
File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 508, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 915, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (InvalidAccessKeyId) when calling the ListObjectsV2 operation: The AWS Access Key Id you provided does not exist in our records.
The issue is not with the AWS Access Key ID or with AWS Secret Access Key. I made sure that these are the latest values. I used these values in Airflow and it works smoothly.
The Airbyte docker container is running locally on my machine. I am not able to locate where am I going wrong. Any help is appreciated. Thanks.
