Summary
The user is trying to connect to a public S3 bucket in Airbyte without authentication and is encountering an authentication error. They are looking for a way to bypass authentication similar to using the
--no-sign-requestargument in the AWS CLI.
Question
hey everybody — i have been trying to connect to a public S3 bucket but i am being asked for authentication. using the aws CLI i would pass the --no-sign-request argument… but idk how to fix it in airbyte.
any idea?
here are the logs:
['Traceback (most recent call last):\n File "/airbyte/integration_code/source_s3/v4/stream_reader.py", line 131, in get_matching_files\n for remote_file in self._page(s3, globs, self.config.bucket, current_prefix, seen, logger):\n File "/airbyte/integration_code/source_s3/v4/stream_reader.py", line 193, in _page\n response = s3.list_objects_v2(Prefix=prefix, **kwargs) if prefix else s3.list_objects_v2(**kwargs)\n File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 553, in _api_call\n return self._make_api_call(operation_name, kwargs)\n File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 989, in _make_api_call\n http, parsed_response = self._make_request(\n File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 1015, in _make_request\n return self._endpoint.make_request(operation_model, request_dict)\n File "/usr/local/lib/python3.9/site-packages/botocore/endpoint.py", line 119, in make_request\n return self._send_request(request_dict, operation_model)\n File "/usr/local/lib/python3.9/site-packages/botocore/endpoint.py", line 198, in _send_request\n request = self.create_request(request_dict, operation_model)\n File "/usr/local/lib/python3.9/site-packages/botocore/endpoint.py", line 134, in create_request\n self._event_emitter.emit(\n File "/usr/local/lib/python3.9/site-packages/botocore/hooks.py", line 412, in emit\n return self._emitter.emit(aliased_event_name, **kwargs)\n File "/usr/local/lib/python3.9/site-packages/botocore/hooks.py", line 256, in emit\n return self._emit(event_name, kwargs)\n File "/usr/local/lib/python3.9/site-packages/botocore/hooks.py", line 239, in _emit\n response = handler(**kwargs)\n File "/usr/local/lib/python3.9/site-packages/botocore/signers.py", line 105, in handler\n return self.sign(operation_name, request)\n File "/usr/local/lib/python3.9/site-packages/botocore/signers.py", line 195, in sign\n auth.add_auth(request)\n File "/usr/local/lib/python3.9/site-packages/botocore/auth.py", line 418, in add_auth\n raise NoCredentialsError()\nbotocore.exceptions.NoCredentialsError: Unable to locate credentials\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/file_based/availability_strategy/default_file_based_availability_strategy.py", line 81, in _check_list_files\n file = next(iter(stream.get_files()))\n File "/airbyte/integration_code/source_s3/v4/stream_reader.py", line 143, in get_matching_files\n self._raise_error_listing_files(globs, exc)\n File "/airbyte/integration_code/source_s3/v4/stream_reader.py", line 147, in _raise_error_listing_files\n raise ErrorListingFiles(\nairbyte_cdk.sources.file_based.exceptions.ErrorListingFiles: Error listing files. Please check the credentials provided in the config and verify that they provide permission to list files. Contact Support if you need assistance.\nsource=s3 bucket=work-sample-mk globs=[\'**/*.csv\'] endpoint=\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/file_based/availability_strategy/default_file_based_availability_strategy.py", line 62, in check_availability_and_parsability\n file = self._check_list_files(stream)\n File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/file_based/availability_strategy/default_file_based_availability_strategy.py", line 87, in _check_list_files\n raise CheckAvailabilityError(FileBasedSourceError.ERROR_LISTING_FILES, stream=stream.name) from exc\nairbyte_cdk.sources.file_based.exceptions.CheckAvailabilityError: Error listing files. Please check the credentials provided in the config and verify that they provide permission to list files. Contact Support if you need assistance.\nstream=aws_s3_bucket_anonymized\n']
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.