Issue with uploading file to internal stage and questions about Airbyte staging files

Slack Knowledge Base Q&A Platform Questions
, , , ,
1

Summary

Encountered an error related to storage access permissions when uploading a file to an internal stage. Questions about how Airbyte handles staging files, support for external storage configurations, and permission handling.


Question

Hi team,
I encountered an issue while trying to upload a file from my local storage to an internal stage using the following command:

H
PUT file:///<path>/<filename>.json @MY_INTERNAL_STAGE```
Here’s the error I received:
```yaml
i
{
   source: 'random9999.json',
   target: 'random9999.json.gz',
   sourceSize: 39050,
   targetSize: 0,
   sourceCompression: null,
   targetCompression: 'GZIP',
   status: 'ERROR',
   message: "Error: Error: <mailto:3xxxxxxxxxxx1-compute@developer.gserviceaccount.com|3xxxxxxxxxxx1-compute@developer.gserviceaccount.com> does not have storage.objects.get access to the Google Cloud Storage object. Permission 'storage.objects.get' denied on resource."
}```
However, when I ran the following command:
```css
 
env -i bash --noprofile --norc```
and executed the `PUT` command again in that bash session, the file successfully uploaded to the internal stage. It seems that the issue is related to storage access permissions from the cloud storage object (GCS in my case).
I have a few questions regarding this:
1. How does Airbyte handle staging files? Are you using Google Cloud Storage (GCS), or is it Amazon S3 in your setup?
2. In my scenario, the staging file might be stored in the client’s bucket. Does Airbyte always use its own internal bucket/storage for staging files, or does it also allow the use of client-specific storage buckets?
3. I tried creating a sample pipeline but didn’t find any field in the Airbyte UI where users can configure external storage. Does Airbyte support external storage configurations, and if so, how do you handle permissions like the one I encountered?
Looking forward to your guidance on this.
Thank you!
Best regards,
Tanuj Vyas

<br>

---

This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. [Click here](https://airbytehq.slack.com/archives/C021JANJ6TY/p1726549911716869) if you want 
to access the original thread.

[Join the conversation on Slack](https://slack.airbyte.com)

<sub>
["upload-file", "internal-stage", "storage-access-permissions", "staging-files", "external-storage", "permission-handling", "airbyte-ui"]
</sub>