- Is this your first time deploying Airbyte?: No
- OS Version / Instance: Ubuntu
- Memory / Disk: 4Gb / 1 Tb
- Deployment: Docker
- Airbyte Version: 0.36.3-alpha
- Source name/version: File/0.2.10
- Destination name/version: Google Cloud Storage (GCS) / 0.2.6
- Step: sync
- Description: I have a connection that connects a File Source to a GCS bucket Destination. Both the connectors individually connected successfully. But when I run
Sync now, the pipeline fails with this error
ERROR i.a.i.d.s.S3StorageOperations(uploadRecordsToBucket):123 - Failed to upload records into storage georgian-results/beam-predictions/2022_06_14_1655173614220_
Based on the error, it looks like the pipeline failed to write to the destination GCS bucket - BUT when i manually checked the GCS bucket via the GCP Cloud console, I can see that the file was successfully created in the GCS bucket.
So the issue here is that the UI and logs show that the file was not written, but infact behind the scenes the file actually got written.
logs-80.txt (50.7 KB)
2 Likes
Hi @coder46,
Is the file fully created on the destination? We use multi-part upload and the failure could happen on a specific part of the file 
.
I would suggest you delete the destination file and try again with another file.
This is very strange if the replication indeed succeeds because the S3 client we use to upload to GCS clearly raises a permission error:
2022-06-14 02:59:12 e[43mdestinatione[0m > 2022-06-14 02:59:12 e[1;31mERRORe[m i.a.i.d.s.S3StorageOperations(uploadRecordsToBucket):123 - Failed to upload records into storage georgian-results/beam-predictions/2022_06_14_1655175545617_2022-06-14 02:59:12 e[43mdestinatione[0m > com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: null; S3 Extended Request ID: null; Proxy: null)
Could you please double-check that you have sufficient permission sto read, list bucket, and perform multi-part upload on your destination bucket?
Hi there from the Community Assistance team.
We’re letting you know about an issue we discovered with the back-end process we use to handle topics and responses on the forum. If you experienced a situation where you posted the last message in a topic that did not receive any further replies, please open a new topic to continue the discussion. In addition, if you’re having a problem and find a closed topic on the subject, go ahead and open a new topic on it and we’ll follow up with you. We apologize for the inconvenience, and appreciate your willingness to work with us to provide a supportive community.
