[File Source -> GCS desination] Connection showing failed on UI, but actually running successfully

[coder46]

• Is this your first time deploying Airbyte?: No
• OS Version / Instance: Ubuntu
• Memory / Disk: 4Gb / 1 Tb
• Deployment: Docker
• Airbyte Version: 0.36.3-alpha
• Source name/version: File/0.2.10
• Destination name/version: Google Cloud Storage (GCS) / 0.2.6
• Step: sync
• Description: I have a connection that connects a File Source to a GCS bucket Destination. Both the connectors individually connected successfully. But when I run Sync now, the pipeline fails with this error

ERROR i.a.i.d.s.S3StorageOperations(uploadRecordsToBucket):123 - Failed to upload records into storage georgian-results/beam-predictions/2022_06_14_1655173614220_

Based on the error, it looks like the pipeline failed to write to the destination GCS bucket - BUT when i manually checked the GCS bucket via the GCP Cloud console, I can see that the file was successfully created in the GCS bucket.

So the issue here is that the UI and logs show that the file was not written, but infact behind the scenes the file actually got written.

logs-80.txt (50.7 KB)

[alafanechere]

Hi @coder46,
Is the file fully created on the destination? We use multi-part upload and the failure could happen on a specific part of the file .
I would suggest you delete the destination file and try again with another file.
This is very strange if the replication indeed succeeds because the S3 client we use to upload to GCS clearly raises a permission error:

2022-06-14 02:59:12 e[43mdestinatione[0m > 2022-06-14 02:59:12 e[1;31mERRORe[m i.a.i.d.s.S3StorageOperations(uploadRecordsToBucket):123 - Failed to upload records into storage georgian-results/beam-predictions/2022_06_14_1655175545617_2022-06-14 02:59:12 e[43mdestinatione[0m > com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: null; S3 Extended Request ID: null; Proxy: null)

Could you please double-check that you have sufficient permission sto read, list bucket, and perform multi-part upload on your destination bucket?