[PRODUCTION ISSUE] Empty sync_id in S3 destination filename pattern

Slack Knowledge Base Q&A Connector Questions
, , , ,
1

Summary

The sync_id in the S3 destination filename pattern is suddenly empty, resulting in a file being stored with an incorrect name. Other sources with similar configurations are working fine. API configurations for the sources are identical.


Question

[PRODUCTION ISSUE] we have an S3 destination with a filename pattern: {sync_id}/{part_number}{format_extension}
Until 2 days ago things worked fine. Suddenly, the sync_id gets an empty value. We don’t get an error. The file is stored as ...//1.jsonl .
We have other identical sources (but different source entity in Airbyte) with identical connection configuration that does work.
I fetched the source/connection configuration via the API and they are identical (except for the name and ID). They all use the same destination object.
Please assist


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

["production-issue", "s3-destination", "filename-pattern", "sync-id", "api", "configuration"]
2

FYI we have created an issue https://github.com/airbytehq/airbyte/issues/45084

3

It seems to be a regression because of https://github.com/airbytehq/airbyte/pull/44933

4

<@U076JJB698A> interesting - why do you think that’s the offending PR? are you using Airbyte on-prem?

5

We are currently using Airbyte Cloud standard offer, and usually every available releases of connectors are deployed immediately once available, without warning, even if it’s adding regression or breaking changes…
We had already 3 complicated months with the Shopify source connector (<Shopify | Airbyte Documentation different updates since May …>), and the S3 destination connector has been updated https://docs.airbyte.com/integrations/destinations/s3#changelog|08-30 and we immediately started to see some bugs in jsonl file naming.
That’s why I suspected this merged release to be the faulty one, because every time we had new problems with Shopify, a new release of the connector was pushed just before.

6

Sorry to be pushy, but, anyone?

7

Hello :plus1: here, the same issue on a Shopify source. We are currently fighting a big mess because of this issue …

8

Hi <@U07HGEJJPR9>. FYI we had an update on the reported issue yesterday : If you are on Airbyte Cloud you should see an improvement