Summary
Custom connector is not applying Add Field transformation in release mode, causing data discrepancy
Question
Custom connector is not using Add Field transformation on release mode
Hello, I have created a custom connector using Airbyte’s no-code builder and initially everything was working fine. However, I recently added Transformations in stream. The problem now is that after adding an Add Field transformation and a Remove Field transformation, everything works fine during the test section of the connector. I can see the transformations being applied on response. But when I release the connector and use it in an actual connection to do a sync, my returned data only contains the Remove Field transformation and not the Add Field transformation. Is it some kind of a bug? We’re using Airbyte Cloud and this is causing some issues with our production releases. Any help will be appreciated.
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want to access the original thread.
Join the conversation on Slack
["custom-connector", "add-field-transformation", "release-mode", "bug", "airbyte-cloud"]
Can you share the YAML file?
Hi <@U02TQLBLDU4> , here it is.
<@U072HQD6DH9> I just tried this out on cloud and the Add Field transformation seems to be working correctly for me.
Can you double check that the new version of the connector you released is set as the Active Version in the Builder tab table of custom connectors?
A couple other things you could try here:
• Go to your connection page, in the Schema tab click Refresh source schema and save the changes, then run another sync to see if the new field is there
• If that ^ doesn’t work, you could try creating a brand new connection, run a sync and see if the new field is there
Sure let me check these, I’ll get back to you with the results.
Hey, it’s working fine now. Just created a fresh connection, and it started working. The older connection still has issues, but the main problem is fixed. Thanks a lot.
I was able to reproduce the issue if I first publish my connector without the Add Field transformation and set up a connection, then I add the Add Field transformation + test + publish.
The solution looks like it is to click the Refresh source schema button and save the changes, so if you want to fix your original connection that should be the solution.
Also, another question. Is there any way to find the definition_id of my custom connector from my cloud workspace? I didn’t see the option anywhere, so I grabbed it by intercepting network calls on Airbyte, but there should be one official way of getting this.
Good question - I don’t think there is an official way to get the definition ID at the moment. Could you explain your use case for that? Is it to use with the Airbyte API?
And would you prefer a way to get the definition ID through the API or through the UI?
Yes, I want the definition ID to create connection via Airbyte API. And it would be better to find the definition ID mentioned somewhere in the UI in builder section. API works great but I think most people will keep that definition ID somewhere in their constraints or in secrets to use that same connector later via API.
Got it, I’ll note down this request. Thank you for the feedback!