Tranformation/Normalization for none-table (events only) sync connection

Connector Development
,
1

Hi, I am quite new to Airbyte and I appreciate any pointers.

  • For my project, I’m currently developing a CleverTap source connector. For my implementation, I’m only reading from CleverTap API to get events in raw json format. Since my target destination only take in events instead of tables, I’m having trouble setting up transformation for the data (raw json) coming in.

  • My current solution is only to expose a few config options for users to fill in and modify certain field names. I implemented this in my source connector. But by doing this, it won’t have the scalability if users want to modify 20 fields.

  • My question is how would you suggest approaching transformation/normalization for this event-based connectors like this?

Thank you for your time!

2

I don’t the CDK destination module will handle this type of situation. But you can our docs about transformations here: https://docs.airbyte.com/connector-development/cdk-python/schemas#type-transformation
Maybe take a look in other connectors like Kafka or Mongodb that are non-sql

1 Like
3

Thank you for you response.
I will update when I progress.

4

Hi there from the Community Assistance team.
We’re letting you know about an issue we discovered with the back-end process we use to handle topics and responses on the forum. If you experienced a situation where you posted the last message in a topic that did not receive any further replies, please open a new topic to continue the discussion. In addition, if you’re having a problem and find a closed topic on the subject, go ahead and open a new topic on it and we’ll follow up with you. We apologize for the inconvenience, and appreciate your willingness to work with us to provide a supportive community.