Tranformation/Normalization for none-table (events only) sync connection

Hi, I am quite new to Airbyte and I appreciate any pointers.

  • For my project, I’m currently developing a CleverTap source connector. For my implementation, I’m only reading from CleverTap API to get events in raw json format. Since my target destination only take in events instead of tables, I’m having trouble setting up transformation for the data (raw json) coming in.

  • My current solution is only to expose a few config options for users to fill in and modify certain field names. I implemented this in my source connector. But by doing this, it won’t have the scalability if users want to modify 20 fields.

  • My question is how would you suggest approaching transformation/normalization for this event-based connectors like this?

Thank you for your time!

I don’t the CDK destination module will handle this type of situation. But you can our docs about transformations here: https://docs.airbyte.com/connector-development/cdk-python/schemas#type-transformation
Maybe take a look in other connectors like Kafka or Mongodb that are non-sql

1 Like

Thank you for you response.
I will update when I progress.