We want to understand whether Airbyte is the right fit for our infrastructure. Currently, we are using singer with a custom scheduler.
I’m trying to develop Airbyte destination that can do the following:
- Access JSON Schema of any incoming data, and convert it into Avro Schema
- Make requests to our API to make metadata checking
- Convert incoming data into Avro Record from Avro Schema in the first step
- Send Avro Record to our API
The question is, can I develop this destination in Java? Or do I need to use Python CDK? Our project is built in Java so it’s preferable.
Hi @gargatuma,
This is definitely something you can achieve in Airbyte.
- Access JSON Schema of any incoming data, and convert it into Avro Schema
A destination connector is able to read the source catalog which is a collection of JSON schemas for each stream.
The rest of your list items are also definitely feasible.
Most of our destination connectors are written in Java, so it’s the right language pick. I can suggest you have a look at our destination-s3 connector for avro management.
1 Like
Hi there from the Community Assistance team.
We’re letting you know about an issue we discovered with the back-end process we use to handle topics and responses on the forum. If you experienced a situation where you posted the last message in a topic that did not receive any further replies, please open a new topic to continue the discussion. In addition, if you’re having a problem and find a closed topic on the subject, go ahead and open a new topic on it and we’ll follow up with you. We apologize for the inconvenience, and appreciate your willingness to work with us to provide a supportive community.