Making Http Request From Java Destination

We want to understand whether Airbyte is the right fit for our infrastructure. Currently, we are using singer with a custom scheduler.
I’m trying to develop Airbyte destination that can do the following:

  1. Access JSON Schema of any incoming data, and convert it into Avro Schema
  2. Make requests to our API to make metadata checking
  3. Convert incoming data into Avro Record from Avro Schema in the first step
  4. Send Avro Record to our API

The question is, can I develop this destination in Java? Or do I need to use Python CDK? Our project is built in Java so it’s preferable.

Hi @gargatuma,
This is definitely something you can achieve in Airbyte.

  1. Access JSON Schema of any incoming data, and convert it into Avro Schema

A destination connector is able to read the source catalog which is a collection of JSON schemas for each stream.

The rest of your list items are also definitely feasible.
Most of our destination connectors are written in Java, so it’s the right language pick. I can suggest you have a look at our destination-s3 connector for avro management.

1 Like