Issues with Redpanda Kafka connector in Airbyte

Slack Knowledge Base Q&A Connector Questions
, , , ,
1

Summary

The user is facing issues with the Redpanda Kafka connector in Airbyte, specifically related to schema detection, data synchronization, and worker management.


Question

Is anyone using Airbyte with Redpanda via the Kafka connector? Set it up and it doesn’t look like its able to pick up the Schema at all. Its writing gibberish to the database its meant to be syncing to. Also seems to be spinning up a ton of workers. We are trying to just get it to stream messages as they go into Redpanda so ideally it’d be just one worker polling and then restarting if it fails. Any help would be appreciated and thanks in advance. Its a bit shocking the Kafka connector is in the state its in.


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

["redpanda", "kafka-connector", "schema-detection", "data-synchronization", "worker-management"]
2

did you ever get this to work? I’ve been messing around with it for a couple of weeks, and I have no failures, it just simply isn’t reading the data from the topic. Sometimes I get the schema error (on the same data) and sometimes I don’t. The connector is kind of shockingly bad for such established technology.

3

Didn’t manage to get it to work. I don’t think anyone is using it

4

wow, that’s nuts. I’m looking through the history on slack and it’s the same complaints for years. This really shouldn’t be that hard. I got kafka working with fivetran in about 10 minutes.