Issues with Redpanda Kafka connector in Airbyte

Summary

The user is facing issues with the Redpanda Kafka connector in Airbyte, specifically related to schema detection, data synchronization, and worker management.


Question

Is anyone using Airbyte with Redpanda via the Kafka connector? Set it up and it doesn’t look like its able to pick up the Schema at all. Its writing gibberish to the database its meant to be syncing to. Also seems to be spinning up a ton of workers. We are trying to just get it to stream messages as they go into Redpanda so ideally it’d be just one worker polling and then restarting if it fails. Any help would be appreciated and thanks in advance. Its a bit shocking the Kafka connector is in the state its in.



This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

["redpanda", "kafka-connector", "schema-detection", "data-synchronization", "worker-management"]