Kafka Connecting but reads no messages

Connector Questions & Issues
, ,
1
  • Is this your first time deploying Airbyte?: No
  • OS Version / Instance: MacOS
  • Memory / Disk: you can use something like 4Gb / 1 Tb
  • Deployment: Docker
  • Airbyte Version: latest
  • Source name/version: Kafka
  • Destination name/version: LocalJSON
  • Step: Sync
  • Description: I have created a source for kafka that actually connects to azure event hub as it is possible to use kafka to connect to these. The connection is successful but the consumer reads no messages. The logs don’t provide much info as it just says 0 messages but I know that there are and can consume them with a kafka consumer I created with python.
2

Hello and welcome to the community, @devhmars. Just to clarify: data is not sent to LocalJSON? Could you provide the logs anyway, they might give me more clarity on the situation. Thanks!

3

Not sure it is related but I had a similar issue and the root cause was in Subscription Method config:
I used Subscribe to all topics matching specified pattern option and specified topic name like topic_1 during connection creation. And after a while changed it to topic_* which is not a proper regexp and connection started to show 0 messages processed.
In the logs I found the line saying Topic list: [] so I fixed the regexp topic_(.*).

4

It looks like I now have the same issue: the topic list is fulfilled, the consumer group appears on Kafka side and shows lag as expected but Airbyte does not read the topics. Every sync ends with the message:

image
image708×132 9.4 KB