Kafka - Total records read: 0 (0 bytes)

Connector Questions & Issues
, ,
1
  • Is this your first time deploying Airbyte?: Yes
  • OS Version / Instance: Debian
  • Memory / Disk: 4GB / 50GB
  • Deployment: Docker
  • Airbyte Version: 0.36.1-alpha
  • Source name/version: Kafka 0.1.5
  • Destination name/version: BigQuery (denormalized typed struct) 0.3.3
  • Step: Sync

When setting up my Kafka source all tests pass successfully, however when launch a sync to BigQuery, it reads 0 messages.

My sysAdmin tells me he’s not seeing any consumer connected to Kafka so I’m not sure how to troubleshoot it.

I tried to no avail :

  • switching Auto Offset Reset from latest to earliest
  • adding a test topic
  • sending data to a different destination (Local CSV instead of BigQuery) in case the issue was with the Kafka-BigQuery sync

Here’s the latest logs if they can help with anything.

logs-672.txt (20.2 KB)

Thanks!

2

Hey @Xavier Rosee,
I found a similar issue on our GitHub repo. Could you please share your logs there too?

My sysAdmin tells me he’s not seeing any consumer connected to Kafka so I’m not sure how to troubleshoot it.

Does this happen on your test topic too?
Are you sure your consumer has the right set of permissions to read the topic?

3

Hi @alafanechere ,

I’ll share my logs on the github issue as well.

My sysadmin sees data exchanged from Airbyte and Kafka, but has a few questions to help troubleshoot…

Could you please let me know :

  • what the minimum version supported for inter.broker.protocol.version (we’re using 0.10.2) and log.message.format.version (we’re on 0.10.2)
  • when the doc states that Airbyte requires A Kafka cluster 1.0 or above we’re talking about the release version? we’re using 1.1.1 and 2.5.1

Thanks!

4

Hi @alafanechere,

My sysadmin told me they could set up a new kafka cluster replicating the topic I’m interested in, however we have a couple questions to make sure it’s the best solution:

  • what the minimum version supported for inter.broker.protocol.version (we’re using 0.10.2) and log.message.format.version (we’re on 0.10.2)
  • when the doc states that Airbyte requires A Kafka cluster 1.0 or above we’re talking about the release version? we’re using 1.1.1 and 2.5.1
  • are there any plans to add a streaming capability to airbyte ? for my current use case I can make do with batch every 5 minutes, but in the long run I4d love to get immediate forwarding of our kafka messages in the warehouse.

Thanks a lot and apologies in advance if I’m nudging too much.

5

Hi,

I’ve got the same problem with Kafka source.
Here is the link to GitHub issue, no progress yet.

6

A workaround was shared on the issue, please give it a try.