Hello, everyone. I’m trying to integrate Airbyte into our ETL with Kafka-destination and I faced a problem with Kafka topics naming convention in our company: their names should have dots as a separator. I found out that Airbyte-kafka-connector always replaces them with underscore if put into names. For example, if I write into field “Topic Pattern” something like “airbyte.test” in actual Kafka I get “airbyte_test”. Is there any way to configure Airbyte to create topics with dots in their names?
Hi @SAmser93, have you taken a look at our documentation for the kafka connector. There is some information about defining naming conventions: Kafka | Airbyte Documentation.
In this case, the period character might be one of the special characters that the StandardNameTransformer replaces, but I am not 100% certain of this.
Hi, @sajarin, thanks for your answer. I actually already figured out about this being a feature rather than bug. So I just added period in this line - airbyte/Names.java at master · airbytehq/airbyte · GitHub, then updated tests in destination-kafka and create custom image to match our needs. With it I can now create topics with periods in names. I guess that workaround solves my question.
But still, I suggest to think about adding possibility to choose conventional symbols between underscores to periods or even removing this limitation at all: in Kafka topic naming conventions periods are as good as underscores - Kafka Topic Naming Conventions: Best Practices | by Kiran Prabhu | Medium, they just don’t recommend mixing them both in names at the same time
1 Like
Hey @SAmser93, glad to hear you were able to find a workaround. I’ve sent your suggestion to the product team and hopefully they’ll think about the possibility of allowing periods in topic names or removing the limitation as you stated.