Multinode ClickHouse destination table creation

Hello there,

I am working on a multi-node destination for ClickHouse. The thing about it is, ClickHouse has a very specific way of dealing with tables across multiple nodes. For the sake of clarity, we will call tabes on individual nodes “local tables”.

Upon local table creation, the create query must contain ON CLUSTER <<cluster_name>> after the table name in order for the table to be created on each node. Otherwise, the table will exist only on the node the load balancer happens to communicate with when it makes the query.

As well as this, in order to insert data across distributed nodes, it is convenient to create a distributed table which points at each of the local tables created in the query mentioned above.

What all of this means is that I require very specific control of how the table where Airbyte writes the data is created. What I have simply been unable to find in the documentation is exactly how and when I am supposed to do this. I would imagine that it is possible, but unfortunately I have not been able to find out how.

Can someone please let me know if I can indeed define how these tables are created and, if so, how I can do so please?

Many thanks,

Hey requested the team will get back on this

Hi there from the Community Assistance team.
We’re letting you know about an issue we discovered with the back-end process we use to handle topics and responses on the forum. If you experienced a situation where you posted the last message in a topic that did not receive any further replies, please open a new topic to continue the discussion. In addition, if you’re having a problem and find a closed topic on the subject, go ahead and open a new topic on it and we’ll follow up with you. We apologize for the inconvenience, and appreciate your willingness to work with us to provide a supportive community.