Hello there,
I am working on a multi-node destination for ClickHouse. The thing about it is, ClickHouse has a very specific way of dealing with tables across multiple nodes. For the sake of clarity, we will call tabes on individual nodes “local tables”.
Upon local table creation, the create query must contain ON CLUSTER <<cluster_name>> after the table name in order for the table to be created on each node. Otherwise, the table will exist only on the node the load balancer happens to communicate with when it makes the query.
As well as this, in order to insert data across distributed nodes, it is convenient to create a distributed table which points at each of the local tables created in the query mentioned above.
What all of this means is that I require very specific control of how the table where Airbyte writes the data is created. What I have simply been unable to find in the documentation is exactly how and when I am supposed to do this. I would imagine that it is possible, but unfortunately I have not been able to find out how.
Can someone please let me know if I can indeed define how these tables are created and, if so, how I can do so please?
Many thanks,
Eric