Destination to Clickhouse cluster

Connector Questions & Issues
1

Hi team!

Here is my issue:

  • Is this your first time deploying Airbyte?: Yes
  • OS Version / Instance: Debian
  • Memory / Disk: 32GB
  • Deployment: Docker
  • Airbyte Version: 0.35.68-alpha
  • Source name/version: Airtable 0.1.1
  • Destination name/version: Clickhouse 0.1.5
  • Step: Synchronisation from Airtable to Clickhouse cluster
  • Description:

I’m currently struggling when pushing data to a Clickhouse cluster.
It’s a 4 servers, 2 shards, 2 replicas/shard with zookeeper setup (3 nodes) and haproxy as load balancer (with 2 servers).
I’d like to push data from Airbyte to CH with replication to all my shards/replicas but I’m getting errors when using my LB url. But it not failing when targeting one single server.

I’m thinking about creating a custom connector from the existing destination-clickhouse and updating the following part:

@Override
  public String createTableQuery(final JdbcDatabase database, final String schemaName, final String tableName) {
    return String.format(
        "CREATE TABLE IF NOT EXISTS %s.%s ON CLUSTER 'datalake'( \n"
            + "%s String,\n"
            + "%s String,\n"
            + "%s DateTime64(3, 'GMT') DEFAULT now(),\n"
            + "PRIMARY KEY(%s)\n"
            + ")\n"
            + "ENGINE = ReplicatedMergeTree ('/clickhouse/tables/{cluster}/{shard}/table', '{replica}');\n",
        schemaName, tableName,
        JavaBaseConstants.COLUMN_NAME_AB_ID,
        JavaBaseConstants.COLUMN_NAME_DATA,
        JavaBaseConstants.COLUMN_NAME_EMITTED_AT,
        JavaBaseConstants.COLUMN_NAME_AB_ID);
  }

What would you recommand?

2

Today the connector only works with single instance rbaron, probably customizing you can achieve it. Please update here if works/

3

Hi there from the Community Assistance team.
We’re letting you know about an issue we discovered with the back-end process we use to handle topics and responses on the forum. If you experienced a situation where you posted the last message in a topic that did not receive any further replies, please open a new topic to continue the discussion. In addition, if you’re having a problem and find a closed topic on the subject, go ahead and open a new topic on it and we’ll follow up with you. We apologize for the inconvenience, and appreciate your willingness to work with us to provide a supportive community.