Configuring Table Naming Conventions for PostgreSQL to Clickhouse Connection

Slack Knowledge Base Q&A Connector Development
, , , ,
1

Summary

User seeks guidance on configuring table naming conventions for a connection from PostgreSQL to Clickhouse in Airbyte, specifically on removing prefixes and setting desired table names.


Question

Hi All

I am creating a connection to extract data from Postgresql and load it into Clickhouse using Airbyte. I need assistance with configuring table naming conventions for my connection.

Here are is an example:

Source:

  • Type: PostgreSQL
  • Database name: bank
  • Table name: customer

Destination:

  • Type: Clickhouse
  • Database name: bank
  • Current table name: bank_raw__stream_customer
  • Desired table name: bank_customer (or simply ‘customer’)

Current configuration:

  • Stream prefix is empty
  • Namespace is set to “Custom format” with value “bank”
  • Unable to find settings to remove “raw__stream_” from the table names

I would like to:

  1. Remove the “raw__stream_” prefix from the table names
  2. Either keep the database name as prefix or remove it completely
  3. Maintain these naming conventions for future syncs

I’ve checked the connection settings but can’t find options to achieve this naming format. Could you please guide me on how to configure this correctly?

Thanks


This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

['postgresql', 'clickhouse', 'table-naming-conventions', 'airbyte-configuration']
2

These are the raw tables Airbyte creates as part of the sync process, not the final destination tables. Usually these live in the airbyte_internal schema and can largely be ignored—once the sync is complete you should have two sets of output for each stream:

  1. The raw tables (which are maintained for both sync history and to assist with merging/deduplicating/incremental syncs depending on your settings)
  2. The final tables (unprefixed)
    I believe with the ClickHouse destination (which is a community/marketplace connector, not an official Airbyte one like the ClickHouse source), the output tables don’t get normalized in any way, so you’ll still have a JSON blob in the _airbyte_data column you’ll need to work with.
3

These are the raw tables Airbyte creates as part of the sync process, not the final destination tables. Usually these live in the airbyte_internal schema and can largely be ignored—once the sync is complete you should have two sets of output for each stream:

  1. The raw tables (which are maintained for both sync history and to assist with merging/deduplicating/incremental syncs depending on your settings)
  2. The final tables (unprefixed)
    I believe with the ClickHouse destination (which is a community/marketplace connector, not an official Airbyte one like the ClickHouse source), the output tables don’t get normalized in any way, so you’ll still have a JSON blob in the _airbyte_data column you’ll need to work with.