Skip to main content
Skip to main content
Edit this page

Integrating Confluent platform with ClickHouse

Prerequisites​

We assume you are familiar with:

The official Kafka connector from ClickHouse with Confluent Platform​

Installing on Confluent platform​

This is meant to be a quick guide to get you started with the ClickHouse Sink Connector on Confluent Platform. For more details, please refer to the official Confluent documentation.

Create a Topic​

Creating a topic on Confluent Platform is fairly simple, and there are detailed instructions here.

Important notes​

  • The Kafka topic name must be the same as the ClickHouse table name. The way to tweak this is by using a transformer (for example ExtractTopic).
  • More partitions does not always mean more performance - see our upcoming guide for more details and performance tips.

Install connector​

You can download the connector from our repository - please feel free to submit comments and issues there as well!

Navigate to "Connector Plugins" -> "Add plugin" and using the following settings:

'Connector Class' - 'com.clickhouse.kafka.connect.ClickHouseSinkConnector'
'Connector type' - Sink
'Sensitive properties' - 'password'. This will ensure entries of the ClickHouse password are masked during configuration.

Example:

Gather your connection details​

To connect to ClickHouse with HTTP(S) you need this information:

Parameter(s)Description
HOST and PORTTypically, the port is 8443 when using TLS or 8123 when not using TLS.
DATABASE NAMEOut of the box, there is a database named default, use the name of the database that you want to connect to.
USERNAME and PASSWORDOut of the box, the username is default. Use the username appropriate for your use case.

The details for your ClickHouse Cloud service are available in the ClickHouse Cloud console. Select a service and click Connect:

Choose HTTPS. Connection details are displayed in an example curl command.

If you are using self-managed ClickHouse, the connection details are set by your ClickHouse administrator.

Configure the connector​

Navigate to Connectors -> Add Connector and use the following settings (note that the values are examples only):

{
  "database": "<DATABASE_NAME>",
  "errors.retry.timeout": "30",
  "exactlyOnce": "false",
  "schemas.enable": "false",
  "hostname": "<CLICKHOUSE_HOSTNAME>",
  "password": "<SAMPLE_PASSWORD>",
  "port": "8443",
  "ssl": "true",
  "topics": "<TOPIC_NAME>",
  "username": "<SAMPLE_USERNAME>",
  "key.converter": "org.apache.kafka.connect.storage.StringConverter",
  "value.converter": "org.apache.kafka.connect.json.JsonConverter",
  "value.converter.schemas.enable": "false"
}

Specify the connection endpoints​

You need to specify the allow-list of endpoints that the connector can access. You must use a fully-qualified domain name (FQDN) when adding the networking egress endpoint(s). Example: u57swl97we.eu-west-1.aws.clickhouse.com:8443

Note

You must specify HTTP(S) port. The Connector doesn't support Native protocol yet.

Read the documentation.

You should be all set!

Known limitations​