Connecting Confluent Cloud To Streambased

Introduction

‍

This guide shows how to connect your Streambased deployment to Confluent cCloud Kafka deployments. In addition to following this guide we encourage you to review the documentation at: https://streambased-io.github.io/streambased/index.html and the Streambased webpage: https://streambased.io/. Feel free to reachout at info@streambased.io with any questions.

Note: We suggest running Streambased in a docker environment. Ensure that your environment has docker and docker-compose installed and is able to run containers with the amd64 platform. If you do not wish to use docker, bare metal steps can be found here: https://streambased-io.github.io/streambased/tutorial.html

‍

Step 1: Download Streambased

‍

‍

‍

Unpack the zip and navigate to the “ccloud” directory. This will be the base directory for our installation. Run the following command to setup the environment:

‍

./bin/setup.sh

‍

Step 2: Grab your Confluent Cloud config

‍

The easiest way to record your Confluent Cloud is from the Confluent Cloud web UI. Log in and navigate to your cluster, then click Clients -> Java

You should see the following listed configs:

‍

‍

‍

Streambased required the following configs, make a note of these:

‍

bootstrap.servers
sasl.jaas.config
schema.registry.url
basic.auth.user.info

‍

Note: These configs should represent a user that has read and describe permissions on all topics you wish to access using Streambased. In addition to this the configs should have read/write permissions on the following Streambased internal topics:

‍

_streambased_markers
_streambased_ranges
_streambased_config

‍

Step 3: Configure Streambased

‍

A template for Streambased config has already been created in the demo. To use your own configurations we’ll first configure Streambased acceleration, change serverConfig/indexer.properties

‍

indexed.topics # -> comma separated list of topics you wish to accelerate with Streambased

schema.registry.schema.registry.url  -> confluent config schema.registry.url

schema.registry.basic.auth.user.info  -> confluent config basic.auth.user.info

consumer.bootstrap.servers  -> confluent config bootstrap.servers

consumer.sasl.jaas.config -> confluent config sasl.jaas.config

‍

Next let’s configure the Streambased Server, change serverConfig/catalog/kafka.properties

‍

kafka.nodes -> confluent config bootstrap.servers

kafka.confluent-schema-registry-url -> confluent config schema.registry.url

kafka.confluent-schema-registry-basic-auth-user-info -> confluent config basic.auth.user.info

‍

Finally, change serverConfig/client.properties

‍

bootstrap.servers ->  confluent config bootstrap.servers

sasl.jaas.config -> confluent config sasl.jaas.config

‍

Step 4: Start Streambased

‍

All that’s left to do is to start the Streambased services:

‍

docker-compose up -d

‍

Now you can connect to all of your Kafka data with JDBC. For instructions on how to do this check out our guides for Superset, DBT, Tableau and Jupyter

‍