Connecting Warpstream To Streambased

Pre-requisites

‍

This guide shows how to connect your Streambased deployment to your Warpstream deployment. In addition to following this guide we encourage you to review the documentation at: https://streambased-io.github.io/streambased/index.html and the Streambased webpage: https://streambased.io/. Feel free to reachout at info@streambased.io with any questions.

Note: We suggest running Streambased in a docker environment. Ensure that your environment has docker and docker-compose installed and is able to run containers with the amd64 platform. If you do not wish to use docker, bare metal steps can be found here: https://streambased-io.github.io/streambased/tutorial.html

Step 1: Download Streambased

‍

The easiest way to get started with Streambased is to use one of our pre-packaged demos. A demo template is available at: https://github.com/streambased-io/streambased-demos to download, locate the “<> Code” button and click “Download Zip”.

‍

‍

‍

Unpack the zip and navigate to the “warpstream” directory. This will be the base directory for our installation.

Run the following command to setup the environment:

‍

./bin/setup.sh

‍

Step 2: Grab your Warpstream config

‍

First select the cluster you want to connect to and then the “Credentials” tab. Generate a set of credentials you wish to use for Streambased and make a note of the username and password.

Note: These configs should represent a user that has read and describe permissions on all topics you wish to access using Streambased. In addition to this the configs should have read/write permissions on the following Streambased internal topics:

‍

_streambased_markers
_streambased_ranges
_streambased_config

‍

Next head to the “Connect” tab and make a note of the following information:

‍

Bootstrap URL

‍

Note: You will need Schema Registry credentials to used Streambased and Warpstream does not offer a Schema Registry service. We recommend to use Confluent Cloud to provide this as per the documentation here:

https://streambased.io/tutorial-guide/connecting-confluent-cloud-to-streambased/

‍

Step 3: Configure Streambased

‍

A template for Streambased config has already been created in the demo. To use your own configurations we’ll first configure Streambased acceleration, change serverConfig/indexer.properties

‍

indexed.topics -> comma separated list of topics you wish to accelerate with Streambased
schema.registry.schema.registry.url -> from Schema Registry config enter: https://{Host}:{Port}
schema.registry.basic.auth.user.info -> from Schema Registry config enter: {User}:{Password}
consumer.bootstrap.servers -> from Apache Kafka config enter: {Bootstrap URL}
consumer.sasl.jaas.config -> from Apache Kafka config enter: org.apache.kafka.common.security.plain.PlainLoginModule required username='{username}' password='{password}';

‍

Next let’s configure the Streambased Server, change serverConfig/catalog/kafka.properties

‍

kafka.nodes -> from Apache Kafka config enter: {Bootstrap URL}
kafka.confluent-schema-registry-url -> from Schema Registry config enter: https://{Host}:{Port}
kafka.confluent-schema-registry-basic-auth-user-info -> from Schema Registry config enter: {User}:{Password}

‍

bootstrap.servers ->   from Apache Kafka config enter: {Bootstrap URL}

sasl.jaas.config -> from Apache Kafka config enter: org.apache.kafka.common.security.plain.PlainLoginModule required username='{username}' password='{password}';

‍

Step 4: Start Streambased

‍

All that’s left to do is to start the Streambased services:

‍

docker-compose up -d

‍

Now you can connect to all of your Kafka data with JDBC. For instructions on how to do this check out our guides for Superset, DBT, Tableau and Jupyter