A Unified Logical Layer for Data with no ETL
Streambased composes a dataset made up of real-time data (from Apache Kafka) and historical data (from Apache Iceberg) with the design goals of:
How Streambased works
Design Goals
A single source of truth
Consistent data across streaming and analytical use cases.
Simplified architecture
Fewer hops, fewer systems, reduced maintenance burden.
Zero-latency data access
Using analytical tooling to query up-to-date “real-time” data without waiting for batch ingestion.
Elimination of ETL and data duplication
Composition eliminates “ahead of time” data movement pipelines, increasing availability and reducing operational overhead.
Cost-efficient storage
Leveraging Apache Iceberg for scalable, low-cost long-term retention for Kafka use cases.
Consistent governance and schema management
Enforcing access control and data structure across underlying storage platforms.
Operational and analytical architecture unified - As Iceberg
Streambased serves Iceberg engines via I.S.K. (Iceberg Service for Kafka). Iceberg tables served by this component seamlessly combine Kafka and Iceberg data into a single logical view. From the Iceberg engine perspective, they behave exactly like standard tables, with no additional complexity or integration overhead. The I.S.K. architecture consists of the following:
An Iceberg catalog
I.S.K. presents a simple, read-only, catalog compliant with the Iceberg REST specification. This is the entrypoint for Iceberg engines and presents 3 namespaces.
A storage gateway
I.S.K. presents an Amazon S3 compatible API for serving Iceberg metadata and table data. When required, the storage gateway will perform any transformation necessary (for instance from Kafka data format to Iceberg native Parquet).
A hotset cache
To reduce impact on the Kafka cluster and improve Iceberg performance, I.S.K. caches files served by the storage gateway. These files represent sections of immutable Kafka log and so can be cached and invalidated as required.
The Iceberg catalog presents 3 namespaces:
Hotset
These tables represent only Kafka data.
Coldset
These tables represent only Iceberg data.
Mergedset
These tables represent a union of Kafka and Iceberg data deduplicated by Kafka partition/offset.
Let's find the right solution for your data
We're here to help you unlock the full potential of your streaming data. Tell us about your challenges or ideas — and let's explore how Streambased can support your business.