Cost Effective Kafka With Iceberg Tiered Storage
Streambased K.S.I. (Kafka Service for Iceberg) unifies long term storage in Apache Iceberg with recent data in Kafka to create a complete view for Kafka clients.
What is Streambased K.S.I.
What You Get with Streambased K.S.I
Cheap Infinite Retention
Store Kafka data forever in cost effective object storage. No more cost/retention compromises mean a richer/more resilient source for your Kafka applications.
No Connector Overhead
Remove deployment, maintenance and evolution overhead that surrounds traditional ETL based approaches to long term Kafka data storage.
Unified Governance
Apply the entire Kafka ecosystem (ACLs, quotas, schemas etc.) to your long term data and see instant updates on changes.
Works With Your Stack
Seamlessly integrates with any Kafka or Kafka compatible distribution (Confluent, Amazon MSK, Aiven etc.) and any Iceberg catalog (Polaris, Unity, LakeKeeper, Nessie).
Single Source of Truth
No duplication or drift. Events are stored only once as either Kafka or Iceberg ensuring no opportunity for inconsistencies can arise.
No Iceberg Maintenance
Combined with Streambased I.S.K., K.S.I.'s view based approach ensures that Iceberg is managed in a way that removes traditional maintenance requirements such as compaction and snapshot expiration.
Getting Started in Minutes, not Months
- 1
Configure
KSI requires only Kafka connection and Iceberg catalog configurations.
- 2
Deploy
K.S.I. is container based, stateless and horizontally scalable. Deploy into your Kubernetes or Docker and be up and running instantly.
- 3
Connect
Point your Kafka clients at the K.S.I. endpoints and experience the full dataset Kafka experience.
Purpose built for Streaming
Cost Reduction
Offload expensive Kafka storage to cheap and universal object storage. With K.S.I. Kafka provides a very small “hottest” storage with the majority of data stored in Iceberg.
State Hydration
Replay stream processing state from a complete dataset employing the complete Kappa architecture pattern.
Lagging Consumers
No longer can a lagging consumer drop off the end of retention and cause data loss. With K.S.I. consumers can access data all the way back to the beginning of time.
Kafka as a System of Record
Kafka, as the typical entry point into an organisation's data estate, is the ideal system of record for data. With K.S.I. this capability can finally be realized providing the single source of truth for ingested data.
Let's find the right solution for your data
We're here to help you unlock the full potential of your streaming data. Tell us about your challenges or ideas — and let's explore how Streambased can support your business.