Painfree transfer from Kafka to Iceberg
Traditional Kafka-to-Iceberg pipelines (Kafka Connect) involve uncomfortable compromises between latency, file layout and differing data structure concepts. The Streambased composed view allows access to Kafka and Iceberg data from all applications without requiring data transfer, so sidestepping these compromises and delivering the full promise of a combined real-time and analytical view.
The Tech
Addressing Transfer Pains
Streambased addresses the common Iceberg transfer pains of small files and snapshots:
A storage gateway
Streaming pipelines write data to Iceberg as it arrives, generating many small, inefficient files. Streambased avoids this by exposing Kafka data as logical Iceberg tables, eliminating the need for immediate physical writes.
Snapshots
When new data is written to Iceberg, associated metadata is created alongside it. Snapshots are metadata recording which data was inserted at which times, enabling Iceberg's time travel feature. Like the small files problem, a large number of snapshots will degrade query performance and is costly to clean up. Streambased treats snapshots as a logical construct, allowing them to be created, merged or removed with minimal overhead.
Let's find the right solution for your data
We're here to help you unlock the full potential of your streaming data. Tell us about your challenges or ideas — and let's explore how Streambased can support your business.