The freshness gap: uniting real-time and batch analytics

A follow-up to The cost structure of near-real-time CDC analytics. That post priced the reconciliation tax on CDC analytics. This one is about why the tax exists at all - and why the way out lives in an awkward seam between two worlds most data teams already own.

If you build CDC analytics today, you work with two worlds of primitives.

On one side is the real-time analytics world - Apache Pinot, ClickHouse, RisingWave, and their kin. These systems treat current-state as a first-class thing: a change stream comes in, and a live, deduplicated, query-ready table of "the latest row per key" is maintained continuously, cheaply, as a core design goal. On the other side is the open lakehouse - Iceberg on object storage, read by Snowflake, Databricks, BigQuery, Spark, Trino. This world is unbeatable at cheap, durable, queryable history, shared openly across every engine you own.

Between them is a gap. And most of what "real-time analytics" costs - in money and in operational drag - is the cost of straddling it.

This isn't anyone's fault. It's the shape of the primitives everyone has to work with, and each system makes sensible use of the ones it has. But it's worth drawing the gap precisely, because once you see it, the reconciliation tax from the cost post stops looking like a Snowflake problem or a Databricks problem and starts looking like what it is: the price of not having fresh current-state and open history in the same place.

Real-time analytics already solved current-state

Start with the good news, because it's the existence proof for everything that follows: maintaining fresh current-state from a change stream is a solved problem. The real-time analytics engines do it well, and they've done it for years.

• Pinot keeps an in-memory map from primary key to the physical location of its latest row, updated as events arrive, so a query returns the current row per key. 
• ClickHouse collapses old versions of a key during background merges (ReplacingMergeTree), or on demand at read time (FINAL). 
• RisingWave maintains streaming materialized views by incremental view maintenance - as inserts, updates and deletes arrive, the stored result is updated in place, no batch refresh. 

Different mechanisms, same design center: current-state is continuous and native, not a batch job bolted on afterward. If your whole workload is real-time serving, these are excellent, and often the right answer. Nothing here argues otherwise.

The catch is where that current-state lives. It lives inside the engine that made it. You reach it by speaking that engine's protocol, inside its ecosystem, on its storage. Even where the raw bytes sit in a familiar columnar format, the part that matters - the continuously-maintained, deduplicated, versioned current-state, and the index that expresses it - stays in that world. It is not a table your Snowflake or your Spark job can simply join against.

Two worlds. And the current-state you want is on the wrong side of the gap.

Why the lakehouse can't simply hold it

The obvious question is: why not just maintain current-state in the open lakehouse directly, and skip the second system? Because the open table formats - Iceberg, Delta - are built for large, immutable, batch commits, and they are hostile to high-cadence streaming updates at every layer of the stack:

• Snapshots and metadata - Committing at streaming cadence explodes snapshot and manifest metadata. The commit protocol is built for periodic large writes, not a steady drip of tiny ones. 
• The file format - Parquet is columnar, batched into row groups, with its footer written last - you cannot simply append a row. And the row-oriented, append-native formats that would suit streaming aren't available where it counts: the big managed engines - Snowflake, Databricks, BigQuery - read and write only Parquet for Iceberg. Open-source Spark can use Avro or ORC; the SaaS warehouses most teams actually live on cannot. Avro, the one format where "just append" is natural, is the one you can't use. 
• Object storage - Immutable objects, per-request cost and latency, eventually-consistent listings - economically hostile to the many-small-writes pattern that high-cadence CDC produces. 
• Deletes - a write/read mismatch - This is the sharpest edge, because CDC is mostly updates and deletes, and current-state means suppressing the old version of every changed key. Iceberg gives you two ways to record a suppression. Equality deletes identify a row by its key - which is all a streaming CDC sink knows - but Snowflake, Databricks, and others can't apply them. Positional deletes identify a row by its physical file and row position - which a streaming sink has no way to know, because it never held the data file. So the format the writer can produce and the format the reader will accept don't meet. 

Everyone who hits this wall builds the same thing to get around it: a hot buffer that absorbs the stream before it has to become immutable lakehouse files. For example, Snowflake's own Snowpipe Streaming is exactly this - an in-memory row buffer, flushed to Snowflake-internal storage, that dodges the object-storage write problem. But it's append-only (you still run a downstream MERGE to get current-state) and it serves only inside Snowflake, because that is precisely its job. The real-time engines build the same buffer, and some even deduplicate it. None of them expose it as open, deduplicated current-state to outside engines - not because they're withholding it, but because that was never their charter.

So two capabilities sit unbundled in the gap: deduplication into current-state, and openness to any engine. Everybody delivers one or the other. Nobody was set up to deliver both, pointed outward.

The usual bridge is Lambda, all over again

Faced with fresh-but-walled on one side and open-but-batch on the other, the standard move is to keep both and query across them - federate the real-time store and the lakehouse behind a single SQL engine like Trino or Starburst, and join fresh to historical at read time.

It works. It's also the Lambda architecture from a decade ago, back in full force: two systems, two sets of semantics, the same business logic maintained twice, reconciled at the seam. Jay Kreps' original critique still lands - keeping code that must produce identical results across two very different distributed systems in sync is exactly as painful as it sounds. You don't have a data pipeline anymore; you have two, plus a federation layer to paper over the fact that they disagree.

And notice the pattern is fractal. The cost post described a query-level Lambda - leaving CDC changes in an append-only bronze table and deduplicating them at read time in every query (its "Mode B"). This is the same shape one level up: a system-level Lambda, a whole second engine standing next to your lakehouse. The gap keeps forcing the same workaround at every scale, because no single open store holds both halves.

Or you stay in the warehouse - and pay the reconciliation tax

The other option is to refuse the second system and do it all in the lakehouse: land the CDC stream as append-only bronze and reconcile it into a current-state table yourself. That is exactly the territory the previous post priced - a continuous MERGE holding every table fresh on the freshness clock (its "Mode A"), or the read-time dedup union (Mode B). Both are the warehouse fighting the immutable format the only ways it can, and both are costed there in detail. The short version: the bill tracks how many tables you keep fresh and how often each commits, not how much data moves, and at ≤5-minute freshness it climbs fast.

Either way - second system or reconciliation tax - you're paying to bridge the same gap.

Streambased ISK (Iceberg Service for Kafka) - don't write current-state - render it through the format's own primitives

This is the seam ISK is built to sit in, and the move is a small one to state: ISK is a gateway over Kafka, not another ingest hop. Kafka is already the streaming-native part of the stack - a durable, ordered log that is already your source of truth, with its durability already paid for. Every other path drains that log into a store of its own at real-time cadence: a second hop, a second durable copy, written onto the very layers that punish fast writes. ISK doesn't take that hop. It exposes the log that is already there - reading the change stream, caching it as ordinary unfiltered Parquet, and presenting it as Iceberg. No ingest, no merge, no second reconciled copy.

Current-state is then expressed not by rewriting data, but by composable delete bitmaps that ISK renders, on demand, into the reader's own native delete primitive - Iceberg v2 positional deletes for Snowflake, v3 deletion vectors for Databricks. The query engine applies them and sees a clean, deduplicated, current-state table. No write-time filtering at streaming cadence, no continuous merge.

That last move is the one that squares the round hole. Recall the delete mismatch: a streaming sink can only write key-based equality deletes, which Snowflake won't read; positional deletes need physical row positions a sink never has. ISK can produce positional deletes from a streaming source precisely because it has seen and indexed the data itself and therefore holds the positions. It hands each engine the exact delete format that engine accepts, derived from a feed the engine could otherwise never consume. Because the deletes live internally as engine-agnostic bitmaps aligned to the data chunks, the same state renders natively to whoever reads it. That is the gap no connector or sink is positioned to close - it requires controlling the data layout and the metadata / delete emission in one place.

And it is incremental and self-amortizing, which is the whole economic point:

• The delete bitmaps form a composable chain - Each new slice of change events adds bitmaps on top; composing the current view for a reader is just OR-ing the existing bitmaps up to that reader's position. Maintaining current-state is incremental work, and serving it is cheap. 
• Readers at different positions get different point-in-time views - over the same untouched data files - an earlier reader composes fewer bitmaps, a later reader composes more. The Parquet underneath is written once and shared. 
• The deduplication work for any range of the stream is done once and cached - every later reader is served from the local cache. The cost is paid once and amortized across all readers and all freshness levels - it doesn't re-accrue per query the way read-time dedup does, and it doesn't re-accrue per commit the way a continuous merge does. 

Durable, compacted writes still happen - but only when you want them. On a cadence you choose, your own ETL runs a MERGE from ISK's current-state table (and a companion projection of deleted keys) into your own compacted cold table, in your own estate. Freshness - what ISK shows live from Kafka - is fully decoupled from how often you durably roll to cold. The fast path never touches the layers that punish fast writes; the durable path runs at the batch cadence those layers are built for. Every layer gets used the way it wants to be.

A missing piece, not a new platform

Step back, and the shape of the fix is simple. The real-time engines built excellent walled current-state. The warehouses built excellent open batch history. The connector vendors built pipes between them. But open, deduplicated, streaming current-state, served natively to any Iceberg engine was nobody's charter - it fell in the seam between those worlds, so it stayed empty. ISK is the piece that fills it, composed from parts the industry already trusts: Kafka as the durable source of truth, a cache of the change log, and current-state expressed through Iceberg's own delete mechanics.

And crucially, it's additive. ISK isn't a replacement for your lakehouse - the compacted history lands in your own Iceberg or your own warehouse, exactly as it would with any traditional ingestion. The fresh current-state simply becomes another table your existing engines already know how to read. It narrows the gap between fresh, available data and the long, batch-cadence history tail.

Closing the gap

The through-line of both posts is the same. Fresh current-state and open history want to be in one place, and the tooling keeps forcing you to choose between them or to straddle both. This post is about the architectural shift. Close the gap - bring real-time close enough to batch that current-state is just another open table - and the second system, the federation seam, and the reconciliation tax all lose the same reason to exist at once.

‍

‍

Sources & method notes

• Real-time engines maintain current-state natively - 
  
  • Pinot upsert (in-memory PK→location map on the consuming segment): Pinot docs. 
  • ClickHouse ReplacingMergeTree + FINAL: ClickHouse docs. 
  • RisingWave incremental materialized views + Iceberg sink: RisingWave. 
• Snowpipe Streaming is an append-only, in-memory buffer flushed to Snowflake-internal storage (BDEC→FDN), current-state still requires a downstream MERGE: Snowflake docs · architecture deep-dive. 
• SaaS engines read/write only Parquet for Iceberg - Snowflake · Databricks UniForm · BigQuery; the Iceberg spec itself permits Parquet/Avro/ORC: spec. 
• Equality vs positional deletes, and the CDC write/read mismatch - RisingWave: the equality-delete problem · delete-method comparison. Snowflake applies positional deletes; Databricks uses v3 deletion vectors - ISK renders to each engine's native primitive. 
• Federation + Lambda - the two-systems pain: Jay Kreps, "Questioning the Lambda Architecture" (O'Reilly, 2014); 
• Trino/Presto federation across lakehouse + real-time sources as a practiced pattern: Trino ecosystem. 
• The reconciliation cost model referenced throughout: The cost structure of near-real-time CDC analytics. 

‍