Read-time current-state: how ISK deduplicates CDC on Iceberg

This is the third in a series. The freshness-gap post argued that fresh current-state and open history belong in one place; the cost post priced what it costs to keep them together. This one is the mechanism: how ISK turns a Kafka CDC stream into a deduplicated, current-state Iceberg table that any engine can read - without re-scanning on every query or generating massive amounts of small writes.

What CDC actually asks of a table

A change-data-capture stream is a log of row mutations: insert, update, delete, keyed by primary key. Turning that log into a current-state table (the latest row per key, deletes removed) is one operation repeated forever: find the prior row for a changed key, and suppress it.

Finding the prior row is the cost. In a warehouse you pay it with a rewrite (copy-on-write rewrites the whole data file) or a merge (a continuous MERGE reconciling a change-log into a silver table on the freshness clock). Both are covered, and priced, in the cost post.

ISK takes a harder constraint: do it at read time, present the result as standard Iceberg that any engine reads unmodified, without rewriting data files. The rest of this post is how that constraint is met. If you want the why (why current-state keeps ending up trapped in a walled real-time engine, and what it costs to bridge that with a second system), start with the freshness-gap post.

Orientation first. ISK is a gateway in front of Kafka. It projects ranges of Kafka offsets as Iceberg data files and serves them through a standard Iceberg REST catalog and an S3-compatible endpoint. No ingest job, no sink connector, no second copy of the data - Kafka is the durable log, and ISK exposes it as Iceberg. The CDC handling architecture below is what happens on top of that projection to make it a current-state table rather than a raw change-log.

The load-bearing invariant: data files are never filtered

Start with the decision everything else hangs from: an ISK data file is a pure function of the Kafka offset range it covers. Fetch offsets [1000, 2000] for a topic and you get the same rows in the same positions every time: the change events in that range, in order, untouched. No deduplication is baked into the bytes. No filtering or any kind of transformation is applied to the data once its read, converted to parquet and cached.

The CDC logic lives in the second half of the design: supersessions live entirely in a separate delete-file layer, composed at read time and applied by the query engine. The data files stay dumb and immutable; the deletes carry the per key suppression and enable the latest value per key resolution.

Two consequences fall straight out of that split:

• Caching is exact. A data file is a deterministic function of its offset range, so a cached copy is always valid - same rows, same positions a fresh Kafka fetch would produce. ISK can cache aggressively, evict freely, and refetch on demand with no staleness and complex invalidation handling. Each event is pulled from Kafka roughly once. 
• Serving is zero-copy. On a cache hit, ISK streams the Parquet bytes opaquely: no deserialization, no filtering, no per-query CDC work on the data path. 

One caveat: a delete event (op=DELETE) isn't written to the data file at all. Its offset simply has no row - a gap. Suppression of a deleted key is expressed not on the delete event's own position, but as a mark on the prior live row of that key.

‍

The serving read-path

When a query engine opens an ISK table, three kinds of requests flow across the Iceberg protocol. The division of labor between them is the design.

1 - Load the table (metadata). The engine asks the REST catalog to load the table. ISK computes the Iceberg metadata on demand from the current state of Kafka: it reads the topic's latest offsets and schema, lays out the offset ranges as Iceberg data files (aligned to fixed chunk boundaries, so the layout is stable and prunable), and returns manifests and a single snapshot. Computed, not stored - the metadata is ephemeral, held in an in-memory cache with a short TTL, never written to a metastore. Nothing has touched Kafka data or Parquet yet; this is pure catalog. When offsets haven't moved since the last load (the common case while dozens of engine tasks open the same table during one scan), ISK short-circuits and returns the already-computed metadata.

2 - Fetch the data files. The engine issues S3 GETs for the data files named in the manifest. On a cache hit, ISK streams the cached Parquet opaquely. On a miss, it fetches exactly that offset range from Kafka, converts the change events to Parquet (streaming, one row group at a time), writes the file to the local cache, and streams it. Same range next time → served from cache. This is the only place Kafka data is touched.

3 - Fetch the delete files. For a CDC table, the manifest also references delete files. The engine GETs those, and this is where current-state is actually produced: ISK composes a positional-delete file on demand for the requested data, and streams it back. The engine applies the deletes (standard Iceberg positional-delete semantics) and sees exactly the latest live row per key.

So the split is clean: the data path is dumb, immutable and heavily cached; the delete path holds all the CDC intelligence, and runs lazily, per read. No background consumer, no pre-materialization, no always-on merge. First touch pays, the rest amortize.

‍

Deduplication: the supersession scan

How does ISK know which row positions to suppress, without rewriting anything? A backward scan over the change events - newest first.

Walk a partition's offset range from the highest offset down. For each event for key K, look for the nearest newer copy of K already seen in the scan. If one exists, the older position is dead: mark a suppression bit there. An update supersedes the row it updates; a delete supersedes the prior live row of its key - the delete event itself being a gap that never occupies a row. Keep a small per-key record of the latest offset seen for K as you go, so lower events can find what superseded them.

Why backward? Correctness first. A row can only know it's been superseded once the thing that superseded it has been placed, so scanning newest-first guarantees that when the walk reaches an older copy of K, every newer copy is already accounted for and the nearest one wins. A forward scan would miss suppressions (or would need to make a list per key and resolve after the scan).

Then serving latency. A suppression can only ever come from a newer event, so once the walk has passed a range, that range's deletes are final (nothing older can change them) and its delete files can be served immediately while the scan continues downward. The walk doesn't have to finish before results start flowing. Backward order isn't an optimization bolted on; it's what makes the scan correct and streamable at once.

Two properties matter here:

• Incremental and lazy. The scan is triggered by a delete-file request, and covers only the range a reader actually asked about, minus whatever earlier reads have already processed. Progress is persisted - the work is done once per range, never repeated. 
• Single writer, many readers. One scan writes suppression state for a partition at a time; reads compose the results without locking. Once frozen for a chunk, that state is never mutated - later work only adds marks in new places. That's what makes lock-free reads safe. 

‍

The index and bitmap maintenance path

This is the core - what makes computing current-state at read time cheap enough to actually do.

Two structures, different jobs:

• A key index - for each key, where is its latest copy, and is it currently deleted? The supersession scan consults it, and it feeds the deleted-keys projection used by the roll (below). It is not a set of delete marks. 
• A suppression bitmap store - the delete bits themselves, one compact bitmap per data chunk, a set bit meaning "the row at this position is superseded." These compose into a delete file at read time. 

The bitmaps are maintained incrementally, as a chain of deltas. Each time a read advances the Kafka high-watermark, ISK adds a delta at the head of the chain, ordered by a monotonic counter. New change events append new bitmaps onto the head; nothing already in the chain is rewritten. Maintaining current-state is append-only work - the expensive scan touches the new tail, never the settled history.

The payoff lands twice.

Point-in-time views come for free. A reader's delete set is the OR of the bitmaps along the chain up to that reader's snapshot. Open the table earlier, compose fewer deltas, see an earlier state; open it later, compose more, see a fresher one - over the same, untouched data files underneath. Engines scanning concurrently at different offsets each get a consistent point-in-time view by construction, with no coordination and no locking. MVCC, falling straight out of deletes being composable bitmaps.

The cost amortizes. The scan that builds a chunk's suppression bits runs once; every later reader at or past that point reuses the result. Dedup cost tracks the volume of change - not the number of queries, the way read-time dedup does, and not the commit cadence, the way a continuous merge does. This is the mechanism behind the claim in the freshness-gap post that ISK's dedup is incremental and self-amortizing.

To keep the chain bounded, retired snapshots' deltas fold into the baseline as they age out - a background reconciliation OR-ing adjacent deltas together, value-preserving for every live reader because the composed result is identical either way. The structure is persisted locally, so a restart resumes from the saved index rather than replaying the topic. In-flight readers re-plan, which is ordinary Iceberg behavior.

Rendering to the engine's native delete primitive

The composed bitmap is abstract - a set of superseded offsets. The last step turns it into something a specific engine applies. ISK walks the chunk's offset range and emits a positional-delete entry (data_file_path, row_position) for every set bit, translating offsets to row positions as it goes. Here the gaps matter: an offset with no row (a delete event) doesn't advance the row position, so the mapping stays correct even though the data file is sparser than the offset range. The delete file is tagged with the same partition spec as its data file, so engines can prune them together.

Because suppression is held as engine-agnostic bitmaps, the same composed state serializes to whatever a reader needs: Iceberg v2 positional deletes for Snowflake, v3 deletion vectors for Databricks. Same bits, different wire format. That's the advantage of keeping deletes as bitmap algebra instead of baking them into the data - one substrate, and the per-engine rendering is a thin projection off it.

The one place Kafka isn't immutable: log compaction

Anyone who knows Kafka has an objection queued by now. Positional deletes point at row positions, and row positions are only stable if the data file is stable - which leans on Kafka being an immutable log. It is, with one exception: log compaction. A compacted topic can physically drop a superseded record once a newer one for the same key exists. If ISK has already materialized, cached, and issued deletes against the chunk that held it, a later refetch returning one fewer row shifts every row after it up by one - and every positional delete pointing past it now points at the wrong row. Silent corruption.

The precision that matters: ISK doesn't need the bytes identical across refetches. It needs the offset-to-row-position layout identical. Row-level, not byte-level.

So the layout is frozen on first materialization. When a chunk is fetched again and an offset present the first time has since been compacted away, ISK re-materializes it as a filler row in its original position - a schema-valid dead row carrying null/default values. Row count and every position match the first fetch. Nothing downstream shifts.

A filler row is dead by construction, so it must always be suppressed - Kafka only compacts away deleted rows (either by key or tombstone). ISK does that with a per-chunk filler mask - itself a bitmap of the filled positions, OR'd into the composed delete at render time alongside the supersession bitmaps. Same bitmap algebra, one more mask. The engine never sees a filler.

Which mirrors the gaps from the top of the post. An op=DELETE is an offset that never had a row - a true gap, advancing no position. A compacted-away record is an offset that did have a row and now doesn't, so rather than let it collapse into a gap and shift the layout, ISK pins it in place with a filler and suppresses it. Two kinds of absence, one stable offset-to-row map.

‍

Freshness, and the roll to cold

Everything so far is the hotset: current-state served live off Kafka. The coldset stays the customer's. On whatever cadence you choose, your own ETL rolls the hot view into your own compacted Iceberg table - reading the current rows plus a deleted-keys projection ISK exposes (so the roll knows which keys to remove, solving the "a MERGE doesn't propagate deletes" problem), then writing your cold table with a normal MERGE/DELETE. ISK never writes the coldset; it exposes the surfaces the roll reads, and the watermark marks the boundary. Freshness (what the hot view shows, live from Kafka) is decoupled from how often you durably roll to cold. The cost post is where that decoupling turns into money.

‍

What's honest about this

In the spirit of the other two posts, the real edges:

• The published scope is hotset deduplication. The coldset is the customer's, held at par; ISK deduplicates the live view, not the customer's historical estate. 
• A restart re-plans in-flight queries. Point-in-time references are in-memory; after a restart an open query re-plans. Standard Iceberg behavior, not a data-loss window. 

None of these touch the core claim, worth restating plainly: current-state from a CDC stream, served as ordinary Iceberg to any engine, deduplicated at read time by composing incrementally-maintained bitmaps over never-rewritten data files. The cost tracks change, not queries. Freshness is a property of the read, not of a batch cadence.

That's the how. The why it matters is the freshness-gap post; the what it costs is the cost post.

‍