Where Event Streaming Fits in Agentic AI Workflows

If you want responsive and delightful traditional applications, you need event streaming. "Your taxi will be there shortly" is good, but "Your taxi is still 10 mins away and you can have another drink with your friends" is better. The same gap exists for AI agents. A reactive agent with narrowly scoped and stale data can still be useful. A proactive agent with full context is better.

TL;DR

This post argues that event streaming belongs at the centre of agentic workflows, both as the trigger and as part of the evidence the agent can query. Getting there takes two practical changes, not a complete rewrite of your platform.

Detection is solved. Investigation is different.

A decade of stream processing has made event streams exceptionally good at detecting when something needs attention. Kafka and its ecosystem can identify threshold breaches, anomalies, suspicious patterns, and operational conditions as they happen and trigger workflows in response.

But detection is not investigation.

An agent responding to a suspicious payment needs to determine whether it is fraud or a false positive. An agent responding to a conversion drop needs to understand whether the problem is isolated to one channel or part of a broader trend. These investigations are unpredictable. They require flexible analytical queries across historical data, multiple domains, and business context that cannot be anticipated when the detection logic is written. This is the key departure from traditional data architecture. Agentic problems cannot be solved by simply building a gold table that holds the data needed.

Flexible analytical access is exactly what the lakehouse is built for. The event stream detects the problem. The agent investigates it using the rich historical context in the lakehouse. Streaming provides the trigger; batch analytics provides the understanding.

The investigation needs the events that triggered it

There is one obvious critical gap: the investigation must include the evidence that caused it to begin.

The triggering event, and potentially other recent events relevant to the investigation, already exist on the stream. But they may not yet exist in the lakehouse.

Streams are continuously ingested into lakehouse tables, but persistence is not instantaneous. Events must be written, committed, and made visible to analytical queries. Even with fast ingestion, there is a window in which the stream contains information that the lakehouse does not. This can be solved by chasing faster ingestion but in most cases this is an expensive and often fruitless exercise.

Consider an unexpected drop in conversion rates. Stream processing detects the anomaly immediately and triggers an investigation. But understanding the cause requires more than examining recent events. The agent might need to correlate changes in customer behaviour with a recent deployment, a pricing experiment, a marketing campaign, or historical patterns across different customer segments.

The agent queries the lakehouse to investigate, but the tables are several hours behind the stream. The latest customer interactions, including those that triggered the anomaly, are missing. The agent has access to years of historical context and the flexibility to explore competing explanations, but not the most recent evidence needed to distinguish between them.

The disconnect between detection and investigation

The system that detects the problem knows something the system investigating it does not.

The result is a broken handoff between detection and investigation. The agent must either wait for the lakehouse to catch up, sacrificing the immediacy of streaming, or proceed with an incomplete picture, sacrificing the quality of its judgement.

Neither is acceptable for agents expected to act on live business events.

Two changes that make it work

Shift bronze to silver left into the stream. Traditionally, raw events land as bronze data and lakehouse jobs turn them into silver tables through ETL. For agentic workflows, cleaning, validation, enrichment and schema work must move left. Streaming infrastructure produces silver-quality streams continuously, then lands them asynchronously into the lakehouse.
Streaming consumers get trustworthy events sooner, and the lakehouse receives silver-grade data instead of repeating bronze-to-silver work. Kappa architecture already points this way, and many organisations are part way there. Agentic use cases raise the priority.
Silver-to-gold work, dashboards and curated products stay in the lakehouse. This evolves medallion architecture rather than replacing it.
Give agents a unified view of stream and table. Agents need one query surface over the silver stream and the silver lakehouse table, spanning from the last millisecond on the stream to years of history in the lakehouse. One analytical question sees live evidence and long-term context together, without stitching systems by hand or waiting for the next commit.

diagram

Today: lakehouse jobs own bronze to silver; freshest evidence appears only after table commits; the triggering event is missing until persisted; agents wait or reconcile systems.

Unified streaming lakehouse: streaming infrastructure owns bronze to silver before landing; events sit with lakehouse history; the triggering event is included when investigation starts; agents use one analytical interface.

Production incident investigation in practice

Consider a SaaS platform that produces application logs, infrastructure metrics, and distributed traces into event streams. Stream processors enrich those events with service metadata and detect anomalies in latency, error rates, and resource utilisation.

During a major customer's monthly billing run, API error rates suddenly spike. Stream processing detects the anomaly and triggers an investigation. The latest logs point to database connection exhaustion, but they don't explain why a workload that normally succeeds is suddenly failing.

An agent queries the unified view, combining years of operational history, customer configurations, previous incidents, and deployment records from the lakehouse with the latest logs and traces still on the stream.

The investigation uncovers a similar failure during an annual reconciliation nine months earlier, involving a different customer with the same unusual billing configuration. Historical performance data reveals that this workload has always been sensitive to external service latency. A connection-pooling change deployed three weeks ago has reduced its tolerance for slow responses.

The decisive evidence comes from the latest traces: billing requests are holding database connections open while waiting for an external tax calculation service. Those traces have not yet reached the lakehouse.

With that context, the agent can identify the interaction between the configuration, the recent change, and the live dependency slowdown. It recommends a targeted mitigation rather than simply increasing the database connection limit, and publishes its findings to the incident management workflow.

Detection happened on the stream. Investigation required the depth of the lakehouse and the immediacy of the stream, queried together.

What agents need

To recap, agents that respond to live business events need to be three things:

Proactive. They must recognise when something needs attention and initiate an investigation without waiting for a human or a scheduled batch job.
Flexible. They must be free to investigate unexpected causes, asking questions that weren't anticipated when the detection logic was written.
Current. They must reason over the latest evidence, including the events that triggered the investigation, rather than an incomplete snapshot of the recent past.

Delivering all three requires two architectural capabilities:

Silver streams provide clean, structured, continuously updated events that can drive reliable detection and trigger agent workflows.
A unified streaming and lakehouse view gives agents the freedom to investigate across deep historical context and the latest events, through the same analytical interface.

Neither capability is sufficient on its own. Streams make agents proactive. The lakehouse makes their investigations flexible. Unifying the two ensures those investigations are current.

Familiar pieces, used differently

The future of agentic AI is about context, but this does not have to be complex, scary or revolutionary. It is about bringing together technologies and practices we know and trust in a new way: streams for detection, lakehouses for history, and a unified view so agents can investigate both at once. Streambased provides that unified view over Kafka and Iceberg when you are ready to put the pieces together.