
From Event to Investigation: How Streaming Data Powers AI Agents
A technical walkthrough of an event-triggered AI investigation, from the arrival of an event in Kafka to an analytical query spanning live and historical data.
November 16, 2026
From Event to Investigation: How Streaming Data Powers AI Agents
TL;DR
A technical walkthrough of an event-triggered AI investigation, from the arrival of an event in Kafka to an analytical query spanning live and historical data.
An AI agent responding to a live event needs more than a notification that something has happened. It needs to investigate that event, understand its context and determine what to do next.
Making this possible requires several components working together, each with a distinct role in the agentic workflow:
- Event streaming (Kafka) provides a continuous flow of live events, making new information available as it happens.
- Stream processing (Apache Flink) continuously evaluates those events, detecting anomalies or conditions that warrant investigation and triggering an AI agent.
- AI agents take responsibility for the investigation, generating hypotheses, asking analytical questions and following the evidence to determine what happened and why.
- The lakehouse (Apache Iceberg) provides the historical and wider scoped data needed to establish baselines, identify patterns and understand events in a broader context.
The critical challenge is that the event triggering an investigation (Kafka) may not yet have been persisted to Iceberg. If the agent queries only the lakehouse, it risks investigating a problem without being able to see the event that caused it.
What’s needed is a unified analytical view (Streambased) that connects live Kafka events with wider Iceberg data, allowing the agent to investigate both through a single query interface.
The sequence diagram below shows how these components come together, from the arrival of an event to an AI investigation grounded in both live and historical data.

Let's follow the sequence from the moment an event arrives.
T0: An event arrives in Kafka
The process begins with an event being published to Kafka.
The event could represent a payment authorisation, an application log, a customer interaction or a change in operational state. For this example, imagine an e-commerce platform publishing checkout events into Kafka.
A checkout event might contain a timestamp, customer identifier, product information, payment status and other relevant attributes.
Crucially, the stream triggering an AI investigation must already be of silver quality. An agent cannot reliably investigate events if the underlying data is inconsistent, incomplete or untrustworthy. The same quality expectations that apply to analytical data in the lakehouse must apply to the live stream.
Fortunately, this is not a new problem to solve. Over a decade of advances in stream processing has given us mature tools and techniques for cleaning, validating, normalising and enriching events as they flow through the streaming infrastructure. These capabilities allow us to produce trustworthy, silver-quality streams before events reach the detection and investigation stages.
The result is a reliable stream of events ready to trigger investigations, without waiting for them to be persisted to Iceberg. Detection happens as data arrives; investigation happens when the agent needs to ask questions of that data.
Step 1: Stream processing consumes the live event
The first message in the diagram passes the event from the event stream to stream processing.
A stream processor, such as Apache Flink, continuously consumes events from Kafka and evaluates them against detection logic. This might involve simple thresholds, aggregations over time windows, comparisons against maintained state or more complex pattern detection.
In our checkout example, the processor could maintain a rolling conversion rate and compare it against an expected baseline.
Suppose conversion normally sits around 4.2%, but the current rate falls to 2.7%.
The processor can identify that the observed change meets the criteria for an investigation.
This part of the architecture is push-based. Events flow continuously through the streaming pipeline, where they are processed and evaluated as they arrive. This contrasts with the pull-based analytical model used later in the investigation, where the AI agent actively queries data to answer questions and follow the evidence.
The important distinction is that the stream processor is responsible for detecting the condition, not explaining it. It can determine that conversion has fallen significantly, but establishing whether the cause is payment failures, product availability, shipping costs or a recent deployment requires further investigation.
Trying to encode every possible explanation into stream processing would mean anticipating every question an investigation might ask. This is much like building a dedicated gold-layer table for every conceivable analytical problem: feasible for a known set of questions, but impractical when AI agents can explore an almost limitless range of hypotheses and analytical paths.
Rather than attempting to predict and precompute every possible answer, the stream processor identifies when something warrants attention and hands responsibility for understanding it to an AI agent. Stream processing determines when to investigate; the agent determines what to investigate and how.
Step 2: Investigating the event
Once a condition has been detected, stream processing triggers the AI agent to begin its investigation.
This marks an important shift in how data is accessed. Detection is push-based: events flow continuously through stream processing, triggering an investigation when a condition is met. Investigation is pull-based: the agent actively queries data, asking questions as it follows the evidence.
The agent receives the relevant context and begins deciding which analytical questions to ask.
In our example, its objective might be:
Investigate the unexpected decline in checkout conversion, identify the likely cause and assess the business impact.
Unlike a predefined diagnostic workflow, the agent determines its investigation path dynamically. It might start by examining payment authorisation rates. If those appear normal, it could investigate conversion by country, device type or product category, before exploring inventory, pricing or recent configuration changes.
The lakehouse may appear to be the perfect place for this investigation to take place but events are persisted to Apache Iceberg asynchronously. An event might already have been processed, contributed to a detected anomaly and triggered an investigation, while still waiting to become visible in the lakehouse.
If the agent queries only the lakehouse, it risks investigating an anomaly without being able to see the very events that triggered it.
Streambased addresses this by providing a unified analytical view across two sources:
- Historical data in Iceberg: persisted analytical data providing baselines, previous behaviour, historical comparisons and context across multiple domains.
- Live data in Kafka: the latest events, including those that have not yet been committed to Iceberg.
The agent queries both through a single analytical interface, without needing to understand where individual events are stored or how to combine the results.
For example, it might query checkout conversion over the last hour, broken down by customer segment, and compare those results against historical behaviour. The query can incorporate both historical events already persisted in Iceberg and recent checkout activity still residing in Kafka.
Crucially, the unified view removes the lakehouse persistence delay from the investigation's critical path. The triggering event can be included in the agent's query results even before it reaches Iceberg.
The agent is free to focus on understanding the problem, while the analytical layer handles the complexity of combining live and historical data.
Step 3: From investigation to action
Now able to leverage a unified view providing the live and historical context it needs to reason accurately about the event. The agent is able to complete its investigation.
Returning to our checkout example, the agent might discover that a recent shipping configuration change increased delivery costs for mobile customers in Germany, explaining the drop in conversion.
With the investigation complete, the agent can determine what happens next. It might take corrective action directly, contact a human for review or approval, or write its findings to a Kafka stream or Iceberg table for downstream processing and further analysis.
Conclusion: Connecting real-time detection with agentic investigation
The architecture brings together two complementary approaches: push-based stream processing to detect when something needs attention, and pull-based analytics to investigate what happened and why.
While the sequence diagram shows a single query, an agent may perform many, following the evidence across different datasets and refining its hypotheses before deciding what action to take.
Making this reliable requires more than simply querying Kafka and Iceberg together. The unified analytical layer must guarantee event visibility, prevent double-counting as data moves between systems, and provide consistent query semantics. Crucially, these complexities belong in the analytical infrastructure, not in the agent.
This approach also preserves the existing lakehouse architecture. Events continue to be persisted asynchronously, and conventional analytical workloads remain unchanged. What changes is that investigations no longer depend on that persistence completing.
Streambased bridges the gap between real-time streaming and historical analytics, giving AI agents the freedom to investigate questions that weren't anticipated in advance, with access to the latest evidence and the historical context needed to reason about it.
The result is a new class of proactive AI agents, capable of identifying problems as they emerge, reasoning with the full context of live and historical data, and taking informed action based on their findings.