Knowing what’s wrong before it gets worse: how Streambased powers AI anomaly detection

An alert fires saying a payment service outage is resolved. The metrics dashboard still shows elevated error rates. The incident platform still says the ticket is active. All three are technically correct – and an AI agent reasoning across all three has no reliable way to know what's actually happening. The fourth post in our six-part series on Streambased and AI. 

Anomaly detection sounds like a solved problem. Set thresholds, watch for breaches, fire an alert. In practice, the alert is rarely the hard part. The hard part is what comes next: understanding what the alert actually means, whether it’s serious, what caused it and what to do about it – all quickly enough for the response to matter.

AI has significant potential here. But so far, most deployments have fallen short of that potential. Why? And what does it take to build anomaly detection that genuinely works in production?

‍

The alert isn’t the problem. The context is.
Detecting anomalies requires two things simultaneously: the live signal that tells you something has changed, and the historical context that tells you whether that change is significant. Without both, AI agents either generate noise – flagging things that aren't really problems – or miss the signals that are.

A single anomalous event is almost never meaningful on its own. A spike in transaction failures might be a serious fraud pattern, a routine system hiccup or the downstream effect of a scheduled maintenance window that someone forgot to log. An unusual vibration reading from a piece of manufacturing equipment might indicate imminent mechanical failure, or it might fall well within the normal range for that machine at that point in a production cycle.

Without historical baselines, an AI agent can’t distinguish. It sees the deviation but has no frame of reference for whether that deviation is significant. The result is alert fatigue – operations teams drowning in notifications that turn out to be routine, and so slower to respond when something genuinely serious arrives.

In financial services, the stakes are particularly high. Fraud patterns often only become visible when a live signal is read against a background of historical behaviour. A single unusual transaction from a long-standing account might be entirely legitimate. But the same transaction from an account that has shown three similar patterns in the past fortnight, combined with a recent change in login location, tells a very different story.

In telecoms, this dynamic plays out at network level. A spike in dropped connections on a cell cluster might look significant in isolation. Against the historical baseline for that location and time of day – for example, taking into account a regular commuter surge, a nearby venue event or a weather pattern that consistently affects signal quality – it might be entirely expected. 

Or it might be the early signature of infrastructure degradation that will cascade into a wider outage if left unaddressed. Without being able to see the historical layer at the same time, the AI can’t tell the difference.

‍

When the data itself disagrees
Another fundamental problem underlies the contextual issue, one we’ve encountered repeatedly in real production environments. Anomaly detection agents don’t just struggle with incomplete data – they often receive contradictory versions of the present from different systems simultaneously.

Take an example from a payments environment. An alert stream reports that a service outage is resolved. But the metrics dashboard still shows an error rate of 18.7%, because it operates on rolling five-minute windows. The incident management platform still shows the ticket as active too, because no one has manually closed it yet.

All three data sources are technically correct. But none of them is giving the agent a reliable picture of what’s actually happening right now.

This is less an AI problem than a distributed systems problem that impacts agentic performance. Agents are being asked to arbitrate between systems that don’t agree on time: a warehouse refreshing hourly, a CRM lagging by minutes, an event stream accurate to the millisecond. In anomaly detection, where the difference between ‘resolved’ and ‘ongoing’ can mean significant financial or operational exposure, that inconsistency can be the difference between a correct diagnosis and a costly one.

‍

How Streambased brings it together
Streambased addresses both the completeness problem and the consistency problem through its unified layer across Kafka and Iceberg. 

Metrics, logs, transaction traces and operational events stream into Kafka as they happen. Those same streams persist to Iceberg, building the historical baseline against which live signals can be compared. Streambased merges both into a single coherent temporal view before the AI agent ever sees the data, so the agent isn’t left arbitrating between conflicting system timestamps or reasoning from a partial picture.

The result is an AI investigative agent that can retrieve correlated context via MCP: not just the anomalous event itself, but related signals from the past few minutes, recent system changes, historical patterns for this entity or account, and the current state across all relevant systems. All consistently timestamped and ready to reason over.

In manufacturing, that might mean an AI agent that spots an unusual vibration signature in equipment telemetry, cross-references it against maintenance history, production schedules and the performance record of that specific machine, and identifies a likely bearing failure before the line goes down. The difference between catching that signal early and missing it can be the difference between a planned maintenance window and an unplanned production stoppage – with all that implies for output, contracts and cost.

In financial services, it means fraud signals that surface in context rather than isolation, giving compliance teams explanations and recommended actions rather than raw alerts to investigate from scratch.

‍

From alerts to answers
The goal of AI anomaly detection isn’t more alerts. It’s faster, more accurate diagnosis – ideally, early identification of patterns before they become incidents at all. That shift from reactive to proactive requires an AI agent that can see both what’s happening and what it means, simultaneously and consistently.

Streambased makes this achievable without the architectural complexity that has traditionally required large, specialist data engineering teams to build and maintain. The historical context and the real-time signal are unified at the data layer, so the agent can focus on reasoning rather than reconciling.

‍