Personalisation is one of the most commercially significant applications of AI in retail, media and financial services. It’s also one of the most commonly misfired – and, when it goes wrong, the damage goes beyond a missed sale.
A brand that makes no attempt to personalise is just a shop. But a brand that tries to personalise and gets it wrong has implied it knows you, then demonstrated it doesn’t.
That disappointed expectation can undermine trust in a relationship, especially in subscription businesses, loyalty programmes and high-consideration retail. Customers don’t just ignore bad recommendations, they start to distrust the ones that follow.
So why do recommendations go wrong so consistently? And what can be done about it?
Recommendations, a two-sided problem
The short explanation for this issue is that context has two dimensions that most personalisation systems fail to serve together: who the user has been, and what they’re trying to do right now.
Most personalisation models lean heavily one way or the other:
- Batch-processed historical data gives you a rich profile – purchase history, category affinities, price sensitivity, long-term preferences – but it reflects behaviour from hours or days ago.
- Real-time signals give you current intent – what someone is clicking on, what’s in their basket, what they’ve just searched for. But without history, those signals lack meaning.
Let’s look at how this issue plays out in a few real-world scenarios.
Retail eCommerce Scenarios
A user browsing running shoes is a good illustration. That single signal is almost uninterpretable on its own.
Is this a casual buyer whose old pair of trainers have finally worn out after years of sporadic use, who’s now looking for something comfy and reasonably priced? Or is this a seasoned runner, the sort who tracks their weekly mileage, researching their next performance shoe?
The real-time signal is identical for both. The history is not. Previous purchases, browsing behaviour across running categories, engagement with technical content are what tells the personalisation engine which conversation it’s actually in.
What’s at stake is your whole downstream relationship with that customer.
The recommendations that might follow diverge dramatically.
The first-time buyer, handled well, becomes a repeat customer through accessible, encouraging suggestions: basic kit, a running top, entry-level accessories. Push them toward a £200 carbon-plate race shoe and you’ll lose them.
The serious runner, handled well, becomes a high-value account, interested in things like Garmin watch upgrades, advanced nutrition, premium shoes, technical apparel. Show them a generic carousel and they’ll go somewhere that does understand them.
Media and finance scenarios
In media and streaming, the same disconnect might play out in a way many will recognise. A user watches a Turkish drama series one evening because a Turkish friend or relative is visiting. The real-time signal – engagement with foreign-language content – looks like a new preference. Without the historical record showing years of very different viewing behaviour and no prior interest in subtitled drama, the algorithm treats it as a pivot and starts surfacing Turkish content across the recommendations feed. The relative goes home. The user is now being served an entirely different platform to the one they chose to subscribe to.
The historical record would have flagged this as an outlier immediately. A personalisation model with access to both dimensions treats a one-off contextual event as exactly that. One working only from recent signals quietly reshapes the entire experience around it.
In subscription media, where disengagement rarely announces itself before it becomes cancellation, that’s a potentially costly mistake to make with your own data.
Financial services faces its own version of the issue. A customer researching fixed-rate savings products might be responding to a recent interest rate movement, or acting on a long-running plan to restructure their savings. The contextually appropriate next step – a rate comparison tool, a savings calculator, a call with an adviser – depends on which it is.
Serve the wrong one and the customer starts to wonder whether the brand actually understands their situation at all.
Where the data problem gets harder
There’s a subtler issue underneath all of this. Personalisation models don’t just struggle with incomplete data, they often receive contradictory versions of the present simultaneously. A warehouse refreshes hourly. A CRM lags by minutes. An event stream is accurate to the millisecond.
When an AI agent pulls from all three, it gets multiple competing pictures of the current moment, and making the right recommendation requires knowing which to trust.
This is less a retrieval problem than a data consistency problem. And it’s one reason why stitching together raw context from disconnected systems at inference time tends to produce unreliable results at scale – even when each individual data source is technically correct.
The solution: context and real-time in a single view
.png)
Streambased addresses the problems of data completeness and consistency by surfacing a unified data view across Kafka and Iceberg. Clickstream events, cart updates, browsing activity and purchase signals stream into Kafka as they happen. Those same streams persist to Iceberg over time, creating a scalable historical foundation without a separate ETL process. Streambased merges real-time and historical data into a single coherent view before the model ever sees it, which means the agent isn't left arbitrating between conflicting system timestamps.
Beyond that unified foundation, Streambased also serves specialist data stores – vector search databases for semantic similarity, graph databases for relationship and affinity modelling – to give AI agents access to the richer signal types that sophisticated personalisation requires. The agent retrieves exactly the context it needs via MCP, drawing on live behaviour, historical preference and semantic context simultaneously.
The commercial effect is straightforward: recommendations adapt instantly to current intent without losing the historical grounding that makes them accurate. Conversion improves. Off-target suggestions that could erode trust are reduced.
Consistency as competitive advantage
In high-volume personalisation at scale – a major retailer, streaming platform or digital bank – marginal improvements in recommendation relevance compound quickly. A system that consistently gets context right across millions of interactions doesn’t just improve conversion rates. It builds the kind of trust that keeps users engaged with the recommendation engine and avoids the specific damage that comes when personalisation misfires.
That consistency should be a baseline requirement for any AI personalisation system. The model's reasoning ability is rarely the limiting factor. What unlocks AI's full potential here is a unified view of live signals and historical data. Streambased delivers that – without the architectural complexity that has traditionally kept it out of reach for all but the largest data engineering teams.
.avif)
.avif)
