Context Engineering vs Bigger Models: Why AI Agents Fail
AI agents usually fail from broken context, not weak models. Here's why context engineering matters more than model size in production.
Why do AI agents fail even when the model is smart enough?
Most agent failures trace back to broken context, not weak reasoning. An agent given the wrong data, stale data, or data that arrives too slowly will produce confident, wrong answers regardless of how capable the underlying model is. Swapping in a bigger flagship model rarely fixes this because the model’s reasoning was never the problem. The fix is context engineering: building the runtime system that decides what information an agent sees at each step, how fresh it is, and how fast it arrives.
TL;DR
- Agent failures are usually a plumbing problem, not a model problem: broken, stale, or slow context causes hallucinations and wrong answers even when the model reasons correctly.
- Context engineering is a layer above prompt engineering: prompt engineering tunes a single instruction string, while context engineering builds the live system that fetches and delivers the right data at each step.
- Four mechanical rules define production-ready context: it has to be navigable, fresh, fast, and able to compound across sessions.
- Disconnected data silos are the root cause in most organizations, where separate teams build isolated vector stores, custom scripts, and one-off tool integrations that never talk to each other.
- Agents need two memory tiers: a working memory scratchpad for the active task and a long-term memory store for durable facts that survive across sessions.
- A demo using the Redis agent memory server showed a coding agent automatically storing project conventions in one session and retrieving them by meaning, not keyword, in a completely fresh session.
Built like a system. Not vibe-coded.
Remy manages the project — every layer architected, not stitched together at the last second.
What is context engineering, and how is it different from prompt engineering?
Prompt engineering is about the instruction you hand the model: phrasing, few-shot examples, output schemas. Everything happens inside a single string, and the goal is to get the best possible response from that one call.
Context engineering operates a layer above that. It’s the infrastructure deciding what the agent sees before it even starts reasoning: which customer record to pull, which live system state to check, and how to get that data to the model fast enough to be useful. Prompt engineering shapes what you say to the model. Context engineering shapes the environment the model operates inside.
This distinction matters because most agentic failures don’t originate in the prompt. They originate in the data pipeline feeding the prompt. An agent can have a perfectly engineered prompt and still fail if the record it retrieves is twelve hours out of date, or if it can’t find a related record at all.
What are the common failure modes in production agents?
Three mechanical failure patterns show up repeatedly in agent systems, particularly in customer-facing use cases like support agents:
Dead ends. The agent pulls one record (say, a support ticket) but there’s no relational path to a connected record (say, the order) because the two live in separate data silos. With no path to follow, the model hallucinates rather than admitting it can’t find the data.
Stale state. The agent finds the right record, but it’s reading from a snapshot or export that’s hours old. A package that shipped that morning still shows as “not shipped” in the agent’s view, so it confidently tells the user wrong information.
Latency. The agent finds the correct data, but doing so requires chaining several sequential calls across multiple APIs. By the time the answer comes back, the user has already given up and closed the session.
In each case, the model’s reasoning was sound. The failure sits entirely in how context was retrieved, how current it was, and how quickly it arrived.
What are the four rules production context has to satisfy?
Context that actually works in production needs to meet four mechanical requirements:
Navigable. The agent needs to traverse relationships between entities (users, orders, tickets) rather than relying on blind keyword lookups over raw text. If related records aren’t linked, the agent hits dead ends.
Fresh. Acting on stale state means acting confidently wrong. Freshness isn’t a nice-to-have; it’s what separates a useful answer from a wrong one delivered with total confidence.
Fast. When a single task triggers dozens of subqueries, each unit of latency compounds. Slow context retrieval turns into slow, broken workflows and abandoned user sessions.
Compounding. This is the rule most setups ignore entirely. When a session ends, most agents wipe their context to zero. The next time the same user shows up, the model has to relearn who they are, what conventions they use, and what already broke last time. A production system that’s actually effective should get smarter with use, not reset to zero every session.
Remy doesn't write the code. It manages the agents who do.
Remy runs the project. The specialists do the work. You work with the PM, not the implementers.
Why do engineering teams struggle to build this?
The pattern shows up repeatedly across organizations: one team builds an agent with an isolated vector database for document search. It works. Another team rolls its own persistence scripts over a separate store. It also works. A third team stitches together a handful of tools for their specific use case. None of these systems talk to each other.
Months later, the organization ends up with a dozen disconnected data pipelines, no one can guarantee freshness across them, and every new agent project spends most of its early development time rebuilding context plumbing that already exists somewhere else in the company, just not in a reusable form.
This is the core reason context engineering hasn’t been treated as its own discipline. It’s not that teams don’t understand the problem. It’s that the plumbing is being rebuilt from scratch every time instead of being built as shared infrastructure.
How do agents compound context across sessions?
Compounding context requires two distinct memory tiers.
Working memory is the active task scratchpad: the immediate tool calls, their state, and execution details for the current session. This gets wiped when the session ends, which is fine, because it’s not meant to persist.
Long-term memory stores durable facts meant to survive across sessions: project decisions, environment configuration, conventions, previously encountered edge cases. This is what lets an agent “remember” a user or a codebase from one day to the next.
The mechanism that matters is how facts move from working memory into long-term memory. Relying on users to manually flag which facts are worth keeping doesn’t scale. An automated memory layer needs to parse conversations in the background and extract durable facts on its own. When a new session starts, the system runs a semantic search over that long-term store and pulls only the facts relevant to the current task, rather than dumping everything back in. The layer also needs to handle deduplication and expiry so memory doesn’t turn into a cluttered junk drawer of outdated or transient information.
Is this approach demonstrated in practice?
One demonstration used a coding agent (DeepSeek’s coder model accessed via API, run through a harness) connected to the Redis agent memory server through an MCP integration. Redis stored each memory as a hash with an attached vector, enabling semantic search instead of keyword matching.
In an initial session, the user told the agent to use specific tooling conventions (a particular dependency manager, a particular linter, and a rule that functions should stay pure and return new tuples rather than mutating state). The agent, finding no existing memory, created three separate long-term memory entries for these facts.
Other agents ship a demo. Remy ships an app.
Real backend. Real database. Real auth. Real plumbing. Remy has it all.
In a second, completely fresh session with zero shared chat history, the agent was asked to add a new function with tests. Before touching any files, it queried long-term memory, retrieved the stored conventions, and wrote code that matched the established style without being told the conventions again. Tests and linting passed. A follow-up test confirmed the retrieval was genuinely semantic: a query using neither of the original keywords still correctly surfaced the relevant stored convention with the closest match. Another test showed the memory layer extracting a durable fact (a stated preference for type hints and descriptive test names) from an ordinary conversation without being explicitly told to store it.
Frequently Asked Questions
What is context engineering in AI agents?
It’s the discipline of building the runtime system that determines what data an agent sees at each step of a task, including which records to fetch, how current they are, and how quickly they can be delivered, as distinct from prompt engineering, which only shapes the instruction text sent to the model.
Why doesn’t upgrading to a bigger model fix agent failures?
Because most agent failures stem from broken, stale, or slow context rather than insufficient reasoning ability. A larger model given the same broken data pipeline will still hallucinate, still act on outdated information, and still be slowed down by the same chain of API calls.
What’s the difference between working memory and long-term memory in an agent?
Working memory is the scratchpad for an active task, including tool calls and execution state, and it gets discarded when the session ends. Long-term memory stores durable facts, like project conventions or past decisions, meant to persist and be retrieved across future sessions.
How does semantic memory retrieval differ from keyword search?
Semantic retrieval stores memories as vector embeddings and matches them by meaning, so a query can surface a relevant stored fact even if it shares no exact keywords with how that fact was originally phrased.
Why do so many organizations end up with disconnected agent context systems?
Different teams tend to build isolated solutions (separate vector databases, custom scripts, one-off tool integrations) for their own projects without a shared context layer, resulting in multiple disconnected data pipelines where no one can guarantee freshness and every new project has to rebuild the same plumbing.
