LLM Agent Memory & Planning: Engineering Patterns

LLM agents have moved rapidly from research curiosity to production staple — but the gap between a demo that impresses and a system that reliably handles thousands of real requests remains surprisingly wide. Most of that gap traces to two components that are chronically underspecified in tutorials and overestimated in benchmarks: memory and planning. How an agent stores context across steps, and how it decides which action to take next, determines whether it actually completes tasks or spirals into confused repetition and tool misuse.

Image: File:Artificial Intelligent Agent.png — Tcharon (CC0), via Wikimedia Commons

The Four Memory Layers Every Production LLM Agent Needs

Memory in an LLM agent system is not a single concept — it exists at multiple layers with different scope, persistence, and retrieval characteristics. Conflating these layers is one of the most common architectural mistakes in agent engineering.

1. In-context (working) memory is what lives in the active prompt window. It is fast, precise, and ephemeral — gone when the context limit hits or the session ends. For most agents, this is also the most expensive memory to fill: every token in context costs inference time and money. The engineering discipline here is ruthless context management — populate it only with what the agent actually needs for the current step.

2. External episodic memory is a persistent store — typically a vector database — that holds summaries or embeddings of past interactions, decisions, and outcomes. When an agent needs to recall what happened two sessions ago, or what a user previously told it, retrieval-augmented memory fills that gap. Retrieval quality is everything: bad chunking strategies or imprecise embedding models produce retrieved context that confuses more than it clarifies.

3. Semantic (knowledge) memory holds structured facts about the world — product catalogs, documentation, policies, entity relationships. This is often populated before the agent runs and accessed via retrieval or direct lookup. The key design decision is whether to store this in a vector store (for fuzzy relevance) or a structured database (for precise lookup). Most production systems benefit from both, with a routing layer that selects the appropriate retrieval path based on query type.

4. Procedural memory encodes how to do things: tool schemas, few-shot examples of successful actions, instructions for handling edge cases. This lives primarily in the system prompt. Unlike the other three layers, procedural memory is curated by engineers rather than accumulated at runtime — which means its quality directly reflects the care taken in prompt engineering.

Planning Architectures That Actually Work

An agent without a planning strategy is a one-step lookup. Adding explicit planning — making the agent reason about what needs to happen before it acts — dramatically improves task completion rates on multi-step problems. The tradeoff is latency and token cost, which means planning depth should match actual task complexity rather than defaulting to the most sophisticated approach.

The ReAct (Reason + Act) pattern, introduced by Yao et al. (2022), is the most widely deployed planning approach in production systems. At each step, the model emits a reasoning trace — "I need to look up the customer account before checking their subscription status" — then selects an action, observes the result, and reasons again. The interleaved reasoning helps the model stay on track and recover from tool errors. Its main weakness is that each step is sequential, making long tasks with many actions slow.

The Plan-then-Execute approach has the model generate a complete plan upfront — a numbered list of steps — then execute them in sequence, potentially with a separate executor model. This works well for stable, well-defined tasks where the steps are known in advance. It breaks down when early steps surface information that should change the plan, because the model committed to its roadmap before seeing the actual data.

The hierarchical agent pattern splits the problem across an orchestrator and specialist subagents. The orchestrator handles high-level decomposition and routing; specialists handle narrow tool domains. This scales better than monolithic agents but introduces coordination overhead and makes debugging harder — a failure in a subagent can be difficult to trace from the orchestrator's output alone.

Engineering Reliable Tool Calling

Tool calling is where production LLM agents most visibly fail. The model decides which tool to invoke and what arguments to pass — and each of those decisions is a probabilistic output, not a guaranteed computation. Engineering reliability requires treating every aspect of the tool interface as a prompt engineering surface, not just documentation.

The most impactful practices for improving tool call reliability in production:

• Write tool descriptions as instructions, not specifications. Instead of "Retrieves customer data", use: "Call this when you need account details for a specific customer. Requires a valid customer_id. Do NOT call if you only have an email — use lookup_customer_by_email first." The difference in correct tool selection on ambiguous inputs is substantial.
• Minimize the tool set exposed at each step. An agent choosing from 40 tools makes worse selections than one choosing from 5–8 contextually relevant tools. Dynamic tool filtering — presenting only the tools relevant to the current subtask — consistently improves decision quality without restricting overall agent capability.
• Validate tool outputs before they re-enter context. A tool returning an error or unexpected schema should not silently pass into the next reasoning step. Validate at the tool-call layer and produce a structured error message the model can reason about explicitly.
• Set explicit retry budgets. Unlimited retries on failed tool calls create infinite loops in production. A hard budget of 2–3 retries per tool call with escalating fallback behavior prevents the most common failure mode in stateful production agents.

Image: File:Workflow of a machine-learning-based AI system.png — Nano Banana 2 (Public domain), via Wikimedia Commons

Memory Persistence: What to Store and When

Naive implementations store everything and retrieve by keyword similarity. That works for demos. In production, the question shifts to what is actually worth storing, in what form, and with what retrieval strategy.

What to always persist: task outcomes with the reason for success or failure, decisions confirmed correct by user feedback, recurring user preferences, and tool results that were expensive to produce and are likely to be needed again in future sessions.

What clutters memory and degrades retrieval: raw tool outputs (store processed summaries instead), reasoning traces from failed action sequences, and high-frequency operational logs — these belong in an observability system, not agent memory.

The retrieval strategy matters as much as the storage strategy. For most production use cases, a hybrid of dense retrieval (vector similarity) with a sparse filter (keyword or metadata matching) outperforms either approach alone. Metadata filtering prevents semantically similar but contextually wrong memories from surfacing — a recurring failure pattern in agents that handle multiple users or topics within the same vector store.

Common Memory Failure Modes to Anticipate

Four failure modes appear repeatedly in production agent systems with persistent memory:

1. Context overload: Retrieved memories fill the context window before the actual task instructions have adequate space. Fix: cap memory injection at 20–30% of available context, prioritized by recency and relevance score.
2. Stale memory poisoning: Old information contradicts current system state and the agent acts on the outdated version. Fix: timestamp every stored item and implement explicit staleness thresholds per memory category.
3. Retrieval hallucination amplification: The model retrieves a weakly relevant memory and, because it is now in context, treats it as authoritative ground truth. Fix: include confidence scores in retrieved context and instruct the model explicitly to discount low-confidence retrievals.
4. Cross-user contamination: In multi-tenant systems, memories from one user's session surface during another user's query. Fix: namespace every vector store entry with a user or session identifier and enforce filtering at query time — never retrieve across namespace boundaries without explicit authorization logic.

Frequently Asked Questions

How much persistent memory does a production LLM agent actually need?

For many use cases, surprisingly little — but it must be the right memory. An agent handling customer support typically needs to recall three things: preferences established in the current session, key facts from the last one or two interactions, and relevant policy snippets. Trying to give the agent access to every historical interaction rarely improves outcomes and often degrades them through retrieval noise. Start with minimal memory and add categories only when you can measure that they improve task completion rate on real workloads.

What is the practical difference between ReAct and Chain-of-Thought for agent tasks?

Chain-of-Thought encourages the model to reason through a problem in writing before producing a final answer — it is a reasoning scaffold for a single output. ReAct extends this to interactive tasks where the model must decide on actions, observe real results from tools, and adjust its reasoning based on what it actually finds. Chain-of-Thought works well for closed reasoning problems like math and logical analysis. ReAct works better when the agent needs to interact with external systems and adapt based on actual tool outputs. In practice, most production agents use a ReAct-style loop for external interactions and Chain-of-Thought reasoning within individual planning steps.

Should teams build their own memory system or use a framework like LangChain or LlamaIndex?

For teams shipping their first production agent, established frameworks reduce time-to-working-system significantly and handle many low-level memory management details correctly out of the box. The tradeoff is opacity — when behavior is unexpected, framework abstractions make diagnosis harder. We recommend using a framework to validate the architecture and reach a working prototype, then evaluating whether the abstraction cost is worthwhile at scale. Many teams find they need to replace specific components — especially retrieval logic and tool call handling — with custom implementations as they tune for production reliability.

The Bottom Line

We find that the teams building the most reliable production LLM agents share a consistent set of practices: they treat memory as a first-class engineering concern with its own data model and retrieval strategy; they pick the simplest planning pattern that their task complexity actually requires rather than the most architecturally impressive one; and they invest heavily in tool interface design, treating every tool description as a critical piece of prompt engineering. The model is only as reliable as the scaffolding around it. Build that scaffolding with the same rigor you would apply to any production system, and the gap between demo and reliable production narrows considerably.

Disclaimer: This article is for informational purposes only. Technology landscapes change rapidly; verify information with official sources before making technical decisions.