Understanding AI Agent Memory Architectures: Short-Term vs Long-Term

"Memory" gets used loosely enough in agent discussions that it's worth being precise about what's actually being implemented, since short-term and long-term agent memory are architecturally different problems with different solutions, not two points on a single spectrum.

Short-term memory: the context window itself

The simplest form of agent memory is just the conversation history included in the model's context window for the current session. This requires no separate storage or retrieval system, it's just accumulating messages and sending them with each new request. Its limitation is the one inherent to any context window: it has a hard size limit, and it disappears entirely once a session ends unless something explicitly persists it elsewhere.

Long-term memory: a separate persisted store

For an agent to remember something across sessions, days or weeks apart, it needs a mechanism outside the context window entirely: information extracted and written to a database, vector store, or similar persistent system, then retrieved and injected back into context when relevant in a future session. This is architecturally a retrieval system, similar in structure to RAG, not simply a bigger context window.

What actually has to happen for long-term memory to work well

Something has to decide what's worth remembering, not every message is worth persisting, and naively storing everything creates a retrieval problem as bad as having no memory system at all, since relevant information gets buried in noise. Something has to decide how to retrieve the right memories at the right time, using the current conversation to search the memory store for genuinely relevant past information, not just the most recent entries. And something has to handle memory that becomes outdated, a user's stated preference from six months ago may no longer be accurate, and a system that treats all stored memories as equally current can act on stale information confidently.

Where teams get this wrong in practice

Treating the context window as if it were long-term memory, expecting an agent to remember something from a session days earlier simply because it was mentioned once, when nothing was actually persisted anywhere durable. And building a long-term memory store that saves everything indiscriminately, without a real strategy for what's worth remembering, which tends to degrade retrieval quality over time as the store fills with low-value entries that compete with genuinely important ones during search.

A practical design pattern that works

Explicit, deliberate memory writes, the agent or a separate process decides specific facts worth persisting (a stated preference, a key project detail, a past decision) rather than dumping raw conversation logs into storage. Retrieval scoped to relevance for the current context, not just recency. And a mechanism, even a simple one, for handling contradictions when new information conflicts with an older stored memory, rather than silently keeping both and letting retrieval surface whichever one happens to rank higher.

The core distinction worth internalizing

Short-term memory is free, it's just context you're already sending. Long-term memory is a real system you have to build and maintain, with its own failure modes, and conflating the two at a design level is one of the more common sources of agents that seem to "forget" things they should reasonably have known.