Memory & persistent context

The model itself doesn't remember anything between conversations. The "memory" features in chatbots are clever scaffolding: notes saved on the side, retrieved and injected into context when needed.

What memory actually does

The model itself doesn't remember a thing between sessions — it's amnesiac. Every "memory" feature in a chatbot is a workaround: a database stores notes about you, and at the start of each new chat, the relevant notes get pasted into the system prompt. The model isn't smarter; it's better-briefed.

That mental model unlocks everything else: it's why you can view what ChatGPT remembers about you, why you can delete it, and why bloated memory makes responses worse (more tokens, more lost-in-the-middle risk). It's database engineering, not magic.

That's the whole concept. Below: the three layers of memory you'll actually meet, and how to build one yourself.

When you'd reach for it

Memory is worth the engineering effort when continuity matters:

• Personal assistants that should know your preferences, projects, allergies.
• Customer support that remembers prior tickets without making the user repeat them.
• Long-running agents working on a multi-day or multi-session task.
• Tutors and companions where ignoring last week's conversation feels rude.

You wouldn't add memory to one-shot tools (extraction, summarisation, classification) — there's nothing to remember. And for shared assistants where every user is anonymous, memory is just risk surface for no benefit.

How it's actually built

Three layers of "memory" you'll encounter:

1. Short-term: the context window

Within a single conversation, everything you've said and the model has replied is in the context window. That's the model's only working memory. When the window fills, older turns either get truncated or summarised.

2. Conversation persistence: chat history

Most products save your conversations in their database, accessible from the sidebar. When you reopen a chat, the whole conversation history is loaded back into context. This isn't really "memory" — it's just persistent state per conversation.

3. Cross-conversation memory: the notebook

This is the interesting layer. ChatGPT Memory, Claude Projects, custom agents — they store facts/preferences/context that persist across separate conversations, and inject them when relevant.

Implementation patterns:

• System-prompt injection — when you start a new chat, the product reads your saved memory entries and prepends them to the system prompt: "Things to remember about this user: [list]".
• RAG-style retrieval — for users with hundreds of memory entries, only the relevant ones get pulled in based on the current question (using embeddings, just like document retrieval).
• User-controllable — most products let you view, edit, and delete memory entries. A blob of text that's just a database row.

Where it bites in real life

Building memory yourself (when you'd want to)

If you're building an agent or chatbot with persistent memory, the architecture looks like this:

This is essentially RAG over your conversation history. Open-source frameworks: mem0, Letta (formerly MemGPT), Zep. Or roll your own with a vector DB.

The interesting design choices:

• What gets remembered — every fact? Only user-confirmed? Only what an LLM judge marks important?
• Memory consolidation — old memories that contradict newer ones should be updated, not duplicated.
• Forgetting — ancient irrelevant memories should age out so they stop polluting context.
• Privacy and trust — users want predictable, reviewable memory, not a black box that surprises them.

Under the hood (optional)

A complete memory layer in ~25 lines: embed the user's message, fetch relevant past memories from a vector DB, paste them into the system prompt, call the LLM, then extract any new facts worth remembering and save them. Skip if you don't code — the whole feature really is just RAG over your own conversation history.

# On every user message:
def chat_with_memory(user_id: str, user_message: str) -> str:
    # Retrieve relevant memories
    q_emb = embed(user_message)
    memories = db.execute("""
        SELECT text FROM memories
        WHERE user_id = %s
        ORDER BY embedding <=> %s::vector
        LIMIT 5
    """, (user_id, q_emb)).fetchall()
 
    # Inject into system prompt
    memory_block = "\n".join(f"- {m['text']}" for m in memories)
    system = f"Things to remember about this user:\n{memory_block}"
 
    # Normal LLM call
    response = llm(system=system, messages=[{"role": "user", "content": user_message}])
 
    # Decide whether to save anything new (often via a separate LLM call)
    new_memories = extract_facts_to_remember(user_message, response)
    for m in new_memories:
        db.execute("INSERT INTO memories (user_id, text, embedding) VALUES (%s, %s, %s)",
                   (user_id, m, embed(m)))
 
    return response

Try it yourself (~5 minutes)

See memory in action:

1. ChatGPT: in a new conversation, say "Remember that I'm allergic to mushrooms." Then start a fresh conversation and ask "What food should I avoid at restaurants?" — it should mention mushrooms.
2. Claude Projects: create a project. Add a system prompt like "The user prefers terse, technical replies. Their name is [your name].". Start a chat in the project; notice the changed style.
3. Check the saved memories: in ChatGPT settings → Personalization → Memory. You'll see a plain-text list of what's stored. Edit or delete any of it.

That's the whole feature, demystified.

Further reading

• OpenAI — Memory and new controls for ChatGPT — design choices behind ChatGPT memory.
• MemGPT / Letta paper (Packer et al. 2023) — research on giving LLMs richer memory architectures.
• mem0 — open-source memory layer for LLM applications.
• Zep — managed memory + temporal knowledge graph.
• Anthropic — Claude Projects — Anthropic's project-scoped context.