Context Window Isn't the Same as Memory

The problem: your agent forgets everything after a few turns

You've built a chatbot or an AI agent that works great in a demo. But the moment you deploy it, users complain that it forgets their name, their preferences, or the details of a conversation from five minutes ago. You check the logs: the model is hitting its context limit and truncating older messages.

The root cause is a fundamental misunderstanding: context window is not memory. The context window is a temporary scratchpad that holds the tokens the model can see right now. Memory is the ability to retain and recall information across sessions or long conversations. If you treat the context window as memory, you will run into token limits, rising costs, and frustrated users.

In this article, I'll show you how to distinguish between the two, and then build a simple but effective memory layer for an AI agent using a vector store and a small script. You'll end with a working pattern you can adapt to your own projects.

What the context window really is

The context window is the maximum number of tokens (roughly words) the model can process at once. For example, GPT-4o has a 128k token window, Claude 3.5 Sonnet has 200k, and Llama 3.1 405B has 128k. When you send a prompt, the model sees all the tokens in that window, and it generates a response based only on those tokens.

Critical limitation: the model has no access to anything outside that window. If a token scrolls out of the window, it's gone. The model cannot 'remember' it unless you include it again. This is why long conversations hit a wall: older messages get truncated or summarized to fit the window.

To illustrate, consider a simple chat history. If you send the entire history every time, you'll eventually exceed the limit. Let's see how that plays out in code.

import openai

client = openai.OpenAI()

def chat_with_history(messages):
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=messages
    )
    return response.choices[0].message.content

# Simulate a long conversation
messages = [{"role": "user", "content": "My name is Alice and I love hiking."}]
for i in range(100):
    messages.append({"role": "user", "content": f"Tell me a fact about {i}."})
    # This will eventually fail with a token limit error
    if len(str(messages)) > 100000:
        print("Token limit reached!")
        break
    print(chat_with_history(messages))

What memory really means for AI systems

Memory, in the context of AI agents, is the ability to store and retrieve information that persists beyond the current context window. This includes user preferences, past decisions, facts learned during a conversation, and even the state of a task across multiple sessions.

There are two main types: short-term memory (within a session) and long-term memory (across sessions). Short-term memory can be managed by keeping a sliding window of recent messages or by summarizing older ones. Long-term memory typically involves an external store like a vector database, a key-value store, or a traditional database.

The key insight: you don't need to stuff everything into the context window. You can selectively retrieve the most relevant pieces and inject them into the prompt. This is called retrieval-augmented generation (RAG), and it's the foundation of most production memory systems.

A simple memory pattern: store, retrieve, inject

The pattern is simple: when a conversation happens, extract important facts and store them. When a new message comes in, retrieve relevant facts and prepend them to the prompt. This way, the model always has the essential context without blowing up the token count.

Here's a high-level architecture: you have a chat endpoint that receives a message, queries a vector store for relevant memories, builds a prompt with those memories plus the recent conversation, sends it to the LLM, and then updates the memory store with new facts extracted from the conversation.

Let's implement a minimal version using Python, a vector store (we'll use Chroma for simplicity), and an LLM call.

pip install chromadb openai python-dotenv

Step 1: Set up a vector store for memories

We'll use Chroma, a lightweight embedded vector database. It's perfect for local development and small-scale apps. Create a script that initializes a collection and provides functions to add and query memories.

Each memory will be a text snippet that we embed using an embedding model (e.g., OpenAI's text-embedding-3-small). We'll store the text and its embedding, along with metadata like timestamp and user ID.

import chromadb
from chromadb.utils import embedding_functions

# Initialize Chroma client
client = chromadb.PersistentClient(path="./memory_db")

# Use OpenAI embeddings
openai_ef = embedding_functions.OpenAIEmbeddingFunction(
    api_key="your-openai-api-key",
    model_name="text-embedding-3-small"
)

# Create or get a collection
collection = client.get_or_create_collection(
    name="conversation_memory",
    embedding_function=openai_ef
)

def add_memory(user_id, text):
    collection.add(
        documents=[text],
        ids=[f"{user_id}-{time.time()}"],
        metadatas=[{"user_id": user_id, "timestamp": time.time()}]
    )

def query_memories(user_id, query, n_results=3):
    results = collection.query(
        query_texts=[query],
        n_results=n_results,
        where={"user_id": user_id}
    )
    return results['documents'][0]

Step 2: Extract facts from conversation to store

We need a way to turn raw conversation into discrete memories. We can use an LLM to extract salient facts. For example, if the user says 'My favorite color is blue', we want to store that as a separate memory.

Here's a function that takes the last user message and the assistant's response, and returns a list of fact strings. We'll use a simple prompt that asks the model to output facts as JSON.

import openai
import json

client = openai.OpenAI()

def extract_facts(user_message, assistant_response):
    prompt = f"""
Extract factual statements from the following conversation. Output a JSON list of strings.
User: {user_message}
Assistant: {assistant_response}
"""
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": prompt}],
        response_format={"type": "json_object"}
    )
    content = response.choices[0].message.content
    facts = json.loads(content).get("facts", [])
    return facts

Step 3: Build the chat loop with memory injection

Now we combine everything. When a user sends a message, we query the vector store for relevant memories, prepend them to the system prompt, and then send the recent conversation (last N messages) to the LLM. After getting the response, we extract facts and store them.

This ensures the model has the needed context without including the entire history. We also keep a short-term buffer of the last 10 messages to maintain conversational flow.

def chat_with_memory(user_id, user_message, conversation_history):
    # Retrieve relevant memories
    memories = query_memories(user_id, user_message)
    memory_text = "\n".join(memories)

    # Build system prompt with memory
    system_prompt = f"You are a helpful assistant. Here are facts you remember about the user:\n{memory_text}"

    # Prepare messages: system + last 10 messages + new user message
    messages = [{"role": "system", "content": system_prompt}]
    messages.extend(conversation_history[-10:])
    messages.append({"role": "user", "content": user_message})

    # Call LLM
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=messages
    )
    assistant_message = response.choices[0].message.content

    # Extract and store facts
    facts = extract_facts(user_message, assistant_message)
    for fact in facts:
        add_memory(user_id, fact)

    return assistant_message

Step 4: Test it with a real conversation

Let's run a quick test. You'll need to set your OpenAI API key as an environment variable. We'll simulate a user telling us their name and a hobby, then later asking if we remember.

This test demonstrates that the agent can recall facts from earlier in the conversation even after many turns, because they are stored in the vector store.

export OPENAI_API_KEY="your-api-key"
python -c """
from memory import chat_with_memory

user_id = "test-user"
history = []

# First interaction
reply = chat_with_memory(user_id, "Hi, my name is Alice and I love hiking.", history)
print(reply)
history.append({"role": "user", "content": "Hi, my name is Alice and I love hiking."})
history.append({"role": "assistant", "content": reply})

# Simulate many turns
for i in range(50):
    reply = chat_with_memory(user_id, f"Tell me a fact about {i}.", history)
    history.append({"role": "user", "content": f"Tell me a fact about {i}."})
    history.append({"role": "assistant", "content": reply})

# Now ask about memory
reply = chat_with_memory(user_id, "What is my name and what do I like?", history)
print(reply)
"""

Verify it worked

If the memory system works, the final response should correctly mention 'Alice' and 'hiking'. If it doesn't, check that the vector store is being populated and that the query is returning results.

You can inspect the Chroma database by opening the persistent directory or using the Chroma client to list all documents.

python -c "
import chromadb
client = chromadb.PersistentClient(path='./memory_db')
collection = client.get_collection('conversation_memory')
print(collection.get())
"

Troubleshooting common issues

If you see token limit errors, your conversation history might still be too long. Reduce the number of recent messages you keep (e.g., from 10 to 5).

If the model doesn't recall facts, the retrieval might be returning irrelevant memories. Try increasing n_results or using a different embedding model.

If the vector store grows too large, consider adding a summarization step that compresses old memories into a summary and stores that instead.

• Check that your OpenAI API key is valid and has access to the embedding model.
• Ensure the user_id is consistent across calls; otherwise memories are partitioned per user.
• Monitor your token usage; retrieval reduces tokens but you still pay for the system prompt and recent messages.
• For production, use a cloud vector database like Pinecone or Weaviate for scalability.
• Add a TTL (time-to-live) or decay mechanism to forget stale memories.

What I would do: a production-ready memory setup

For a real application, I would not build a custom memory system from scratch. Instead, I would use a framework like LangChain or LlamaIndex that provides memory abstractions. But if you want to keep it minimal, here's my recommended setup:

Use a managed vector database (e.g., Pinecone) for persistence, use an embedding model like text-embedding-3-small, and store memories as JSON blobs with metadata. Use a background job to extract and store facts asynchronously to avoid latency.

Here's a starter configuration for a Docker Compose setup with a vector DB and your agent service.

version: '3.8'
services:
  qdrant:
    image: qdrant/qdrant:latest
    ports:
      - "6333:6333"
    volumes:
      - ./qdrant_storage:/qdrant/storage

  agent:
    build: .
    environment:
      - OPENAI_API_KEY=${OPENAI_API_KEY}
      - QDRANT_URL=http://qdrant:6333
    ports:
      - "8000:8000"
    depends_on:
      - qdrant

FAQ

• Q: Can I just increase the context window instead of adding memory? A: You can, but it's expensive and doesn't scale. Costs grow linearly with tokens, and even with 200k tokens, long-term conversations will exceed it. Memory is more efficient.
• Q: What's the difference between memory and caching? A: Caching stores exact responses to avoid recomputation. Memory stores semantic information that can be retrieved and used in new contexts.
• Q: How do I handle multiple users? A: Use a user_id in the metadata and filter queries by it, as shown in the example.
• Q: What if my agent needs to remember code or structured data? A: Store structured data in a traditional database and retrieve it based on intent, not just text embeddings.
• Q: Is RAG the same as memory? A: RAG is a technique for retrieving relevant information from a knowledge base. Memory is a broader concept that includes storing conversation facts. They often use similar infrastructure.

Next step: try it on code.live

You can experiment with the JSON Diff tool on code.live to compare different memory structures, or use the JSON to YAML converter to format your memory config. These tools help you debug your data formats quickly.