NOA Labs

Memory & Conversational Chains

Buffer, window, summary and token-limited memory patterns, and a complete conversational chain example.

Why a chain needs memory at all

A plain chain (prompt → model → parser, covered on the earlier chains-and-prompts page in this track) treats every .invoke() call as a fresh, isolated request — the model has no idea a previous call ever happened, because nothing from it is included in the new prompt. That's fine for a one-shot summarization or translation task, but it breaks down immediately for a conversational assistant: ask "What's the capital of France?" and then "What's its population?" and a memory-less chain has no way to resolve "its" — the word only makes sense in light of the previous turn. Memory is the piece of a conversational chain responsible for carrying relevant history from earlier turns into the prompt for the current turn, so the model can actually use it.

Memory patterns

Different memory strategies trade off completeness of context against prompt size (and therefore cost and the risk of hitting the context window limit covered in this site's LLM tutorials):

  • Buffer memory — keeps the entire raw conversation history and re-sends all of it on every turn. Simplest to reason about and loses nothing, but grows without bound — eventually too large for the context window, and increasingly expensive well before that.
  • Window memory — keeps only the last k turns (for example, the last 5 exchanges) and drops anything older. Bounded size, but a reference to something discussed 10 turns ago silently stops working once it falls outside the window.
  • Summary memory — periodically compresses older turns into a running summary (itself generated by an LLM call) instead of dropping them outright, and sends that summary plus the most recent raw turns. Keeps some memory of old context indefinitely, at the cost of losing exact wording and an extra summarization call.
  • Token-limited buffer — keeps as much raw history as fits within a configured token budget, dropping the oldest turns once the budget is exceeded. A practical middle ground between window and buffer memory, sized directly against the actual constraint (the context window) rather than an arbitrary turn count.
Strategy What it keeps Bounded size? Loses
Buffer Full raw history No Nothing, until it eventually breaks
Window (last k turns) Most recent k turns Yes Anything older than k turns, entirely
Summary A running summary + recent turns Mostly Exact wording of older turns
Token-limited buffer As much raw history as fits a token budget Yes Oldest turns first, once over budget

A complete example: a conversational chain with memory

Python
from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder
from langchain_core.output_parsers import StrOutputParser
from langchain_core.chat_history import InMemoryChatMessageHistory
from langchain_core.runnables.history import RunnableWithMessageHistory
from langchain_openai import ChatOpenAI

prompt = ChatPromptTemplate.from_messages([
    ("system", "You are a concise, helpful assistant."),
    MessagesPlaceholder("history"),
    ("human", "{input}"),
])

model = ChatOpenAI(model="gpt-4o-mini")
chain = prompt | model | StrOutputParser()

# One in-memory history store per conversation ("session")
store = {}

def get_history(session_id: str):
    if session_id not in store:
        store[session_id] = InMemoryChatMessageHistory()
    return store[session_id]

conversational_chain = RunnableWithMessageHistory(
    chain,
    get_history,
    input_messages_key="input",
    history_messages_key="history",
)

config = {"configurable": {"session_id": "user-42"}}

conversational_chain.invoke({"input": "What's the capital of France?"}, config=config)
# -> "The capital of France is Paris."

conversational_chain.invoke({"input": "What's its population?"}, config=config)
# -> "Paris has a population of roughly 2.1 million people." -- "its" correctly
#    resolved to Paris, because the history was injected into this call's prompt

Each call to conversational_chain.invoke looks up the message history for the given session_id, injects it into the MessagesPlaceholder("history") slot in the prompt, runs the chain, and then appends the new turn back into that same history store — so the next call sees this turn too. Swapping InMemoryChatMessageHistory for a window, summary, or token-limited variant changes only what gets stored and replayed; the rest of the chain is unaffected, which is exactly the point of keeping memory as a separate, swappable component.

Common mistakes

  • Using buffer (full-history) memory in a long-running production chat feature without any bound — it works fine in a demo and then silently starts failing (or getting expensive) once real users have long conversations.
  • Keying memory by the wrong identifier (for example, one shared history across all users instead of one per session/user) — this leaks one user's conversation into another's context, a serious correctness and privacy bug, not just a quality one.
  • Assuming summary memory is "free" — it costs an extra LLM call to (re-)generate the summary, and a poor summary can silently drop a detail a later turn actually needed.