Why ChatGPT Forgets: It Never Remembered

People describe a context window as the model’s memory: fill it up and the model starts forgetting. That gets the mechanism backwards, and everything practical that follows from it stays mysterious.

Here is what actually happens, from OpenAI’s own documentation:

While each text generation request is independent and stateless, you can still implement multi-turn conversations by providing additional messages as parameters to your text generation request.

The model does not remember your previous message. It cannot. Between one turn and the next it retains nothing at all. What creates the illusion of a conversation is the client: on every single turn, it re-sends the entire transcript.

The context window is not memory. It is the maximum size of the parcel you can hand over each time you speak.

Watch the parcel fill

Add exchanges and watch what gets sent. The system prompt is pinned; when the transcript stops fitting, the oldest turns fall out of the request before it is even dispatched.

Nothing is lost on the model’s side, because there was never anything on the model’s side. A dropped turn is a turn that did not get put in the parcel.

Three things this explains

Why it forgets the beginning of a long chat. Once the transcript exceeds the window, the client has to drop something, and the usual choice is oldest-first. The assistant then answers with genuine ignorance of what you said an hour ago — not because it forgot, but because those words were not in the request.

Why the reply competes with the history. Per the key concepts page, “the prompt and the generated output combined must be no more than the model’s maximum context length”. The answer is drawn from the same budget as the question. Fill the window to the brim and there is no room left to reply in — which is why the demo above lets you reserve space for the response.

Why long conversations get expensive out of all proportion. This is the one almost nobody mentions, and the documentation is blunt about it:

Even when using previous_response_id, all previous input tokens for responses in the chain are billed as input tokens in the API.

You pay for the whole history, every turn, forever.

The cost arithmetic, which is worse than it looks

Say each question and each answer runs about ten words, with a ten-word system prompt. Turn one sends thirty words. Turn two sends the system prompt, the first exchange, and your new question. Turn twenty sends everything that came before it.

SYSTEM = 10      # words in the pinned system prompt
QUESTION = 10    # words you type
ANSWER = 10      # words it replies

print(f"{'turn':>5}{'sent this turn':>16}{'running total':>15}{'if it only sent your message':>31}")
total = naive = 0
for turn in range(1, 21):
    # Every request carries: system prompt + all previous exchanges + your new question.
    sent = SYSTEM + (turn - 1) * (QUESTION + ANSWER) + QUESTION
    total += sent
    naive += QUESTION
    if turn in (1, 5, 10, 20):
        print(f"{turn:>5}{sent:>16}{total:>15}{naive:>31}")

print()
print(f"20 turns cost {total} words sent, not {naive}.")
print(f"That is {total / naive:.1f}x more than the naive expectation.")

 turn  sent this turn  running total   if it only sent your message
    1              20             20                             10
    5             100            300                             50
   10             200           1100                            100
   20             400           4200                            200

20 turns cost 4200 words sent, not 200.
That is 21.0x more than the naive expectation.

The per-turn cost grows linearly, so the total grows with the square of the conversation length. Twenty turns of small talk costs twenty-one times what the words you typed would suggest.

This has a practical shape. A long-running assistant is not expensive because its answers are long; it is expensive because it keeps re-reading its own diary aloud. Trimming old turns, summarising them, or starting a fresh conversation are not tidiness — they are the cost control.

What the demo simplifies

• It counts words, not tokens. We do not ship a real GPT vocabulary, and inventing a token count would be a fabricated measurement rather than a demonstration. The eviction and re-send mechanics are identical either way. To see real subword tokenization, the tokenizer on What Is a Token? trains on text you supply.
• Eviction is oldest-first here. Real clients vary: some summarise the dropped section, some keep a rolling summary plus recent turns, some use retrieval to pull back only the relevant parts. All of them are answering the same question — what goes in the parcel?
• The system prompt is pinned. That is the common choice, not a rule. Dropping it would change the assistant’s behaviour mid-conversation, which is worse than dropping an old turn.
• Real requests carry more than text. Tool definitions, function schemas and images all consume the same budget.

What to take from this

When a chat assistant loses the thread, the useful question is not “why did it forget?” but “what was actually in the request?” Almost every mystery about long-conversation behaviour resolves into that, and so does most of the bill.