What a Context Window Actually Does in an LLM

You paste a long document into ChatGPT, ask a question near the end, and it answers as if it never read the first half. Or you have a great back-and-forth, come back the next day, and the model has no idea who you are or what you decided. Both of those are the context window doing exactly what it is built to do. The context window is the single most misunderstood part of how a large language model works, and once you get it, most of your "why is the AI ignoring me" problems make sense.

This guide explains what a context window actually is, what fits inside it, why the model forgets the moment the window fills or the chat resets, and what to do about it so your AI works from the real details instead of guessing.

Key takeaways

• A context window is the maximum amount of text (measured in tokens) an LLM can read and hold in mind for a single response. It is working memory, not long-term memory.
• Everything counts against it: the system prompt, your instructions, the chat history, pasted documents, tool outputs, and the answer being generated.
• When the window fills, older content gets pushed out or truncated. The model does not "remember" it anymore, even inside the same chat.
• A bigger context window helps, but it does not solve staleness. A fresh chat starts empty, and re-pasting the same context every time is the tax most people are quietly paying.
• The real fix is not a larger window. It is a way to feed the model the right, current context on demand, so the window is always full of what matters.

---

What a context window actually is

A context window is the fixed span of text an LLM can consider when it produces a response. Think of it as the model's field of view. Whatever is inside the window, the model can use. Whatever is outside it, the model cannot see at all, no matter how important it was two minutes ago.

That span is measured in tokens, not words or characters. A token is a chunk of text, roughly 3 to 4 characters of English on average, so a token is often a short word or a piece of a longer word. "context" is one token. "tokenization" might be three. As a rough rule, 1,000 tokens is around 750 words of English. When a model advertises a "200K context window," it means it can hold about 200,000 tokens of combined input and output at once.

The critical word is combined. The window is a single shared budget. It is not "200K for your input plus unlimited output." Everything the model reads and everything it writes has to fit inside the same number.

What actually counts against the window

People assume the context window is just "the message I typed." It is not. In a real conversation, the window is filled by several things at once, and most of them are invisible to you:

• The system prompt: hidden instructions the app sends before your message (tone, rules, tool definitions). This can be thousands of tokens before you type a word.
• Your custom instructions or profile: any standing preferences you set once.
• The full chat history: every earlier message in the thread, yours and the model's, is re-sent on each new turn.
• Pasted or attached content: documents, code, transcripts, spreadsheets.
• Tool and search results: anything a connected tool, browser, or file reader returns.
• The response being generated: the answer itself reserves space in the same budget.

That last point trips up a lot of people. If you fill a window almost to the brim with a giant document, there may be very little room left for a long answer, and the model will cut itself short or refuse. The window is shared by the question and the answer.

Why the model forgets mid-conversation

Here is the part that feels like a bug but is not. As a chat grows, the running history keeps eating into the window. Once the total crosses the model's limit, something has to give. Most systems drop or summarize the oldest messages to make room for the newest ones.

So when you have a long session and the model suddenly "forgets" a decision you made an hour ago, it is not being lazy. That text has literally fallen out of the window. It is no longer in the model's field of view, so it may as well never have been said. The model is not searching a memory of your conversation. It is re-reading whatever slice of the conversation still fits, every single turn.

This is also why a brand-new chat starts from zero. A new thread is an empty window. Nothing carries over unless the product has a separate memory feature that re-injects a summary of you into the new window. The window itself has no persistence. It resets to blank every time.

A context window is short-term working memory. It holds what is in front of the model right now, and it clears the moment the room fills or the chat ends.

Context window vs memory: not the same thing

Because both sound like "remembering," people mix them up. They are different mechanisms doing different jobs, and knowing the difference tells you why your AI behaves the way it does.

The context window is where thinking happens for a single turn. Memory is what decides what gets loaded into that window in the first place. A big context window with nothing useful in it still produces generic output. A modest window filled with the exact facts the task needs produces sharp output. That gap between the two is the whole game, and it is why a bigger model alone rarely fixes bland results. If you have felt this, the deeper version of the argument is in how to give your AI a memory that lasts.

Do bigger context windows solve this?

Partly, and this is where a lot of hope gets misplaced. Larger windows are genuinely useful. You can drop a whole report in and ask about all of it. But a bigger window does not fix three real problems:

1. It still resets. A 1-million-token window is still empty at the start of every new chat. Size does not create persistence.
2. It rewards dumping, not curating. Filling a huge window with everything you own buries the 5 facts that matter under 500 that do not, and models pay less attention to the middle of a very long context. More is not automatically better.
3. Someone has to keep re-supplying it. If the useful context lives in your head or scattered across ten apps, you are the one re-pasting it into every fresh window, forever. That is the hidden tax.

The honest answer: a bigger window raises the ceiling, but it does not answer the real question, which is what should go in the window this turn, and where does it come from without you fetching it by hand every time.

The practical fix: stop feeding the window by hand

If the window resets constantly and only holds what you put in it, the move is obvious. You want the right, current context to land in the window automatically, right when the model needs it, without you re-pasting it.

Two things make that work in practice:

• A place your real context lives, kept current, so the facts are true today and not whatever you wrote six months ago.
• A way for the AI to pull only the relevant slice into the window on demand, instead of you dumping everything or nothing.

This is exactly the job Locul does. It builds a second brain from what you already produce (your markdown notes, PDFs, dictation, Notion, your LinkedIn profile) and keeps it current on its own. Then it serves that brain to your AI over MCP, so tools like Claude can call something like recall_memories or search_notes and pull the exact facts a task needs straight into the context window. You are not pasting. The window fills with what matters, per turn, and the facts stay true because when something changes, the old version is marked superseded and the current one takes over. That "stays current" behavior is what most static setups miss, and it is the difference between a window full of stale context and a window full of the truth. For the packaged version of that idea, curated fact bundles you can install into a brain, see what a memory pack is.

A concrete before and after

Say you are asking an AI to write an outreach message. Here is the difference between hand-feeding a window and having the right context loaded for you.

Before, hand-fed and thin:

Write a cold outreach message introducing my product.

The window has almost nothing true in it, so the model invents a generic SaaS pitch. Bland, wrong, and you rewrite it.

After, with real context available in the window:

Write a cold outreach message introducing my product. Use my actual product details, my positioning, and how I usually open a message.

Now the AI pulls the facts it needs (what the product actually is, the one-line pitch, your usual opener, who it is for) into the window from your brain, and writes something specific enough to send. Same model. Same window size. The only thing that changed is what was inside the window when it answered.

That is the entire lesson of the context window in one example. The model is only ever as good as what fits in its field of view at the moment you ask.

---

Frequently asked questions

What is a context window in simple terms?

It is how much text an AI can read and keep in mind for a single answer, measured in tokens. Anything inside it, the model can use. Anything outside it, including earlier parts of a long chat, the model cannot see. It is working memory for one response, not a memory of you that carries across sessions.

How big is a context window?

It depends on the model. Limits are quoted in tokens, and roughly 1,000 tokens is about 750 words of English. Models today range from a few thousand tokens up to hundreds of thousands or more. Remember the number covers input and output combined, so a giant pasted document leaves less room for the answer.

What is the difference between a context window and memory?

The context window is temporary working memory that clears when the chat resets or overflows. Memory is a persistent store that survives between chats and re-injects facts about you into new windows. The window decides what the model can think about right now. Memory decides what gets loaded into it. More on that in how to give your AI a memory that lasts.

Why does the AI forget things I said earlier in the same chat?

Because the conversation grew past the window limit. To make room for new messages, the system drops or summarizes the oldest ones, so earlier details fall out of the model's field of view. It is not ignoring you. That text is no longer in the window it reads each turn.

Does a bigger context window make the AI smarter?

It lets the model consider more at once, which helps for long documents, but it does not make output better on its own. A large window full of irrelevant text still produces generic answers, and models attend less to the middle of very long context. Quality comes from the right context being in the window, not just more of it.

How do I get the right context into the window without pasting it every time?

Keep your real context in one place that stays current, and let your AI pull the relevant slice on demand. A tool like Locul builds that brain from what you already produce and serves it to your AI over MCP, so the useful facts land in the window automatically instead of you re-supplying them each chat.

---

The context window is not the thing to optimize. It is a fixed pane of glass that resets constantly and only shows what you put in front of it. The real work is upstream: have a current, real record of you and your work, and let your AI pull the right piece into that window every time it answers. Locul builds that record from what you already do and keeps it current, free to start, 500 memories, local AI. That is the difference between a window full of guesses and a window full of the truth.