Explore strategies for effectively curating and managing the context that powers them
Introduction
Context engineering is becoming increasingly important.
Finding the right words and phrases for prompts -> what configuration of context is most likely to generate the desired behavior from our model?
For modern LLMs, the overall context matters more than any single word or phrase.
Context refers to the set of tokens included when sampling from a large-language model (LLM).
Context refers to all of the tokens included in the model when it generates the next segment of text. It can be thought of as everything the AI agent remembers about what has happened and what is currently happening.
Goal: find ways to get the model to do the right thing within practical constraints such as limited context windows, attention limits, and the fact that overly long prompts can introduce noise.
Context engineering vs. prompt engineering
Context engineering is a natural extension of prompt engineering.
Prompt engineering refers to methods for writing and organizing LLM instructions for optimal outcomes.
Prompt engineering refers to the design and organization of instructions given to an LLM in order to achieve better results.
Context engineering refers to the set of strategies for curating and maintaining the optimal set of tokens (information) during LLM inference.
The key question is: which information should the model see at the moment it acts? This information may include not only the prompt itself but also tool results, external data, conversation history, task state, MCP resources, and other runtime context.
In the early days of engineering with LLMs, prompting was the biggest component of AI engineering work.
This is because most early tasks were one-shot tasks such as classification, summarization, translation, and text generation. In those cases, users typically provide one input and receive one response, so a well-written prompt often leads to good performance.
However, as we move towards engineering more capable agents that operate over multiple turns of inference and longer time horizons, we need strategies for managing the entire context state.
An agent is not just answering once. It continuously understands the task, plans steps, calls tools, fetches data, organizes intermediate results, and revises its direction based on new information. Therefore, the relevant content includes:
- system instructions
- tools
- Model Context Protocol (MCP)
- external data
- message history
- task state
These pieces of information jointly determine how the model thinks and acts next.
There are two important points here:
- An LLM’s context window is finite, so not everything can be included.
- An agent continuously generates new information, so context must be constantly refined and updated.
Key point: Prompt engineering is about designing instructions. Context engineering is about designing the agent’s working memory. In the early LLM era, most workloads were one-shot tasks, so prompt quality mattered a lot. But once AI agents started handling multi-turn reasoning, extended task horizons, tool use, and external data integration, the critical engineering challenge shifted to managing the entire context state. Because agents keep producing new information, they need ongoing filtering, compression, and refinement to ensure that each step sees only the most valuable information.
Why context engineering is important to building capable agents
The fact that the context window gets larger does not mean the model is better at using all the information.
LLMs, like humans, lose focus or experience confusion at a certain point.
context rot: as the number of tokens in the context window increases, the model’s ability to accurately recall information from that context decreases.
Long contexts are not always higher-quality contexts. In fact, when too much information is included, the model may fail to identify what is truly important.
Context, therefore, must be treated as a finite resource with diminishing marginal returns.
At first, adding useful information can significantly help the model. But as more information is added, each additional piece contributes less value and may even reduce clarity.
LLMs have an “attention budget” that they draw on when parsing large volumes of context.
Because of the transformer architecture, every added token consumes part of the model’s attention budget. The goal of context engineering is to include only information that is truly valuable.
Key point: Although context windows are growing, context remains a scarce resource. More information can lead to distraction and context rot. Good context engineering is therefore about selecting, compressing, and organizing only the information that most helps the model complete the task.
The anatomy of effective context
Good context engineering aims to find the smallest set of high-signal tokens.
System prompts should be extremely clear and use simple, direct language
The right altitude is the Goldilocks zone
System prompts should be neither too narrow nor too vague.
- Too detailed and the prompt becomes brittle, difficult to maintain, and overfit to specific cases.
- Too abstract and the model lacks clear guidance.
The ideal system prompt is in the “Goldilocks zone”: specific enough to guide behavior, but still flexible enough to allow the model to adapt.
Practical recommendations:
- Structure system prompts into clear blocks such as
<background_information>,<instructions>,## Tool guidance, and## Output description. - Use a minimal viable prompt and test it on the strongest model available.
- Add only the missing instructions or examples needed to fix observed failure patterns.
System prompt: the pre-configured set of rules and background information that is set before the conversation begins.
minimal does not necessarily mean short
A good prompt should remove unnecessary material, but it still needs to provide enough information for the agent to behave reliably.
Tools allow agents to operate with their environment and pull in new, additional context as they work.
Tool design should be clear and efficient, and the output they return should be compact. Otherwise, large amounts of irrelevant data may be inserted into the agent’s context.
tools should be self-contained, robust to error, and extremely clear with respect to their intended use.
If there are too many tools or functions overlap too much, the agent may not know which one to use.
examples are the “pictures” worth a thousand words.
A small number of highly representative examples can often teach the model more effectively than large amounts of prose.
Key point: Effective context should be informative but tight. The goal is not to stuff everything into the prompt. Instead, the goal is to retain only the highest-signal information: clear system instructions, well-designed tools, and a small set of a few strong examples that illustrate the expected behavior.
Context retrieval and agentic search
An agent can be understood as an LLM that uses tools in a loop.
LLMs autonomously using tools in a loop.
The agent decides what to do next, calls a tool, reads the result, and then continues based on the new information.
smarter models allow agents to independently navigate nuanced problem spaces and recover from errors.
As models improve, agents become more autonomous. Better models can explore complex problem spaces, recover from mistakes, and adjust their strategy during the task.
just in time context strategies
Just-in-time context means not injecting all information up front. Instead, the agent loads relevant information only when it is needed. This helps avoid drowning the model in unrelated content.
maintain lightweight identifiers (file paths, stored queries, web links, etc.)
An agent can keep lightweight references such as file paths, query strings, or web links rather than loading the full content immediately. It can fetch the actual relevant content only when needed.
This approach mirrors human cognition. People do not memorize every fact in full; they keep folders, bookmarks, index cards, or search traces and retrieve the relevant material later.
metadata of these references provides a mechanism to efficiently refine behavior
Metadata such as filenames, directory locations, or timestamps can itself be highly informative. For example, tests/test_utils.py and src/core_logic/test_utils.py may have the same name but very different roles.
progressive disclosure
Progressive disclosure means the agent gradually uncovers relevant context as it explores the problem. It does not need to know everything at the start.
runtime exploration is slower than retrieving pre-computed data.
The downside of just-in-time loading is that it is slower than using already indexed or precomputed data. The agent has to explore the environment, run queries, and read results during execution.
Without proper guidance, an agent can waste context by misusing tools, chasing dead-ends, or failing to identify key information.
Without good tool design and good instructions, agents may waste context by using tools incorrectly, following irrelevant paths, or failing to identify the true bottleneck.
hybrid strategy
Some tasks are best solved with a hybrid strategy:
- keep some information preloaded because it is important and stable;
- load large or uncertain information just-in-time as needed.
This combines speed and efficiency with flexibility.
Key point: For agents, context is not just a static block of text; it is a runtime resource that should be fetched and curated dynamically. Lightweight references such as paths, queries, and metadata can help the system find the right content without overloading the prompt. In practice, a hybrid strategy often works best: pre-seed the essential context, then let the agent explore and fetch additional context when needed.
The right level of autonomy depends on the task itself.
Context engineering for long-horizon tasks
Long-horizon tasks are tasks with many steps over a long period, causing the total amount of information to exceed the LLM context window.
Examples: large code migrations, long running research projects, multi-hour task workflows, or other continuous agent tasks.
The central challenge is that the model cannot keep all important information in its context window at once.
| Method | Best suited for | Benefits | Risks / limitations |
|---|---|---|---|
| Compaction | Workflows with long back-and-forth exchanges that still need continuity | Preserves conversational continuity while reducing window pressure | Over-aggressive summarization can lose details |
| Structured note-taking | Tasks with milestones, dependencies, and long-term state tracking | Keeps task state and dependencies outside the context window | Notes can become messy and difficult to retrieve if poorly organized |
| Sub-agent architectures | Complex research, analysis, or parallel exploration tasks | Allows each agent to work in a clean context and only return the essentials | Requires the main agent to integrate results well |
Compaction
Compaction is the practice of taking a conversation nearing the context window limit, summarizing its contents, and reinitiating a new context window with the summary.
Compaction means that when a conversation is approaching the context window limit, the system summarizes what has happened and starts a new context window using that summary.
This is a high-fidelity way to preserve the most important information, such as architecture decisions, unresolved bugs, implementation details, and key conclusions, while removing duplicated or low-value tool outputs.
However, compaction carries the risk of losing important details. For example, a prior model once discarded a critical instruction like “ask for human approval before deleting email.” To fix this, a common strategy is to have one LLM summarize while another LLM evaluates whether the summary dropped essential instructions, then feed that feedback into the next compaction pass.
This is a form of iterative refinement: the model learns what must be preserved and what can be omitted.
Another approach is to compress long content into a lightweight placeholder such as “there is a section about X here,” rather than including the full text. In some cases, this performs similarly to full summarization while being cheaper.
The key question is when to compact. Some systems do this according to a fixed rule, because models are not always good at deciding when to compact their own memory without training or reinforcement.
Structured note-taking
Structured note-taking, or agentic memory, is a technique where the agent regularly writes notes persisted to memory outside of the context window.
The agent periodically writes important information into an external memory store or notes file, then retrieves it when needed.
This allows the agent to maintain continuity over long tasks without keeping everything in the main context window.
A major design question is when to fetch information back from external memory and how to do it efficiently. This is an active area of research.
Sub-agents
Sub-agent architectures provide another way around context limitations.
Rather than having a single agent keep all information in memory, the system can create specialized sub-agents for distinct subtasks. Each sub-agent works in a clean, narrow context and only returns a condensed summary to the main agent.
This is essentially a form of autonomous context compression: while the sub-agent works, most of the details stay outside the main agent’s context. The main agent only receives the distilled final result.
The same pattern is used in reinforcement learning-style training: a sub-agent produces a result, returns it, and the main agent continues. The system is designed to penalize overly long sub-agent traces or overly long parent-agent memory, which encourages compact and useful summaries.
Key point: Long-horizon tasks require strategies that preserve continuity without requiring the model to keep everything in its working memory. Compaction compresses old context, structured note-taking persists task state externally, and sub-agents isolate subtasks so the main agent only sees the essential results.
Other strategies
Filtering
Filtering means deciding what to read and include in the first place, rather than letting all available information pile up in the context. Research suggests that a large fraction of an agent’s context can be observational noise; only a smaller portion is actually used for reasoning and decisions.
Agentic context engineering
This broader idea treats context as an active object that an agent manages continuously: gather, prioritize, summarize, retrieve, and prune.
Conclusion
Context engineering is the discipline of managing the information that an agent sees at each step. It is a core capability for building reliable agents because LLMs do not benefit from blindly stuffing more information into the prompt.
The main principles are:
- keep context high-signal and compact;
- load only what is needed at the moment;
- preserve important state outside the working context when tasks are long;
- use structure, summaries, and tool-based retrieval to keep reasoning coherent.
In other words, the goal is not to maximize context size—it is to maximize useful context.