Context engineering is most of the job

The demo is always easy. You wire a model to a few documents, ask it something, and it answers well. Then you put it in front of real inputs and the quality collapses in ways that have nothing to do with the model.

Almost all of it comes down to one thing: what was in the context window, and what wasn't.

The context window is a budget, not a bucket

The instinct when a system answers badly is to add more — more documents, more instructions, more examples. This usually makes it worse, for two reasons that pull in the same direction.

First, relevant signal gets diluted. A model attending to forty thousand tokens of mostly-irrelevant material is doing a harder task than one attending to two thousand tokens of exactly the right material.

Second, and less appreciated: everything in the window is treated as endorsed. If you retrieve a stale document, you have not given the model a document to evaluate. You have told it a fact.

The four questions

For any system I'd ask these before touching a prompt:

What must always be present? The stable frame: the task, the constraints, the output contract, the tools available. This is small, hand-written, and changes rarely.

What must be fetched per request? The dynamic part. The discipline here is precision over recall — it is better to retrieve three correct passages than twenty passages containing those three.

What must be excluded? Rarely asked and often decisive. Superseded documents, other tenants' data, internal debate that reads as conclusion, anything whose presence would be read as endorsement.

What must survive compression? On long tasks the window will overflow and something has to be summarised away. Deciding in advance what can never be lost — the original goal, the constraints, the decisions already taken — prevents the characteristic failure where an agent forty steps in is confidently working on the wrong problem.

Order is not neutral

Position affects what the model attends to. The practical consequence is that the arrangement is a design decision, not a serialisation detail:

[ system frame        ]  stable, hand-written, short
[ retrieved evidence  ]  each chunk labelled with its source
[ working state       ]  what's been decided and done so far
[ the actual request  ]  last, so it's what the model is holding

Labelling each retrieved chunk with where it came from does double duty: it improves attribution in the answer, and it gives you something to inspect when the answer is wrong.

Make the context inspectable

The highest-value piece of infrastructure in any of these systems is embarrassingly unglamorous: log the exact final context for every call.

Not the prompt template. The rendered text, as sent. Almost every "the model is being stupid" investigation I've seen ends with someone reading that log and finding a truncated document, a duplicated instruction, a stale cache entry, or an empty retrieval that silently returned nothing.

Where agents make this sharper

A single-turn system builds context once. An agent builds it repeatedly, and each turn's output becomes the next turn's input. That changes the failure mode from "bad answer" to "compounding drift" — a small misreading at step three is load-bearing by step fifteen.

Two things help more than anything else:

• Re-anchor every turn. Restate the original goal and constraints rather than relying on them surviving in the history.
• Keep tool output structured and small. A tool that returns eight thousand tokens of HTML has just spent your budget on markup.

The short version

If you're building with AI and looking for where the leverage is, it isn't in prompt phrasing and it usually isn't in the model choice. It's in deciding — deliberately, and with the ability to inspect the result — what the model is allowed to see.