Context Engineering

Part of a series: Read Part One: From Prompt Engineering to Eval Engineering

Chapter Two

Context Engineering

Intelligence Is Only As Good As What It Knows

Imagine walking into the office of the world's most accomplished heart surgeon.

Decades of experience.

Thousands of successful operations.

An encyclopedic understanding of medicine.

Now imagine asking them to diagnose a patient.

They smile confidently.

Then you refuse to let them see the patient's X-rays.

You hide the blood test.

You don't tell them the patient's age.

You don't mention the symptoms.

You won't even tell them why the patient came to the hospital.

How good will their diagnosis be?

Not very.

Not because they lack expertise.

Because expertise without information is powerless.

The problem isn't intelligence.

The problem is context.

This simple idea explains one of the biggest misconceptions surrounding large language models.

People often assume that if an AI produces a poor answer, it must not be intelligent enough.

Sometimes that's true.

More often, however, the model simply didn't have access to the information it needed.

That realization fundamentally changed how engineers began thinking about AI systems.

Instead of endlessly rewriting prompts, they started asking a different question:

How do we make sure the model always has the right information before it starts thinking?

That question gave birth to what many practitioners now call Context Engineering.

Prompts Tell The Model What To Do

Context Tells The Model What To Work With

At first glance, prompts and context seem almost identical.

After all, they're both pieces of text placed into the model's context window.

So why separate them?

Because they serve entirely different purposes.

A prompt is an instruction.

It defines the task.

Context is evidence.

It provides the information needed to complete that task.

Imagine asking two software engineers to fix the same production bug.

You tell both of them:

"Find the issue causing checkout failures and submit a fix."

The instruction is identical.

But one engineer receives:

• the source code,
• production logs,
• stack traces,
• recent commits,
• API documentation,
• failing tests.

The other receives...

nothing.

Who finishes first?

Who produces the more reliable fix?

The difference isn't skill.

It's information.

AI systems work exactly the same way.

A brilliant prompt cannot compensate for missing evidence.

The Internet Changed How We Think About Knowledge

When ChatGPT first became popular, many people imagined language models as giant encyclopedias.

Ask a question.

Receive an answer.

Repeat.

For general knowledge, this worked surprisingly well.

Ask,

"Who discovered penicillin?"

and the model can answer because that information existed during training.

But real work is rarely about public knowledge.

Companies don't ask AI to explain gravity.

They ask it questions like:

"Summarize yesterday's board meeting."

"Review this pull request."

"Search our customer support tickets."

"Generate a report using this month's sales figures."

"Find every API endpoint using our legacy authentication system."

The model wasn't trained on any of that.

It can't possibly know.

Not because it's incapable.

Because the information didn't exist when it was trained—or because it's private to your organization.

That realization marked an important shift.

Instead of trying to store more knowledge inside the model, engineers started focusing on delivering the right knowledge at the moment it was needed.

Context Is Dynamic

One of the easiest mistakes to make is thinking of context as static.

It isn't.

Every request deserves a different view of the world.

Suppose you're building an AI assistant for a law firm.

When helping a tax lawyer, the assistant should retrieve tax legislation.

When helping someone in intellectual property, it should retrieve patent law instead.

The underlying model doesn't change.

The prompt may barely change.

Only the context changes.

The same is true for a coding assistant.

When you're editing a single function, the model probably doesn't need your entire repository.

It needs:

• the file you're editing,
• the functions that call it,
• the interfaces it implements,
• recent changes,
• relevant documentation,
• perhaps the failing test.

Good context engineering is not about giving the model more information.

It's about giving the model the right information.

Key Idea

Better context usually beats a longer prompt.

Most production AI systems don't become more reliable because engineers discover magical prompting techniques. They become more reliable because engineers improve what the model sees before it generates a response.

Retrieval: Giving Models Access To Fresh Knowledge

This is where one of the most influential ideas in modern AI systems appears:

Retrieval-Augmented Generation, more commonly known as RAG.

Rather than expecting the model to memorize everything during training, a retrieval system searches for relevant information first.

The workflow looks something like this:

User asks a question
        │
        ▼
Search relevant knowledge
        │
        ▼
Retrieve documents
        │
        ▼
Inject them into the context
        │
        ▼
Generate an answer

The model is no longer relying solely on memory.

It's reading before answering.

Much like a human engineer opening documentation before making a code change.

This seemingly simple idea transformed enterprise AI.

Instead of retraining enormous language models whenever company information changed, organizations could simply update their knowledge base.

The model remained the same.

The context evolved.

Memory Is Context Across Time

Not all useful information comes from documents.

Sometimes the most valuable information is the conversation itself.

Imagine telling your AI assistant:

"I'm planning a trip to Japan next spring."

Twenty minutes later you ask,

"What's the weather likely to be?"

You naturally expect the assistant to know you're still talking about Japan.

That's conversational memory.

Now imagine returning a week later.

Should the assistant still remember your travel plans?

Maybe.

Maybe not.

That depends on the product.

Modern AI systems often distinguish between different kinds of memory.

Short-term memory keeps track of the current conversation.

Long-term memory stores information across conversations, such as user preferences, recurring projects, or previously established facts.

Managing these memories—deciding what to keep, what to summarize, and what to forget—is increasingly becoming part of context engineering.

Because context windows are finite.

Every sentence you keep is a sentence you cannot replace with something potentially more useful.

More Context Isn't Always Better

A common misconception is that if some context helps, then more context must help even more.

Unfortunately, that's rarely true.

Imagine trying to find a single paragraph inside a thousand-page PDF.

The information exists.

But it's buried.

Language models face a similar challenge.

As context grows, irrelevant information can distract the model from what's actually important.

This is sometimes called context dilution.

Effective context engineering isn't about maximizing the amount of information.

It's about maximizing the signal-to-noise ratio.

That means carefully selecting, ranking, summarizing, and sometimes discarding information before it ever reaches the model.

In practice, one highly relevant document often outperforms twenty vaguely related ones.

Context Engineering Is Becoming Its Own Discipline

As AI systems evolved from chatbots into agents, context stopped being a technical detail hidden inside a prompt.

It became an engineering problem.

How should documents be retrieved?

How much conversation history should be preserved?

When should memories be summarized?

How should conflicting sources be handled?

Which information is trustworthy enough to include?

These questions aren't answered by the language model itself.

They're answered by the systems surrounding it.

That's why many practitioners increasingly view context engineering as a distinct discipline rather than simply an extension of prompt engineering.

The prompt defines the task.

The context provides the evidence.

Both are necessary.

Neither replaces the other.

The Next Layer

Imagine you've built the perfect prompt.

You've retrieved exactly the right documents.

You've supplied every relevant piece of information.

Your AI now knows everything it needs to know.

There's just one problem.

It still can't do anything.

It can't browse a website.

It can't call your company's API.

It can't open a terminal.

It can't inspect a Git repository.

It can't execute code.

It can't send an email.

Knowledge alone isn't enough.

To become useful, an AI system needs access to the outside world.

And that brings us to the next layer in the modern AI engineering stack:

Harness Engineering.

Further Reading

Foundational Papers

• Lewis et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.
• Gao et al. (2023). Retrieval-Augmented Generation for Large Language Models: A Survey.

Recent Reading

• Context Engineering (arXiv:2607.27250)
• Anthropic Engineering Blog: Building Effective AI Context Systems (when discussing practical context management)
• OpenAI documentation on long-context models and retrieval architectures