About this book

This book is a project to document how to use AI well in ways that are maintainable and preserve human knowledge. The primary principle is to offload work from where it's scarce (maintainer attention, human time) to places where it's cheap (burning more tokens).

Many people online have said how AI encourages carelessness, similar to self-driving cars: they're correct just enough of the time that you're tempted not to review their work, but incorrect just often enough that you can't let them do their own thing. We agree. Therefore this book is strongly oriented around process engineering, not how to be more careful while using AI.

This book and all its chapters are currently stubs. This project is a work in progress. If you're interested in contributing, please join our discord.

Making AI do the work

LLMs are very eager to spit out an answer. Just a like a human, their first answer is often "off-the-top of their head", and not particularly detailed.

By telling them to design, research, and test their work in detail, we can achieve significantly better results. A good rule of thumb is this: forcing an LLM to explain exactly how things work helps it catch issues.

We are used to working with other humans who "know what they're doing". LLMs do not know what they're doing. In fact, LLMs are probably the furthest from "knowing what they're doing" that exists:

Instead, they ~blindly follow instructions. They have no memory, do not learn, and every change they make is a drive-by contribution.

This is a curse, but also a blessing. LLMs can be programmed, just like any other part of the computer. However, this programming is closer to process engineering than to writing a program. The way to get LLMs to do good work is to have really really good contributing documentation, and to a lesser extent to give them good tooling. As a bonus, this documentation and tooling is generally useful for humans too.

What is a harness?

Humans interact with the world through their eyes and ears and hands. AI interacts with the world through writing text and receiving text back.

A harness is a model's body. It turns text into actions in the world. For example, a model might call a tool that runs a bash command, or reads or writes a file, or sends a message to discord.

What makes a harness good or bad?

A model is a kind of "tabula rasa". It has background knowledge of the world, and it's shaped by its training and reinforcement-learning to provide helpful and coherent answers ¹, but by itself it's closer to a Broca's area that knows every language in the world than to a whole brain.

To make a model an "agent", it needs two main things:

• A way to act in the world (tools).
• A way to respond to those actions ("agent loop"). Harnesses react to events and re-prompt the model with the results.

We call this an agent because it can act by itself without a human constantly prompting it to continue.

Most harnesses will have additional features:

• Peripheral awareness ("context management"). Files such as AGENTS.md and SKILL.md tell the agent what to pay attention to before it ever runs a tool.
• Pause and resume ("session persistence"). Most harnesses let you stop, start, branch, and undo conversations with an agent.
• Interaction with the human operator (interruptions, "steering", questions, tool output).
• Mid-term memory ("compaction"). Models have a limited short-term memory ("context-window"), and every so often it needs to be summarized so the model only remembers what's important.
• Physical constraints (sandbox). Sandboxes prevent agents from taking dangerous actions, kinda, sorta. We'll talk more about sandboxes in upcoming chapters. ²
• Unconscious reflexes (hooks).
• Delegating to a colleague ("subagent"). These are new agents created and controlled by the "root" agent, rather than directly by a human. The root agent doesn't automatically know what a subagent does except in the reports they exchange with each other.

Some advanced harnesses have even more features:

• Long-term memory ("memory store"). Compaction summaries only last for the current "session" (conversation, roughly); memory stores persist them between sessions.
• Configurability, allowing you to change the harness itself.
• Persistent undo: Claude Code allows you to undo not just the conversation itself, but also the files the agent changed on disk.

1. for more on this, see https://www.anthropic.com/research/persona-selection-model ↩

2. "flinch reaction" might sound weird if you're already familiar with sandboxes. My reasoning here is that most sandboxes are not actually very effective at preventing agents from doing things they really want to do. See https://www.frogandtoad.ai/ for more discussion. ↩

Process engineering

First, let's talk about the options you have available to you.

• Plain old documentation
• Agent-specific documentation
• Skills
• Hooks
• Tools (in the CLI)
• Tools (in the harness)

Plain old documentation

"Plain old documentation" is exactly what it sounds like. I generally have a doc/*.md layout for my repos, but you can write your docs in any form you choose. LLMs will read your READMEs, CONTRIBUTING, etc. You just have to write them!

To make agent-specific docs more effective, your normal documentation should be well-organized. Each file should be narrow and targeted, easy to navigate by section headings, and cover most things a contributor might need to think about.

If you're not sure how to get to that state, you can use jyn's reorganize-docs skill on your existing docs to have an LLM give you suggestions.

Agent documentation

"Agent-specific documentation" is files like AGENTS.md: normal markdown files that are the first thing the agent sees, even before it sees your initial prompt.

Put things that an LLM should always think about in your AGENTS.md. For example, you could add this:

Use concise and clear language. Avoid jargon. If jargon is necessary, define it.

The rest of your AGENTS.md should be almost exclusively routing. LLMs are lazy, and if you have a lot of documentation, it won't read it all (and even if it does, that burns a lot of tokens). By associating events or tasks to documentation files, you can have it read documentation "on-demand", exactly when it's necessary for the task.

Skills

"Skills" are regular markdown files that have a name and description as YAML frontmatter. The description (or "trigger") acts similarly to the routing in AGENTS.md: it tells the LLM when to read the skill file.

Skills should not repeat your normal docs. Link freely to human-facing docs in your skills, and if a human would benefit from a bit of info, put it in the documentation instead of a skill.

Instead, they should be a procedural, like a runbook. Usually they will have checklists, phases, or exact command invocations. Use them when steps need to be done in a certain order, in an exact way, or if they're useful across projects. For example, jyn has a jj-workflow skill teaching agents how to use jj, and a simplify skill telling them to make code less Like That. The jj skill is useful across repos, and the simplify skill is so procedural that it would be rude to show to a human, but it's fine for an agent.

Hooks

A "hook" is a command that runs when an LLM does a certain action. For example, you can configure rustfmt to run whenever an LLM edits a Rust file. These are useful when some problem you want to catch is mechanically checkable.

CLI tools

LLMs love specialized tools. They can work with general tools, but the more restricted the interface, the better they do. The nice thing about using LLMs is you can get them to write their own tools. Consider getting an LLM to write a linter, then attaching the linter to a hook.

Harness tools

This is a fancy word for an API. It's just an API. The LLM is writing TypeScript (or sometimes JSON, or Python, or ...), calls a function, and the function returns some data to it.

Harness tools can sometimes be useful if whatever it's doing is different from the way you would do things. For example, an LLM has trouble using a web browser, so it makes sense to give it a specialized tool for searching Google. In general though, CLI tools are preferable because they're more general and can be used by human contributors, not just LLMs.

Sandboxing

Agents are, generally, well-meaning but extremely tunnel-visioned. Giving them credentials and permissions is dangerous, not because they're malicious, but because they don't know what is and isn't a good idea.

The confused deputy problem

The post What Are Capabilities? introduces the confused-deputy problem like so:

At heart, the FORTRAN compiler was deputized by two different masters: the customer and the system operators. To serve the customer, it had been given permission to access the customer’s files. To serve the operators, it had been given permission to access the accounting file [which recorded usage billing so the customer could be charged]. But it was confused about which master it was serving for which purpose, because it had no way to associate the permissions it had with their intended uses. It couldn’t specify “use this permission for this file, use that permission for that file”, because the permissions themselves were not distinct things it could wield selectively – the compiler never actually saw or handled them directly. We call this sort of thing “ambient authority”, because it’s just sitting there in the environment, waiting to be used automatically without regard to intent or context.

We have this problem in spades in modern computing. For example, when you run git push --force, the remote server cannot distinguish between you, a Git GUI acting on your behalf, a confused LLM who messed up which branch is which, and a malicious program that is publishing a worm to NPM.

We can solve this in one of three ways:

1. Distinguish between you and the user agents acting on your behalf. For example, give the LLM a separate user account on your computer.
2. Prompt each time you use the credential. For example, use the 1password desktop app or macOS' Keychain.
3. Use a capabilities-based model, where programs cannot even access your SSH key unless you as the user explicitly delegate that permission. For example, use Fuschia on Google Nest Hub ¹.

A true capabilities-based model generally requires redesigning underlying infrastructure and is hard to migrate to incrementally. We don't discuss it much in this book.

Alert fatigue

Claude Code used to default to prompting each time an LLM wanted to leave the sandbox. It recently changed the default to having another LLM review the request:

Data suggests that manual review can become habitual: users approve 97% of permission prompts in Claude Code. While most prompts are likely for safe, routine commands, an approval rate that high suggests many users are clicking through reflexively rather than reviewing each command.

This is known as alert fatigue: when you get too many alerts, and most alerts are just noise, you start ignoring them.

Therefore, this book discourages alert-based security.

How secure is secure?

We are used to thinking of security in terms of "patches": "is your system up-to-date with the latest security fixes?" As LLMs become more-and-more capable, they increasingly find security vulnerabilities that are new and have never before been discovered, called "0-days" ². We can no longer assume that running the latest software version is sufficient for your LLM to be sandboxed. Instead, we have to use defense-in-depth approaches that are correct-by-construction.

Takeaways

• LLMs should not be trusted not to break anything.
• Repeated approval requests cause alert fatigue.
• Distinguish between an LLM and a human using separate credentials, not approvals.
• Build systems that are correct-by-construction.

1. It might be clear that I had to search pretty hard to find an example where capabilities are really used ... ↩

2. "0-days" means 0 days of time for the software maintainer to respond before it starts being exploited. ↩

Scoped credentials

In the main sandboxing page, we talk about the confused deputy problem. Here, we expand more on distinguishing between you and the LLM acting on your behalf.

Our primary suggestion is to use scoped credentials, as described in this blog post.

If you are talking to a web service, this is often supported natively. for example Github will let you fine-grained personal access tokens to talk to its REST API rather than using a single token that allows any access to your account.

But now you have a follow-up problem: how do you give the LLM access to that fine-grained token without also removing your ability to run gh yourself? You can give the LLM a separate account, but this is annoying to set up and maintain. We suggest instead sandboxing the LLM: running it either in a Docker container or in a full virtual machine. This ensures that the only thing in the LLM is what you've given it access to.

Some providers do not support fine-grained tokens at all. For example, if you want your LLM to be able to use JJ, it needs write-access to the .jj directory so that JJ can snapshot the current working directory. But that in turn allows the LLM to run rm -rf .jj, either intentionally or accidentally.

In this case we suggest brokers: introduce an isolation boundary ¹ between the LLM and the credential. For example, jyn has set up a jj-proxy which:

1. Runs the LLM and the broker in separate docker containers
2. Mounts the .jj directory read-only for the LLM and read-write for the broker
3. Adds a jj wrapper command in the LLM's container which proxies to the broker. The broker disallows destructive commands such as --ignore-working-copy or jj abandon.

This allows the LLM to use jj like normal (from its perspective) while not being able to break anything.

Takeaways

• Use narrow-scoped credentials, preferring read-only tokens whenever possible.
• Use virtual machines or containerization to avoid giving LLMs access to your own wide-scoped tokens.
• If your provider does not support narrow-scoped credentials, use "brokers" across an isolation boundary to limit what the LLM can do with a credential.

1. usually a network boundary ↩