Comparing Agent Instructions Before and After a Change

When you work with AI agents, you end up editing instructions like CLAUDE.md, AGENTS.md, or .claude/rules/ a lot.

After an edit, I had to run a prompt by hand and check the result, or open the session file and check whether the agent read the relevant files in the order I wanted, ran the task I intended, and produced the output I wanted. That got fairly tedious (handing the analysis itself to an AI does work well, though).

To know whether a changed instruction had any effect, I also had to test before and after it, but the results vary a bit from run to run, so one or two runs felt like too little to judge from.

It was also hard to explain to other people any qualitative grounds for “which one is better”. I should have set clear success criteria and shown results against them, but doing that every time is a hassle, so by my own standard I mostly looked at the edited instructions and judged on the expectation that it would probably be better. For example, if it ran fine two or three times, I’d call that a success.

While looking for a better way, I remembered that Anthropic’s official skill-creator skill, which I’ve been using for a while, lets you compare results with and without the skill on a report page it serves.¹ I’d always found that comparison itself quite useful, so I borrowed the approach and built a skill that shows what changes in the results when you edit the instructions.

The way it works is simple. It takes the original environment A and the changed environment B, keeps everything else the same, runs several prepared prompts multiple times in each, then shows the results as A / B. It then shows which tool calls happened and which instruction files were read in A and B respectively.

For example, you can test situations like these.

Whether compressing a 60-line instruction file down to 40 lines still gives the same or a similar level of results
Whether a specific rule is still followed after the instructions get long from adding many unrelated rules
What difference the presence or absence of a specific piece of context (a document or code) makes in the results
Whether the agent actually uses a tool after adding an instruction like “use this tool in this situation”
Whether the results include what you expected after the change

I think I mostly use it when I want to see whether an instruction change, or the presence or absence of some context, makes a meaningful difference in the results. Since the model and other variables are hard to control completely, I use it as a rough basis for seeing what trend shows up and judging which one looks better qualitatively.

References

It runs the same prompts with and without the skill and gives a report comparing how many tokens are used, how the run time differs, and what output comes out. ↩

References

Footnotes