Claude Code plugin evals, measured on a real skill — the case that scored 1.00 was the wrong answer

The 30-second version

On September 11 Anthropic added claude plugin eval to Claude Code. The skill-creator plugin has had its own eval format for a while; what is new is a first-class command with a no-plugin baseline built in. That baseline is the interesting part: each case is run under two conditions — with your plugin loaded, and with nothing loaded — three times each by default, and the headline number is Δ, the difference between the two scores.

We pointed it at a real, publicly published, actively maintained skill: Stripe’s own stripe-best-practices — specifically the copy installed on this machine back in May and never updated since, which is the state most installed skills are actually in. Three cases, 18 runs, 237 seconds, $1.79, on Claude Opus 5.

Every case passed with a perfect 1.00. Mean Δ +0.33.

Then we read the runs, and the result inverts:

• The entire Δ came from one case — and the answer it scored 1.00 for is four months stale. Upstream replaced that string in July, and again in August.
• The two cases where the skill’s advice still holds scored Δ 0.00 — both arms passed, so the skill is not what made them pass.
• Those two cases cost 56% and 84% more per run with the skill loaded, and took four to five agent turns instead of one.

Δ is a score difference measured against graders you wrote yourself. Write the obvious grader — does the reply contain what my skill says? — and it cannot tell a useful skill from one that quietly went stale. If anything it favours the stale one, because the stale value is the one the bare model won’t produce.

What the command actually runs

A case is a directory: a prompt.md holding the message Claude receives, and a graders/ folder where each file is one pass/fail check. Six grader types exist. Four of them — regex, tool_used, tool_order, file_exists — are computed from the transcript and cost nothing. Two — llm and baseline — call a judge model and bill for it.

Each run starts a fresh headless session in an empty directory with only your plugin loaded: no CLAUDE.md, no user settings, no other skills. Tools beyond a read-only set, and a plugin’s MCP servers, are opt-in per run. Our cases granted only Skill, so neither arm could reach the network — that matters for reading the results below.

By default each case runs three times in each arm, so one case is six runs. The case score is the mean of its runs; a run’s score is the fraction of its graders that passed.

One scoring rule is easy to miss. A grader like “the skill was invoked” can never pass in the without-arm, so counting it would drag the baseline toward zero and inflate Δ. Claude Code therefore excludes every tool_used: Skill grader from the score in both arms and reports it in the with-arm as a pass/fail indicator only. That is why a case can show the skill firing on every single run and still report Δ 0.00.

The suite: one real skill, three questions

We used stripe-best-practices, Stripe’s own published skill, exactly as it sat installed on this machine — an unmodified copy from May 16, four months behind upstream. Two things make it a good test subject. It is not a toy, and it mixes two very different kinds of content: general integration advice, and specific facts like the current API version string. Everything below is a measurement of that installed copy on Claude Opus 5, not of the skill Stripe ships today; we did not test the current version.

Three cases, each phrased the way a user would type it, with no mention of the skill:

Every grader is a regex over the final message, so nothing here depends on a judge model’s mood. Each case also carries a tool_used: Skill grader as the plugin-fired indicator. Each case’s allowed_tools was [Skill], which adds the Skill tool to the read-only set a run always has — so Read and Grep were available, and nothing that reaches the network was. That makes this a test of what each configuration knows, not what it could look up.

The whole api-version-pin case is three short files, if you want to rerun it:

evals/api-version-pin/
├── prompt.md
└── graders/
    ├── names-current-version.md
    └── skill-fired.md

<!-- prompt.md -->
---
max_turns: 6
timeout_seconds: 180
allowed_tools: [Skill]
---

I'm starting a fresh Stripe integration today and I want to pin the API
version explicitly. Which exact version string should I pin to?

<!-- graders/names-current-version.md -->
---
type: regex
pattern: "2026-04-22\\.dahlia"
---

Then claude plugin eval <your-plugin> from the directory above it.

The result

Three of three cases passed. Overall score 1.00. Mean Δ +0.33. 18 runs, 237 seconds, $1.79. The skill fired on all nine with-arm runs.

By every number the tool puts in front of you, this is a healthy suite and a skill that works. Read on.

The case that scored 1.00 is four months stale

The api-version-pin case scored a perfect +1.00 because the with-arm answered with the version string the skill contains, and the bare model never produced that string. Here is the with-arm reply, verbatim from the run transcript:

Pin to:

2026-04-22.dahlia

That’s the latest Stripe API version.

It is not the latest Stripe API version. The copy installed on this machine dates from May 16. Upstream has revised that line at least twice since:

Stripe is not being careless here. That file has nineteen upstream commits since the day this copy was installed, eight of them between August 5 and September 11 alone. The skill is maintained. The installed copy simply froze, the way every installed copy freezes, and nothing in the eval noticed.

That is the first half of the problem. The second half is worse.

The bare model gave the better answer and scored zero

The without-arm scored 0.00 on this case. Not because it answered wrong — because it declined to answer at all, for exactly the right reason:

I can’t give you a trustworthy exact string here, and this is one where a wrong guess is worse than no guess. […] My knowledge cutoff is May 2026, and today is 2026-09-13. Stripe ships new dated API versions regularly (roughly every few months), so any string I recall is at best stale and at worst invented.

It then told the user where to get the authoritative value: Dashboard, Developers, API versions. A second baseline run added that the sandbox had no network access and no Stripe code to read, so there was nothing to infer from.

So the scoring, stated plainly:

• A calibrated refusal that correctly identifies its own staleness risk and routes the user to the source of truth: 0.00.
• A confident assertion of a four-month-old string, presented as current: 1.00.

Δ +1.00. The metric did not malfunction. It measured precisely what it claims to measure — whether the plugin changed the answer — and the plugin changed the answer a great deal.

Same run, two graders, opposite verdicts

To see how much of that +1.00 was the grader’s doing, we re-ran the case with a second grader alongside the first: one checks for the string the skill contains, the other for the actually current string, 2026-08-26.dahlia. Both are applied to every run, so each transcript gets two verdicts.

Case score: WITH 0.50, W/OUT 0.00, Δ +0.50, $0.42 for four runs.

Two graders, the same transcripts, opposite verdicts. Point a grader at your skill’s copy of a fact and you measure whether the skill got its way. Point it at a value you verified yourself and you measure whether the answer was right. The harness gives you either, in the same column, in the same format — nothing in the output tells a reader which one you chose.

Note the second row: 0.00 in both arms. On that grader — the one checking for the string that is actually current — the skill moved nothing, and Δ would have been 0.00. The +1.00 in the headline suite came entirely from the grader that asks whether the skill got its way.

Δ 0.00 is not free

Now the other two cases, the ones that looked like a tie. They are not a tie. Per-arm figures from aggregate-result.json, which the summary table does not show:

On both Δ 0.00 cases the bare model answered in a single turn on every one of its three runs. With the skill loaded the same answer took four to five turns and cost 56-84% more. We kept one run’s transcript to see where the turns went: Claude invoked the skill, then read references/payments.md out of it, then grepped the same file, and only then gave the answer the bare model had produced in one shot. The percentages come from unrounded per-run costs, so recomputing them from the rounded figures in the table lands a point or two off.

The one case where the skill made things cheaper is the one where it supplied a specific version string, however stale: across three runs the baseline arm spent 84 seconds in total against the with-arm’s 19, working out that it could not answer.

So the skill earns its keep exactly where it supplies something the model won’t assert on its own, and costs money everywhere else. None of that reaches the summary, which reports the two expensive cases as Δ 0.00, prints a per-case cost that sums both arms rather than separating them, and passes all three at threshold 1.0.

Five things worth knowing before you run it

It may be gated. On Claude Code 2.1.266 the command exists and prints full help, then refuses with plugin eval is currently in early access. Upgrading to 2.1.270 enabled it. If the command is there but won’t run, update before debugging anything else.

The flag set moved between those builds too. --trust-plugin, which the CI recipe depends on, is absent from --help in 2.1.266 and present in 2.1.270. Pin your Claude Code version in CI, not just your models.

Put the target before the flags. --json, --tag and --allow-tools all take values, so claude plugin eval --json my-plugin reads your plugin name as the output path. Write claude plugin eval my-plugin --json instead.

A run started by an agent won’t publish. Claude Code detects when the eval was launched from inside a Claude Code session rather than by a person, and keeps the HTML report local with a note saying so. Add --publish-report if you want it published anyway. Useful to know before you wonder why your Published: line never appeared.

The cheap mode scores differently. Under --ablation none the tool_used: Skill grader is scored rather than reported as an indicator. In our check both graders passed so the number happened to match, but a case where the skill fails to fire will report a lower score in the cheap mode than in the full one. Same suite, two different numbers — don’t chart them together.

What to do with this

The tool is genuinely good at the thing it was built for. A skill that stops firing on natural phrasing is a real, common, expensive defect, and it is invisible to claude plugin validate, which checks manifest syntax rather than behaviour. The skill-creator plugin has carried its own evals/evals.json format for a while; what is new here is a first-class command with a no-plugin baseline built in. Δ plus the fired-indicator catches non-firing precisely.

The trap is reading Δ as a quality score. It is a score difference, and the graders that produce it are ones you wrote. Four rules follow from the numbers above:

1. Grade against the truth, not against your own file. If your skill hardcodes a value that changes — a version, a price, a limit, an endpoint — type the currently correct value into the grader by hand, and cite where you checked it. When the skill goes stale, that grader fails. A grader that quotes your skill back to itself never can. Its own value will drift too, so date it and re-check it on a schedule; for the case above the right reference is Stripe’s API versioning page, not a second copy of the same skill.
2. Read Δ 0.00 as a bill, not a tie — after checking which way it went. Δ 0.00 with both arms at 1.00 means the model got there alone. Δ 0.00 with both arms at 0.00 means nothing worked and the case tells you nothing. They look identical in the Δ column, so read WITH and W/OUT, not the difference. When both arms pass, open aggregate-result.json, compare costUsd and turns across them, and decide whether that content earns what it costs to load.
3. Gate on the indicator as well as the score. This is the part that is easy to get backwards: tool_used: Skill is excluded from the score in a two-arm run, so a skill that stops firing does not necessarily drag the case below your threshold — both of our non-firing-proof cases would still have passed at WITH 1.00. Check the fired-indicator explicitly in CI rather than assuming --threshold covers it, and pin --model and --judge-model so the number means the same thing next month.
4. A suite that only reaches Δ 0.00 is a result, not a bug in your suite — but confirm it before acting. Weak graders, easy prompts and thin coverage all produce zero gain too. Establish that both arms actually pass on a realistic prompt before concluding the model already does this, and before deleting anything.

The narrow version, which is all our data supports: Δ scores your skill against graders you chose, and the obvious grader — does the reply repeat what my skill says? — rewards a stale skill exactly as much as a useful one.

Companion reading

• What Anthropic’s 80% prompt deletion means for your CLAUDE.md — the same finding arrived at from the other direction: most of what you write down, the model already does.
• CLAUDE.md size and performance — what loading instructions costs before anything is measured.
• Spotify’s 90% token cut, reproduced — the previous measurement in this series, and the same lesson about headline numbers.
• CLAUDE.md token mistakes — the specific content patterns that buy nothing.
• The Claude Code ecosystem’s gaps — where plugin tooling still has holes.

Sources

1. Anthropic — Test plugins with evals. Primary source for case layout, the six grader types, the two-arm ablation, the scoring exclusion for tool_used: Skill, the CI recipe, and exit codes.
2. Anthropic — Claude Code changelog. Version 2.1.269, September 11, 2026: “Added claude plugin eval”.
3. Stripe — stripe-best-practices skill, in the stripe/ai repository. The skill under test, and the commit history used to date the version-string changes.
4. Stripe — Skills for agents. Installation and intended use of the skill.
5. Anthropic — Agent Skills documentation. The description frontmatter that determines whether a skill fires.
6. Anthropic — Claude Code costs. How the list-price estimates the eval reports are derived.

Our own measurements are described in full above: Claude Code 2.1.270 on macOS, the default model for these runs being Claude Opus 5 (claude-opus-5), stripe-best-practices as installed 2026-05-16 and not updated since, three cases, default three runs per arm, free regex and tool_used graders only, allowed_tools: [Skill] on top of the standard read-only set, and no network-capable tool in either arm. Totals of 18 runs, 237 seconds and $1.79 for the main suite, plus a four-run follow-up at $0.42. Per-arm cost and turn figures are read from aggregate-result.json.

FAQ

What is claude plugin eval and what does it measure? A command added in Claude Code 2.1.269 on September 11, 2026. Each case is a prompt plus graders, and runs three times with your plugin and three times with nothing loaded. You get WITH, W/OUT, and Δ. Δ is what the plugin contributed against the graders you wrote — nothing checks whether an answer is correct unless one of your graders does.

Does a high score mean my skill is good? Not on its own. High Δ means your skill made the graders pass where the bare model did not. If the grader looks for the string your skill contains, a stale string earns that as reliably as a useful one — ours scored +1.00 on a version string superseded twice upstream, while the bare model scored 0.00 for declining to guess.

What does Δ 0.00 mean? Read WITH and W/OUT before the difference — both arms at 1.00 means the model got there alone, both at 0.00 means the case failed everywhere and tells you nothing. Ours were the first kind, and still cost 56% and 84% more per run on average, at four to five turns instead of one. The summary table prints that as a tie.

How much does it cost? Our three-case suite was 18 runs, 237 seconds, $1.79 at list price — about $0.60 per case, $0.10 per run, using only free graders. llm and baseline graders add three judge calls each per run. Runs scale as cases × runs × 2. --ablation none halves the runs but not necessarily the bill, since the arms cost different amounts; --max-cost-usd stops new runs at the ceiling, though an in-flight run still finishes.

Should I gate CI on the score? Gate on it, but check the fired-indicator separately: tool_used: Skill is excluded from the score in a two-arm run, so a skill that stops firing will not necessarily drop below your threshold. A score gate also can’t catch stale content, which raises Δ rather than lowering it. Add a grader asserting a hand-checked current value, and pin both --model and --judge-model.

What should a case actually test? Somewhere your skill supplies something the model won’t produce alone, with the grader written against a value you verified rather than your skill’s copy of it. If no such case exists, a near-zero Δ may be a finding about the skill — but check that both arms pass on a realistic prompt first, because weak graders produce zero gain too.