# Claude Code plugin eval: gate skills on delta before you merge

> Use claude plugin eval WITH/WOUT delta and CI thresholds to prove a skill helps before you merge it into an owned kit repo.
> By Dave · 2026-09-14
> Source: https://otf-kit.dev/blog/claude-code-plugin-eval-ci

Claude Code now ships a first-party way to prove a plugin or skill actually helps. The command is `claude plugin eval`. It runs your suite with the plugin loaded and again without it, then reports `WITH`, `W/OUT`, and `Δ` — the lift the plugin contributed. That changes what production teams should do before they merge a skill into a kit repo or pin it in CI.

This is not another “write better prompts” essay. It is a builder workflow for Claude Code v2.1.269+: create cases under `evals/`, grade outcomes, and fail the build when the score drops. If you already ship agent docs in an owned monorepo, pair this with [Claude Code on an OTF kit: the CLAUDE.md and prompts that ship with the repo](/blog/claude-code-otf-kit-docs). For feature-level evals outside plugins, keep using [A practical LLM evaluation loop for AI features that need to ship](/blog/llm-evaluation-loop) — different layer, different artifact.

## What changed for plugin authors

Anthropic’s [plugin evals docs](https://code.claude.com/docs/en/plugin-evals) describe a dedicated CLI path for plugin and skill authors. The important claims for builders:

- You need Claude Code **v2.1.269 or later** (`claude --version`, then `claude update` if needed).
- Cases live under an `evals/` directory next to your plugin manifest (`plugin.json` or `.claude-plugin/plugin.json`).
- Each case is a realistic user prompt plus one or more graders (regex, tool_used, tool_order, file_exists, llm, baseline).
- By default each case runs **three times with the plugin and three times without it**. `Δ` is with-arm score minus without-arm score.
- `claude plugin eval init` can propose cases and graders interactively; `claude plugin eval init --bare <case>` writes a blank template.
- CI can gate on `--threshold`, pin `--model` / `--judge-model`, keep reports local with `--no-publish`, and cap spend with `--max-cost-usd`.

That is the DO: stop merging skills because a demo chat looked fine. Measure contribution, then gate.

![Dex and Luna compare WITH versus WOUT scores while Nova highlights a positive delta](https://cdn.otf-kit.dev/blog/claude-code-plugin-eval-ci/inbody-delta-decision-20260914b.jpg)

## Why WITH alone is not enough

A case that scores `1.00` with the plugin loaded can still be useless. Claude might already solve the prompt without your skill. The docs call this out: if `WITH` and `W/OUT` are both high, `Δ` near zero means the plugin did not move the outcome.

Production implications:

1. **Trigger failures show up as skill graders failing.** A common first finding is `Δ ≈ 0` with a `tool_used: Skill` grader failing — Claude never chose your skill on natural phrasing. Fix the skill `description`, re-run, compare.
2. **Skill-fired graders are indicators, not free Δ inflation.** In two-arm mode, `tool_used` graders on `Skill` are excluded from both arms’ scored totals so you do not invent lift by checking something impossible without the plugin.
3. **Judge graders cost money.** `regex`, `tool_used`, `tool_order`, and `file_exists` are free transcript/file checks. `llm` and `baseline` call a judge model and add to the run’s list-price estimate.
4. **Non-determinism is assumed.** Default three runs per arm exist because one agent run is noise. Cheap iteration uses `--runs 1 --ablation none`; trust needs the default three.

If you only ever run the with-arm, you are grading absolute behavior. That is fine while drafting graders. Before you call a skill “done,” turn the baseline back on and read `Δ`.

## Minimal suite layout you can commit

From the plugin root (the directory that contains the manifest):

```bash
claude --version   # need >= 2.1.269
claude plugin eval init --bare commit-message
```

You get something shaped like:

```text
evals/commit-message/
├── prompt.md
└── graders/
    └── criteria.md
```

Edit `prompt.md` so the body is a request a user would type — do not name the skill in the prompt:

```markdown
---
max_turns: 10
allowed_tools: [Read, Glob, Grep, Skill]
---

Write me a commit message for this change: I renamed getUser to fetchUser and updated the three call sites.
```

Add a result grader (`llm` or `regex`) and a skill-fired grader:

```markdown
---
type: tool_used
tool: Skill
input_match: '"skill"\s*:\s*"(?:[\w-]+:)?your-skill-name"'
---
```

Then run:

```bash
claude plugin eval .
```

Expect six runs for one case at defaults (3× with, 3× without). Open the HTML report path printed at the end. Iterate on description and graders until `Δ` is positive for the prompts you care about.

For MCP-backed skills, put mocks under `evals/mocks/<server>/<tool>.md` so CI does not need the real service. Use `--scaffold` only for suites you trust — scaffold scripts run as you, outside the agent sandbox.

## Gate the suite in CI

The docs’ recommended CI shape is explicit:

```bash
claude plugin eval . \
  --trust-plugin \
  --json results.json \
  --threshold 0.8 \
  --model claude-sonnet-5 \
  --judge-model claude-haiku-4-5 \
  --no-publish \
  --max-cost-usd 20
```

Exit codes that matter:

- `0` — every case met `--threshold`
- `1` — a case scored below threshold, files failed to load, or trust was missing without `--trust-plugin`
- `2` — partial run (`--max-cost-usd` hit or auth failed); JSON still written with `partial: true`

Pin both models so a provider rollout is not mistaken for a plugin regression. Keep every-change suites on free graders when you can; reserve `llm` judges for short outputs with concrete PASS/FAIL rubrics. Leave `partial: true` results out of trend charts.

Credentials and install still belong on the runner (`ANTHROPIC_API_KEY` or your normal Claude Code auth). Without `--trust-plugin`, a non-interactive job against an untrusted checkout exits `1`.

![Byte pins a CI gate on plugin eval threshold while Nova holds a report card](https://cdn.otf-kit.dev/blog/claude-code-plugin-eval-ci/inbody-ci-gate-20260914c.jpg)

## How this fits an owned kit repo

OTF’s wedge is not “another sandboxed chat.” It is an owned repo agents keep editing — `CLAUDE.md`, prompts, and skills that travel with the product. Plugin eval is the missing acceptance gate for those skills:

| Layer | Question | Artifact |
|---|---|---|
| Repo conventions | Can an agent find the seams? | `CLAUDE.md` / `.cursorrules` (see the kit docs post above) |
| Change acceptance | Did this PR do what we asked? | [AI coding agent acceptance checklist](/blog/ai-agent-acceptance-checklist) |
| Plugin contribution | Did the skill raise the score vs no plugin? | `claude plugin eval` + `Δ` |
| Product AI features | Does the user-facing model path hold? | LLM evaluation loop |

Put the suite next to the plugin you ship. Fail CI when `Δ` collapses after a description tweak or a model pin change. That is how skills stay production assets instead of chat folklore.

Security note from the same docs: pointing `claude plugin eval` at a plugin is the same trust decision as `claude --plugin-dir`. Only evaluate plugins you trust. Hooks and real MCP servers you opt into with `--allow-real-servers` or `--mocks off` run outside the agent sandbox.

## Practical checklist before you merge a skill

1. Upgrade to Claude Code ≥ 2.1.269.
2. Add at least one natural-language case that should trigger the skill and one that should not.
3. Include one outcome grader and one `tool_used: Skill` indicator.
4. Run with baseline on; require a positive `Δ` for the should-trigger cases.
5. Wire the CI command with `--trust-plugin`, pinned models, `--threshold`, and `--max-cost-usd`.
6. Mock MCP tools for CI; do not call production services from eval runs.
7. Treat rate-limit errors mid-suite as invalid scores — re-run after the limit resets before you trust the table.

If you are still choosing between a sandboxed MVP host and a repo you own, start with the kit spine, then add this gate. Skills without `Δ` are demos. Skills with a CI threshold are release machinery.

## Sources

- [Test plugins with evals (Claude Code docs)](https://code.claude.com/docs/en/plugin-evals) — primary: CLI requirements, WITH/WOUT/Δ, grader types, CI flags, exit codes, trust and isolation notes (verified 2026-09-14).