Evaluator Optimizer

---

name: evaluator-optimizer description: A generator-plus-critic loop that improves output against an explicit rubric before acceptance. Use when you have clear acceptance criteria and iteration measurably helps (specs, prompts, important docs, tricky implementations). Includes a seed eval set for regression-testing prompt and rule changes.

Evaluator-Optimizer Loop

Generate, then have an independent critic score against an explicit rubric, then refine, until it passes or the budget is spent. One of Anthropic's five effective-agent patterns, and the one most starter kits omit. It is how you get measurably better output instead of one-shot output you hope is good.

When to use this

Invoke /evaluator-optimizer when both are true:

1. You can state acceptance criteria. There is a rubric, a spec, a set of must-haves, or a clear definition of done.
2. Iteration helps. A first pass is rarely the best pass for this kind of artifact: a specification, an API design, a security-sensitive change, an important document, a prompt you will reuse.

Do not use it when there is no clear evaluation signal, or when a single pass is obviously good enough. Adding a critic loop to a trivial task is the over-engineering this kit warns against (see .claude/rules/ai-orchestration-decision-gate.md).

The loop

1. RUBRIC      Define explicit, checkable acceptance criteria up front.
2. GENERATE    A generator produces the artifact against the rubric.
3. EVALUATE    The `evaluator` agent (frontier, read-only) scores each criterion: PASS / FAIL + concrete gap.
4. DECIDE      All blocking criteria pass        -> accept.
               Gaps remain and rounds remain     -> feed the evaluator's gaps back to the generator, go to 2.
               Budget spent and gaps remain       -> stop, hand back with the open gaps named (do not pretend done).

Keep it to 1 to 2 refinement rounds by default. More than that usually means the rubric is wrong or the task is mis-scoped, not that another round will help.

Roles

• Generator: the implementation or authoring agent (or the main thread). Produces and revises.
• Critic: the evaluator agent (.claude/agents/evaluator.md). Read-only, frontier model, scores against the rubric, never edits. Keep these two separate; a generator grading its own work is not an evaluation.

Writing a good rubric

A rubric is a short list of criteria, each independently checkable and ideally backed by a command or a concrete check:

• Binary where possible ("all exported functions have explicit return types": yes/no), not vague ("code is clean").
• Tied to evidence ("npx tsc --noEmit exits 0", "the failing test from step 1 now passes").
• Separated into blocking (correctness, security, meets-the-requirement) and advisory (style, naming). Only blocking criteria gate acceptance.

LLM-as-judge and the seed eval set

For prompts, rules, and skills you will reuse, you want to know whether a change made them better or worse, not just whether one example looked fine. This skill ships a small seed set of example tasks in seed-tasks.md. Use it as a lightweight regression harness:

1. When you change a rule, a skill, or an agent's prompt, run a sample of the seed tasks through the affected path.
2. Have the evaluator agent score the outputs with a fixed rubric (LLM-as-judge).
3. Compare against the previous version's scores. A change that lowers scores on the seed set is a regression, even if the one case you were looking at improved.

This is the same principle behind held-out test suites: trust the measured signal, not self-assessment. Expand seed-tasks.md with the cases your project actually cares about; twenty good examples beat two hundred shallow ones.

High-stakes work gets a human checkpoint

Automated evaluation reduces error; it does not eliminate it. For changes that are expensive to get wrong (migrations, security boundaries, anything touching money or auth, production mutations), the loop ends at a human approval, not at the critic's PASS. The evaluator's job there is to surface the risks clearly so the human decision is well-informed.

Anti-patterns

• The generator evaluating its own output (not an independent signal).
• A rubric of vague adjectives instead of checkable criteria.
• Looping more than twice (fix the rubric or re-scope instead).
• Treating the critic's PASS as a substitute for human sign-off on high-stakes changes.
• Skipping the seed set when changing a reused prompt or rule, then being surprised by a regression elsewhere.

Related

• .claude/agents/evaluator.md, the critic that scores against the rubric
• .claude/rules/ai-orchestration-decision-gate.md, when the loop is worth it vs over-engineering
• .claude/skills/verification-before-completion/SKILL.md, the per-change definition-of-done gate
• .claude/rules/deterministic-review.md, the evidence-strength bar for the critic's findings
• seed-tasks.md (in this skill folder), the regression-test example set