Evaluation
Prompt Regression Suite
Treat prompts like code: keep a small regression set before tuning the clever wording.
Use when
A prompt, system instruction, tool policy, or model setting change could silently break workflows that used to work.
Cadence
Before changing a production prompt, agent instruction, or model setting
Verification
Must-pass cases stay green, known failure cases do not regress, and any changed behavior is documented with an accept or reject decision.
Structured loop spec
| Field | Value |
|---|---|
| Name | Prompt Regression Suite |
| Category | Evaluation |
| Trigger | Before changing a production prompt, agent instruction, or model setting |
| Objective | Treat prompts like code: keep a small regression set before tuning the clever wording. |
| Allowed inputs | Relevant files, source notes, logs, tests, screenshots, metrics, or task state for this loop |
| Allowed actions | Select 8-20 representative cases: normal requests, edge cases, refusal boundaries, formatting requirements, and past failures.; Record the expected behavior, allowed variance, required citations or tool evidence, and disallowed outputs for each case.; Run the baseline prompt and the proposed change against the same cases with model, tools, and temperature held constant where possible.; Compare outputs using the written rubric, not vibes, and mark each case pass, fail, degraded, improved, or needs human review.; Ship only if must-pass cases remain green and degraded cases are accepted, fixed, or rolled back. |
| Verification | Must-pass cases stay green, known failure cases do not regress, and any changed behavior is documented with an accept or reject decision. |
| Stop condition | Stop when the verifier passes, the budget is exhausted, no progress is made, a blocker appears, or approval is required. |
| Budget | Set a time, turn, token, retry, file, or dollar cap before running the loop. |
| Approval boundary | Human approval required before publishing, sending, deleting, spending, changing accounts, touching production, or making reputational/legal/financial commitments. |
| Safe output | Draft, report, checklist, table, or approval-gated recommendation |
| Works with | Claude, ChatGPT, Gemini, any tool-using AI assistant |
Steps
- Select 8-20 representative cases: normal requests, edge cases, refusal boundaries, formatting requirements, and past failures.
- Record the expected behavior, allowed variance, required citations or tool evidence, and disallowed outputs for each case.
- Run the baseline prompt and the proposed change against the same cases with model, tools, and temperature held constant where possible.
- Compare outputs using the written rubric, not vibes, and mark each case pass, fail, degraded, improved, or needs human review.
- Ship only if must-pass cases remain green and degraded cases are accepted, fixed, or rolled back.
Prompt
Run the Prompt Regression Suite loop before changing a production prompt, system instruction, model setting, or agent policy. Build or reuse 8-20 representative cases with expected behavior, allowed variance, required evidence, and disallowed outputs. Run baseline and candidate under comparable settings, score against the rubric, and report pass/fail/degraded/improved cases. Do not ship the prompt change unless must-pass cases stay green and any degradation has an explicit accept, fix, or rollback decision.Run in Claude Code
Paste this into Claude Code (or any tool-using agent) to run the loop bounded: one change per round, the same verification every round, durable state files, and explicit stop conditions.
Run the "Prompt Regression Suite" loop from AI Loop Library (https://ailooplibrary.com/loops/prompt-regression-suite/) as a bounded loop.
Goal: Treat prompts like code: keep a small regression set before tuning the clever wording.
Rules: one change per round; run the same verification every round (Must-pass cases stay green, known failure cases do not regress, and any changed behavior is documented with an accept or reject decision.); append each round to docs/loops/prompt-regression-suite/progress.md and update docs/loops/prompt-regression-suite/state.json; stop on verifier pass, 8 rounds, 3 consecutive failed verifications, no progress, a blocker, or anything needing human approval (money, production, outbound, deletion). Finish with a proof report: rounds used, changes made, verification output, remaining risk, and the next human decision.