Evaluation
Model Output Drift Watch
Catch quiet AI behavior drift before users become the monitoring system. They charge more.
Use when
A production AI workflow can change behavior because the model, prompt, tool results, retrieval index, or upstream data shifted under it.
Intermediatedifficulty
Evaluationcategory
Mia daily expansionsource
Cadence
Weekly or after model, prompt, retrieval, or tool changes
Verification
Representative outputs are compared against a baseline, regressions are classified with evidence, and risky changes are rolled back or queued for approval.
Structured loop spec
| Field | Value |
|---|---|
| Name | Model Output Drift Watch |
| Category | Evaluation |
| Trigger | Weekly or after model, prompt, retrieval, or tool changes |
| Objective | Catch quiet AI behavior drift before users become the monitoring system. They charge more. |
| Allowed inputs | Relevant files, source notes, logs, tests, screenshots, metrics, or task state for this loop |
| Allowed actions | Select a stable sample of real or synthetic cases covering common tasks, edge cases, refusal boundaries, formatting needs, and high-risk customer or internal workflows.; Run the current workflow with model, prompt, tools, retrieval filters, temperature, and input data recorded alongside the previous baseline where available.; Score differences as acceptable variance, improvement, regression, privacy risk, policy risk, formatting break, tool-use failure, or needs human review.; Patch one cause at a time: prompt, tool schema, retrieval metadata, guardrail, fixture, or model setting, then rerun the failed cases plus nearby controls.; Do not ship prompt/model/tool changes, publish generated content, or update customer-facing claims until must-pass cases are green or an explicit approval accepts the risk. |
| Verification | Representative outputs are compared against a baseline, regressions are classified with evidence, and risky changes are rolled back or queued for approval. |
| Stop condition | Stop when the verifier passes, the budget is exhausted, no progress is made, a blocker appears, or approval is required. |
| Budget | Set a time, turn, token, retry, file, or dollar cap before running the loop. |
| Approval boundary | Human approval required before publishing, sending, deleting, spending, changing accounts, touching production, or making reputational/legal/financial commitments. |
| Safe output | Draft, report, checklist, table, or approval-gated recommendation |
| Works with | Claude, ChatGPT, Gemini, any tool-using AI assistant |
Steps
- Select a stable sample of real or synthetic cases covering common tasks, edge cases, refusal boundaries, formatting needs, and high-risk customer or internal workflows.
- Run the current workflow with model, prompt, tools, retrieval filters, temperature, and input data recorded alongside the previous baseline where available.
- Score differences as acceptable variance, improvement, regression, privacy risk, policy risk, formatting break, tool-use failure, or needs human review.
- Patch one cause at a time: prompt, tool schema, retrieval metadata, guardrail, fixture, or model setting, then rerun the failed cases plus nearby controls.
- Do not ship prompt/model/tool changes, publish generated content, or update customer-facing claims until must-pass cases are green or an explicit approval accepts the risk.
Prompt
Run the Model Output Drift Watch loop. Build or reuse a stable sample covering common tasks, edge cases, refusal boundaries, required formats, and high-risk workflows. Run the current AI workflow and compare against the previous baseline with model, prompt, tools, retrieval filters, temperature, and data version recorded. Classify differences as acceptable, improved, regressed, privacy risk, policy risk, formatting break, tool failure, or needs review. Patch one cause at a time and rerun failed cases plus controls. Do not ship risky prompt, model, tool, retrieval, or customer-facing changes until must-pass cases are green or explicitly approved.Run in Claude Code
Paste this into Claude Code (or any tool-using agent) to run the loop bounded: one change per round, the same verification every round, durable state files, and explicit stop conditions.
Run the "Model Output Drift Watch" loop from AI Loop Library (https://ailooplibrary.com/loops/model-output-drift-watch/) as a bounded loop.
Goal: Catch quiet AI behavior drift before users become the monitoring system. They charge more.
Rules: one change per round; run the same verification every round (Representative outputs are compared against a baseline, regressions are classified with evidence, and risky changes are rolled back or queued for approval.); append each round to docs/loops/model-output-drift-watch/progress.md and update docs/loops/model-output-drift-watch/state.json; stop on verifier pass, 8 rounds, 3 consecutive failed verifications, no progress, a blocker, or anything needing human approval (money, production, outbound, deletion). Finish with a proof report: rounds used, changes made, verification output, remaining risk, and the next human decision.