Evaluation
RAG Grounding Audit
Test the retrieval chain before polished nonsense reaches users.
Use when
A chatbot, agent, search page, or knowledge workflow answers from docs and could sound confident while grounded in the wrong source.
Intermediatedifficulty
Evaluationcategory
Mia daily expansionsource
Cadence
Before shipping or refreshing a retrieval-backed assistant
Verification
Representative answers cite the right source chunks, abstain when retrieval is weak, and expose no private or stale material.
Structured loop spec
| Field | Value |
|---|---|
| Name | RAG Grounding Audit |
| Category | Evaluation |
| Trigger | Before shipping or refreshing a retrieval-backed assistant |
| Objective | Test the retrieval chain before polished nonsense reaches users. |
| Allowed inputs | Relevant files, source notes, logs, tests, screenshots, metrics, or task state for this loop |
| Allowed actions | Select 10-20 real questions covering common asks, edge cases, stale topics, and questions the system should refuse or escalate.; For each case, record required sources, acceptable answer shape, refusal conditions, and private or outdated material that must not appear.; Run the current system with model, prompt, filters, and temperature held as constant as possible, capturing retrieved chunks, citations, and final answer.; Score each case as grounded, partially grounded, hallucinated, stale, private-risk, or correct no-answer, with the evidence attached.; Patch one retrieval, chunking, metadata, or prompt issue at a time, then rerun the failed cases and the nearest passing controls.; Do not publish updated answers, docs, or customer-facing claims until failed cases are cleared or explicitly queued for approval. |
| Verification | Representative answers cite the right source chunks, abstain when retrieval is weak, and expose no private or stale material. |
| Stop condition | Stop when the verifier passes, the budget is exhausted, no progress is made, a blocker appears, or approval is required. |
| Budget | Set a time, turn, token, retry, file, or dollar cap before running the loop. |
| Approval boundary | Human approval required before publishing, sending, deleting, spending, changing accounts, touching production, or making reputational/legal/financial commitments. |
| Safe output | Draft, report, checklist, table, or approval-gated recommendation |
| Works with | Claude, ChatGPT, Gemini, any tool-using AI assistant |
Steps
- Select 10-20 real questions covering common asks, edge cases, stale topics, and questions the system should refuse or escalate.
- For each case, record required sources, acceptable answer shape, refusal conditions, and private or outdated material that must not appear.
- Run the current system with model, prompt, filters, and temperature held as constant as possible, capturing retrieved chunks, citations, and final answer.
- Score each case as grounded, partially grounded, hallucinated, stale, private-risk, or correct no-answer, with the evidence attached.
- Patch one retrieval, chunking, metadata, or prompt issue at a time, then rerun the failed cases and the nearest passing controls.
- Do not publish updated answers, docs, or customer-facing claims until failed cases are cleared or explicitly queued for approval.
Prompt
Run the RAG Grounding Audit loop. Build 10-20 representative questions, including common asks, edge cases, stale topics, and should-refuse cases. For each, define required sources, acceptable answer shape, refusal rules, and private or outdated content that must not appear. Run the current retrieval-backed system with settings held constant, capture retrieved chunks, citations, and answers, then score grounding quality. Patch one retrieval, chunking, metadata, or prompt issue at a time and rerun failed cases plus nearby controls. Do not publish revised answers, docs, or customer-facing claims until failures are cleared or approval is explicit.Run in Claude Code
Paste this into Claude Code (or any tool-using agent) to run the loop bounded: one change per round, the same verification every round, durable state files, and explicit stop conditions.
Run the "RAG Grounding Audit" loop from AI Loop Library (https://ailooplibrary.com/loops/rag-grounding-audit/) as a bounded loop.
Goal: Test the retrieval chain before polished nonsense reaches users.
Rules: one change per round; run the same verification every round (Representative answers cite the right source chunks, abstain when retrieval is weak, and expose no private or stale material.); append each round to docs/loops/rag-grounding-audit/progress.md and update docs/loops/rag-grounding-audit/state.json; stop on verifier pass, 8 rounds, 3 consecutive failed verifications, no progress, a blocker, or anything needing human approval (money, production, outbound, deletion). Finish with a proof report: rounds used, changes made, verification output, remaining risk, and the next human decision.