{"id":77098,"plugin_id":"plugins_6ab42163e9048191ab6e8c14258a48c3","kind":"skill","collection_source":"plugin_package","comparison_source":null,"observed_at":"2026-11-03T00:12:26.355Z","digest":"0578ab63f88dd9272c9737ab42e4f519f9fcd5ce007f3c12d09fdc47ba7a22e6","against":null,"payload":{"description":"Build measurable RAG evaluations for retrieval relevance, grounding, citations, abstention, security, latency, and cost.","included_files":[],"name":"evaluate-rag","skill_md_contents":"---\nname: evaluate-rag\ndescription: Build measurable RAG evaluations for retrieval relevance, grounding, citations, abstention, security, latency, and cost.\n---\n\n\n# Design and run RAG evaluations\n**Trigger:** Evaluation design, regression gates, quality scoring, testing retrieval settings or checking an architecture change.\n\n1. Name the product decision: compare retrievers, catch regressions, choose a model, validate authorization or accept a launch.\n2. Build a versioned dataset with query, expected evidence IDs, answerable/no-answer label, tenant/scopes, source version, language and category; avoid synthetic-only data.\n3. Separate retrieval (recall@k, MRR, nDCG), context (evidence present after packing), generation (groundedness, relevance, completeness), citation claim-support, abstention, access/security and cost/latency.\n4. Define metric denominators, missing-evidence handling and per-slice thresholds. Pair automatic judges with targeted human review; validate judge reliability.\n5. Establish baseline/holdout, paired comparisons, uncertainty and regression gates. Do not cherry-pick cases or claim live run without artifacts.\n6. Execute only with actual tools/data. For deterministic, provided JSON fixtures, `scripts/score_retrieval.py` can compute basic metrics; it cannot measure groundedness or security.\n7. Deliver evaluation spec, data manifest, result table if run, failure slices, deployment gate and follow-up experiments.\n\nRead `../../references/evaluation-metrics.md`, `../../references/evidence-contract.md` and `../../templates/eval-plan.md`.\n"},"changes":[],"summary":"First saved snapshot. No earlier version is available for comparison.","summary_kind":"deterministic","summary_metadata":{}}