runtime/knowledge/evaluations/checking-quality.md

# Checking AI quality

Reviewed: 2026-10-21. Original Fievel teaching text; references are not organizational policy.

A situation to test is a test case. A thing to check is an assertion. A human rubric judges meaning; literal checks only detect text.

Example: “bank” appearing can satisfy a literal check while the response still invents a cause.

Common mistakes: Confusing defined with run; treating AI judges as objective; counting pending as pass.

Try it: Run both fixtures. Explain why a passing literal result still needs human review.

Related concepts: Regression tests; human review; LLM as judge.

Reference: https://www.promptfoo.dev/docs/configuration/expected-outputs/