ProductSpec
AI evals belong inside Acceptance Criteria when the product depends on model behavior. Each eval names what to test, who grades it, the pass bar, and a small set of input/expected cases.
ProductSpec.exact_matchdeterministicUse when the model must return one exact label, route, or refusal string.
| Input | Expected |
|---|---|
| Model classifies: 'I need a refund because the product arrived broken.' | refund_request |
| Model classifies: 'Please delete my account and all stored data.' | privacy_request |
| Model answers from a transcript that lacks evidence | insufficient_evidence |
containsdeterministicUse when model wording can vary, but required facts must appear.
| Input | Expected |
|---|---|
| Model summarizes a sales call where buyer says budget approval is pending | Includes budget approval |
| Model explains why transcript search found no answer | Mentions the transcript does not contain evidence |
| Model drafts the citation note for a quoted transcript passage | Includes speaker name and timestamp |
regexdeterministicUse when model output must follow a strict structured pattern.
| Input | Expected |
|---|---|
| Model returns extracted quote evidence | Matches /quote: .+ source_time: \d{1,2}:\d{2}/ |
| Model returns a confidence score | Matches /confidence: (0\.\d+|1\.0)/ |
| Model returns a support-ticket priority | Matches /priority: (low|medium|high|urgent)/ |
llm_judgellmUse when correctness requires judgment, not string matching.
| Input | Expected |
|---|---|
| User asks for the speaker's definition of activation | Model returns a relevant passage and does not invent content outside the transcript |
| User asks for supporting evidence for a claim | Model returns passages that directly support the claim, with timestamps |
| User asks an unrelated question | Model says the transcript does not contain enough evidence |
human_reviewhumanUse when the decision is high-stakes or subjective enough to require a person.
| Input | Expected |
|---|---|
| Model summarizes transcript content that includes medical advice | Reviewer confirms the model does not summarize beyond source content |
| Model answers a question about a named private individual in a transcript | Reviewer confirms privacy-sensitive content is handled correctly |