# Eval harness — how we know an answer is good

The assistant is graded against a **golden set** (human-labelled questions) on every change.
The judge grades against the **approved sources** (an answer key), never its own knowledge.

## Pipeline
```
golden_set.jsonl   (human-labelled: question, expected behaviour, [expected source], [program/platform])
      │
      ▼  for each case, run the REAL /chat pipeline → {answer, escalate, reason, sources, best_sim}
      │
      ├─ DECISION   (code)  did it answer / escalate / decline as expected, with the right reason?
      ├─ RETRIEVAL  (code)  was the expected approved entry actually retrieved (recall@k)?
      ├─ FAITHFUL   (judge) is every claim in the answer supported by the retrieved approved sources?
      └─ RELEVANT   (judge) does the answer actually address the question?
      │
      ▼  aggregate → scorecard + list of failures with reasons → eval/last_results.json
```

## Why the judge is trustworthy
1. **It grades against the approved sources, not from memory** — "is this answer supported by THESE sources?" is a checkable task (marking with the answer key), not a knowledge test.
2. **Decision + retrieval need no LLM** — pure code, 100% reliable. Only faithfulness/relevance use the judge.
3. **Human-anchored** — humans define the golden labels once; we spot-check judge-vs-human agreement, and a stronger model (Gemini Pro) can be set via `GEMINI_JUDGE_MODEL` to reduce correlated errors.

## Metrics & targets
| Metric | Means | Target |
|---|---|---|
| Decision accuracy | answered/escalated/declined correctly, right reason | high |
| Retrieval hit@k | the right approved answer was retrieved | high — caps everything |
| Faithfulness | answer fully supported by source, nothing invented | ~100% (non-negotiable) |
| Relevance | answer addresses the question | high |
| Safety recall | medical/crisis correctly escalated | ~100% (non-negotiable) |

## The flywheel
Every production 👎 and every `no_confident_answer` escalation becomes a **new golden case** →
the golden set grows more representative over time → eval gets stricter as real usage reveals gaps.

Run: `python3 eval/run_eval.py`