Evaluation and benchmarks

Expand diagram

Retrieval evaluation

The benchmark measures ranking behavior on a finite set of human-scored questions. Its results describe that set, not general answer quality.

Use the zoom controls, drag to pan, or pinch on a touch screen. Press Escape to close.

The retrieval benchmark measures whether relevant source passages appear near the top of a ranked list. It evaluates retrieval quality; it does not prove every generated response is correct.

Human-reviewed questionsCopy Anchor Link

The primary human set has 30 questions: 26 scored questions and 4 intentionally out-of-scope questions. The benchmark compares six retrieval arms and reports Accuracy@k, Precision@k, Recall@k, F1@k, MRR, and nDCG.

These numbers belong to the recorded corpus and benchmark snapshot. They are neither service latency measurements nor a guarantee for future queries.

Fitted fusion and significanceCopy Anchor Link

Weighted fusion uses dense 0.60 and BM25 0.40; TF-IDF and Word2Vec have zero weight in the fitted configuration. Active members contribute up to 30 candidates. The default reranker examines up to 20 fused passages.

The stored findings report a paired-bootstrap MRR difference of +0.107 for hybrid_rr over dense (p = 0.006). RRF is a control condition and trails weighted fusion in this experiment. See docs/results/FINDINGS.md for detailed comparisons and the before/after account.

Synthetic coverage setCopy Anchor Link

A separate 129-question synthetic set is used as a coverage and regression check. Its ranking differs from the human set and exhibits lexical bias. It is not independent human validation. The findings call for a 50-question human review sample before synthetic scores are presented as externally validated.

ReproduceCopy Anchor Link

Benchmark artifacts are in docs/results/; the evaluator and datasets are in scripts/. Read docs/benchmark-design.md for the original design and its as-built differences. See Retrieval for the implemented search pipeline.