evaluation.md

# Evaluation

The executable synthetic evaluation is `evals/run_eval.py`; its dataset is `evals/cases.json`. It calls the same `BookAgent` Core used by CLI/MCP and creates an isolated PDF knowledge package with existing fixed 2D fixture vectors. No original example archive is modified or copied into the distributable dataset. The synthetic vectors exercise adapter/ranking mechanics and do not represent a validated real embedding model.

Run from the project directory in PowerShell:

```powershell
& .\.venv\Scripts\python.exe .\evals\run_eval.py --top-k 3 --repetitions 3
```

Use `--output` to choose independent evaluation runtime storage and `--dataset` for a compatible case file. Each run creates a new folder; it does not delete or overwrite an earlier package. The default is `.work/evals/`. A complete machine-readable `report.json` is saved in each run directory and copied to `latest-report.json`.

The two modes use identical package/version, query vectors, query runtime fingerprint, dense/lexical limits, candidate limit, text budgets and top_k. The control records full candidate-text fingerprints. The custom loopback HTTP MOCK compares its submitted documents to those fingerprints and reports mismatches as failed checks. Its lexical scoring rule is deliberately simple and transparent. It is neither a remote model nor a local CrossEncoder. No API keys, external calls or downloads are used. Query representation is marked `caller_vector_unverified`; the no-evidence question uses explicit lexical fallback because no query encoder is verified.

Metrics are defined as follows:

| Metric | Definition |
| --- | --- |
| retrieval hit@k | Fraction of answerable cases with at least one expected record among top k |
| MRR | Mean reciprocal rank of the first expected record; zero when absent |
| source locator hit | Fraction of source-labelled cases retrieving an expected record and verifying its expected PDF file-page index |
| visual fallback precision | True fallback decisions / all predicted fallback decisions |
| visual fallback recall | True fallback decisions / all expected fallback decisions |
| search latency | Wall-clock Core search duration in milliseconds, p50/p95/min/max across repetitions |
| original integrity | Exact before/after file hashes, sizes, modification times and directory inventory |

Fallback is measured using `source_fallback.recommended`: labels mean the Agent should inspect or verify original evidence. They include image questions, a Chinese “state graph” question and insufficient textual evidence. Text insufficiency may need original-source verification without proving an image exists. Reasons remain available per case. Metrics with no denominator return null. Expected-no-evidence cases are excluded from retrieval/MRR/source denominators and receive separate no-hits/status contract checks.

The dataset covers graph automata, temporal property checking, Chinese short terms, publication citations, no evidence and two visual questions. A unique value occurs solely as raster pixels on the parameter page. The runner confirms it is absent from PDF native text, Markdown and SQLite body text. It follows Core locate → read → render for visual cases, checks the correct verified file page, inspects original image presence and validates PNG bytes. The result explicitly records `model_visual_understanding_verified=false`, `answer_read_from_image=false` and `ocr_used=false`.

Passing checks establish consistent candidate generation, stable ranking for this fixture, unchanged query-runtime fingerprint, original-input preservation and successful location/PNG generation. They do not establish real-service availability, real-model accuracy, Harness image delivery, image-only answer correctness or general performance improvement. Latency describes a small warmed local fixture. Model/provider and Harness latency require separate authorized real acceptance runs.

The report retains both modes' metrics even when one mode ranks poorly. This project does not assume reranking always improves retrieval. Real-book evaluations should hold book hashes, compatible query encoder/version and candidate settings fixed, add independently checked expected records/source ranges, and label any questions requiring an actual visual model. Never copy copyrighted local example text or model weights into the public evaluation fixture.

The development run on 2026-10-02 used seven cases, top_k=3 and three repetitions. Both `none` and the loopback MOCK produced hit@3=1.0, MRR=1.0 and source locator hit=1.0. Fallback precision was 0.8 and recall 1.0 in both modes: the temporal-property question also retrieved figure-referencing context, causing one extra fallback recommendation. No ranking-quality improvement was demonstrated. Initial search latency p50/p95 was about 9.8/16.2 ms without rerank and 97.9/130.4 ms with the localhost MOCK; timing is machine/run specific and should be read from the current report. All original integrity, candidate match, runtime fingerprint, stable ranking, explicit case and location/PNG checks passed.

原文 SHA-256：1f57af0386ce69a7f54f45bf67636a64d928bcf053c14e5f91981a499d653527