# Distilled modality router (`evals/distilled_router/`)

A training-utility demonstration built **from this dataset's train split**: three
small students are fine-tuned to route a query to a modality-specific index,
using generation-modality weak labels, then evaluated end-to-end against the
frozen released ranked lists. Metrics: `distilled_router_metrics.csv`.

## Provenance (read this first)

**(a) Label source — train split only, video-disjoint from the benchmark.**
All 500,000 sampled rows (490,000 train / 10,000 val) come exclusively from
`dist/retrieval_queries-train.jsonl`. There are **zero shared videos** between
that pool and the 2,195 evaluation queries (train pool: 25,995 distinct
videos; evaluation: 978; intersection: 0), so the students never saw a frame,
caption, or query from an evaluation video. 32 of the 500,000 sampled rows
happen to carry a query *string* that also occurs in the evaluation set (e.g.
generic phrasings such as "man giving presentation on stage with TED sign");
these originate from different videos and carry their own train-item label.
That is 0.0064% of the sample and is reported here for completeness.

**(b) Sampling procedure and regenerability.** Per-class reservoir sampling in
a single pass over the train-split file, 125,000 queries per class
(asr / ocr / visuals / dense), `random.Random(20260829)`, then a shuffled 2%
val holdout. Class totals available in the train split were
visuals 3,231,888 / asr 2,860,700 / ocr 2,233,228 / dense 3,262,756. The
sampler (`prep_p3_router_data.py`) was re-run against the released file and
reproduced both `train.jsonl` and `val.jsonl` **byte-identically**, so the
training data is regenerable from this container alone.

**(c) Evaluation inputs.** No new retrieval was run. Each predicted route
selects a saved ranked list from `evals/retrieval/ranked/`, using the a-priori
mapping asr→asr, ocr→ocr, visuals→dvd, dense→fuse (an alternate mapping and
an oracle-label row are included in the CSV). All ten ranked lists consumed by
the evaluation (`{te3l,MiniLM}__{asr,ocr,dvd,fuse,all}`) were verified
identical to the released blobs in this container.

## Results

| student | zero-shot acc | fine-tuned acc | e2e R@5 (te3l, primary) | R@100 | bs1 latency |
|---|---|---|---|---|---|
| Ettin-68M (encoder head) | n/a | .768 | .749 | .925 | 31 ms GPU |
| Gemma-3-270m-it | .236 | .788 | .748 | .918 | 61 ms GPU / 161 ms CPU |
| Qwen3-0.6B | .251 | **.805** | **.764** | **.931** | 166 ms GPU / 2.0 s CPU |

- **Zero-shot, the decoders are at or below the .25 chance floor and collapse
  to a single class** — base Qwen3-0.6B routes 2,195/2,195 queries to `asr`;
  base Gemma-3-270m-it routes 2,062/2,195 the same way. Their end-to-end R@5
  is ~.46. Fine-tuning on the weak labels therefore accounts for essentially
  all of the routing skill (.25 → .79–.81) and lifts end-to-end R@5 to ~.75.
  (Ettin has no zero-shot point: a randomly initialized classification head is
  chance by construction.)
- Fine-tuned accuracy is monotonic in student size while end-to-end R@5 is
  nearly flat (.748–.764), so the spread across the spectrum is a cost /
  latency axis rather than a quality axis. Gemma-3-270m-it is the CPU sweet
  spot at 161 ms/query.
- The 0.6B student's .764 R@5 exceeds the flat te3l `all` index (.759) at no
  API cost.
- Routing-error structure: OCR and dense are the confusable classes in every
  student (Ettin per-class: asr .84 / visuals .80 / dense .81 / ocr .62).

## Caveats

- **Weak labels distil a heuristic, not the router.** The supervision signal is
  each query's generation modality, so label≈route is confounded; these
  students imitate provenance, not an oracle routing policy.
- **Judge disjointness is not affected.** Using a Qwen student as the *router*
  does not violate the QA judge-disjointness protocol: the router emits an
  index choice that feeds deterministic retrieval metrics, and no
  Qwen-generated text ever reaches the QA judge.
- **Inference gotcha.** Qwen3's chat template defaults to thinking mode; with
  it enabled, 62% of generations are unparsable and accuracy reads .43.
  Evaluation must pass `enable_thinking=False` to match the training template.
- Comparisons to published API-router rows should be made with care — those
  route over different index sets.

## Files

| file | contents |
|---|---|
| `distilled_router_metrics.csv` | all rows: per student x encoder family x mapping x split, plus `zeroshot-*` and oracle-label rows |
| `prep_p3_router_data.py` | weak-label sampler (seed 20260829; byte-reproducible) |
| `make_p3_sft_dataset.py` | converts sampled rows into decoder SFT format |
| `p3_router_common.py` | shared routing prompt and route parser |
| `train_p3_router.py` | encoder-head student training |
| `run_p3_sft_decoder.sh` | decoder student fine-tuning |
| `eval_p3_router.py` | end-to-end evaluation against the released ranked lists |