# VLM fine-tuning pilot (`evals/finetune_vlm/`)

Artifacts for the supplementary section on training utility: an 8B
vision-language model was LoRA-fine-tuned on AVC captions, then evaluated in
two directions — vision-grounded QA (eval B) and caption-based retrieval
(eval A). The two results have opposite signs, and reconciling them is the
point of this release.

Everything reported below is recomputable from the files in this directory.

## Training configuration (complete)

| setting | value |
|---|---|
| base model | `OpenGVLab/InternVL3-8B-hf` |
| adapter | LoRA, rank 16, alpha 32, `target_modules = all-linear` |
| vision tower | frozen |
| epochs | 1 |
| batch size | 8, gradient accumulation 2 (effective 16) |
| learning rate | 1e-4, warmup ratio 0.03 |
| max sequence length | 4096 |
| precision | bfloat16, gradient checkpointing enabled |
| attention | sdpa |
| framework | ms-swift 4.5.2 |
| training samples | 50,000-sample pilot subset of the AVC **train** split |
| validation samples | 2,000 held out |
| target layer | `visual_caption` (the dense-visual "dvd" layer) |
| input per sample | ten 1 fps frames at 448 px |
| prompt | identical to the released re-captioning-probe prompt, so tuned and zero-shot conditions are directly comparable |
| hardware | 1x A100 80GB, 19 h 39 m wall clock (22.6 s/it, 0.71 samples/s) |
| final metrics | train loss 0.495, val loss 0.492, val token accuracy 83.2% |

The validation curve is flat from step 2800 to 3125 (0.4924 → 0.4921): the run
is converged at one epoch, not undertrained.

`fused_caption` was deliberately **not** used as the target. It embeds speech
content that is not visible in the pixels, so training on it would teach
audio hallucination; the dvd layer is the pixels-grounded target.

## Eval B — vision-grounded QA (frames-only): **+7.1 points**

Frames-only QA over the benchmark's 2,042 questions: the model sees the gold
window's ten 1 fps frames and the question, with no captions. Judged by the
same local open-weights judge, with the same prompt, sampling parameters, and
3-vote majority as the published frames-only runs (configuration pinned in the
container's judging documentation); the answerer is InternVL, so judge-family
disjointness holds.

Paired on the 2,035 items both models answered (7 skipped for missing frames):

| slice | n | zero-shot | tuned | delta |
|---|---|---|---|---|
| **overall** | 2035 | .3627 | **.4339** | **+.0713** |
| visuals | 530 | .4208 | .5396 | +.1189 |
| dense | 526 | .3289 | .4278 | +.0989 |
| ocr | 456 | .5987 | .6535 | +.0548 |
| asr | 523 | .1319 | .1415 | +.0096 |
| hard | 657 | .2648 | .3349 | +.0700 |
| normal | 1378 | .4093 | .4811 | +.0718 |

McNemar exact two-sided **p = 1.05e-15** (239 items correct only after tuning,
94 only before). Paired bootstrap over items, 10,000 resamples: mean delta
**+.0713, 95% CI [+.0540, +.0885]**. Both are recomputable from
`qa_judgments/`.

Context against the published frames-only answerers: GPT-4.1 .4691; **tuned
InternVL3-8B .4339**; Nova-Pro .4094; zero-shot InternVL3-8B .3624. One epoch
of LoRA on a 50K pilot moves an open 8B model above Nova-Pro and to within
3.5 points of GPT-4.1.

## Eval A — caption-based retrieval: **−14.6 points**

Tuned re-captions were generated for the test corpus and swapped into the
released MiniLM dvd index (88,201 of 88,566 rows; 365 windows lacked usable
frames and kept their original captions in every condition). Overall R@5 falls
**.3631 → .2173**.

Because the earlier zero-shot probe swapped only the 4,117 archived windows,
the tuned model was additionally re-scored at exactly that coverage
(4,100 shared rows) for a like-for-like comparison:

| slice | original | zero-shot recaption | tuned recaption |
|---|---|---|---|
| overall | .3631 | .2784 | **.2014** |
| visuals | .4833 | .2056 | **.2197** |
| ocr | .4369 | .5709 | **.3165** |
| asr | .1164 | .0945 | **.0545** |
| dense | .4153 | .2638 | **.2210** |

Tuning helps exactly one slice — `visuals`, the layer it was trained on — and
hurts the rest, most severely `ocr`.

### Mechanism: target style learned, target substance undershot

Three-way comparison on the 161 windows where released, zero-shot, and tuned
captions all exist (`paired_caption_stats_161.csv`):

| caption version | words | mentions >1 shot | proper nouns | quoted text |
|---|---|---|---|---|
| released dvd (training target) | 92.1 | 26.7% | 2.08 | 37.3% |
| zero-shot | 110.7 | 32.3% | 3.08 | 61.5% |
| **tuned** | **93.0** | **12.4%** | **1.40** | **26.7%** |

The tuned model matched the target's length almost exactly (93.0 vs 92.1
words, down from 110.7) — it learned the surface convention — but fell *below
the layer it was trained on* on every specificity axis: half the multi-shot
coverage, a third fewer proper nouns, less quoted on-screen text. It regressed
toward describing only the dominant visual scene.

Retrieval depends on that specificity: benchmark queries are short and
proper-noun-heavy. A caption can be fluent, accurate, and correctly styled
while being unretrievable.

`ocr_hit_to_miss_55.csv` lists the 55 OCR-born queries that fell from rank <=5
to rank >50, with both ranks. `worked_examples.json` gives five of them with
all three caption versions side by side — including cases where an entire
text-bearing shot disappears from the description, not merely the words on
screen.

### How the text-mention rates were counted

Over a fixed 4,000-caption sample per condition (and, for the paired row, the
161 shared windows): a caption "mentions on-screen text" if it matches the
case-insensitive regular expression
`"[^"]{2,}"|\btext\b|\bwords?\b|\breads?\b|\bcaption\b|\blogo\b|\bsign\b|\btitle\b`,
and "contains a quoted string" if it matches `"[^"]{2,}"`. Rates: 84.4% →
28.3% (mentions) and 59.1% → 22.9% (quoted) from zero-shot to tuned; on the
161 paired windows, 86.3% → 34.2% and 61.5% → 26.7%. "Mentions >1 shot"
matches `\b(then|transitions?|cuts? to|next|followed by|later|afterwards?|subsequently|scene changes)\b`.
Proper nouns are counted as a proxy: capitalised tokens longer than two
characters that are not sentence-initial.

### Alternative explanations ruled out

- **Degenerate generation**: no. Captions are fluent and on-topic; samples are
  in `tuned_recaptions_probe4117.jsonl`.
- **Embedding collapse**: no. Mean pairwise cosine among tuned caption
  embeddings is .4082, *lower* (more spread) than the released dvd captions
  (.4133) and the zero-shot re-captions (.4388).
- **Truncation at the decoding cap**: no. 1.75% of captions lack terminal
  punctuation; maximum length 299 words against a cap of roughly 240.

## Reconciling the two results

Eval A shows the OCR slice of *retrieval* collapsing; eval B shows the OCR
slice of *QA* improving. These are consistent, and together they identify what
changed: fine-tuning altered the model's **captioning convention**, not its
**visual capability**. Asked directly about on-screen text, the tuned model
still reads it; it simply no longer volunteers text transcription when asked
to describe a clip, because the layer it imitated does not.

The pair is also an argument for the multi-layer annotation design: benchmark
queries are drawn from all four generation sources, so a generator optimised
to imitate a single layer loses what the other layers carry.

## Scope

One base model, one target layer, one epoch, a 50,000-sample pilot, a single
judge, and a single query-side encoder (MiniLM) for the retrieval condition.
These results are not a general claim about vision-language fine-tuning.

## Files

| file | contents |
|---|---|
| `finetune_vlm_e10.csv` | eval B accuracies: full-set rows and paired 2,035-item rows for both conditions |
| `finetune_vlm_retrieval.csv` | eval A metrics: original, tuned full-corpus swap, tuned matched-coverage swap |
| `qa_judgments/tuned_internvl3__frames.jsonl` | per-item judge verdicts and votes, tuned (2,035) |
| `qa_judgments/zeroshot_internvl3__frames.jsonl` | per-item judge verdicts and votes, zero-shot (2,042) |
| `tuned_recaptions_probe4117.jsonl` | tuned re-captions for the archived probe windows |
| `tuned_recaptions_all.jsonl.gz` | tuned re-captions for all 88,201 re-captioned windows |
| `paired_caption_stats_161.csv` | three-way caption statistics on the shared windows |
| `ocr_hit_to_miss_55.csv` | OCR-born queries that fell from rank <=5 to rank >50 |
| `worked_examples.json` | five queries with released, zero-shot, and tuned captions side by side |