Accuracy Without Grounding

Diagnosing Visual Dependency Dissociation in Video LLM Benchmarks

Jae Joong Lee
Department of Computer Science, Purdue University
ACM International Conference on Multimedia (ACM MM), 2026

A state-of-the-art video model is asked “what is the right order of the tools?” — and answers correctly with the video replaced by a black screen.

VDG = Original accuracy − Black-screen accuracy

The Visual Dependency Gap turns that failure into a benchmark-quality diagnostic.

A four-condition diagnostic ladder

We re-ask every benchmark question under four visual conditions and watch where the answer breaks. If nothing breaks, the video was never needed.

1

Original

The full, untouched video — the benchmark as intended.

2

Single frame

One frame, repeated. Removes motion & time, keeps a snapshot.

3

Shuffled

Every frame present, order scrambled. Isolates temporal order.

4

Black screen

No visual input at all. Surviving accuracy is pure language prior.

Ordering the four gives a diagnostic ladder: black ≤ single frame ≤ shuffled ≤ original.

See it for yourself

Each clip is what the model actually saw. The chip under every video is its real Qwen2-VL-7B answer for that condition — green = correct, red = wrong.

8-second, down-sampled excerpts of Video-MME clips. Predictions are the verbatim outputs in results/videomme/.

The Visual Dependency Gap, by task and model

VDG by task type Qwen2-VL-7B · Video-MME 600-Q

Temporal Reasoning collapses to VDG 0.045 — statistically indistinguishable from ignoring the video.

Accuracy ≠ grounding selected models

ModelOrig.BlackVDG

Higher accuracy does not buy higher VDG; InternVL2-76B even goes negative.

BibTeX

@inproceedings{lee2026accuracy,
  title     = {Accuracy Without Grounding: Diagnosing Visual Dependency
               Dissociation in Video LLM Benchmarks},
  author    = {Lee, Jae Joong},
  booktitle = {Proceedings of the 34th ACM International Conference on Multimedia (MM '26)},
  year      = {2026},
  publisher = {ACM}
}