arXiv:2610.32411 · Multimodal retrieval

A Broader Look into MLLM's Retrieval Heads

Jia Wang^1,2* · Yuqing Li^1,2* · Jiangnan Li^3* · Zheng Lin^1,2† · Mo Yu³

¹Institute of Information Engineering, CAS ²UCAS ³Tencent

^*Equal contribution · ^†Corresponding author

How does an MLLM locate evidence when that evidence is spoken, shown, or embedded in a long document?

01 Overview

Evidence localization is compact, shared, and trainable.

We study whether the small set of attention heads that retrieves evidence in text and images also supports audio-visual understanding—and whether these functional heads can be directly optimized.

Overview of paired audio-visual evidence and retrieval-head scoring — From paired audio-visual evidence to measured retrieval-head scores. The same answer is either spoken or displayed while the remaining input is held fixed.

Abstract

Across 18 language, vision-language, and omni-modal checkpoints, retrieval heads remain sparse as input modalities expand. A small subset is reused for matched spoken and displayed evidence, preserved during modality adaptation, and necessary for audio-visual retrieval. We then train selected-head attention with a ranking objective, improving document retrieval across three model families.

02 Findings

One mechanism, across modalities.

Sparse by design

For both speech and displayed evidence, high retrieval scores concentrate in a small set of heads. In every tested omni-modal model, fewer than one in ten heads accounts for half of positive score.

Shared across audio and vision

When the evidence content is matched, 69–83% of the top 5% heads are selected under both audio and visual evidence—far above independently selected sets.

Retained through adaptation

Omni-modal models preserve the retrieval-head structure of their language or vision-language backbones as they gain new modalities.

Sparse retrieval-head score maps — Few heads receive high scores under audio evidence.

Overlap of audio and visual retrieval heads — Matched spoken and displayed evidence recruits overlapping heads.

Retrieval-head structure retention across adaptation — Retrieval-head structure remains correlated through adaptation.

03 Method

Turn functional attention into a ranking signal.

We select retrieval heads once in the base model, use their question-to-candidate attention to score pages, then train LoRA adapters with a within-document ranking objective.

Select

Detect the highest-scoring retrieval heads under multimodal conditions and keep the set fixed.

Score

Aggregate question-to-visual-token attention from those heads for every candidate page.

Rank

Optimize the selected attention to favor relevant pages over hard negatives from the same document.

04 Results

Better document retrieval,
without sacrificing generality.

MMDocIR · Page Micro Recall@1

56.45→62.48

+6.03 points with retrieval-head ranking training

MMDocIR Micro Recall@1 (%)
Model	Page base	Page trained	Gain	Layout base	Layout trained
Qwen3-VL-8B	56.45	62.48	+6.03	47.14	48.60
Qwen2.5-VL-7B	52.41	58.10	+5.69	31.79	32.60
Gemma3-12B	44.75	46.92	+2.17	23.88	26.04

Transfer beyond pages. Page-level supervision also improves layout retrieval, even though the model never sees layout labels during training.

Stable out of domain. Scores on MMMU, MMMU-Pro, MathVista, and MMLongBench-Doc change by less than one point.

Out-of-domain performance is nearly unchanged by ranking training — Ranking training has little effect on four out-of-domain benchmarks.

05 Citation

Use this work?

Please cite the arXiv preprint and link to this project page.

@article{wang2026retrievalheads,
  title   = {A Broader Look into MLLM's Retrieval Heads},
  author  = {Wang, Jia and Li, Yuqing and Li, Jiangnan and
             Lin, Zheng and Yu, Mo},
  journal = {arXiv preprint arXiv:2610.32411},
  year    = {2026},
  url     = {https://arxiv.org/abs/2610.32411}
}

Download from arXiv