ICML 2026 · Multimodal Large Language Models

MoDA: Modulation Adapter for Fine-Grained Visual Understanding in Instructional MLLMs

1 Dartmouth College 2 KAUST † Equal senior authorship

Dartmouth College King Abdullah University of Science and Technology (KAUST)
Paper arXiv Code BibTeX
MoDA · channel modulation σ(W · F(T, V)) ∈ [0,1]
instruction What color is the dog’s ear?
modulation mask   M = σ(W · F(T, V)) ∈ [0,1]N×E  ·  each visual token gets its own channel-wise gate
visual tokens · N ≈ 576
embedding channels · E = 4096  ·  downsampled view (real map is N≈576 × E=4096)
Valigned ∈ ℝB×N×E  ·  M ∈ [0,1]N×E suppress 0 1 keep

MoDA reads the instruction and dials each visual feature channel up or down, sharpening what an MLLM sees, with under 1% extra compute.

Abstract

Multimodal Large Language Models integrate pretrained visual encoders with LLMs, yet they struggle with fine-grained visual grounding because of semantic entanglement in visual patch representations: a single patch blends multiple distinct elements, making it hard to focus on instruction-relevant detail.

We propose MoDA (Modulation Adapter), a lightweight module that enhances grounding through instruction-guided, channel-wise modulation. Unlike token-level methods such as Q-Former that perform additive feature selection, MoDA operates at the channel level via multiplicative (Hadamard) modulation on already-aligned features, giving fine-grained control over which embedding dimensions matter for each instruction, without architectural modifications or extra supervision.

Across 12 benchmarks and three architectures, MoDA delivers consistent gains: +12.0 on MMVP (LLaVA-1.5), +4.8 on ScienceQA (LLaVA-MoRE), and +4.9 ScienceQA / +4.1 RealWorldQA / +3.8 GQA on Qwen3-VL, confirming the gains generalize beyond CLIP-based encoders, with minimal overhead.

Every patch is a blur of many things

A Vision Transformer cuts an image into a grid and compresses each cell into a single embedding. Most cells end up holding several things at once, so the model reasons over a mixture instead of the actual detail. Hover the patches.

A sleeping French bulldog with a plush toy, overlaid with a 3×3 patch grid

MoDA modulates channels, not tokens

MoDA sits at the adapter-to-LLM interface. It cross-attends the instruction to the pre-aligned visual features, then emits a soft mask in [0,1] that scales each feature channel, boosting the relevant dimensions and suppressing the rest.

MoDA architecture: visual and language tokens enter a cross-attention block, then linear projection and sigmoid, multiplied element-wise to produce MoDA tokens.
The Modulation Adapter. Visual tokens (target) cross-attend the language tokens (memory); a projection + sigmoid yields a channel mask, applied by element-wise multiplication.
Channel-wise modulation

F(·) is a stack of cross-attention layers; W a learned projection; σ a sigmoid bounding the mask to [0,1]. ⊙ is the Hadamard product along the embedding dimension.

Training objective

Standard autoregressive cross-entropy. No new data, no extra supervision. MoDA drops into the two-stage LLaVA instruction-tuning pipeline.

01

Granularity

Operates on channels within each token, not whole tokens, reweighting embedding dimensions rather than selecting or dropping tokens.

02

Operation

Multiplicative Hadamard modulation, not additive residuals, enabling selective suppression of irrelevant dimensions.

03

Position

Acts post-alignment on already-projected features, complementing existing adapters instead of replacing them.

Two stages. Nothing retrained from scratch

Stage 1 pre-training: the adapter is trained for visual-language alignment with the vision encoder and LLM frozen.
Stage 1 Pre-train the visual adapter for alignment, vision encoder and LLM frozen.
Stage 2 instruction tuning: MoDA and the LLM are fine-tuned jointly while MoDA refines adapter outputs.
Stage 2 Introduce MoDA (Xavier-init) and fine-tune MoDA + LLM jointly.

Consistent gains across families

Improvements are strongest on fine-grained, vision-centric and hallucination tasks, and scale with the quality of the visual encoder.

+12.0MMVPLLaVA-1.5 · 24.0 → 36.0
+4.9ScienceQAQwen3-VL · 79.3 → 84.2
+4.1RealWorldQAQwen3-VL · 64.7 → 68.8
+3.8GQAQwen3-VL · 59.4 → 63.2

+ MoDA (ours) Δ shown vs. the matched baseline. green = gain, amber = drop. All metrics are percentages; higher is better.

Picking the right answer, not the long one

On MMVP, the baseline (LM) often produces lengthy free-form answers that miss the question format. MoDA consistently selects the correct alternative through better fine-grained grounding.

Qualitative comparison on MMVP: baseline LLaVA-MoRE vs. MoDA across keyboard key recognition, snake tongue contact, and police vehicle text.
LLaVA-MoRE SigLIP-S2 (LM) vs. + MoDA on MMVP. ✗ incorrect, ✓ correct.

BibTeX

@inproceedings{barrios2026moda,
  title     = {MoDA: Modulation Adapter for Fine-Grained Visual
               Understanding in Instructional MLLMs},
  author    = {Barrios, Wayner and Villa, Andr\'es and
               Leon Alcazar, Juan C. and Jin, SouYoung and
               Ghanem, Bernard},
  booktitle = {Proceedings of the International Conference on
               Machine Learning (ICML)},
  year      = {2026}
}