MoDA: a guide to visual grounding

Two questions about the same image can require different visual evidence. Identifying an object, comparing its size, and reading a nearby label place different demands on a model's visual representation. When a visual feature mixes several kinds of information, passing the same representation to the language model can make those distinctions harder to preserve.

MoDA investigates instruction-guided modulation as a way to address this problem. The research by Wayner Barrios and collaborators focuses on fine-grained visual grounding: connecting the question to the image details that support an answer. The module operates on features that the existing visual adapter has already aligned with the language model. It can therefore complement that adapter within the evaluated model architectures.

The model's visual adapter first produces aligned image features. MoDA then conditions on the language instruction to predict a soft mask over individual feature channels. Multiplying the aligned features by that mask changes the contribution of different embedding dimensions before autoregressive decoding begins. This is a channel-level operation on the representation, rather than simply choosing a subset of image tokens.

The MoDA paper describes the modulation network and training setup. In the documented LLaVA training protocol, visual alignment is learned first; MoDA is introduced during instruction tuning and trained jointly with the language model. This distinction matters when reproducing the work: using a trained adapter involves the corresponding model weights and preprocessing, not just adding a new prompt to an unchanged model.

The study covers LLaVA-1.5, LLaVA-MoRE, and Qwen3-VL across 12 benchmarks. These evaluations include visual question answering, tasks that require detailed image understanding, and checks for hallucinated visual content. The reported computational overhead is below 1% additional FLOPs. That is a compute measure for the evaluated configurations, not a statement that model memory or every inference latency remains unchanged.

Selected comparisons in the official results table include an MMVP increase from 24.0 to 36.0 for the LLaVA-1.5 family and a ScienceQA increase from 77.1 to 81.9 for LLaVA-MoRE with SigLIP-S2. The Qwen3-VL-2B comparison reports ScienceQA improving from 79.3 to 84.2. These are benchmark score changes for named configurations; consult the full evaluation when selecting a baseline.

Begin with the MoDA repository and choose the implementation that matches the model family you want to study. Its model table links released checkpoints and identifies the language model and vision encoder used in each configuration. Matching those components matters because feature dimensions, image processing, and evaluation settings differ across families.

The LLaVA-MoRE implementation provides its own training and evaluation instructions. The Qwen3-VL implementation has a separate workflow for integration and evaluation. Follow the selected family's instructions and reproduce a documented comparison before changing encoders, training data, or checkpoints. For an initial inspection, compare answers to questions that refer to different details in the same image.

MoDA targets the interface between visual features and the language model. Better grounding on the tested benchmarks does not establish that every generated explanation is complete or that the model will generalize to an unfamiliar image domain. Evaluation should distinguish changes in visual evidence use from changes in answer style and should retain the same protocol when comparing model variants.

The CRYSTAL reasoning guide addresses a complementary question: whether the intermediate steps in an explanation are relevant and correctly ordered. The vLLM-MLX serving guide discusses efficient local inference, a separate systems concern. Find the paper among the research publications and browse the open source projects for the other implementations.