CRYSTAL: multimodal reasoning evaluation

A vision-language model can return the right answer while overlooking the visual evidence needed to support it. Final-answer accuracy alone cannot show whether an explanation identifies the relevant objects, follows the required relationships, or skips an essential inference. This matters when comparing models that sound equally confident but arrive at their answers through very different reasoning traces.

In CRYSTAL, Wayner Barrios and SouYoung Jin study that gap through explicit reference steps. The benchmark contains 6,372 instances drawn from visual, scientific, and mathematical reasoning tasks. Each example pairs a question with an ordered reference explanation. This makes the evaluation useful for inspecting a failure: a model may misunderstand the image, omit a needed step, or describe appropriate steps in an inconsistent order.

CRYSTAL compares a model's predicted reasoning steps with the reference sequence using semantic matching. Match F1 combines precision and recall: precision asks whether the generated steps are relevant, while recall asks how much of the reference reasoning the model covers. Reading these measures together helps expose explanations that mention a few correct observations but leave out most of the necessary reasoning.

Ordered Match F1 also considers the sequence of matched steps. A model that identifies useful facts but arranges them incoherently can therefore receive a lower ordered score. The metric definitions explain the matching procedure and its settings. Keep those settings fixed when comparing runs, and examine individual traces alongside aggregate scores.

The study evaluates 20 multimodal models. In the reported comparison, 19 show the pattern of high reasoning precision with low recall. Among competitive models, none preserves more than 60% of the matched steps in the correct order. That ordering result is distinct from final-answer accuracy: it measures the organization of the reasoning that the evaluator can match.

The work also introduces Causal Process Reward and a curriculum that increases reasoning difficulty during training. For Qwen2.5-VL-3B, the reported curriculum experiment raises Match F1 from 0.480 to 0.633, approximately a 32% relative improvement, while answer accuracy rises from 39.85% to 47.52%. The CRYSTAL paper provides the experimental setup and the comparisons with answer-only and additive rewards.

Start with the CRYSTAL dataset and inspect a few examples before running a full evaluation. The project repository provides a model-agnostic inference script that connects to an OpenAI-compatible endpoint. Its output separates reasoning steps from the final answer so the metrics can evaluate both parts.

The metrics quickstart describes the expected input format and how to score predictions with the Python package or command line. For a useful comparison, record the model, prompt, decoding settings, and metric configuration. Begin by scoring a small sample and reading the explanations where precision, recall, and answer correctness disagree.

CRYSTAL measures agreement with reference reasoning traces. It does not reveal a model's hidden computation or prove that a written explanation caused an answer. Semantic matching and the choice of reference steps also affect the scores. Treat the benchmark as a diagnostic tool, with examples to inspect and a reproducible protocol to compare, rather than a single certificate of reasoning ability.

For the broader research context, see the publication list. The MoDA visual grounding guide examines how instructions can influence visual features, while the vLLM-MLX inference guide covers the systems needed to serve models locally. The open source collection brings these implementations together.