What Generation Knows Can Unlock Fine-Grained Understanding in Unified Multimodal Models
GAFU uses native generation-side features to improve fine-grained visual understanding across BAGEL, Janus-Pro, and Lance without image generation.
Key Takeaways
- GAFU is a post-training framework that uses native generation-side representations in unified multimodal models to improve fine-grained visual understanding without requiring explicit image generation at inference.
- The study shows that generation-side representations are more sensitive to fine-grained semantic changes and capture visual variations less reflected in understanding-side representations, with advantages closely tied to spatial organization.
- GAFU works by compressing generation features into compact G tokens, task-conditioned retrieval and injection of complementary information, and pairwise preference learning to encourage effective use of generation-side signals.
- Across BAGEL, Janus-Pro, and Lance on five benchmarks, GAFU outperforms matched-data SFT baselines in all 15 model-benchmark combinations, with relative improvements up to 20.00%.
- The key takeaway is that generation capability can provide internal representations that complement and strengthen understanding, rather than being only an output function or a competing use of model capacity.
GAFU Turns Generation Representations Into Better Multimodal Understanding
Meta description: GAFU uses native generation-side representations in unified multimodal models to improve fine-grained visual understanding across BAGEL, Janus-Pro, and Lance.
Tags: GAFU, unified multimodal models, fine-grained visual understanding, post-training, multimodal AI
Featured image: abstract friendly illustration of layered visual tokens connecting generation and understanding pathways, no real people, no logos, no text
A new paper titled "What Generation Knows: Unlocking Fine-Grained Understanding in Unified Multimodal Models" introduces GAFU, a post-training framework that uses native generation-side representations inside unified multimodal models (UMMs) to improve fine-grained visual understanding. The paper is available on arXiv at https://arxiv.org/abs/2610.23987, and the accompanying code is available at https://github.com/LRuca/GAFU.
Why the study matters
Unified multimodal models are systems that can both understand visual information and generate images with rich visual details. A central question in the paper is whether the internal representations that support image generation can also help the model understand small visual differences. That matters because many current methods that use generation to assist understanding rely on generative supervision or explicitly generated intermediate outputs. The new work instead asks what the generation pathway already knows inside the model, even when no image is produced during inference.
The paper describes this native generation-side representation as largely unexplored for understanding. By testing it, the authors aim to find a route from a model's generation capability to better fine-grained multimodal understanding.
What the team found
To answer that question, the study systematically compares generation-side and understanding-side visual representations across six UMMs. The comparison focuses on three properties: fine-grained semantic sensitivity, representational complementarity, and spatial dependence.
The findings show that generation-side representations are more sensitive to fine-grained semantic changes. They also capture visual variations that are less effectively reflected in understanding-side representations. The paper links these advantages closely to spatial organization, meaning that how the model organizes visual information across space appears to be an important part of the generation pathway's value.
This comparison gives the basis for GAFU. The framework is built on the observation that generation-side representations can carry complementary information that understanding-side representations may not use as effectively.
How GAFU works
GAFU, short for Generation-Assisted Fine-grained Understanding, is a post-training framework. It is designed to use native generation-side representations without requiring explicit image generation at inference. In other words, the model can benefit from generation-related internal features while performing understanding tasks, rather than having to generate intermediate images or outputs first.
The framework has three main parts. First, it compresses and aligns generation-side features into compact G tokens. These tokens act as a more usable form of the generation pathway's internal visual information. Second, GAFU enables task-conditioned retrieval and injection of complementary information, so the understanding pathway can draw on generation-side signals that are relevant to the current task. Third, the framework incorporates pairwise preference learning, which encourages the model to make effective use of the generation-side representations.
Together, these steps turn the generation side from a source of image outputs into a source of internal representations that can support understanding.
Benchmark results
The authors evaluate GAFU on three unified multimodal architectures: BAGEL, Janus-Pro, and Lance. The evaluation covers five benchmarks that include general, spatial, and domain-specific understanding tasks. Because three architectures are tested on five benchmarks, the paper reports results across 15 model-benchmark combinations.
In all 15 combinations, GAFU outperforms matched-data supervised fine-tuning (SFT) baselines. The reported relative improvements reach up to 20.00% on VStar-Bench, 17.94% on OmniSpatial, and 8.62% on VSR-Bench. These results support the paper's central claim that generation-side representations can improve fine-grained visual understanding when used directly during post-training.
What readers can take away
For readers following multimodal models, the paper points to a useful distinction: a model's ability to generate visual content is not only a feature of its output. It may also reveal internal representations that are valuable for understanding. GAFU suggests a path where generation and understanding are not separate capabilities competing for the same model capacity, but two pathways whose representations can complement each other.
The code is available at https://github.com/LRuca/GAFU, and the authors welcome feedback and discussions. The paper remains focused on improving fine-grained visual understanding inside UMMs, and its results across BAGEL, Janus-Pro, and Lance indicate that generation-side signals can be a practical source of complementary information.

Comments (0)