A Single Block, Many Depths: Recurrent Vision Transformers with Depth-Programmed Experts
reViT loops one Transformer block, using shared FFN experts for depth, to match DeiT III and DINOv2 distillation with fewer stored parameters.
Key Takeaways
- reViT replaces a full-depth vision Transformer with one recurrent block, using shared FFN experts combined by depth to restore layer-specific behavior with fewer stored parameters.
- In supervised ImageNet-1k training, reViT-B/16 matches DeiT III accuracy with about 70% fewer stored parameters, while keeping inference compute comparable to a full-depth encoder.
- In DINOv2 distillation, an 8-expert reViT model uses only teacher output features, avoids intermediate feature distillation, and nearly matches the teacher’s linear-probe accuracy.
- Weight-space merging is identified as the strongest Mixture-of-Experts design under a one-FFN-per-depth budget, outperforming token-dispatch and output-mixture alternatives.
- The method supports elastic-depth training from one checkpoint, but fixed-depth deployment can require materializing merged weights as a dense graph, increasing storage.
reViT Uses One Recurrent Transformer Block to Match Vision Encoders
Meta description:
reViT-B/16 uses one recurrent Transformer block and shared FFN experts to match DeiT III accuracy while storing about 70% fewer parameters in supervised tests.
Tags:
reViT, Vision Transformer, MoE, ImageNet-1k, DINOv2
Featured image:
An abstract friendly illustration of one Transformer block looping through several depth positions, with small expert modules blending into a smooth path, no text, no logos, no people.
This update is for AI engineers and vision researchers who want smaller vision models without retraining full encoders: it explains how reViT, a recurrent Transformer design, can match the accuracy of a full-depth vision encoder at comparable inference FLOPs, or floating-point operations, without intermediate feature distillation. The work presents the method as a way to reuse one Transformer block across depth while restoring the depth-specific behavior that normally comes from many separate layers.
How reViT Replaces Depth With Shared Experts
A standard vision Transformer usually stacks many blocks, each with its own feed-forward network, or FFN, the component that transforms tokens after attention. reViT instead reuses a single Transformer block recurrently. To avoid making every recurrent step identical, it represents the FFN at each recurrent depth as a convex combination of a small shared expert bank, a Mixture-of-Experts idea.
In plain terms, each depth selects a weighted average of shared FFN experts, with the weights forming a continuous normalized-depth coordinate. That coordinate describes a resampleable trajectory through FFN parameter space, so different depths can be generated by sampling the same path rather than storing a separate FFN for every layer.
What the Results Show
The method is evaluated in two regimes: supervised ImageNet-1k training, a common image classification benchmark, and distillation from a DINOv2 teacher, a vision model used as a training target. In the supervised setting, the work reports that trained from scratch, reViT-B/16 attains DeiT III accuracy with about 70% fewer stored parameters. The comparison is about storage, not a claim of fewer inference FLOPs: the design is framed as matching full-depth encoder accuracy at comparable inference compute.
In the distillation setting, an 8-experts model uses only the teacher's output features, not intermediate feature distillation, yet retains nearly all of the DINOv2 teacher's linear-probe accuracy. Linear-probe accuracy measures a frozen encoder by training a simple classifier on top. The distilled model is also reported to transfer across classification, segmentation, and depth prediction.
Mixture-of-Experts Choices and Deployment Trade-offs
The work also compares controlled adaptations of a Mixture-of-Experts design under a matching one-FFN budget. It identifies weight-space merging as the strongest tested MoE family, ahead of token-dispatch and output-mixture alternatives. Token-dispatch sends tokens to selected experts, while output-mixture combines multiple expert outputs; weight-space merging instead builds a depth-specific FFN by combining expert parameters. This is the mechanism that lets a recurrent block restore depth-specific transformations without adding a full FFN at every depth.
Elastic-depth training gives one checkpoint the ability to operate at multiple tested depths by resampling the same normalized coordinate interval. For fixed-depth deployment, the recurrent block can be materialized as a conventional dense graph. That removes online routing and merging at runtime, without changing the one-FFN-per-depth compute, but it expands deployment storage because the merged weights are written out as separate parameters.
Why This Matters for Builders
For builders, the value proposition is a smaller stored model and a flexible depth knob, not a universal drop-in replacement for full-depth encoders. The result suggests that depth-specific behavior in vision Transformers can be represented by a continuous path through a shared expert bank, and that parameter merging can be a competitive Mixture-of-Experts choice when compute is fixed.
The next practical question is how far these results extend beyond the tested ImageNet-1k and DINOv2 regimes, and how much storage savings remain attractive when fixed-depth deployment requires materializing a dense graph.

Comments (0)