Pumpire: A Unified Benchmark for Estimating Metric Distances
Pumpire benchmarks 29 baselines for metric point-pair distances using 6,400 frames from 100 scenes, testing scale and cross-view consistency.
Key Takeaways
- Pumpire is a unified benchmark for estimating real-world metric point-pair distances in image- and video-level 3D foundation models, using 6,400 frames from 100 physically measured scenes.
- It evaluates 29 baseline configurations with and without depth priors to determine whether models can infer scale directly or rely on added depth hints.
- The benchmark assesses both metric accuracy and cross-view consistency, checking whether the same point-pair distance remains stable when estimated from different viewpoints.
- It addresses a practical need for 3D models in robotics, AR, autonomous systems, digital twins, and measurement tools, where real-world distances—not just relative shapes—must be reliable.
- The article emphasizes the benchmark’s shared task, measured dataset, and evaluation framework, while detailed baseline scores and model-specific results are not yet provided.
Meta description: Pumpire is a unified benchmark for metric point-pair distance estimation in image and video 3D foundation models, using 6,400 frames from 100 scenes.
Tags: 3D foundation models, Pumpire benchmark, point-pair distance, cross-view consistency, depth priors
Featured image: Abstract, friendly illustration style showing multiple camera viewpoints connected by measured distance markers, with no real people, logos, or text.
Pumpire is a unified benchmark for evaluating metric point-pair distance estimation in image- and video-level 3D foundation models, with or without depth priors. It is aimed at researchers and developers who need 3D models to produce distances that can be trusted in real-world units, not only relative shapes. The benchmark uses 6,400 frames drawn from 100 real-world scenes, annotated with physically measured point-pair distances, and assesses 29 baseline configurations for metric accuracy and cross-view consistency.
What the benchmark tests
A metric point-pair distance is the physical separation between two selected points in a scene, such as the distance between two objects, corners, or locations in a room. For 3D foundation models, this is harder than estimating depth or geometry alone: the model must understand scale. A reconstruction can have the right shape but be too large or too small. Pumpire’s task is to test whether image- and video-level models can estimate the actual distance between point pairs.
Depth priors are extra assumptions or signals about how far away parts of a scene are. Some models use them; some do not. Pumpire evaluates both settings. That gives readers a clearer picture of whether a model’s distance accuracy comes from its own reasoning or from an added depth hint.
Dataset and evaluation design
The benchmark’s data are manually collected: 6,400 frames from 100 real-world scenes. Each scene is annotated with point-pair distances that have been physically measured. This matters because many 3D datasets are built from synthetic scenes or camera calibration assumptions, which may not reflect real-world scale reliably. Pumpire instead uses measurements tied to the physical world.
The evaluation covers 29 baseline configurations. Rather than ranking only final 3D reconstructions, it analyzes metric accuracy and cross-view consistency. Cross-view consistency checks whether the same point-pair distance estimated from different viewpoints remains stable. A model that gives one distance from one angle and a different distance from another angle may look reasonable in a single view but fail when used for navigation, measurement, or multi-view reconstruction.
The available summary does not include detailed scores, individual model names, or performance tables for those 29 baselines. The core contribution is the benchmark structure: a shared task, a measured real-world dataset, and evaluation criteria that separate scale accuracy from view consistency.
Why this matters for 3D models
3D foundation models are used to infer scene structure from images and videos. For many applications, relative geometry is not enough. Robotics, augmented reality, autonomous systems, digital twins, and measurement tools need distances in real units. If a model estimates the distance between two objects incorrectly by a large factor, downstream systems can fail: a robot might plan a path through an object that is actually farther away, an AR app might place furniture at the wrong scale, or a reconstruction pipeline might combine views inconsistently.
Pumpire’s focus on point-pair distances provides a practical test. It is not just asking whether a model can create a 3D shape. It asks whether the model can answer a specific, measurable question: how far apart are these two real-world points?
What readers should watch next
For researchers, Pumpire offers a common evaluation target for comparing image-level and video-level 3D models. The with-or-without depth priors setting is especially useful because it separates models that rely on depth hints from models that infer scale directly. For developers, the benchmark can guide model selection when a product requires real-world measurements, not just visual reconstruction.
The next step is to see how the 29 baselines perform on metric accuracy and cross-view consistency, and where each model type struggles. The benchmark also creates room for follow-up work: new models can be trained to improve scale estimation, and future datasets can expand the range of scenes, camera conditions, and distance types while keeping the same physical-measurement standard.

Comments (0)