RefineAny3D: Depth Refinement as Semantic Alignment for Monocular 3D Detection

RefineAny3D overview: depth is the bottleneck, reason-act-refine loop, semantic alignment, and results — **(a)** Object depth is the bottleneck: ground-truth depth adds +29.92 AP_3D on Omni3D, while a depth foundation model (DFM) *lowers* it by 3.68. **(b)** A VLM emits action tokens that iteratively adjust depth. **(c)** Depth error has a visual signature: a too-close box projects too large. **(d)** Plug-and-play gains on closed-set detection, open-vocabulary detection, and auto-labeling.

TL;DR

Project a candidate 3D box onto the image: if it is too far, the wireframe looks too small; if it is too close, it looks too large. RefineAny3D turns depth refinement into this visual judgment. A vision-language model reasons about the wireframe and emits action tokens (direction + magnitude) instead of a number, and repeats until the box fits.

+3.49

AP_3D Mod. on KITTI val
over MonoCoP

+4.35

AP_3D on Omni3D
over DetAny3D

+2.37

AP_3D Mod. from refined
LabelAny3D labels

1.62

VLM calls per object
on average

Key Insight

Once a 3D box is projected onto the image, camera intrinsics and metric scale are absorbed into the projection. What is left is a purely 2D question: does the wireframe tightly enclose the object? Drag the box depth, then let the refinement loop fix it.

Object (true depth) Candidate box

Candidate depth d16.0 m

True depth: 10.0 m · object size s_obj = 2.40 m

Visual evidence

—

Action tokens

—

An illustrative simulation of the refinement loop, not the trained model: the action here is computed from the known true depth, and the step sizes are placeholder fractions of s_obj. In RefineAny3D, the VLM makes this judgment from the rendered image alone.

Abstract

Monocular 3D object detection spans two regimes: closed-set detectors operating within a fixed category vocabulary, and open-vocabulary detectors that localize arbitrary categories by leveraging depth foundation models for 3D geometry. We find that current depth foundation models, despite their strong zero-shot generalization, lack the object-level precision 3D detection demands: substituting a state-of-the-art depth foundation model for a strong detector's predicted depth degrades accuracy. Rather than pushing detectors or depth models to be more accurate end-to-end, we treat object-level depth refinement as a stand-alone task and present RefineAny3D, a vision-language model that corrects depth without ever predicting a numerical value. Our key insight is that depth error has a direct visual signature in image space: when projected onto the image, a correctly placed box tightly encloses the object, while a too-far box projects too small and a too-close box projects too large. Depth refinement thus reduces to a visual alignment problem rather than a metric regression problem, which we instantiate by extending the VLM's vocabulary with action tokens that replace numerical depth output with categorical decisions, and by supervising the model on a large-scale chain-of-thought dataset that grounds each decision in explicit visual evidence. Applied as a single post-hoc step, RefineAny3D delivers consistent gains across closed-set detectors, open-vocabulary detectors, and 3D auto-labeling tools, and generalizes to novel categories, scenes, and cameras without retraining.

Method

Overview of the RefineAny3D framework — **Overview of RefineAny3D.** The candidate box is rendered as a wireframe on the image. The VLM reasons in four steps (identify, recall, evidence, decision) and emits a direction token and a magnitude token. The depth is updated and the loop repeats until `<depth_ok>` or a step cap.

Action tokens

6 new tokens

VLMs are unreliable at numerical regression, so RefineAny3D never outputs a number. Its vocabulary is extended with a direction token and a magnitude token, each with its own learnable embedding:

<depth_closer><depth_ok><depth_farther> <step_small><step_medium><step_large>

Steps are relative to the object's own size: 0.5 m is negligible for a distant truck but huge for a nearby cup. Precision lost to discretization is recovered by iterating.

3.0M CoT samples

from Omni3D

Training data is curated from Omni3D (six datasets, indoor and outdoor, 50 categories). After geometric and VLM-based filtering, 335K annotations remain. Their depth is perturbed along the camera ray with object-relative, action-balanced magnitudes, giving about 3.0M samples.

Each sample carries a chain-of-thought rationale before the action, so every decision is grounded in visual evidence rather than a shortcut. Training has two stages: warm up the six token embeddings with the VLM frozen, then fine-tune the LLM with the vision encoder frozen.

Examples of chain-of-thought supervision — **Chain-of-thought supervision.** Each training sample pairs a rendered wireframe with a reasoning trace (identify → recall → evidence) that ends in the action tokens.

Results

Applied post hoc, with every other box attribute held fixed, RefineAny3D improves closed-set detectors, open-vocabulary detectors, and 3D auto-labeling.

KITTI val, Car AP_3D at IoU_3D ≥ 0.7. RefineAny3D refines the depth of MonoCoP's predictions.

Method	Easy	Mod.	Hard
MonoDETR	28.84	20.61	16.38
MonoCD	26.45	19.37	16.38
FD3D	28.22	20.23	17.04
MonoMAE	30.29	20.90	17.61
MonoTAKD	34.36	22.61	19.88
MonoDGP	30.76	22.34	19.02
MonoCoP	32.06	23.98	20.64
+ RefineAny3D (ours)	35.62	27.47	21.31
Δ vs. MonoCoP	+3.56	+3.49	+0.67

Omni3D test, AP_3D per sub-dataset. RefineAny3D refines DetAny3D conditioned on ground-truth 2D boxes, its strongest variant; only depth changes.

Method	KITTI	nuScenes	SUN RGB-D	ARKit	Objectron	Hypersim	Overall
Cube R-CNN	32.50	30.06	15.33	41.73	50.84	7.48	23.26
OVMono3D	25.45	24.33	15.20	41.60	58.87	7.75	22.98
DetAny3D	31.61	30.97	18.96	46.13	54.42	7.17	24.92
DetAny3D w/ GT 2D box	38.68	37.55	46.14	50.62	56.82	15.98	34.38
+ RefineAny3D (ours)	43.47	44.13	48.58	54.20	59.29	17.80	38.73
Δ vs. DetAny3D w/ GT 2D	+4.79	+6.58	+2.44	+3.58	+2.47	+1.82	+4.35

KITTI val AP_3D of a detector (MonoCoP) trained on each set of labels. Refining LabelAny3D's pseudo-labels narrows the gap to ground truth.

Training labels	Easy	Mod.	Hard
Ground truth (upper bound)	32.06	23.98	20.64
LabelAny3D	13.57	11.78	10.27
+ RefineAny3D (ours)	15.76	14.15	10.99
Δ vs. LabelAny3D	+2.19	+2.37	+0.72

Refine3D, our standalone benchmark on Omni3D. DirAcc: direction correct; FullAcc: direction and magnitude correct; DepthErr: residual depth gap (lower is better).

Method	Standard			Novel category			Novel camera
Method	DirAcc	FullAcc	DepthErr	DirAcc	FullAcc	DepthErr	DirAcc	FullAcc	DepthErr
Qwen3-VL-8B	28.7	11.4	0.32	27.5	10.9	0.37	28.3	11.1	0.33
Qwen3-VL-8B w/ our data	75.6	55.4	0.21	56.3	41.2	0.27	68.7	49.8	0.23
RefineAny3D (ours)	89.2	76.7	0.16	74.4	58.5	0.19	85.6	73.2	0.17

Refining MonoCoP's depth on KITTI val (AP_3D, IoU_3D ≥ 0.7). Geometric fitting even uses the ground-truth 2D box.

Method	Easy	Mod.	Hard
MonoCoP	32.06	23.98	20.64
Geometric fitting	29.89	21.55	18.36
Geometric fitting (GT dims + yaw)	30.56	21.79	18.43
DINOv2 action classifier	24.71	17.62	12.78
RefineAny3D (ours)	35.62	27.47	21.31

Both simpler judges make MonoCoP worse. The gain comes from the VLM's visual judgment, not the discrete action space alone.

AP3D versus number of refinement iterations on KITTI Moderate — **Fast convergence.** On KITTI Moderate, one step recovers about 85% of the gain (+2.95); a second adds +0.54, after which accuracy plateaus. We cap refinement at two steps.

What matters in the design?

Refine3D standard split, FullAcc (direction and magnitude both correct). Each row changes one choice.

Plain-text actions55.4

One joint token per action72.5

No chain-of-thought70.2

Unfrozen vision encoder69.6

RefineAny3D (dir + mag)76.7

Factored direction and magnitude tokens each see about a third of the training samples, versus about a seventh for joint tokens. Chain-of-thought and a frozen vision encoder both matter too.

Qualitative Results

RefineAny3D refining DetAny3D predictions with single and multiple steps

RefineAny3D refines DetAny3D's initial prediction (red) to the final box (green), with the emitted action tokens. Most cases need one step; some take two (yellow is the intermediate box).

Additional qualitative refinement examples on Omni3D

More examples on Omni3D, across indoor and outdoor scenes and many categories. Click to enlarge.

BibTeX

@inproceedings{zhang2026refineany3d,
  title     = {RefineAny3D: Depth Refinement as Semantic Alignment
               for Monocular 3D Detection},
  author    = {Zhang, Zhihao and Zhang, Gengwei and Chen, Tianlong and
               Liu, Xiaoming},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
  year      = {2026}
}