1. Select
2D attention identifies dominant tokens with strong semantic saliency.
Project Page
Merge3D: Efficient 3D Multimodal LLMs via Joint 2D-3D Token Merging
Geometry-aware token compression for dual-encoder 3D video MLLMs.
1National University of Singapore 2The Hong Kong Polytechnic University
Overview
Abstract
Multimodal Large Language Models (MLLMs) with 3D geometry priors show strong capability in 3D scene understanding, but long multi-view visual token sequences make them expensive to run. Merge3D introduces a geometry-aware token merging framework that combines 2D semantic saliency and 3D geometric consistency.
Our Semantic–Geometric Token Merger (SemGeo Merger) uses 2D attention to select dominant tokens, then aggregates contextual tokens with a hybrid 2D+3D similarity. This preserves spatially critical structure and inter-frame correspondences even under aggressive compression.
Merge3D reduces visual tokens by up to 70% and delivers up to ~3.1× faster inference while retaining strong performance on 3D grounding, captioning, and spatial reasoning benchmarks including Scan2Cap, CV-Bench, and BLINK.
Method
Semantic–Geometric Token Merger
Merge3D is built on top of a dual-encoder 3D video MLLM. A 2D visual encoder captures semantic detail, while a 3D geometry encoder captures spatial structure and cross-frame correspondence.
Instead of compressing tokens with only semantic similarity, Merge3D first keeps semantically important dominant tokens using 2D attention, then merges remaining contextual tokens according to a hybrid similarity that respects both appearance and geometry.
- 2D attention selects dominant, query-relevant tokens.
- 3D features preserve geometry-aware neighborhoods.
- Hybrid merging balances grounding fidelity and reasoning performance.
- The base visual encoders and decoder stay frozen; Merge3D is inserted as a token compression module.
2. Group
3D-aware similarity encourages merging within spatially coherent regions.
3. Preserve
Geometry and semantics are fused to keep spatial reasoning and grounding robust under compression.
Qualitative
Qualitative comparison
Citation
BibTeX
@misc{pan2026merge3d,
title = {Merge3D: Efficient 3D Multimodal LLMs via Joint 2D-3D Token Merging},
author = {Tianbo Pan and Xingyi Yang and Xinchao Wang},
year = {2026},
note = {Project page / manuscript}
}