NeurIPS 2026

HiLoc.

A Hierarchical Representation Method for Spatial Localization in Multimodal Large Language Models

Find the region. Refine the position.
Precise localization through shared Grid–Cell tokens.

Evelyn Zhang^1,2,† Fufu Yu¹ Hanjun Li¹ Aoqi Wu³ Ke Yan^1,* Shouhong Ding¹ Tianhe Ren⁴ Xinting Hu⁴ Xiaojuan Qi^4,* Li Jiang⁵ Jiaya Jia²

¹Tencent Youtu Lab²The Hong Kong University of Science and Technology³The Hong Kong Polytechnic University⁴The University of Hong Kong⁵The Chinese University of Hong Kong, Shenzhen

* Co-corresponding author
† This work was completed during Evelyn Zhang’s internship at Tencent Youtu Lab.

Read the paper Code

64shared coordinate tokens

4,096positions per axis

Grid → Cellcoarse-to-fine prediction

FIGURE 1Flat and hierarchical coordinate representations, the original design comparison, and benchmark results.Full size ↗

THE IDEA

A small vocabulary.
A finer spatial representation.

Multimodal large language models often express object locations as sequences of text or flat coordinate tokens. Increasing the resolution of a flat token representation requires a larger coordinate vocabulary.

HiLoc factorizes each coordinate into two levels. A Grid index selects a coarse region, and a Cell index refines the position within it. Both levels share the same index-token vocabulary. With 64 tokens per level, the representation encodes 4,096 discrete positions along each axis.

Built on Qwen3-VL-4B-Instruct, HiLoc combines supervised fine-tuning and GRPO post-training. HiRef-500K adds complex referring expressions and non-existence queries to support description grounding and rejection.

At equal resolution and with the same decoder, flat and hierarchical encodings have the same quantization error. HiLoc’s vocabulary advantage is representational; learned accuracy and optimization behavior are evaluated experimentally.

01 / REPRESENTATION

From a region to a point.

Coarse Grid indices locate the region.
Fine Cell indices refine the position within it.

FIGURE 2The original Grid–Cell representation and four design motivations. Cell-error locality assumes a correct Grid; optimization behavior is evaluated empirically.Full size ↗

Try the Grid–Cell encoding

Play a moving-point example, or use the sliders to explore.

Image coordinate space64 × 64 coarse grid

Use the sliders to move the point. Each coarse grid contains 64 × 64 fine cells.

Select the Grid

Predict the coarse x and y indices.

<grid> <20> <35> </grid>

Refine with the Cell

Reuse the same 64 tokens for the local position.

<cell> <0> <0> </cell>

X coordinate 1310Y coordinate 2293

q = 64 × Grid + Cellq ∈ {0, …, 4095}

Interactive illustration of the encoding. Coordinates range from 0 to 4095 on each axis; q = 64 × Grid + Cell.

From coordinates to structured predictions.

HiLoc pairs phrase and box tags with Grid–Cell coordinates, allowing a single response to ground multiple queries and instances. Supervised fine-tuning teaches the output structure; GRPO then refines localization with task-specific rewards.

FIGURE 3Original input image, structured output tokens, HiLoc vocabulary, supervised fine-tuning, and GRPO training workflow.Full size ↗

02 / SUPPORTING DATA

HiRef-500K

Complex descriptions. Multiple instances.
And knowing when there is no match.

Built from LVIS training annotations and COCO train2017 images, HiRef-500K connects generated referring expressions to annotated instances. Five expression types support complex grounding and explicit rejection.

AttributeStateRelationSet logicNon-existence

Data engine for HiRef-500K — FIGURE 4 · INTERACTIVEThe HiRef data engine redrawn as an animated workflow. The interactive schematic illustrates how expressions map to one or more targets—or to no target.

Explore HiRef-500K examples

Examples of HiRef-500K referring expressions and their annotated targets across expression types and object categories. — 03 / HIREF-500K Only non-existence expressions have zero targets. Other expression types refer to one or more instances.

03 / EVIDENCE

Measured across localization tasks.

Six benchmarks, from objects to descriptions.
Scores are percentages; higher is better.

COCO · F1 @ IoU 0.95

21.3Rex-Omni: 15.9

RefCOCOg test · F1 @ IoU 0.95

53.9Rex-Omni: 36.6

D³ · Full AP

53.1Rex-Omni: 21.5

These are complete-system comparisons. Backbones, data, and training recipes differ. The controlled studies in the analysis section examine specific factors.

OBJECT DETECTION

Common categories and the long tail.

COCO and LVIS evaluate localization across common and long-tailed object categories. The full comparisons below report recall, precision, and F1 at both loose and strict overlap thresholds.

Bold and underlined values follow the paper. On smaller screens, swipe tables horizontally to view every metric.

TABLE 1 COCO · Common object detection
Method	IoU 0.5			IoU 0.95			Mean IoU
Method	Recall	Precision	F1	Recall	Precision	F1	Recall	Precision	F1
Closed-set
Faster RCNN-R50	60.4	60.7	60.6	4.9	12.6	7.1	43.2	54.2	48.1
DETR-R50	59.6	73.9	65.9	10.6	19.0	13.6	42.9	55.3	48.3
DyHead-R50	58.1	76.6	66.1	11.9	20.6	15.0	44.8	60.1	51.3
DAB-DETR-R50	59.2	77.4	67.1	10.5	18.5	13.4	43.8	58.8	50.2
Deformable-DETR-R50	62.6	78.5	69.7	14.8	17.7	17.7	48.7	62.4	54.7
DINO-R50	62.6	76.5	68.8	17.8	25.8	21.1	50.0	62.4	55.6
DINO-Swin-L	69.8	82.5	75.6	22.2	29.7	25.4	57.0	68.2	62.1
Open-set
Grounding DINO-Swin-T	62.8	78.4	69.8	20.5	26.2	23.0	54.6	58.8	56.6
MLLM
DeepSeek-VL2-Tiny	29.4	55.0	38.2	5.9	13.0	8.1	20.0	38.6	26.3
OVIS2.5-9B	60.0	45.3	51.6	8.4	8.0	8.2	39.3	31.0	34.6
Mimo-VL-7B	56.2	56.9	56.5	6.1	7.4	6.7	35.3	36.4	35.9
OVIS2.5-2B	63.1	50.7	56.2	10.4	10.1	10.3	42.6	35.4	38.7
DeepSeek-VL2-Small	52.1	73.3	60.9	12.5	18.5	14.9	39.1	55.4	45.9
Qwen2.5-VL-7B	55.4	75.8	64.0	10.6	15.5	12.6	39.5	54.2	45.7
Qwen2.5-VL-3B	55.7	77.2	64.7	12.7	18.3	15.0	41.0	56.8	47.6
SEED1.5-VL	65.3	78.6	71.3	12.7	16.4	14.3	46.8	56.9	51.4
Qwen3-VL-4B	58.6	70.5	63.7	13.7	16.1	14.8	43.4	52.0	47.1
Qwen3-VL-8B	58.4	71.4	64.2	14.1	19.1	16.2	43.4	53.8	48.1
Rex-Omni	68.1	76.3	72.0	14.5	17.5	15.9	49.8	56.5	52.9
HiLoc	69.6	77.0	73.1	20.1	22.8	21.3	53.4	59.6	56.3

TABLE 2 LVIS · Long-tailed object detection
Method	IoU 0.5			IoU 0.95			Mean IoU
Method	Recall	Precision	F1	Recall	Precision	F1	Recall	Precision	F1
Open-set
Grounding DINO-Swin-T	39.1	61.2	47.7	16.9	34.8	22.7	31.9	49.6	38.8
MLLM
DeepSeek-VL2-Tiny	22.4	55.7	32.0	7.2	25.7	11.2	14.2	41.9	21.2
MiMo-VL-7B	43.3	57.8	49.5	6.5	13.7	8.8	25.4	41.1	31.4
OVIS2.5-2B	53.2	55.6	54.4	13.3	19.4	15.8	34.0	41.6	37.4
OVIS2.5-9B	52.2	54.1	53.1	11.2	19.9	14.4	32.2	40.1	35.8
Qwen2.5-VL-7B	47.1	74.3	57.7	12.7	29.0	17.6	32.0	54.3	40.2
Qwen2.5-VL-3B	44.8	73.9	55.8	14.3	29.8	19.3	31.9	54.6	40.3
DeepSeek-VL2-Small	46.1	72.2	56.2	15.8	31.2	21.0	33.7	55.0	41.8
SEED1.5-VL	54.7	82.0	65.6	15.0	28.1	19.5	38.5	59.3	46.7
Qwen3-VL-4B	49.3	81.4	61.4	15.7	32.8	21.2	35.7	61.7	45.3
Qwen3-VL-8B	46.5	80.3	58.9	14.7	32.3	20.2	33.4	60.2	43.0
Rex-Omni	54.7	78.1	64.3	16.5	27.6	20.7	39.6	57.5	46.9
HiLoc	58.3	80.5	67.6	19.3	34.8	24.8	42.9	61.1	50.4

What changes as localization gets stricter?

The IoU curves and object-size breakdown complement the aggregate scores, showing how the relative gains vary with overlap threshold and target size.

Performance scaling with IoU and object size — FIGURE 6Relative F1 gains on COCO and LVIS as the IoU threshold increases, and LVIS results by object size. These comparisons evaluate complete systems.Full size ↗

SMALL OBJECTS & REFERRING EXPRESSIONS

From aerial scenes to language-guided grounding.

VisDrone tests small-object localization; RefCOCOg tests whether a natural-language expression can be matched to the correct region. Qualitative examples connect these measurements to predicted boxes.

TABLE 3 VisDrone and RefCOCOg
Method	VisDrone			RefCOCOg val			RefCOCOg test
Method	F1 @ 0.5	F1 @ 0.95	Mean F1	F1 @ 0.5	F1 @ 0.95	Mean F1	F1 @ 0.5	F1 @ 0.95	Mean F1
Open-set
Grounding DINO-Swin-T	28.0	16.5	25.2	52.9	20.9	45.9	53.8	22.9	46.8
MLLM
DeepSeek-VL2-Tiny	4.3	0.1	1.8	67.4	16.1	50.5	69.3	16.9	52.1
OVIS2.5-2B	21.0	0.1	9.2	87.4	29.3	73.4	87.6	30.5	73.8
OVIS2.5-9B	15.8	0.1	6.5	88.8	23.5	72.1	88.7	24.2	72.6
Qwen2.5-VL-3B	31.5	1.9	20.4	83.5	30.7	69.2	83.8	31.8	70.1
MiMo-VL-7B	27.7	0.3	14.3	84.9	14.4	65.3	84.6	14.9	65.5
Qwen2.5-VL-7B	34.5	1.6	21.7	86.2	27.2	70.0	85.7	28.4	70.4
DeepSeek-VL2-Small	35.8	1.7	23.3	92.4	45.6	81.4	91.8	47.0	81.6
SEED1.5-VL	55.9	0.6	27.4	84.7	30.9	71.9	85.2	32.1	73.2
Qwen3VL-4B	26.8	1.1	17.3	87.5	35.3	74.2	88.3	36.7	75.6
Qwen3VL-8B	25.5	1.1	16.3	88.3	36.7	75.6	87.5	35.5	74.8
Rex-Omni	61.6	1.5	35.8	86.6	35.3	73.6	86.8	36.6	74.3
HiLoc	59.9	3.9	37.9	89.0	55.6	81.5	88.8	53.9	80.6

Qualitative comparison with baseline methods — FIGURE 5Examples from LVIS, VisDrone, RefCOCOg, and D³, reproduced from the paper.Full size ↗

DESCRIPTION-CONDITIONED DETECTION

Find the match—or recognize its absence.

OmniLabel and D³ include descriptions without a corresponding object. The evaluation therefore tests both grounding and rejection. AP-d denotes average precision for descriptions; dashes indicate unreported values.

TABLE 4 OmniLabel and D³ · Description-conditioned detection
Method	OmniLabel · AP-d					D³ · AP
Method	All	Positive	Small	Medium	Large	Full	Presence	Absence
Specialists
RegionCLIP	2.6	3.2	3.6	2.7	2.3	—	—	—
OFA-DOD	—	—	—	—	—	21.6	23.7	15.4
Grounding DINO-Swin-T	—	—	—	—	—	20.7	20.1	22.5
OWLv2	21.9	26.9	32.9	20.5	14.4	22.8	22.1	24.7
GLIP-T	16.4	25.8	29.4	14.8	8.2	19.1	18.3	21.5
FIBER-B	22.3	34.8	38.6	19.5	12.4	22.7	21.5	26.0
Generalists
InternVL2-76B	15.3	19.3	21.9	15.2	11.3	25.3	25.7	23.5
InternVL2-8B	4.5	5.2	6.2	4.2	3.9	9.8	11.0	6.2
InternVL3-8B	4.0	15.1	4.8	4.0	4.2	14.3	14.3	14.3
Griffon(13B)	7.2	12.7	12.5	6.3	4.5	12.3	12.4	12.2
Groma(7B)	15.6	27.2	21.4	15.6	11.3	16.0	15.9	16.3
ROD-MLLM(7B)	25.3	30.9	31.8	24.5	21.0	29.7	30.0	28.7
Qwen3VL-4B	12.6	27.6	21.1	11.9	10.0	28.8	29.9	25.4
Qwen3VL-8B	9.24	19.59	10.90	9.12	10.71	30.6	31.5	27.9
Rex-Omni (3B)	12.9	33.6	20.0	12.2	11.0	21.5	22.1	19.8
HiLoc (4B)	23.0	41.4	38.5	21.4	16.8	53.1	55.3	46.6

04 / ANALYSIS

Understanding the gains.

Representation, resolution, and post-training each play a role.

The ablations examine GRPO, coordinate formats, and training dynamics. Two controlled comparisons then separate the coordinate vocabulary from resolution and compare representations under shared data and reward settings.

GRPO, coordinate representations, and training dynamics — FIGURE 7Original three-panel ablation figure. The coordinate-format comparison uses different resolutions; the gradient curves are empirical optimization diagnostics.Full size ↗

Resolution and vocabulary, disentangled.

TABLE 5 COCO validation · shared Qwen3-VL-4B backbone and training settings
Representation	Tokens L	Resolution K	F1 @ 0.5	F1 @ 0.95	Mean F1
Flat-64	64	64	54.5	8.6	37.7
Flat-1024	1,024	1,024	56.2	13.4	41.8
Grid32 × Cell32	32	1,024	66.5	14.6	48.6
Grid64 × Cell64	64	4,096	72.3	18.7	54.8

+6.8 mean F1 at matched resolution. Grid32 × Cell32 improves over Flat-1024 at K = 1,024 with 32 coordinate tokens instead of 1,024. These detection-only experiments use a different training mixture from the full model.

Shared data. Shared reward. Different representations.

TABLE 6 D³ Full AP · matched backbone, HiRef data, output grammar, and stage-specific training settings
Representation	Resolution K	SFT	SFT + GRPO
Flat-1000	1,000	25.8	42.6
HiLoc · Grid64 × Cell64	4,096	34.2	53.1

+10.5 AP after GRPO. Data and reward are controlled, but resolution differs. GRPO improves both representations, so the gains cannot be attributed to representation alone.

REFERENCE

Cite HiLoc.

If you find this work useful, please cite our paper.

View the paper on OpenReview ↗

@inproceedings{zhang2026hiloc,
  title = {HiLoc: A Hierarchical Representation Method for
           Spatial Localization in Multimodal Large Language Models},
  author = {Zhang, Evelyn and Yu, Fufu and Li, Hanjun and Wu, Aoqi
            and Yan, Ke and Ding, Shouhong and Ren, Tianhe
            and Hu, Xinting and Qi, Xiaojuan and Jiang, Li and Jia, Jiaya},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2026},
  url = {https://openreview.net/forum?id=V5T2vPmvxO}
}