Object-Centric Alignment and Anchor Distillation for Weakly Supervised Referring Expression Comprehension

Abstract

Weakly supervised Referring Expression Comprehension (WREC) aims to localize referred objects

using only image-text pairs without box-level annotations. Existing one-stage methods predominantly

rely on anchor-level alignment, which suffers from

two fundamental limitations: (1) anchors represent

local visual patches rather than holistic objects, and

(2) they lack the capability to model inter-object

relations. To address these issues, we propose

OCAAD, an Object-Centric Alignment and Anchor Distillation framework. Our key insight is that

different self-attention heads in DINOv2 naturally

attend to distinct semantic regions, effectively capturing object-level information. Building on this,

OCAAD introduces two complementary modules:

(1) Anchor-Object Distillation Module (AODM),

which bridges the semantic gap between anchors

and objects by transferring object-level knowledge

from attention heads to anchors via overlap-aware

contrastive learning; and (2) Intra-Modal Relation Consistency (IRC), which explicitly models

inter-object relations by enforcing the relational

structure among linguistic entities to match that of

their visual counterparts. Extensive experiments on

RefCOCO, RefCOCO+, and RefCOCOg demonstrate that OCAAD achieves new state-of-the-art

performance, validating the effectiveness of objectcentric alignment for WREC. Code is available at

https://github.com/VILAN-Lab/OCAAD.