Object-Centric Alignment and Anchor Distillation for Weakly Supervised Referring Expression Comprehension

1 Introduction

Localizing objects through natural language without box annotations remains a fundamental yet challenging problem. Weakly supervised Referring Expression Comprehension (WREC) tackles this challenge by learning to ground referred objects using only image-text pairs as supervision [Deng et al., 2021]. Recent one-stage methods [Jin et al., 2023; Chen et al., 2025] formulate this task as anchor-text contrastive learning, where anchor features from a pre-trained detector are directly matched with text features. Despite their efficiency, these methods share two fundamental limitations that hinder their performance on complex expressions.

∗Contact Author.

First, existing methods treat anchors as isolated visual patches rather than complete semantic entities. As illustrated in Figure 1(a), anchor features capture only local appearances, lacking holistic object-level information. This makes it difficult to distinguish objects with similar local patterns but different overall semantics, such as two persons wearing similar clothes.

Second, without explicit relation modeling, current methods fail to reason about structural dependencies among multiple objects. For expressions like "the player holding a bat", understanding the relation between "player" and "bat" is crucial for accurate grounding. However, anchor-level alignment provides no mechanism for such relational reasoning.

To overcome these limitations, we propose OCAAD, an Object-Centric Alignment and Anchor Distillation framework. Our approach is motivated by a key observation: different self-attention heads in DINOv2 [Oquab et al., 2023] naturally attend to distinct semantic regions within an image [Barsellotti et al., 2025]. These attention maps exhibit remarkable object-level consistency without any supervision, making them ideal for bridging the gap between local anchors and holistic objects.

Building on this observation, OCAAD introduces two complementary modules that address the aforementioned limitations respectively:

• Anchor-Object Distillation Module (AODM) bridges the semantic gap between anchors and objects. Given a text-guided selected anchor, AODM identifies attention heads with maximum spatial overlap and performs contrastive distillation to transfer object-level semantics into the anchor representation. This enables anchors to capture holistic object information beyond local patches.
• Intra-Modal Relation Consistency (IRC) explicitly models inter-object relations. Each linguistic entity is paired with its most similar attention head, and the pairwise relation matrices in both modalities are constrained to be consistent. This enables the model to learn structural dependencies crucial for complex expressions.

We evaluate OCAAD on three standard benchmarks: RefCOCO, RefCOCO+, and RefCOCOg. Extensive experiments demonstrate that our method outperforms previous state-ofthe-art, with improvements of up to 3.29% on RefCOCO testA. Notably, OCAAD shows particularly strong gains on RefCOCO+, which relies heavily on relational descriptions, confirming the effectiveness of our relation modeling.

Our contributions are summarized as follows:

• We identify two fundamental bottlenecks inWREC: the lack of object-level semantics in anchors and the absence of relational reasoning, and propose object-centric alignment as a unified solution.
• We design two complementary modules, AODM and IRC, to inject object-level semantics and relational awareness into anchor representations respectively, enabling precise and structured grounding.
• OCAAD achieves state-of-the-art on three REC benchmarks, demonstrating the effectiveness of object-centric alignment for weakly supervised visual grounding.

2 RelatedWork

2.1 Referring Expression Comprehension

Referring Expression Comprehension localizes image regions described by natural language [He et al., 2023]. Twostage approaches [Hu et al., 2016; Liu et al., 2019a; Yu et al., 2018; Rohrbach et al., 2016; Zhao et al., 2024] generate region proposals (e.g., via Faster-RCNN [Ren et al., 2015]) and rank them by text similarity, while one-stage methods [Liao et al., 2020; Luo et al., 2020; Zhu et al., 2022; Deng et al., 2021] perform end-to-end prediction for higher efficiency. Recent works leverage content-structure modeling [Zheng et al., 2025], auto-focus mechanisms on locked pre-trained vision models [Zheng et al., 2026], Transformer architectures [Vaswani et al., 2017; Kim et al., 2021], and large language models [Sui et al., 2023] for stronger crossmodal reasoning. However, both paradigms require dense box annotations, motivating weakly supervised alternatives.

2.2 Weakly Supervised REC

WREC learns object grounding using only image-text pairs. Early two-stage methods [Liu et al., 2019b; Liu et al., 2019c; Wang et al., 2021] employ sentence reconstruction [Liu et al., 2021] or contrastive learning [Gupta et al., 2020; Zhang et al., 2020] as weak supervision, but suffer from high computational costs due to dense region proposals.

One-stage alternatives have recently gained attention for their efficiency. RefCLIP [Jin et al., 2023] introduces anchortext contrastive learning; APL [Luo et al., 2024] improves anchor representations via prompt learning; DViN [Chen et al., 2025] fuses multiple pre-trained encoders through dynamic visual routing. More recently, AlignCAT [Wang et al., 2025] and MC-Bench [Xu et al., 2025] extend to settings with extra category labels or MLLM-based multi-image grounding, both beyond our scope. Existing methods treat anchors as isolated patches, lacking object-level semantics and relational reasoning. Our OCAAD addresses this by introducing objectcentric alignment and explicit relation modeling.

3 Preliminary

We briefly review DViN [Chen et al., 2025], a recent onestage WREC method that serves as our baseline.

Problem Formulation. Given an image I ∈ RH×W×3 and a referring expression T = {w1, . . . ,wL}, the goal is to localize the referred object by predicting a bounding box b∗ ∈ R4 using only image-text pairs as supervision.

Visual Feature Extraction. DViN extracts anchor features {F i v }3i=1 fromDarkNet [Redmon and Farhadi, 2018] at three scales (13×13, 26×26, 52×52 for 416×416 input). Each spatial location generates anchor features fa ∈ Rd that can be decoded into bounding boxes via the detection head D(·).

Language Feature Extraction. An LSTM encoder processes the expression into word-level features {f l t }Ll=1 and a global sentence feature ¯ ft ∈ Rd from the last hidden state.

Dynamic Visual Routing. To enhance fine-grained visual capability, DViN incorporates multiple visual encoders (DINOv2, EfficientSAM) through a vision-conditioned router. The routing weights are computed as:

where Fv0 denotes DarkNet features. Only the top-1 expert is selected along with DarkNet to maintain efficiency:

The routed anchor feature is a weighted combination:

Anchor-Text Matching. Best-matching anchor satisfies:

where ϕ(·, ·) denotes cosine similarity. The final bounding box is obtained as b∗ = D(a∗ ).

Training Objective. DViN adopts anchor-text contrastive learning with additional alignment losses:

where Lc is the contrastive loss, Lr is the diversity-aware routing loss, and Ldt, Ldv are text-guided and vision-guided alignment losses respectively.

Limitations. Despite the progress, DViN suffers from two limitations. First, anchor features represent local patches rather than complete semantic entities, lacking holistic object-level information. This makes it difficult to distinguish objects with similar local appearances but different overall semantics. Second, without explicit relation modeling, DViN cannot reason about structural dependencies among objects. For expressions like "the giraffe with mouth closed", understanding the relation between "giraffe" and "mouth" is crucial for accurate grounding, yet anchor-level alignment provides no mechanism for relational reasoning.

4 Method

Given an image I ∈ RH×W×3 and a referring expression T = {w1, . . . , wL}, our goal is to localize the referred object using only image-text pairs as supervision. As shown in Figure 2, OCAAD builds upon a dual-branch architecture and introduces two key modules: Anchor-Object Distillation Module (AODM) and Intra-Modal Relation Consistency (IRC). In the following, we first describe the feature extraction process, then detail our object-centric visual representation, and finally present the two proposed modules.

4.1 Feature Extraction

Following DViN [Chen et al., 2025], we adopt the same feature extraction pipeline for visual and language modalities to ensure fair comparison. The key difference lies in our use of DINOv2 attention maps for object-centric representation.

Visual Features. We extract multi-scale visual features {F i v }3i=1 from DarkNet [Redmon and Farhadi, 2018] for anchor-based detection. Following DViN [Chen et al., 2025], we also incorporate features from EfficientSAM [Xiong et al., 2024] and DINOv2 [Oquab et al., 2023] through a vision-conditioned router. The routed features are fused with DarkNet features, yielding enhanced multi-scale representations { ˆ F i }3i=1 . Additionally, we extract self-attention maps {Ak}Nh k=1 from the last transformer layer of DINOv2, where Ak ∈ RNp captures the attention weights from the CLS token to spatial patches for head k, and Np denotes the number of spatial patches.

Language Features. An LSTM encoder encodes the expression into word-level features {f l t }Ll=1 and a global sentence feature ¯ ft ∈ Rd . To enable object-level reasoning, we employ an entity extractor to identify object noun indices O = {o1, . . . , oM} from the expression, yielding entity-level features {f om t }Mm=1 .

4.2 Object-Centric Visual Representation

A key observation motivating our approach is that different self-attention heads in DINOv2 naturally capture diverse semantic regions within an image [Barsellotti et al., 2025]. Research on bridging self-supervised vision backbones with language has demonstrated that these attention maps computed between the CLS token and spatial tokens align with relevant objects, exhibiting remarkable semantic and local consistency. We exploit this property to construct object-centric visual representations that bridge the gap between local anchors and holistic objects.

Head-Weighted Visual Embeddings. For each attention head k, we compute a weighted embedding by aggregating the routed features according to the attention distribution:

where vAk ∈ Rd encapsulates the semantic content captured by head k. This yields Nh distinct visual embeddings {vAk }Nh k=1 , each attending to different image regions.

Entity-Head Hard Selection. To associate each linguistic entity with its corresponding visual region, we compute the similarity between entity features and all head embeddings, then perform hard selection:

where sim(·, ·) denotes cosine similarity. The selected embedding vAk∗m serves as the object-centric visual representation for entity om.

4.3 Intra-Modal Relation Consistency

Complex referring expressions often describe target objects through their relations with other entities, such as "the player wearing a helmet and holding a bat". To capture such structural dependencies, we introduce IRC to enforce consistency between relational structures in visual and linguistic modalities, as illustrated in the left part of Figure 2.

Cross-Modal Entity Matching. Each linguistic entity om is first paired with its most similar visual head embedding:

Pairwise Relation Modeling. For samples with M ≥ 2 entities, we construct pairwise relation matrices that capture relationships among entities within each modality:

where τrel is the relation temperature parameter and softmax normalization is applied row-wise.

Relation Alignment Loss. The IRC loss enforces that the linguistic relation structure guides visual relation learning:

where V denotes the set of valid samples containing at least two entities, and ∥ · ∥F denotes the Frobenius norm. The key insight is that if two entities are semantically close in language space, their visual counterparts should also exhibit similar proximity.

4.4 Anchor-Object DistillationModule

Although anchor features enable efficient one-stage detection, they represent local visual patches rather than complete semantic entities. This creates a fundamental gap between anchor representations and holistic object understanding. To address this issue, AODM transfers object-level knowledge from attention heads to anchors through spatially overlapaware contrastive learning.

The detailed process of AODM consists of four steps, as illustrated in the right part of Figure 2.

Step 1: Text-GuidedAnchor Selection. Given anchor features {f j a }Na j=1 and the global text feature ¯ ft, we first identify the anchor most relevant to the text query:

Step 2: Spatial Overlap Computation. For the selected anchor j ∗ , we retrieve its associated anchor box bj ∗ with the highest objectness score. Then, we compute the spatial overlap between this anchor box and each attention map to identify which heads best capture the target region. The attention maps are reshaped to spatial grids ˆ Ak ∈ RHa×Wa , and the overlap score is computed as:

where B denotes the set of spatial positions within the anchor box region.

Step 3: Top-2 Head Selection. Based on the overlap scores, we select the top-2 attention heads:

The head ktop 1 with maximum overlap serves as the positive target for distillation, while ktop 2 serves as a hard negative since it typically captures nearby or partially overlapping regions that are easily confused with the target.

Step 4: Overlap-Aware Contrastive Distillation. Finally, we train the anchor feature to align with the best-matching head while contrasting against negative samples. Let s+ = sim(f j ∗ a , vAktop 1 ) denote the positive similarity. The distillation loss is formulated as:

where the negative term N = ∑n̸=b exp(s(1) n /τ) + ∑Bn=1 exp(s(2) n /τ). Here, s(1) n and s(2) n represent similarities to the top-1 and top-2 heads of other samples in the batch, B is the batch size, and τ is the temperature parameter. Importantly, the head embeddings vAk are detached during training, making DINOv2 heads serve as a frozen teacher that distills object-level knowledge into learnable anchor representations. This formulation encourages anchors to learn holistic object semantics while maintaining cross-sample discrimination.

4.5 Training and Inference

TrainingObjective. The overall training loss combines the anchor-text contrastive loss with our proposed modules:

where LATC is the standard anchor-text contrastive loss [Jin et al., 2023], and λ1, λ2 are balancing weights.

Inference. During inference, we select the anchor with the highest text similarity and regress the final bounding box through the detection head:

5 Experiments

5.1 Experimental Setup

Datasets. We evaluate OCAAD on three standard benchmarks: RefCOCO [Yu et al., 2016], RefCOCO+ [Yu et al., 2016], and RefCOCOg [Mao et al., 2016]. RefCOCO and RefCOCO+ contain over 140K expressions for 50K objects, with RefCOCO+ excluding absolute spatial references. RefCOCOg features longer expressions (average 8.4 words) covering attributes and spatial relations.

Implementation Details. We build upon WeakMCN [Cheng et al., 2025], which shares its architecture with DViN [Chen et al., 2025]. For fair comparison, we adopt the single-task WREC setting without auxiliary segmentation. The detection backbone is YOLOv3-DarkNet [Redmon and Farhadi, 2018], with EfficientSAM [Xiong et al., 2024] and DINOv2 ViT-B/14 [Oquab et al., 2023] as visual experts in the router. We extract attention maps from the 12 heads (Nh = 12, d = 768) of DINOv2's last transformer layer. An LSTM encodes language features. All visual encoders are frozen; only the LSTM is trainable. Images are resized to 416 × 416, with maximum text length 15 for RefCOCO/RefCOCO+ and 20 for RefCOCOg. Entity nouns are extracted via Stanza [Qi et al., 2020]. We train for 25 epochs using Adam (learning rate 1 × 10-4 , batch size 64) with cosine decay. Loss weights are λ1 = 0.1 and λ2 = 10; temperatures are τ = 0.07 and τrel = 0.1. All experiments use a single NVIDIA RTX 4090 GPU.

EvaluationMetric. We use Precision@0.5 (IoU≥ 0.5).

5.2 Comparison with State-of-the-Art

As shown in Table 1, OCAAD achieves state-of-the-art on most splits, with improvements of 1.59% on RefCOCO val, 3.29% on testA, and 1.06% on testB over the baseline. On RefCOCO+, gains reach 3.12% on val and 2.66% on testA.

Notably, gains are more pronounced on testB splits, which feature non-person objects requiring fine-grained attribute understanding, validating that OCAAD captures object-level semantics better than anchor-level methods. On RefCOCO+, which relies on attributes and relative relations rather than absolute spatial references, OCAAD's strong gains confirm IRC's effectiveness in modeling inter-object relations.

5.3 Ablation Study

Component Analysis

Table 2 presents the ablation results of AODM and IRC.

Effect of AODM. AODM achieves notable gains on person-centric splits, improving RefCOCO testA by 2.10% and RefCOCO+ val by 1.92%. This validates that distilling object-level semantics from DINOv2 attention heads enriches anchors for better holistic object understanding.

Effect of IRC. IRC shows strong improvements on RefCOCO+, with gains of 2.70% on val and 1.86% on testB. Since RefCOCO+ excludes absolute spatial references, expressions must rely on relative relations, demonstrating IRC's effectiveness in modeling inter-object relations.

Complementary Nature of AODM and IRC. The full model outperforms individual components with synergistic effects. On RefCOCO testA, AODM alone provides +2.10% and IRC alone +1.85%, while their combination yields +3.29%. This complementarity stems from their distinct roles: AODM enhances object identity recognition while IRC captures structural dependencies among entities.

AODM Design Analysis

To validate AODM design choices, we ablate the head selection strategy in Table 3 and Figure 3.

Importance of HardNegatives. Using only the top-1 overlapping head (without the second-best as hard negative) decreases performance by 0.46% on val and 1.17% on testA. As shown in Figure 3, the top-2 overlapping head typically attends to nearby or similar regions, serving as an effective hard negative that helps the model learn more discriminative anchor representations.

Spatial Overlap vs. Text Similarity. Replacing spatial overlap with text-similarity for head selection leads to performance drops of 0.94% on val and 1.45% on testA. Figure 3 illustrates that text-similarity may select heads that are semantically related but spatially misaligned with the target object, while spatial overlap provides more accurate correspondence between anchors and attention heads.

Robustness to Noisy Anchors. Although early-stage anchors can be noisy, AODM remains robust: frozen DINOv2 heads provide stable supervision, and the overlap score (Eq. 13) varies smoothly with box shifts. Empirically (Table 4), anchor-head overlap and accuracy co-evolve in a selfreinforcing loop. Better anchors yield better head selection, which refines anchors in return.

IRC Design Analysis

Table 5 compares token selection strategies in IRC.

Necessity of Disentangled Head Features. Replacing disentangled head features with raw anchor features ("w/o Head features") causes significant drops across all splits (0.94% on val, 1.28% on testA, 1.72% on testB). This confirms that DINOv2 attention heads provide superior object-centric representations for relation modeling.

Entity Extraction Improves Precision. Using all word tokens introduces noise from non-entity words (e.g., prepositions, articles), resulting in 0.37% lower performance on val. Filtering to all nouns improves results but still includes irrelevant nouns. Our parsed entity extraction achieves the best performance by precisely identifying object-relevant tokens, enabling more accurate entity-visual correspondence.

5.4 IRC Effectiveness on Different Entity Counts

Grouping RefCOCO+ val samples by entity count (Table 6), IRC yields progressively larger gains as the count increases: +1.62% for one entity, +2.00% for two, and +3.19% for three or more, confirming that IRC's effectiveness scales with relational complexity.

5.5 Cross-Dataset Transfer

To assess generalization, we transfer OCAAD across datasets without fine-tuning. As shown in Table 7, OCAAD consistently outperforms WeakMCN, with the largest gains under diverse linguistic styles (e.g., RCg→RC: +5.2% on testA), confirming that object-centric and relational features transfer better than anchor-level ones. Cross-dataset evaluation thus warrants wider adoption in WREC, since in-domain accuracy can mask poor generalization.

5.6 Visualization of DINOv2 Attention Heads

Figure 5 visualizes the attention maps from the last transformer layer of DINOv2 on a sample image. Different heads consistently focus on distinct object-level concepts (e.g., A4: full body, A6: head, A11: ball, A12: racket) without supervision, empirically supporting the object-centric assumption underlying both AODM and IRC.

5.7 Qualitative Results

Figure 4 compares DViN and OCAAD on six examples. Action understanding (Exp-1): for "woman playing tennis", the baseline encompasses multiple people, while OCAAD identifies the active player. Spatial relation reasoning (Exp2, 3): for "white pants behind batter" (Exp-2), the baseline focuses on the batter; OCAAD localizes the person behind. IRC enables correct grounding for "striped couch nearest tan chair" (Exp-3). Multi-entity relational grounding (Exp-4): for "man in blue shirt by guy in red shirt", the baseline fails to associate entities; IRC correctly grounds the target by modeling inter-object relations. Attribute-based distinction (Exp5): for "white shirt oww my toe", AODM captures holistic clothing semantics that the baseline misses. Object-person association (Exp-6): for "old man with cooler", OCAAD identifies the person-object association. These cases confirm that AODM enhances object-level semantics while IRC enables structured relational reasoning.

6 Conclusion

In this paper, we proposed OCAAD, an Object-Centric Alignment and Anchor Distillation framework for Weakly supervised Referring Expression Comprehension (WREC). Our key contribution is identifying that anchor-level alignment fundamentally limits fine-grained reasoning, and proposing object-centric alignment as a solution. Specifically, AODM transfers object-level semantics from DINOv2 attention heads to anchors through overlap-aware contrastive distillation, while IRC enforces relation consistency between linguistic and visual entity pairs. Experiments demonstrate that OCAAD achieves state-of-the-art performance on three benchmarks, validating the effectiveness of object-centric alignment for weakly supervised visual grounding.

Ethical Statement

There are no ethical issues.

Acknowledgments

This work was supported by the National Natural Science Foundation of China (62276072), the Guangxi Natural Science Foundation Key Project (2025GXNSFDA069017), and the Bagui Scholar Program of Guangxi (2025).