MAgSeg: Segmentation of Agricultural Landscapes in High-Resolution Satellite Imagery using Multimodal Large Language Models

1 Introduction

Agricultural Landscape Segmentation involves identifying the polygonal boundaries and constituent areas of agricultural features, such as crop fields, trees and water bodies, using satellite imagery. It is a foundational step for large-scale mapping and analysis of agricultural environments [FAO et al., 2023; OECD, 2023], which, in turn, support advanced, field-level analytics for multiple downstream precision agriculture applications such as crop type and yield management, policy making and resource allocation [Rudel et al., 2009; Kang and ¨Ozdo˘gan, 2019]. Effective solutions in these applications can have significant impact on mitigating climate change and managing global food security [FAO et al., 2023; OECD, 2023]. This task is especially challenging for smallholder farms in the Global South, which are characterized by an intricate mosaic of very small fields (< 2 ha) [Ministry of Agriculture and Farmers Welfare, 2024], diverse land uses, and interspersion with natural vegetation and water resources [Lesiv et al., 2019; Samberg et al., 2016; Rada and Fuglie, 2019], and are responsible for more than half of the global agricultural production [Samberg et al., 2016; Sylvester and others, 2015].

∗Work done as a Student Researcher at Google DeepMind.

The nature of smallholder systems pose unique technical challenges in landscape segmentation. These include high variance within the field class (due to diverse crops, shapes, and sizes), and low variance between fields and other classes like forests; and large distribution shifts with varying landscape characteristics across different regions, climates, and seasons [Wang et al., 2022; Kerner et al., 2023]. Accurate delineation of small-sized fields and other features necessitate the use of very-high-resolution (VHR) satellite imagery (e.g., WorldView-3 data at 0.31 to 2 m/pixel). However, adapting standard image segmentation models, primarily developed for natural images, for this task remains challenging due to the high scale variation, densely packed small-scale objects and computational challenges arising from high dimensional inputs [Zhu et al., 2017; Rolf et al., 2024]. Moreover, nearly all previous approaches use specialized supervised learning architectures and rely on labelled data. Obtaining such labeled data is labor-intensive, costly, and difficult to scale across the diverse geographies of the Global South.

Multimodal Large Language Models (MLLMs) harness the inherent reasoning capabilities and broad world knowledge of LLMs, and have been recently used for various segmentation tasks with natural images [Zhou et al., 2024]. MLLMs offer the potential for scalability and accessibility with relatively lesser labeled data requirements to address the smallholder landscape segmentation problem. However, existing MLLM-based segmentation approaches perform poorly on VHR satellite images, particularly on this challenging task. The dominant 'embedding-asmask' paradigm, exemplified by LISA [Lai et al., 2024] and GSVA [Xia et al., 2024], relies on auxiliary decoder heads to generate pixel-level masks. This design introduces parameter overhead and creates an alignment bottleneck between the LLM's text-centric latent space and the pixel decoder. The other 'text-as-mask' paradigm is a decoder-free approach proposed by Text4Seg [Lan et al., 2025]. It encodes masks as text sequences and seamlessly integrates segmentation into the standard autoregressive generation process of MLLMs.

However, this approach also fails when applied to VHR satellite imagery for two reasons. Encoding a full VHR mask (e.g., 615 × 615 px) would require about O(3.6 × 105 ) tokens, far exceeding the context windows of many MLLMs. Text4Seg relies on severe downsampling (e.g., to 32×32 px) to keep token counts of the encoded masks low. While acceptable for natural images, this destroys critical high-frequency details in satellite data required for boundary delineation. The second reason is the domain alignment gap: general-purpose MLLMs lack intrinsic understanding of overhead satellite imagery. E.g., while they can identify 'trees' in a photograph, they struggle to distinguish similar features like 'crops' versus 'pasture' in satellite views without explicit pixel-level grounding [Quenum et al., 2025].

In this work, we present a decoder-free MLLM-based Agricultural Landscape Segmentation approach (MAgSeg) that systematically overcomes these limitations. We create an instruction tuning dataset with a novel format, which is designed to enable efficient fine-tuning and aims at retaining both local or global context in VHR masks. This resolves the token length problem through a patch-based strategy: MAgSeg receives the full VHR image as context but is tasked with generating a text-based mask for only a specific sub-patch. Supervised fine tuning on this data helps, but is insufficient since it is optimized for next-token prediction (for the text-based mask) and not spatial accuracy. We address this domain alignment gap via Group Relative Policy Optimization (GRPO) using the mean-DICE score as a direct reward signal. This enables MAgSeg to prioritize pixel-level accuracy over mere textual fluency on the encoded masks.

We use another metric, overhead, to measure the parameter efficiency of MLLM-based segmentation models which quantifies the additional learnable parameters (e.g., in pixel decoders and segmentation heads) beyond the base LLM backbone; and serves as a proxy for computational cost and memory footprint during inference. MAgSeg, which has zero overhead, shows dramatic gains in segmentation performance over previous zero-overhead decoder-free approaches in our experiments. Moreover, MAgSeg also significantly outperforms previous decoder-based methods which achieve better performance, relative to previous zero-overhead methods, at the cost of high overhead.

Our contributions are summarized as follows:

1. We propose an architecturally efficient, zero-overhead approach, MAgSeg, that enables standard MLLMs to perform segmentation of complex smallholder agricultural landscapes from very-high-resolution (VHR) satellite imagery, without requiring auxiliary vision decoders.
2. MAgSeg overcomes critical context length bottlenecks and domain alignment gaps in previous decoder-free approaches through instruction tuning with a novel format, tasked to generate text tokens for only a patch within a VHR image, and RL-based post-training to refine pixellevel understanding.
3. On challenging benchmark smallholder agriculture datasets spanning three countries in the Global South, MAgSeg significantly outperforms state-of-the-art LLM-based baselines.

2 Background and RelatedWorks

MLLMs for Vision Tasks. MLLMs demonstrate exceptional capabilities in visual reasoning, captioning, and visual question answering. However, applying MLLMs to dense prediction tasks like segmentation remains non-trivial due to the inherent modality gap between language and vision. A common approach, exemplified by LISA [Lai et al., 2024] and GLaMM [Rasheed et al., 2024], involves appending a specialized pixel decoder (e.g., SAM [Kirillov et al., 2023]) to the LLM. The LLM outputs a special token (e.g., <seg>) which cues the decoder to generate a mask. While effective, this 'embedding-as-mask' approach increases training complexity, limits model scalability and disconnects the segmentation process from the LLM's native text-generation capabilities. Other approaches like VisionLLM [Wang et al., 2023; Wu et al., 2024] output polygon coordinates directly, but often struggle with complex geometries in satellite imagery.

To address these limitations Text4Seg [Lan et al., 2025] introduced a decoder-free 'text-as-mask' approach by casting segmentation as a sequence-to-sequence problem, which aligns with the autoregressive training pipeline of LLMs and enables scalable, flexible modeling with minimal architectural changes. The key idea is to split the mask into a grid of fixed-size patches, flatten the patches and represent each patch with a semantic descriptor, which can be a word, phrase or a more complex textual description. Thus, we obtain a sequence of text tokens whose length can become exceedingly large with both number of patches and more complex descriptions. To compress this description, they apply Run-Length Encoding (RLE) to the sequence, and empirically find that applying RLE row-wise (RRLE), with each row separated by a newline character, to the mask patches works well in practice. This transformed visual instruction data is used to fine tune the MLLM.

However, Text4Seg, which achieves state-of-the-art performance on several segmentation tasks on natural images, fails on the challenging task of smallholder landscape segmentation (as seen in our experiments). These landscapes have both large number of objects to be delineated and high scale variation, e.g., ranging from small to very large fields. Hence, the masks lead to complex and long descriptions with long sequences even after RRLE. The approach of downsampling the masks used by [Lan et al., 2025] leads to loss of critical high-frequency spatial details required for boundary delineation, whereas increasing the number of patches leads to loss of contextual information for each instance, which, in turn, deteriorates segmentation performance. Our approach, MAgSeg, specifically addresses these problems, while retaining the advantages of their decoder-free approach.

Agricultural Landscape Understanding. Segmentation of agricultural landscapes has been extensively studied in remote sensing - see [Zheng et al., 2025] for a recent survey. Most previous works have focussed on field boundary delineation, particularly in the global North, e.g., [Masoud et al., 2020; Waldner and Diakogiannis, 2020; Waldner et al., 2021]. A few recent works have recognized and addressed the challenges of smallholder agriculture by adapting classical computer vision techniques such as panoptic segmentation [Dua et al., 2024], object detection [Mei et al., 2022] and transfer learning based methods [Wang et al., 2022; Kerner et al., 2023]. All these methods rely on supervised learning (mainly CNN-based) architectures.

To harness their robust reasoning capabilities, visual understanding and world knowledge, MLLMs are being actively explored for various remote sensing tasks such as visual question answering and retrieval [Li et al., 2024]. However, their use in segmentation tasks is relatively underexplored. Two recent segmentation models, LISAt [Quenum et al., 2025] and FSVLM [Wu et al., 2025] follow the same embedding-asmask architecture of LISA [Lai et al., 2024]: a CLIP-encoded image [Radford et al., 2021] and a text embedding are passed to a MLLM (both use LLaVA [Liu et al., 2023b]); segmentation is triggered by a dedicated <seg> token within the LLM's vocabulary; upon generation, the token's embedding is mapped to a SAM-based decoder through an MLP; the decoder subsequently combines this embedding with the base image features to reconstruct the pixel-level mask. The model is trained end-to-end with loss functions to optimize text generation and segmentation. FSVLM was specifically evaluated for farmland segmentation while LISAt demonstrated improvements over many geo-spatial foundation models on visual question answering, visual grounding and image captioning tasks. In our experiments, we demonstrate significant improvements over this state-of-the-art approach on smallholder landscape segmentation datasets.

3 Our Approach: MAgSeg

Let D = {xi, yi}Ni=1 denote a (training) dataset, where xi ∈ Rhx×wx×3 represents a high-resolution satellite RGB image with height hx, width wx, 3 channels, and yi is the corresponding multi-class instance map. Our aim is to train a base MLLM, πθ, with parameters θ, using D, for the task of instance segmentation.

We first transform D into an instruction-tuning dataset with a novel format that preserves both local and global context in high-resolution images. Since supervised fine-tuning (SFT) alone lacks the fine-grained spatial reasoning needed for pixel-perfect segmentation of complex satellite imagery, we employ a Reinforcement Learning (RL) based post-training step to maximize pixel-level semantic alignment. At inference, the trained model generates semantic masks that are post-processed into instance masks. Figure 1 provides an overview of MAgSeg.

3.1 Confronting High Resolution: Patch-based Instruction Tuning with Global Context

Given a dataset sample (xi, yi), we first derive the semantic segmentation map, si from the instance map yi. To facilitate patch-based processing, we extract a random crop from the high-resolution input. We sample top-left coordinates (j, k) uniformly such that j ∼ U(1, hx - hp) and k ∼ U(1,wx - wp). Thus, from si, this yields an image patch pi of dimensions hp × wp and a spatially corresponding semantic mask patch ri.

The mask patch ri is subsequently converted into a textbased RRLE representation, denoted as ti = RRLE(ri), following the semantic descriptors devised for Text4Seg [Lan et al., 2025] (cf. Algorithm 1 in Appendix A). We then construct an instruction-tuning sample consisting of the prompt input and the target response:

Here, Itext is the textual prompt instructing the LLM to generate the RRLE mask for the provided patch. The full highresolution image xi is passed alongside each patch pi to provide global context, while targeting only the patch tokens ti keeps us within the MLLM's context window. This lets us apply Text4Seg's decoder-free seq-to-seq framework to HR satellite images without downsampling or losing critical highfrequency details.

To fine-tune πθ we employ Low-Rank Adaptation (LoRA) [Hu et al., 2022] in conjunction with a standard autoregressive objective. LoRA optimizes a significantly smaller set of parameters, denoted as ∆θ, by freezing θ and injecting trainable rank decomposition matrices into the linear layer of the MLLM's transformer. This approach ensures scalability, data efficiency as well as generalizability.

Formally, let the ground-truth RRLE sequence be represented as a series of tokens ti = {w1,w2, . . . ,wT}. The trainable parameters ∆θ are optimized by minimizing the negative log-likelihood of these target tokens conditioned on the multimodal context-

where, πθ+∆θ(·) denotes the probability distribution of the next token predicted by the adapted model. Minimizing this objective aligns the model's visual reasoning with the textual RRLE structure, enabling segmentation of the target patch pi.

After this fine-tuning stage, the MLLM is (a) conditioned to generate masks in the specified RRLE format, and (b) instilled with a foundational understanding of segmentation semantics and class labels, as established in [Lan et al., 2025].

3.2 Mitigating Domain Gap with GRPO

SFT optimizes for next-token prediction rather than spatial accuracy, often failing to achieve pixel-perfect alignment. To bridge this gap, we employ Group Relative Policy Optimization (GRPO) [Shao et al., 2024] as a post-training step, shifting the objective from likelihood maximization to a pixellevel semantic reward.

For a given input tuple {Itext, xi, pi}, we sample a group of G distinct outputs from the policy model πθ, denoted as {ti,g}Gg=1 . Each output ti,g represents a candidate RRLE sequence for the target patch pi. To evaluate the quality of these candidates, we first decode them into 2D semantic masks:

where, ˜si,g is the predicted segmentation map for the g-th sample (cf. Algorithm 2 in Appendix A). To quantify the semantic accuracy, we compute the reward ri,g as the Mean DICE score (mDICE) between the predicted mask ˜si,g and the ground truth mask si. Let C denote the set of classes (e.g., fields, ponds, trees). The reward is defined as:

where, ∩ denotes the intersection of active pixels for class c, and ϵ is a smoothing term. GRPO optimizes the policy by estimating the advantage of each sample relative to the group average, eliminating the need for a separate value function critic. The advantage Ai,g for the g-th sample is computed as:

where, µr and σr are the mean and standard deviation of the rewards within the group of G samples. The GRPO loss objective is formulated as:

where ρi,g = πθ+∆θ(ti,g) πθ+∆θSFT (ti,g) is the probability ratio. The final post-training objective is the GRPO loss with KL penalty:

This objective encourages the model to increase the probability of generating RRLE sequences that yield higher DICE scores, thereby explicitly refining the model's pixel-level understanding via the GRPO loss while at the same time avoids diverging from the reference policy obtained from Stage I, πθ+∆θSFT .

3.3 Inference

During inference, we patchify the high-resolution input image into a grid of disjoint spatial patches of dimensions hp ×wp (cf. Inference pipeline in Figure 1). These patches are collated into a single batch to enable parallel processing, which significantly reduces computational latency compared to sequential execution. Once the individual patch masks are generated, they are spatially concatenated to reconstruct the full-resolution semantic map. To remove stitching artifacts and improve boundary precision, we apply the postprocessing steps recommended in EPOC [Chen et al., 2025]. This includes (i) boundary detection using a SegFormer [Xie et al., 2021] based refinement model and (ii) the Watershed algorithm [Vincent and Soille, 1991] on the refined semantic maps to extract discrete instance segmentation maps.

4 Experiments

4.1 Experiment Settings

Datasets. We conduct our experiments on two datasets which span across 3 countries in the Global South with smallholder agriculture systems. The first is the ALU dataset [Dua et al., 2024] which contains 2009/430/431 images in its train/val/test splits. This data is from India, annotated with five distinct semantic classes: fields, trees, clouds, ponds, and wells. The second is a publicly available benchmark dataset, AI4SmallFarms [Persello et al., 2023], for agricultural field boundary delineation. The train/val/test splits contain 1962/90/304 images for Vietnam and 1860/605/177 images for Cambodia. In both cases, VHR imagery from Google Maps Satellite View are used. We take images sampled at 0.5 m/pixel GSD, from two satellite sources: Maxar Worldview, Airbus Pleiades.

Baselines. We benchmark our approach against state-of-theart segmentation MLLM-based methods: LISA [Lai et al., 2024], GSVA [Xia et al., 2024], which are decoder-dependent and the decoder-free Text4Seg [Lan et al., 2025]. We consider two variants of Text4Seg - coarse predictions (C) and refined predictions (R) that uses SAM refinement. Further, we also consider LISAt [Quenum et al., 2025] which extends LISA for segmentation in satellite images. In addition, we include Transformer-based models specialized for segmentation, LAVT [Yang et al., 2022] and GRES [Liu et al., 2023a]. Evaluation Metrics. Following the evaluation protocol in [Dua et al., 2024] for smallholder farms, we report the instance-wise Mean Intersection-over-Union (mIoU), Median IoU, False Positive Rate (FPR), and False Negative Rate (FNR). More details are in Appendix C.

In addition to standard performance metrics, we use a parameter efficiency metric, denoted as Overhead. This metric quantifies the additional learnable parameters required by a method beyond the base LLM backbone. For decoderdependent baselines, this includes parameters of the external segmentation heads such as SAM-H, pixel decoders, and any projection layers. For decoder-free methods such as ours and Text4Seg, the overhead is zero. Concretely, total number of parameters of a method is denoted as -

where, Φbase, Φprojection and Φdecoder denote the parameters in the base model, projection layers and decoder heads respectively. Then, the overhead is defined as -

This metric is vital for two reasons: (1) it highlights the efficacy of the base LLM's native reasoning capabilities without reliance on auxiliary decoder heads, and (2) it serves as a proxy for the computational cost and memory footprint required during inference 1 .

Implementation. For MAgSeg, we experiment with MLLMs of two sizes - 4B and 12B - versions of Gemma3 [Team et al., 2025]. We refer to these two variants as MAgSeg (4B) and MAgSeg (12B) respectively. These are comparable to the model sizes used in the MLLM-based baseline methods: LISA (13B), LISAt (7B), GSVA (13B) and Text4Seg (7B). We use hp = wp = 32 for patch dimension as mentioned in Section 3. Appendix B has other details of MAgSeg and all the baselines.

1 Here, we refer solely to the additional overhead incurred during training; inference overhead is excluded.

4.2 Results

Segmentation Performance

India. The quantitative comparison on the panoptic segmentation dataset from India is summarized in Table 1. Our proposed method, MAgSeg, advances the state-of-the-art, outperforming all MLLM-based and transformer-based baselines by a substantial margin. Specifically, we observe an improvement of 21 mIoU points over GRES, the closest competitor.

Vietnam and Cambodia. These regions are characterized by dense wetland rice cultivation, offering a different spatial distribution compared to India. As shown in Table 2, MAgSeg consistently achieves state-of-the-art results when trained on these diverse landscapes. The performance gap is distinct in Cambodia split. Here, baseline methods struggle to learn effective segmentation boundaries; notably, even strong decoder-based models like GRES and LISA plateau at approximately 0.12 mean IoU. Text4Seg fails completely (0.04 mean IoU) due to the small, fragmented nature of agricultural fields which are lost during mask downsampling. In contrast, MAgSeg successfully learns the complex landscape features, achieving a mean IoU of 0.43. This result - nearly quadrupling the performance of the nearest baseline - demonstrates that our GRPO-based training objective enables the model to converge on difficult pixel-level tasks where standard crossentropy or decoder-alignment losses fail. In the Vietnam data, while baseline performance improves (with GRES reaching 0.40 mean IoU), our MAgSeg-12B model maintains better performance with a mean IoU of 0.42 and a better median IoU of 0.35.

We attribute the significantly lower performance of prior methods to their inherent architectural limitations, which are addressed in MAgSeg: (a) Text4Seg: This method exhibits poor performance on satellite imagery. As noted in [Lan et al., 2025], Text4Seg relies on downsampling masks (e.g., to 32 × 32) to maintain tractable token counts. In the context of high-resolution satellite data, this compression causes a severe loss of high-frequency spatial details - such as narrow field boundaries - which cannot be recovered during inference. (b) Decoder-Based Models (LISA, GSVA, GRES): While methods like GRES perform better than Text4Seg, they still struggle to match MAgSeg. Specifically, for field segmentation on the ALU dataset, our method outperforms the best decoder-based model (GRES) by 12 and 33 points in mIoU and median IoU, respectively. On the Cambodia dataset, the gains are even larger-31 and 40 points in mIoU and median IoU. These substantial improvements validate our claim. We attribute this to the 'domain alignment gap': the disconnect between the LLM's textual representation and the separate pixel decoder's latent space, which often leads to inconsistent segmentation masks.

Qualitative Analysis

Illustrations in Figure 2 show that MAgSeg produces sharp, instance-aware boundaries for agricultural fields where competitors often output noisy or over-smoothed/undersegmented masks. The visual comparisons corroborate our quantitative findings: Text4Seg outputs suffer from severe blockiness due to the aforementioned downsampling artifacts. Similarly, decoder-based methods like LISA often generate artifacts or fail to capture crisp boundaries, likely due to latent misalignment. In contrast, our proposed method generates aesthetically clean, pixel-precise masks that closely align with the ground truth, effectively capturing the nuanced geometries of agricultural fields. The consistency of these results across both India (ALU) and Southeast Asia (Cambodia and Vietnam) confirms that MAgSeg's superiority is intrinsic to the method, and performs equally well across different landscapes. Figures 5, 6 and 7 in Appendix D.4 show additional illustrations for India, Cambodia and Vietnam.

Zero Shot Generalization

To evaluate the generalization capabilities of MAgSeg across different geographical regions, we conduct a zero-shot evaluation on the AI4SmallFarms dataset. Specifically, models trained solely on the ALU dataset (India) were evaluated directly on the Cambodia and Vietnam test sets without any fine-tuning. This setting is particularly challenging due to differences in agricultural landscapes between the source and target domains. The results are presented in Table 3. We observe that MAgSeg, the 4B version itself, significantly outperforms decoder-dependent baselines in this regime. For Cambodia, our model achieves a mean IoU of 0.40, surpassing the strongest baseline, LISA, which achieves 0.25. Notably, the median IoU for our method is 0.43, whereas baselines struggle to exceed 0.15. A similar trend is observed for Vietnam, where our method achieves a mean IoU of 0.41, compared to 0.36 for LISAt. These results indicate that decoder-based architectures may overfit to the visual statistics of the training domain, leading to poor performance when subjected to domain shifts. In contrast, our generative approach appears to learn more robust, semantic-aware representations that transfer effectively to unseen agricultural landscapes.

Architectural and Training Efficiency

A critical advantage of our approach is highlighted by the Overhead metric (Table 1, top row). We observe a clear dichotomy in prior work: decoder-free methods like Text4Seg offer low overhead but poor performance, whereas decoderbased methods achieve better performance at the cost of high overhead. Our method successfully breaks this trade-off, delivering superior segmentation performance with zero decoder overhead. This confirms that proper instruction tuning (via LoRA) and reinforcement learning (via GRPO) are sufficient to ground LLMs in pixel-level tasks without parameterheavy auxiliary networks.

We also analyze the computational efficiency of MAgSeg with the best-performing baselines in Table 4. To ensure a fair comparison across different base model sizes, we report a parameter-normalized training time, defined as the average training duration per token divided by the total number of model parameters. Our decoder-free approach demonstrates superior efficiency on both metrics. First, regarding architectural complexity, our method incurs zero overhead, whereas decoder-dependent baselines introduce substantial additional parameters, ranging from 9.35% (GSVA) to 15.82% (GRES). Second, this streamlined architecture translates directly to training speed. Our method achieves a normalized training time of 1.33 × 10-8 s, significantly outperforming LISA/LISAt (1.69), GRES (1.80), and GSVA (2.04). This reduction in latency confirms that eliminating auxiliary vision decoders not only simplifies deployment but also enhances the computational throughput of the training process.

Additional Results

We evaluate the impact of GRPO post training and EPOC refinement during inference in MAgSeg through an ablation study. Table 1 in Appendix D.1 shows that, while each of them individually improves the mean IoU, the combination of both components yields the best performance. We qualitatively demonstrate the importance of GRPO in suppressing spurious boundaries in figure 4 (Appendix D.3). We also present (in Appendix D.2) the comparative performance on field segmentation stratified by various criteria: MAgSeg outperforms the best baselines across 5 different field size ranges (< 100m2 to 5 acres; and by a large margin for smaller field sizes < 1 acre), 5 climatic regions and 7 ecological regions. MAgSeg's robustness to field sizes is particularly important in smallholder landscapes, where plot sizes are highly heterogeneous.

Limitations

Our experiments also highlight limitations of MAgSeg. Performance drops on minority classes (e.g., wells, ponds) due to inherent class imbalance, a challenge shared by most segmentation baselines. Additionally, our patch-based inference lacks cross-patch spatial context, which can produce stitching artifacts and geometric discontinuities in final maps (see Figure 8 in Appendix D.5). Future work can address both limitations within our framework.

5 Conclusion

In this work, we presented MAgSeg, the first MLLM framework specifically adapted for smallholder agricultural landscape segmentation from VHR satellite imagery. MAgSeg successfully addresses the critical bottlenecks of prior decoder-free MLLM approaches - token-length limitations and domain alignment inefficiencies, without using parameter-heavy auxiliary decoders. Extensive evaluations on smallholder agriculture datasets spanning 3 countries demonstrate the superiority of our method over previous MLLM-based methods, achieving up to 4× improvement over the nearest competitor while incurring zero parameter overhead. Our evaluation also confirms the robustness of MAgSeg across diverse geographies including in zero-shot settings. Ultimately, this work illustrates that with appropriate grounding strategies, general-purpose foundation models can be effectively adapted for specialized, high-precision agricultural landscape segmentation, paving the way for scalable, accessible, and accurate global agricultural monitoring.