AlgoSimBench: Identifying Algorithmically Similar Problems for Competitive Programming

1 Introduction

Large Language Models (LLMs) trained on both natural language and code have shown remarkable capabilities in automating various aspects of code generation and software engineering [Chen et al., 2021; Jimenez et al., 2024]. Although solving competitive programming (CP) problems has become a popular and challenging benchmark to evaluate LLMs' code reasoning abilities, it remains largely focused on code generation [Li et al., 2022; Shi et al., 2024]. This raises an open question: can models trained on problem-solving generalize their reasoning to more abstract or alternative forms of algorithmic understanding?

Recent works have highlighted several limitations of LLMs' code reasoning abilities. Gu et al.[2024] find that LLMs often struggle to predict program outputs given specific inputs. Similarly, Wei et al.[2025] shows that LLMs struggle to determine whether two programs are functionally equivalent. In this paper, we propose to study an underexplored but crucial subskill that underlies many forms of code reasoning: the ability to identify algorithmically similar problems.

In the context of programming and problem-solving, "algorithmically similar problems" (ASPs) are those that-despite differences in context, surface wording, or background story-can be solved using similar algorithmic strategies(Figure 1). This notion of similarity goes beyond superficial textual resemblance; it lies in the shared algorithmic structure, solution path, and problem-solving "tricks" required.

To curate AlgoSimBench, we collect and filter humanexpert-annotated CP problems with fine-grained labels from 4 CP websites, from which we develop a set of 402 multiplechoice questions (MCQs). Each MCQ requires identifying an ASP for a reference problem from 4 options: one ASP and 3 algorithmically-dissimilar but textually similar distractors.

We intentionally design the task to challenge superficial textual similarity analysis, emphasizing true structural algorithmic reasoning. Inspired by adversarial benchmarks such as Winograd and WinoGrande [Levesque et al., 2011; Sakaguchi et al., 2019], AlgoSimBench deliberately inverts the usual correlation between surface semantics and problem similarity in terms of solving. By making distractors highly semantically similar yet algorithmically different, we enforce that success does not come from shallow lexical overlap but requires deep reasoning about problem-solving and algorithmic properties.

AlgoSimBench can be applied to evaluate LLMs in both an end-to-end selection setting and under a retrieval-based setting (see Sec. 5.1). AlgoSimBench can encourage the development of better LLMs to reason independently of superficial textual features, as well as better embedding models to represent entailed logical features in text.

Our initial study shows that program solutions are better for identifying ASPs than natural-language problem descriptions. This motivates our proposed method: attempted solution matching (ASM), in which LLMs first generate an attempted natural-language or programming-language solution for each candidate problem, and then these solutions are compared instead of problem statements to identify ASPs. ASM leverages attempted solutions as a bridge to problem similarity, yielding robust gains on AlgoSimBench for both End-to-End and Retrieval-Based selection.

Finally, we found that ASM can help improve the selection of exemplars for In-Context Learning (ICL) in the context of competitive programming. Unlike previous methods [Xiong et al., 2026; Shi et al., 2022], ASM does not require any information from human-generated oracle solutions, and yet achieves comparable performance.

2 RelatedWorks

Solving Competitive Programming Problems with LLMs. Competitive programming presents a significantly more challenging landscape than general code generation tasks [Chen et al., 2021; Austin et al., 2021; Liu et al., 2023], as it often demands deep algorithmic reasoning, problem decomposition, and planning before implementation. Li et al.[2022] first framed competitive programming as a benchmark for evaluating language-and-code models and it was later advocated as a way to evaluate LLMs by Huang et al. [2024]. CodeLMs usually pre-train and post-train LLMs on real or synthesized code data, relying on supervised fine-tuning [Rozi`ere et al., 2024; Guo et al., 2024; Hui et al., 2024] or policy optimization with program correctness as the reward [DeepSeek-AI et al., 2025a]. Multi-agent frameworks [Islam et al., 2024; Li et al., 2025] assign separate agents to reasoning, implementation, and debugging, reflecting how humans tackle problems step by step. Shi et al.[2024] shows that selecting exemplar problems and solutions from top human coders for ICL provides significant gains, suggesting that good ICL examples can help LLMs better solve programming problems.

Code/Text Retrieval and Code Similarity. Early code similarity detectors relied on lexical overlap and syntax. Tools like MOSS [Schleimer et al., 2003] used token matching to flag plagiarism. However, lexical and syntax match could be easily evaded by superficial code modifications [ZakeriNasrabadi et al., 2023]. To overcome these limitations, semantic code search techniques emerged [Gu et al., 2018; Feng et al., 2020]. Suresh et al. [2025] construct code retrieval contrastive datasets through mining hard negatives. However, Wang et al. [2025] has found that retrievers struggle to fetch relevant and helpful context for code generation. Wei et al.[2025] showed that LLMs can perform poorly on the deceptively simple task of determining functional equivalence between two programs. Unlike prior works on code similarity studying snippet-level functionality (e.g., sort an array of floats), AlgoSimBench focuses on higher-level logical similarity, which lies in the problem structure and potential solutions.

Disentangling Logical Reasoning in Context. Disentangling semantic understanding from logical or formal/symbolic reasoning has been explored across a range of tasks and domains. Zhong et al.[2023] separated semantic representation from multi-step reasoning by decomposing general QA into two dedicated modules. Hua et al. [2024] introduced ContextHub to analyze the role of context in logical problems, showing that reasoning performance is strongly affected by natural-language-rich context. [Valentino et al., 2024] examined this issue in syllogistic reasoning by injecting content bias, benchmarking whether LLMs can judge an argument's formal validity independently of its plausibility. Related efforts in math word problem solving similarly aim to remove linguistic confounds by transforming problems into a more purely logical form [Calais et al., 2025]. To the best of our knowledge, AlgoSimBench is the first work to tackle this disentanglement in the code-and-algorithms domain, and the first to explicitly contrast textual and algorithmic similarities to enforce deeper logical reasoning.

3 AlgoSimBench Dataset

AlgoSimBench is a challenging benchmark, designed to evaluate LLMs' algorithmic-reasoning ability beyond code gener-

• (a) Top 15 Codeforces broad algorithm tags.

Figure 2: Comparison between Codeforces tag distribution and our fine-grained tag distribution.

ation, code embedding and code retrieval. It is collected from competitive programming problems, written, solved, labeled, and categorized by human experts. We collect problems and expert annotations from competitive programming communities, constructing 402 MCQs for ASP identification: Each question contains a given reference problem and 4 options, one algorithmically similar problem and three algorithmically dissimilar distractors. In this section, we introduce how we collect the labeled problems and the method used to construct adversarial MCQs.

3.1 Data Sources and Statistics

Many competitive programming platforms provide algorithm tags, but these are usually too general (e.g., dynamic programming) to fully capture the solution strategy. To identify ASPs, we need more specific tags, such as Longest Increasing Subsequence or 0/1 Knapsack, characterizing a specific technique needed to solve the problem. A comparison between the distributions of AlgoSimBench's labels and Codeforces' tags are given in Figure 2. From four different CP community websites, we collect problems with corresponding correctnessverified human-written solutions (subsequently called oracle solutions) and fine-grained algorithmic labels. We manually unified labels from these different sources and removed duplicates and overly broad or ambiguous tags, resulting in 1,317 cleanly-labeled problems. The final 402 MCQs utilized 903 problems and 231 labels from the filtered set.

3.2 Adversarial Task Design

As Jain et al. and Wei et al.[2021; 2025] found, language and code models are sensitive to functionality-preserving attacks and struggle to disentangle program logic from natural language semantics.

Competitive programming problems usually contain a background story, connecting the algorithms to a real-life scenario. We intentionally invert the correlation between algorithmic similarity and semantic textual similarity. That is, algorithmically similar problems should differ as much as possible in wording and surface descriptions, while dissimilar problems should appear similar in text but differ in algorithmic structure. Specifically: The similar option shares the closest match in problem topics and algorithmic approach but differs significantly in text and overall linguistic meaning. The distractors belong to different algorithmic categories but are chosen to be textually close to the original problem. The idea underlying this setting is that models should be able to ignore features irrelevant to problem solving and base their judgment of algorithmic similarity solely on problem logic and structure.

3.3 Multiple-Choice Question Construction

We obtain a problem set, S, each with a problem statement, si, a corresponding solution program, ci, and a set of algorithmic labels Yi = {y1, · · · , yk} We create multiple-choice questions as follows: Given a reference CP problem p ∈ S, we find an algorithmically similar problem, q ∈ S, and three algorithmically dissimilar problems {d1, d2, d3} ⊂ S.

Algorithmic Similarity Filter. To ensure algorithmic similarity between the reference problem p and the similar choice q, they should have the exact same set of labels Yp = Yq. On the other hand, the distractors di should have no label overlap with the reference problem p. To further enforce the minimal algorithmic similarity of each di, we apply the following conditions to di's labels: 1. Any Ydi does not share label subcategories with Yp; 2. Any Ydi ' is lexically dissimilar with Yp; We then manually filter and remove edge cases.

Textual Similarity Filter. Once we select candidates for the correct option as well as distractors, we ensure textual dissimilarity and similarity as follows. For the correct ASP option, q, among all candidates, we select the problem with the lowest textual similarity to p. For the distractors di, we select the n options with the highest textual similarity to p, ensuring Sim(p, di) > Sim(p, q). We compute textual similarity using a dense embedding model [Lewis et al., 2020].

4 Methods

Given the definition of algorithmic similarity, we hypothesize that it is better reflected in the solution code than in the problem statement - a finding supported by Shi et al.[2024].

4.1 Hypothesis: solutions manifest algorithmic features

To test this hypothesis, we apply a direct similarity-based retrieval approach where the goal is to retrieve ASPs (from the options in our MCQ problems) using either problem statements or oracle solution code as the problem representation. We treat the 4 options q, d1, d2, d3 as a mini corpus and use the reference problem p as the query. Retrieval is successful when q is the highest-ranking option. A variety of dense and sparse retrieval methods were tested using this setting. We use cosine similarity to calculate the similarity scores. As shown in Table 1, retrieval based on oracle code solutions consistently achieves higher accuracy, significantly better than random. This supports our hypothesis: Solutions explicitly encode algorithmic features that remain implicit or latent in the problem.

Since oracle solutions are generally not available in realworld use cases, we propose Attempted Solution Matching (ASM)--a method that uses an LLM to generate a solution attempt for each problem, and then compares these attempted solutions to identify ASPs. Figure 3 illustrates the approach.

4.2 Attempted SolutionMatching

Episodic Retrieval [Shi et al., 2024] uses similarity among solution programs to identify similar problems, but it compares LLM-generated code for the query problem with oraclesolution code for the potential retrieval targets. This assumes that the initial solution contains the correct algorithmic signal. However, this breaks down when the LLM selects the wrong approach where such errors can accumulate, decreasing LLMs' capacity to identify algorithmically similar problems.

To mitigate this mismatch between initial attempts and oracle solutions, we draw insights from the "generate-toretrieve" paradigm [Gao et al., 2023] and propose matching first-attempt solutions to other first-attempt solutions. Our method comes with a simple assumption: when attempting to solve a problem, the algorithmic ideas and problem-solving strategies are naturally reflected in both the model's natural language reasoning and its generated code, revealing the underlying methods being applied. As a result, algorithmically similar problems tend to produce similar solution attempts.

In ASM, each problem statement (i.e., problem description) is first mapped to a first-attempt solution using an LLM, and ASPs are identified by evaluating the similarities between attempted solutions. The attempted solution can take two forms: A natural language explanation of the solution strategy (NL-solution), denoted as ASM-NL; or a solution written in a programming language (PL-solution), ASM-PL. While both should contain algorithmic information, ASM-NL can include content on problem analysis, strategy exploration, algorithm selection, and problem solving; on the other hand, ASM-PL focuses on the complete implementation, whose content is rich in details but lacking in problem analysis.

5 Experiments

In this section, we start with evaluating LLMs' abilities to identify ASPs with AlgoSimBench under End-to-End Selection. Experiments are conducted to analyze how different factors affect LLMs' performance. Next, we explore using AlgoSimBench to test the abilities of various retrieval methods under the Retrieval-Based Selection setting. We evaluate several baselines as well as ASM-NL and ASM-PL under both settings. Finally, we apply ASM to in-context learning to retrieve better examples for aiding the generation of solutions to programming problems.

5.1 Experimental Settings

Datasets&Metrics. AlgoSimBench, consists of 402MCQs, where we measure the accuracy as the rate of selecting the correct ASP out of 4 options. To evaluate the performance of ICL exemplar selection, we test ASM and various baselines on USACO [Shi et al., 2024], a dataset with 307 "olympiad" competitive programming problems. We calculate the increase in the pass@1 solution rate [Chen et al., 2021] over random exemplar selection for different exemplar retrieval methods.

Models. For End-to-End Selection, we evaluate 12 open and closed LLMs: Qwen3-0.6B, 4B, 8B and Qwen3-Coder480B 1 . Llama-3.1-8B-instruct 2 , GPT-4o-mini, GPT-4o 3 , o3mini-medium 4 , Deepseek-R1 [DeepSeek-AI et al., 2025a], Deepseek-V3 [DeepSeek-AI et al., 2025b], Claude-3.5Sonnet 5 and Gemini 2.0 Flash. 6 For Retrieval-Based Selection, we tested various metrics for measuring the similarity between the query and candidate answers. We used a sparse retriever, BM25 [Robertson et al., 1995], and cosine similarity of dense embeddings from a text embedding model BART [Lewis et al., 2020], code embedding models GraphCodeBert [Guo et al., 2021], CodeSage-v2 [Zhang et al., 2024], SFR-Code400M R [Liu et al., 2024], Jina-Code-V2 [G¨unther et al., 2024], CodeRank Embedding [Suresh et al., 2025].

1 https://huggingface.co/collections/Qwen/qwen3

2 https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct

3 https://openai.com/index/hello-gpt-4o/

4 https://openai.com/index/openai-o3-mini/

5 https://www.anthropic.com/news/claude-3-5-sonnet

6 https://deepmind.google/technologies/gemini/flash/

Baselines. We compare ASM to using the following baseline problem representations. Statement: Original naturallanguage descriptions are used to represent problems, Summary: To mitigate aspects unrelated to problem-solving, an LLM is first prompted to summarize and abstract the problem, minimizing narrative elements and formatting details irrelevant to the structure of the problem. Solution: Each problem is replaced with a correct solution written by an expert programmer, which serves as an "oracle" upper baseline.

ASMSettings. As discussed before, for Attempted Solution Matching, we have a Natural Language setting, ASM-NL, where LLMs are asked to describe the solution in English given the problem, which is later used as reference and options; a Programming Language setting ASM-PL, where LLMs are asked to first generate Python-implemented solutions for the reference and options.

Evaluation Settings. We evaluate models under two task settings: 1) End-to-End Selection: where a full MCQ is presented to the LLM in a single prompt with a reference problem and four candidate options, and models are directly asked to select the most algorithmically similar problem. 2) RetrievalBased Selection: where each MCQ is framed as a retrieval task, where the query is the reference problem and the 4 candidates form a corpus of potential retrievals. For both settings, it is considered correct if the retrieved option is the algorithmically similar one.

5.2 End-to-End Selection

We apply the same prompts to all End-to-End Selection experiments, with a definition of ASPs and encouragement to use chain-of-thought reasoning. The only difference is to provide problems or attempted solutions as the options.

Main Results

MCQ accuracies of all seven models using End-to-End selection are given in Table 2. We found that none of the models perform particularly well when only given the original problem Statements. The weaker model Llama3.1-8B-Instruct has near-random performance across all methods. Reasoning LLMs like Deepseek-R1 and o3-mini perform better than other models, achieving accuracies around 60%. Given oracle solutions (Solution) instead of problem statements, LLMs can identify algorithmic similarity much better, yielding improvements as high as 26.3%.

Summarizing problems actually hurts the performance of all models except 2. We hypothesize that SoTA LLMs can already ignore superficial similarities when comparing programming problems. Our methods, ASM-NL and ASM-PL, consistently improve absolute performances across all models with an average improvement of 9.7% and 8.3%, respectively. ASM-NL outperforms ASM-PL on 10 out of 12 models, indicating that most models can generate and compare solutions described in natural language better than actual code solutions. For o3-mini-medium, both ASMs even outperform the oraclesolution baseline and give the best performances overall. The performance gaps between Statement and ASM also suggest that LLMs cannot generalize their code reasoning ability to similar tasks when they are out of a code generation context.

Effects of Problem Attributes

We further analyzed failure cases in AlgoSimBench to answer two research questions: 1. Do models make more mistakes on more difficult problems? 2. Are there some algorithms that are especially difficult to identify from problem descriptions?

Difficulty-level. The Pearson correlation between a problem's Codeforces' difficulty rating and the probability that an LLMwill get it wrong is -0.15 with p-value = 0.45, indicating that models do not perform worse on ASP if the problems are harder. This highlights an important distinction: identifying a problem's category and the appropriate algorithm is not the same as being able to solve it. For instance, recognizing that a Segment Tree is needed does not necessarily imply the ability to implement a correct solution for range updates and queries. Some problems are easy to classify but difficult to solve-reinforcing the need to evaluate models beyond final solution correctness.

Algorithm Labels. The over-represented and underrepresented algorithmic labels characterizing problems that models tend to get wrong (i.e., highest increment and decrement in label proportion) are shown in Figure 4. We found that some algorithmic types, though often seen as difficult (e.g., Centroid Decomposition) are not hard to identify. Some other simple problems, like shortest-path/Dijkstra, are also easy to identify. On the other hand, algorithms like DP and Fast-Fourier-Transform can be applied in very different circumstances, making such problems difficult to identify.

In a nutshell, LLMs' ability to completely solve programming problems and their ability to identify relevant algorithmic strategies are two fairly independent axes. The latter ability is more determined by a problem's type and topic than its difficulty level.

Effectiveness and Efficiency of ASM

Effectiveness. As discussed earlier, models' abilities to identify ASPs are not directly reflected by their ability to fully solve the problems. With Attempted Solution Matching, models are able to identify ASPs based on solutions that are not fully correct. To illustrate this robustness, we tested the quality of the generated programs for ASM-PL and surprisingly found that even when programs are incorrect, they can still substantially help identify algorithmically similar problems. The quality of generated programs in ASM-PL is evaluated by pass@1 [Chen et al., 2021]. In Table 3, we find that for weaker models, with pass@1 ranging from 1.8% to 19.6%, matching mostly-incorrect attempts still gives significant performance gains compared to directly matching problems statements, For Qwen models, ASM performs close to matching oracle solutions. While Claude-3.5-Sonnet has similar performance to Deepseek-R1 when solving problems and matching oracle solutions, it performs much worse when matching its own generated programs. This suggests that it does not identify the correct algorithm during failed solution attempts.

Efficiency. To illustrate the efficiency of ASM, we compare the cost (in generated tokens) of ASM-NL and ASM-PL against simply matching problem statements. We find that ASM significantly reduces the number of reasoning tokens used to solve problems. ASM-NL costs 80% as many tokens on average across all models. ASM-PL costs 190% as many tokens as it needs a longer CoT to generate a complete solution than a high-level NL strategy. Even so, to obtain similar accuracy, ASM-PL is more efficient than either relying on the built-in thinking mode of LLMs or using a larger model within the same model family. In Table 4, we define cost as # gen tokens × unit price. ASM-NL can achieve better accuracy while costing only 15% to 26% as much. ASM-PL also costs less than comparing statements. In other words, both ASMs can achieve better accuracy in non-thinking mode or with smaller models, yielding a lower overall cost.

5.3 Retrieval-Based Selection

In this setting, we treat AlgoSimBench as a retrieval problem as discussed in section 5.1. Results for retrieval based on problem statements were shown in Table 1. Here, we use an LLM to first map each statement to a Summary, an NL solution (ASM-NL), and a program solution (ASM-PL). We evaluate how well retrieval using these different representations is able to identify the correct ASP. Results are shown in Table 5.

ASM-NL again outperforms other methods. Unlike retrieval based on statements (see Table 1), using a Summary is better than random guessing. Since AlgoSimBench is constructed so that the textual problem-statement similarity is a misleading indicator of ASPs, summarization is helpful but still provides limited explicit information about algorithms.

An interesting finding is that simple term-matching (BM25) performs better than dense embedding models. A possible reason could be that algorithmic similarity is often better indicated by keywords like algorithmic terms or descriptions of the core idea, rather than a full implementation.

Although Retrieval-Based Selection performs significantly worse than End-to-End Selection, it scales much better to retrieving ASPs from a large corpus. End-to-end selection requires fitting all potential retrievals into an LLM input window and is difficult to scale beyond choosing from only four options in an MCQ. In contrast, retrieval-based selection only requires running an LLM on individual problems once to produce more effective problem representations that can then be efficiently retrieved from a large corpus using standard IR techniques (such as BM25) or dense retrieval with compressed text representations [Izacard et al., 2022; Li et al., 2023].

5.4 ICL Exemplar Selection with ASM

Since ASM can help find algorithmically similar problems, we apply it to enhance code generation. Shi et al. [2024] showed that including in-context exemplars of solved problems can enhance LLMs' abilities to solve competitive programming problems. They proposed Episodic Retrieval, using an LLM's solution along with the problem statement to retrieve the most similar human solutions using BM25. Then, the retrieved problem is included as an ICL exemplar, hopefully improving the ability to solve the given problem. ASM, on the other hand, directly matches LLM attempted solutions for both the given problem and problems in the corpus. Utilizing their methodology, we explored how ASM could be used for improved selection of ICL exemplars.

Table 6: ICL enhanced by different exemplar selection methods. Results are 1-shot Pass@1 on the USACO Benchmark. *Episodic Retrieval requires human-generated oracle solutions.

We also applied two baselines that select either a Random exemplar or retrieve the most similar exemplar using problem statements. Table 6 shows that with the same Retriever (BM25, which outperformed various dense-embedding methods), ASM methods achieve the best pass@1 [Chen et al., 2021] performance. While the absolute improvement is modest, this might be limited by how much performance gain ICL with a single example can bring to competitive program-

ming [Tang et al., 2023; Patel et al., 2024].

6 Conclusion and Discussion

This paper introduced AlgoSimBench, a novel benchmark designed to evaluate models' ability to reason about algorithmic similarity between competitive programming problems. This benchmark: 1) provides a focused evaluation of the algorithmic reasoning abilities of LLMs, decoupled from full solution generation; and 2) enables the study of retrieval methods that go beyond surface-level textual similarity, instead capturing deeper structural and problem-solving-related semantics.

We propose Attempted Solution Matching (ASM), which leverages LLM-generated first-attempt solutions-either in natural language or code-to better reflect the core algorithmic aspects of each problem. In both end-to-end selection and retrieval settings, ASM improves the accuracy of identifying ASPs and improves exemplar selection for in-context learning.

Some key findings were: 1) the performance gap between comparing statements and attempted solutions suggests limited generality of the algorithmic reasoning skills learned from code generation alone; 2) Identifying ASPs depends more on algorithm type rather than problem difficulty, making it a separate axis from problem-solving; 3) BM25 outperforms dense retrievers, indicating that keyword signals capture algorithmic features more effectively than generic semantic embeddings.

We hope AlgoSimBench inspires future work on identifying algorithmic similarity, code retrieval, and retrieval-augmented reasoning.