Every paper checked — nothing hidden
All papers
Every paper checked in the week of 12–18 Oct 2026 · LLMs for code — nothing is hidden. Brief papers first.
All1,106In brief3More (on-topic)144Not relevant928Not assessed6Older42Could not confirm7Blogs with no paper39All blogs42
| Paper | Topic | Relevance | Feedback |
|---|---|---|---|
| AgentSWE: Can Coding Agents Build the Agent You Actually Want? Introduces AgentSWE, a 25-task benchmark evaluating whether coding agents can build, edit, and optimize other software agents against user requirements. Ranked #1Surprising resultPreprint9/10 suggested | LLMs for code via arxiv · 12 Oct, 16:28 | On-topic Focuses directly on coding agents generating, editing, and repairing agent software codebases. | |
| SWE-Duel: Towards a Self-Scaling Arena for Coding Agents Presents an arena benchmark where pairs of coding agents dynamically generate, test, and debug code against each other. Ranked #2New testPreprint9/10 suggested | LLMs for code via arxiv · 12 Oct, 16:28 | On-topic Centers entirely on software engineering coding agents, pull requests, and automated program repair. | |
| SWEeper-Bench: Can Agents Discover Bugs in Interactive Software? This paper presents a benchmark evaluating whether software agents can autonomously find and fix bugs in interactive web applications. In brief · #3New testPeer-reviewed8/10 suggested | LLMs for code via arxiv · 15 Oct, 06:19 | On-topic Introduces a benchmark evaluating coding agents on finding and fixing software bugs. | |
| SWE-MISBEHAVIOR: Benchmarking and Mitigating Boundary Violations in Software Coding Agents Introduces a benchmark evaluating coding agent misbehaviors across code generation, file management, and system environment integrity. In brief · #4New testPreprint8/10 suggested | LLMs for code via arxiv · 15 Oct, 06:17 | On-topic Benchmarks misbehaviors and mitigation strategies in software engineering agents. | |
| Athanor: An Agent-Native System for Autonomous Post-Training Research Develops an open agent environment for evaluating how autonomously language agents can conduct post-training machine learning experiments. In brief · #5Surprising resultPreprint8/10 suggested | LLMs for code via arxiv · 17 Oct, 06:17 | On-topic Agents autonomously write, modify, and debug machine learning training scripts and code. | |
| VulnBump: A Benchmark for Fixing Vulnerable Dependencies in Real Projects While Keeping the Build and Tests Green This paper introduces a benchmark testing whether coding agents can upgrade vulnerable dependencies while preserving project builds and test suites. Ranked #6New testPreprint8/10 suggested | LLMs for code via arxiv · 15 Oct, 06:18 | On-topic Evaluates coding agents fixing vulnerable dependencies across real software repositories. | |
| Co-Modifying Application Code and Deployment Infrastructure: A Causal Study of AI Agent Pull Requests Evaluates pull requests from software agents modifying both application code and deployment infrastructure in continuous integration pipelines. Ranked #7Surprising resultPreprint8/10 suggested | LLMs for code via arxiv · 12 Oct, 16:28 | On-topic Analyzes AI coding agents modifying application code and CI/CD pipelines in pull requests. | |
| (When) Should a Software Factory be Lights-off? Benchmarking Code Review Agents on Bug Detection Introduces a benchmark pairing pull requests with human-verified bugs to evaluate code review agents on bug detection accuracy. Ranked #8New testPreprint8/10 suggested | LLMs for code via arxiv · 12 Oct, 16:29 | On-topic Introduces a benchmark evaluating LLM-based coding agents on reviewing code and detecting software bugs. | |
| TestSuiteBench: Coding Agents Struggle to Generate Strong Test Suites from Specifications Evaluates coding agents on generating full application test suites from specifications using a benchmark of open-source projects and mutants. Ranked #9Surprising resultPreprint8/10 suggested | LLMs for code via arxiv · 12 Oct, 16:29 | On-topic Introduces a benchmark evaluating coding agents on generating complete software test suites from specifications. | |
| SpecGuardBench: Can Coding Agents Construct Reliable Validation Harnesses from Natural Language Specifications? This paper introduces a benchmark to test if coding agents can build reliable validation test harnesses from natural language specifications. Ranked #10New testPreprint8/10 suggested | LLMs for code via arxiv · 15 Oct, 06:17 | On-topic Evaluates coding agents on generating reliable validation harnesses and tests from specifications. | |
| ANCHOR: Assessing Naturalistic Constraint Hierarchies and Ordering in Repository-Grounded Agentic Coding This paper introduces a repository-grounded benchmark to evaluate how well coding agents follow engineering constraints and developer workflows. Ranked #11New testPreprint8/10 suggested | LLMs for code via arxiv · 15 Oct, 06:17 | On-topic Assesses coding agents on following realistic developer constraints in repository tasks. | |
| On the Risks of using LLM-Generated Tests for Regression Testing Investigates the risks of using LLM-generated regression tests, showing they can inadvertently preserve and enforce code faults. Ranked #12Surprising resultPreprint8/10 suggested | LLMs for code via arxiv · 12 Oct, 16:29 | On-topic Examines risks of using LLMs to generate regression tests for software repositories. | |
| SWE-DeceptionBench: Benchmarking Deceptive Behaviors of LLMs Across the Software Development Lifecycle Introduces SWE-DeceptionBench, a benchmark measuring active and passive deceptive model behaviors across software development lifecycle tasks. Ranked #13New testPreprint8/10 suggested | LLMs for code via arxiv · 12 Oct, 16:28 | On-topic Evaluates LLM behaviors across the software engineering lifecycle including requirement feasibility and code tasks. | |
| Do Modifications Made by Coding Agents Make Subsequent Software Maintenance Harder? The paper evaluates whether initial code changes made by coding agents make subsequent software maintenance tasks harder than human changes. Ranked #14Surprising resultPreprint8/10 suggested | LLMs for code via arxiv · 15 Oct, 06:19 | On-topic Studies whether code generated by coding agents harms subsequent software maintenance. | |
| What Should a Software Team Spend on Agents? Simulation-Based Cost Optimization of Mixed AI-Human Teams The paper models mixed human-agent software teams to find cost-optimal spending allocations balancing human developer hours and automated coding agents. Ranked #15New way to see a problemPreprint8/10 suggested | LLMs for code via arxiv · 15 Oct, 15:07 | On-topic Investigates allocating budgets and roles between human developers and AI coding agents. | |
| BenchChecker: Assessing the Credibility of Bug-Fixing Benchmarks for LLMs This paper proposes a framework to detect training data contamination in bug-fixing benchmarks for language models. Ranked #16Surprising resultPeer-reviewed8/10 suggested | LLMs for code via arxiv · 15 Oct, 15:07 | On-topic Investigates data contamination detection in automatic program repair benchmarks for LLMs. | |
| TurnCost: Turns, Not Observations: Cache-Aware Loop Engineering for Coding Agents Analyzes coding-agent trajectories to model execution costs under prompt caching, showing turn count matters far more than observation token length. Ranked #17Surprising resultPeer-reviewed8/10 suggested | LLMs for code via arxiv · 15 Oct, 06:17 | On-topic Investigates execution traces, loop engineering, and cost optimization of software engineering agents. | |
| Executable Prefix Values for Locating Agent Failures The paper evaluates judge labels on agent steps by restarting failed trajectories from intermediate states to identify true failure points. Ranked #18Surprising resultPreprint8/10 suggested | LLMs for code via arxiv · 15 Oct, 06:17 | On-topic Evaluates failure localization on SWE-bench Verified coding agent trajectories. | |
| CWE-bench: Refocusing Agentic Security from Exploitation to Defense The paper presents CWE-bench, a benchmark of 120 held-out software tasks to evaluate AI agents on finding and repairing security vulnerabilities. Ranked #19New testPreprint8/10 suggested | LLMs for code via arxiv · 15 Oct, 06:17 | On-topic Tests LLMs on automated code vulnerability repair, patching, and repository-level software security. | |
| SWE-Stream: Scaling Executable Training Data for SWE Agents via Progressive Construction and Elastic Streaming The paper introduces a framework that progressively constructs and streams executable training tasks from code repositories for software engineering agents. Ranked #20New way to see a problemPreprint8/10 suggested | LLMs for code via arxiv · 15 Oct, 15:06 | On-topic Constructs validated tasks and trajectories to train LLMs for autonomous software engineering. | |
| P2Patch: Exploit-Guided Vulnerability Repair The paper presents an adversarial framework where an exploitation agent iteratively attacks and refines automated patches generated by a repair agent. Ranked #21Surprising resultPreprint8/10 suggested | LLMs for code via arxiv · 15 Oct, 15:06 | On-topic Proposes an adversarial agentic system to generate and validate code patches for software vulnerabilities. | |
| FDE-Gym: The Last Mile of AI Delivery The authors introduce FDE-Gym, a benchmark of 95 customer delivery tasks evaluating AI agents on ambiguous requirements and live software environments. Ranked #22New testPreprint7/10 suggested | LLMs for code via arxiv · 17 Oct, 06:17 | On-topic Evaluates agents modifying production code and live software systems for customer engineering tasks. | |
| Doc2Harness: Benchmarking LLMs on Long-Horizon Harness Engineering of Coding Agents The paper introduces a benchmark evaluating how well language models engineer isolated execution harnesses for coding agents using specification documents. Ranked #23New testPreprint7/10 suggested | LLMs for code via arxiv · 15 Oct, 15:06 | On-topic Evaluates language models on constructing execution harnesses for coding agents. | |
| ProgramGym: Scaling Training Environments for Long-Horizon Program Reproduction Agents Introduces ProgramGym, a framework providing verifiable sandbox environments and black-box test feedback for training long-horizon program reproduction agents. Ranked #24New way to see a problemPreprint7/10 suggested | LLMs for code via arxiv · 12 Oct, 16:28 | On-topic Directly targets software engineering agents building programs from scratch across multiple programming languages. | |
| A Decade in a Day: Synthesizing Agent-Native Interfaces from Human-Oriented Software Evolution The paper introduces a benchmark measuring how software interface versions affect the task performance of desktop language model agents. Ranked #25New way to see a problemPreprint7/10 suggested | LLMs for code via arxiv · 15 Oct, 06:17 | On-topic Analyzes software interfaces and synthesis to support automated agents interacting with software. | |
| SWE-at-Scale: Scalable Issue Synthesis from Historical Bug Fixes The authors propose a method to generate realistic software engineering benchmarks at scale by inverting historical bug fixes into issues. Ranked #26New testPreprint7/10 suggested | LLMs for code via arxiv · 17 Oct, 06:19 | On-topic Presents a framework to synthesize software engineering bug-fixing tasks for coding agents. | |
| PreciseCoder: Faithful and Generalizable Debugging Beyond Unit-Test Rewards The paper introduces a debugging training algorithm that rewards localized semantic code edits alongside unit test execution. Ranked #27Surprising resultPreprint7/10 suggested | LLMs for code via arxiv · 15 Oct, 06:17 | On-topic Trains models for precise software debugging and repository-level program repair via localized edits. | |
| Diff Diffusion: Predicting Edit Paths with Diffusion Models for Efficient Code Editing Proposes Diff Diffusion, a diffusion-based model that predicts localized code edit sequences rather than regenerating entire source files. Ranked #28New way to see a problemPreprint7/10 suggested | LLMs for code via arxiv · 15 Oct, 06:17 | On-topic Proposes a diffusion-inspired LLM framework for predicting code edit paths in software development. | |
| Beyond Binary Verifiable Rewards: Behavioral Self-Rewarding for SWE Agents Introduces behavioral self-rewarding to refine reinforcement learning feedback for software engineering agents by comparing execution trajectories. Ranked #29New way to see a problemPreprint7/10 suggested | LLMs for code via arxiv · 12 Oct, 16:28 | On-topic Focuses on training software engineering (SWE) agents to improve debugging and coding trajectories. | |
| TestPrism: Rethinking Test Evaluation Beyond a Single Reference Proposes evaluating generated test suites across pools of valid and invalid implementations rather than against a single reference implementation. Ranked #30New testPreprint7/10 suggested | LLMs for code via arxiv · 12 Oct, 16:29 | On-topic Studies LLM coding agents generating test suites and benchmarks multi-reference test evaluation. | |
| Humans vs. Machines: A Comparison With AI Agents in Software Reverse Engineering This paper benchmarks language model agents against skilled humans on binary software reverse engineering tasks using interactive tools. Ranked #31Surprising resultPreprint7/10 suggested | LLMs for code via arxiv · 15 Oct, 06:17 | On-topic Analyzes agent performance and workflows on software reverse engineering and binary analysis tasks. | |
| Scaling Inference-Time Compute with Teams of Coding Agents The paper studies compute allocation across multi-agent coding teams to identify Pareto-optimal trade-offs between cost and performance. Ranked #32Surprising resultPreprint7/10 suggested | LLMs for code via arxiv · 15 Oct, 06:17 | On-topic Evaluates multi-agent coordination specifically for programming tasks on ProgramBench. | |
| DuoDiscover: Scaling Autonomous Discovery Beyond the Evaluation Bottleneck The paper proposes an agent framework that discovers lightweight proxy evaluators to bypass costly benchmark evaluations during autonomous engineering discovery. Ranked #33New way to see a problemPreprint7/10 suggested | LLMs for code via arxiv · 15 Oct, 06:17 | On-topic Deploys a fleet of coding agents to autonomously optimize software and engineering systems. | |
| Don't Guess, Just Ask: Evaluating Interactive Requirement Clarification for Repository-Level Software Development Introduces a benchmark to evaluate how effectively LLMs clarify ambiguous user requirements in repository-level software development tasks. Ranked #34New testPreprint7/10 suggested | LLMs for code via arxiv · 12 Oct, 16:28 | On-topic Evaluates interactive requirement clarification specifically for repository-level software development. | |
| Can LLMs Fix It Without Code? Toward Automated Verification of No-Code Bug Fixes Presents an execution-based evaluation pipeline using browser agents to verify language model solutions for no-code software bug reports. Ranked #35New testPreprint7/10 suggested | LLMs for code via arxiv · 12 Oct, 16:28 | On-topic Addresses automated resolution and verification of bug reports in software workflows using LLM agents. | |
| HarnessSQL: Harness-Native Training for SQL Agents in Realistic Database Environments Introduces a post-training framework that trains text-to-SQL agents directly inside interactive database execution environments. Ranked #36Surprising resultPreprint7/10 suggested | LLMs for code via arxiv · 12 Oct, 16:28 | On-topic Focuses on generating, executing, and revising SQL code in realistic database environments. | |
| When the Approved File Is Unchanged: Authority-Set Integrity for Deployment and Coding-Agent Approvals The paper analyzes integrity risks in deployment workflows when secondary repository files change without modifying the approved primary configuration file. Ranked #37New way to see a problemPreprint7/10 suggested | LLMs for code via arxiv · 15 Oct, 06:18 | On-topic Investigates authority and approval integrity vulnerabilities when coding agents execute and modify repository files. | |
| SecReason: Vulnerability Detection and In-Stream Self-Repair for LLM Code Generation The paper presents an in-stream monitoring tool that detects security vulnerabilities and prompts repairs during reasoning-model code generation. Ranked #38New way to see a problemPreprint7/10 suggested | LLMs for code via arxiv · 15 Oct, 06:18 | On-topic Introduces in-stream security vulnerability detection and self-repair for LLM code generation. | |
| Passing Tests, Missing Semantics: Evidence of Interpretive Lock-In in Plan-Based DataFrame Code Generation The paper examines interpretive lock-in where planning agents generate code passing self-generated tests while missing the actual specification requirements. Ranked #39Surprising resultPreprint7/10 suggested | LLMs for code via arxiv · 15 Oct, 06:17 | On-topic Investigates interpretive lock-in failures during plan-based DataFrame code generation. | |
| JevCoder: When Does Decision Delegation Pay Off in Code Repair? The paper evaluates delegating code-repair decisions to low-cost probabilistic judges to reduce token costs without losing accuracy. Ranked #40Surprising resultPreprint7/10 suggested | LLMs for code via arxiv · 15 Oct, 15:06 | On-topic Directly addresses reducing token cost and improving automated program repair on SWE-bench. | |
| Beyond Functional Correctness: Static Code Quality Profiling of LLM-Generated Code This paper profiles the maintainability and structural code quality of model-generated Python programs using static analysis tools. Ranked #41New way to see a problemPeer-reviewed7/10 suggested | LLMs for code via arxiv · 15 Oct, 15:07 | On-topic Evaluates LLM-generated code quality using static analysis metrics like maintainability. | |
| Priming Base Models for Agentic Post-Training: An Open Midtraining Recipe The paper outlines a long-context midtraining stage using software engineering data to prepare base language models for agentic tasks. Ranked #42New way to see a problemPreprint7/10 suggested | LLMs for code via arxiv · 17 Oct, 06:17 | On-topic Trains models on software repositories, pull requests, and commit histories to improve SWE-bench benchmarks. | |
| A Self-Improving Agent that Grows its Own DSL: Oracle-Free Operator Induction for Exact Question Answering The paper presents an agent that expands its domain-specific language for exact answering without external rewards or judges. Ranked #43New way to see a problemPreprint7/10 suggested | LLMs for code via arxiv · 17 Oct, 06:17 | On-topic Induces and synthesizes executable DSL code operators to solve tasks. | |
| OPRD: On-Policy Reminder Distillation for Long-Horizon Coding Agents Uses multi-scope reminders and on-policy distillation to help coding agents maintain constraints over long software engineering interactions. Ranked #44New way to see a problemPreprint7/10 suggested | LLMs for code via arxiv · 17 Oct, 06:17 | On-topic Addresses long-horizon repository-level code generation and tool feedback on SWE-bench. | |
| SolBehave: Evaluating the Behavioural Correctness of LLM-Generated Smart Contracts The paper introduces a benchmark using fuzzing and transaction replays to evaluate the behavioral correctness of generated smart contracts. Ranked #45New testPreprint7/10 suggested | LLMs for code via arxiv · 15 Oct, 06:17 | On-topic Evaluates and repairs LLM-generated Solidity code using behavioral testing. | |
| Instructions Are Evidence, Not Objectives: Measuring Outcome Alignment of Agents This paper introduces an approach and benchmark for evaluating whether language model agents can infer unstated true requirements instead of following incomplete instructions. Ranked #46New way to see a problemPreprint7/10 suggested | LLMs for code via arxiv · 15 Oct, 06:17 | On-topic Evaluates agents on discovering unstated software engineering requirements using code repository context. | |
| SpecCloak: Probing Harmful Code Generation through Engineering Specifications Finds that framing malicious coding prompts as engineering specifications systematically reduces language model refusal rates across multiple models. Ranked #47Surprising resultPreprint7/10 suggested | LLMs for code via arxiv · 15 Oct, 06:17 | On-topic Investigates how engineering specifications bypass safety refusals in LLM code generation. | |
| Interaction-as-a-Judge Presents Interaction-as-a-Judge, using computer-use agents to evaluate interactive web and desktop software applications through direct dynamic interaction. Ranked #48New testPreprint7/10 suggested | LLMs for code via arxiv · 15 Oct, 06:17 | On-topic Evaluates interactive applications and coding agent outputs such as websites and games. | |
| SWEPatchTest: Evaluating Software Tests Beyond Reproduction SWEPatchTest provides a benchmark to evaluate whether agent-generated test suites correctly accept valid patches and reject invalid ones. Ranked #49New testPreprint7/10 suggested | LLMs for code via arxiv · 15 Oct, 06:17 | On-topic Benchmarks test generation and patch validation for coding agents and program repair. | |
| EvoWeb: A Benchmark for Coding Agents on Evolving Web Applications Introduces a benchmark evaluating coding agents on evolving web applications across cumulative feature requests. Ranked #50New testPreprint7/10 suggested | LLMs for code via arxiv · 15 Oct, 06:17 | On-topic Proposes a benchmark specifically assessing coding agents maintaining web applications. | |
| Sequent-Prover: Training Agents for Formal, Checkable SMT-based Reasoning The paper introduces Sequent-Prover, training language model agents to perform multi-turn formal reasoning using an external SMT solver scratchpad. Ranked #51New way to see a problemPreprint7/10 suggested | LLMs for code via arxiv · 15 Oct, 06:17 | On-topic Focuses on generating, inspecting, and repairing executable formal SMT-LIB code. | |
| LintEval: Benchmarking LLMs on Linter-Guided File Refactoring The paper presents a multilingual benchmark to evaluate how language models resolve linter alerts while preserving program functionality. Ranked #52New testPreprint7/10 suggested | LLMs for code via arxiv · 15 Oct, 15:06 | On-topic Introduces a benchmark evaluating LLMs on linter-guided code refactoring and repair. | |
| CodeHealth Predicts the Token Cost of Agentic Code Review: A Pre-Registered Study of 465 Agent-Authored Pull Requests Examines whether source code quality metrics predict the token costs of multi-agent pull request review pipelines. Ranked #53Surprising resultPreprint7/10 suggested | LLMs for code via arxiv · 17 Oct, 06:17 | On-topic Measures token consumption and cost dynamics of LLMs reviewing software pull requests. | |
| Poison Valleys: Diagnosing Path-Dependent Failure in LLM Code Repair Analyzes how misconceptions in early planning stages cause cascading failures during automated code repair by language model agents. Ranked #54New way to see a problemPreprint7/10 suggested | LLMs for code via arxiv · 17 Oct, 06:17 | On-topic Directly studies LLM failure modes and mechanisms in automated code repair. | |
| PremiseSQL: Evaluating Premise Critique and Abstention in Text-to-SQL Systems Introduces a benchmark to evaluate whether text-to-SQL systems can detect flawed query premises and abstain from generating invalid SQL. Ranked #55New testPreprint7/10 suggested | LLMs for code via arxiv · 17 Oct, 06:17 | On-topic Evaluates text-to-SQL systems on identifying flawed user premises and abstaining. | |
| WEAVE: Weaving Real-World Code Evolution into Dependency-Structured Tasks for Benchmarking and Training Code Agents Introduces a benchmark that structures multi-pull-request code changes into dependency graphs to evaluate long-horizon software engineering agents. Ranked #56New testPreprint7/10 suggested | LLMs for code via arxiv · 17 Oct, 06:17 | On-topic Builds dependency-structured benchmark and RL training tasks from multi-PR code repositories. | |
| Beyond Confidence: Rethinking Uncertainty Calibration in Deep Code Models This study evaluates whether confidence scores and calibration methods effectively identify failure cases in deep learning models for code tasks. Ranked #57Surprising resultPreprint7/10 suggested | LLMs for code via arxiv · 17 Oct, 06:19 | On-topic Evaluates uncertainty calibration and selective prediction for deep code models. | |
| Plausible Is Not Enough: Trust and Evidence in AI-Authored Pull Request Review The authors survey software practitioners to analyze differences in trust, evidence requirements, and review practices for AI-authored pull requests. Ranked #58New way to see a problemPreprint7/10 suggested | LLMs for code via arxiv · 17 Oct, 06:19 | On-topic Surveys practitioner verification and trust criteria when reviewing AI-authored pull requests. | |
| Skip the Compressor: Coding Agents Fix More Issues When They Pick Whole Files The paper shows that providing whole selected files to coding agents outperforms prompt compression and retrieval under fixed token budgets. Ranked #59Surprising resultPreprint7/10 suggested | LLMs for code via arxiv · 17 Oct, 06:19 | On-topic Examines context management strategies to improve coding agent performance on SWE-bench. | |
| Waypoint-OPD: Transferring RL-Induced Policy Shifts via Gated On-Policy Distillation Gates reinforcement learning policy-shift distillation using intermediate checkpoints to ensure student models only learn consistent teacher shifts. Ranked #60New way to see a problemPreprint6/10 suggested | LLMs for code via arxiv · 17 Oct, 06:17 | On-topic Evaluates gated policy distillation on code reasoning tasks. |
1–60 of 1,106Next →