All papers

Introduces AgentSWE, a 25-task benchmark evaluating whether coding agents can build, edit, and optimize other software agents against user requirements.

This paper introduces a benchmark evaluating whether coding agents produce functionally correct code that also avoids real-world security vulnerabilities.

Analyzes filesystem privilege escalation vulnerabilities in Model Context Protocol servers and develops a tool to detect them.

This paper presents a benchmark evaluating whether software agents can autonomously find and fix bugs in interactive web applications.

The paper examines whether non-thinking mode reinforcement learning gains in hybrid reasoning models stem from leakage of thinking-mode activations.

The paper introduces a benchmark evaluating agent susceptibility to prompt and script injections across multi-tool shared workspace environments.

Introduces a benchmark evaluating coding agent misbehaviors across code generation, file management, and system environment integrity.

The paper demonstrates that internal activation states can forecast and block unauthorized actions and deceptive reports by language model agents.

The paper investigates decision failures in fast single-pass agent models, reporting that wrong actions often occur despite correct reading of the environment state.

The paper introduces an automated weak-to-strong hacker-fixer loop to find and repair vulnerabilities in agent evaluation verifiers.

The authors develop a reinforcement learning method to calibrate an agent's per-step confidence against the final outcome of an episode.

Evaluates a label-free rubric clarification method, finding it increases consensus among LLM judges without improving agreement with human evaluations.

Audits legal chain-of-thought faithfulness by substituting cited authorities and observing whether model verdicts causally change.

Introduces a benchmark pairing pull requests with human-verified bugs to evaluate code review agents on bug detection accuracy.

The paper proposes executable contracts to verify that adversarial benchmark edits preserve the core task definition when evaluating tool-using agents.

The paper introduces a testing framework that turns documented model safety claims into specifications to find counterexamples via mutated prompts.

The paper shows that large language model agents fail to retrieve relevant skills when large numbers of irrelevant skills populate the metadata library.

This paper analyzes methodological failure modes in data science agents, such as data leakage and invalid evaluation, beyond simple code accuracy.

The authors reproduce and compare five language model factuality alignment methods on a standardized benchmark across identical base models and evaluation protocols.

The paper investigates on-policy indirect manipulation attacks that steer autonomous agent purchasing and ranking decisions via web retrieval.

The paper evaluates typed decision models against standard language model judges to measure how emotional tone and biases affect their evaluation accuracy.

Demonstrates that knowledge distillation transfers safety dispositions and behavioral tendencies from teacher models to student models on unrelated tasks.

Identifies passage copying as a failure mode in long-form retrieval systems and introduces sentence-level credit assignment to encourage genuine grounding.

Studies agent-to-agent negotiations over money, finding that the receiving agent's characteristics primarily dictate the final transfer amounts.

This paper introduces a benchmark to test if coding agents can build reliable validation test harnesses from natural language specifications.

This paper introduces a repository-grounded benchmark to evaluate how well coding agents follow engineering constraints and developer workflows.

Investigates the risks of using LLM-generated regression tests, showing they can inadvertently preserve and enforce code faults.

Measures output variability across repeated runs at temperature zero in semantic query engines backed by language models.

Introduces SWE-DeceptionBench, a benchmark measuring active and passive deceptive model behaviors across software development lifecycle tasks.

Demonstrates that reward-hacking strategies can propagate across multiple agents through shared memory and code in multi-agent coding systems.

Analyzes security incidents from frontier AI agent evaluations and proposes a framework for proactive boundary assurance.

Audits the reliability of LLM judges across multiple prompt formats, temperatures, and orderings, introducing a trustworthy verdict metric.

Evaluates how schema ordering impacts LLM performance under structured output constraints, showing reasoning-first designs prevent accuracy drops.

Evaluates whether LLM compliance systems adjust verdicts when governing rules are perturbed, finding many verdicts are insensitive to rule changes.

Evaluates typed decision models used as agent guardrails, uncovering vulnerabilities and asymmetric error rates in allow-or-block actions.

Introduces a benchmark evaluating guard models on identifying unfulfilled safety obligations in language model agent trajectories alongside forbidden actions.

Measures how easily user-created web posts become cited sources in commercial AI search engines.

The paper shows that backdoor mechanisms can cryptographically hide arbitrary neural network capabilities until activated by secret triggers.

The paper introduces an automated framework that fuzzes and identifies semantic security bugs across Model Context Protocol server tools.

The paper introduces a multilingual benchmark evaluating safety and discrepancies between verbal refusals and tool actions in tool-calling language models.

The paper introduces an external reference monitor that enforces purpose-bound capabilities to secure multi-agent delegation against confused-deputy attacks.

The paper proposes a deterministic gating architecture enforcing contextual integrity access control for enterprise document retrieval in language models.

The paper investigates confused-deputy vulnerabilities where injected web page text causes browser agents to approve unauthorized access requests.

The paper proposes a control plane enforcing structured contracts and single-use allowances to prevent unauthorized actions by enterprise AI agents.

The paper proposes an object-bound sandboxing broker to confine compromised AI coding agents and prevent unauthorized system actions.

This paper investigates how unverified claims by competitors manipulate agent decisions and proposes checking authority rather than truth.

The paper evaluates 19 safety guards across multiple non-English languages to measure how translation alters step-level safety decisions for agents.

The paper shows mean benchmark accuracy misses tail risks in fine-tuned models, proposing recipe certification cards for deployment decisions.

The paper audits multimodal clinical machine learning studies, finding that incomplete unimodal baselines frequently inflate claimed gains.

The paper evaluates whether initial code changes made by coding agents make subsequent software maintenance tasks harder than human changes.

The paper introduces a benchmark to study why frontier models prioritize protecting subjects over preventing larger physical property damage.

The authors evaluate how language models alter their refusal behavior when informed they are being monitored in English and five Indic languages.

Demonstrates a graph-topology-aware poisoning attack against large-scale retrieval systems using approximate nearest-neighbor indexing.

Presents an agentic system with human-governed adaptation to automate root-cause analysis across complex, large-scale infrastructure environments.

Analyzes authorization security flaws in agent human-in-the-loop approval mechanisms and proposes a tamper-resistant approval gateway.

The authors introduce a benchmark to evaluate whether personal agents can distinguish accepted user arrangements from unacknowledged proposals.

This paper shows chain-of-thought monitoring can fail even when reasoning is necessary, proposing a multi-level hierarchy to measure monitorability.

The authors show that medical vision-language models frequently abandon correct clinical answers when prompted with fabricated medical documents.

The authors introduce Bemula, a benchmark designed to evaluate how accurately language model agents erase specific user memories upon request.

The paper presents a taxonomy that locates agent workflow failures across specific workflow stages and agent functional capabilities.