Every paper checked — nothing hidden
All papers
Every paper checked in the week of 12–18 Oct 2026 · LLM evaluation and benchmarks — nothing is hidden. Brief papers first.
All3,345In brief3More (on-topic)587Not relevant2,721Not assessed24Older45Could not confirm25Blogs with no paper54All blogs57
| Paper | Topic | Relevance | Feedback |
|---|---|---|---|
| AgentSWE: Can Coding Agents Build the Agent You Actually Want? Introduces AgentSWE, a 25-task benchmark evaluating whether coding agents can build, edit, and optimize other software agents against user requirements. Ranked #1Surprising resultPreprint9/10 suggested | LLM evaluation and benchmarks via arxiv · 12 Oct, 16:28 | On-topic Introduces AgentSWE, a novel benchmark and evaluation protocol assessing coding agents. | |
| SWE-Duel: Towards a Self-Scaling Arena for Coding Agents Presents an arena benchmark where pairs of coding agents dynamically generate, test, and debug code against each other. Ranked #3New testPreprint9/10 suggested | LLM evaluation and benchmarks via arxiv · 12 Oct, 16:28 | On-topic Introduces SWE-Duel, a self-scaling dynamic evaluation arena and benchmark for coding agents. | |
| More Methods, Better Agents? A Reality Check on Self-Distillation in Agentic Reinforcement Learning The paper benchmarks ten self-distillation methods in agentic reinforcement learning across varied model sizes, tasks, and computational costs. Ranked #4Surprising resultPreprint8/10 suggested | LLM evaluation and benchmarks via arxiv · 15 Oct, 15:06 | On-topic Provides a comprehensive empirical audit and comparison of self-distillation methods in agentic RL. | |
| SWE-MISBEHAVIOR: Benchmarking and Mitigating Boundary Violations in Software Coding Agents Introduces a benchmark evaluating coding agent misbehaviors across code generation, file management, and system environment integrity. Ranked #7New testPreprint8/10 suggested | LLM evaluation and benchmarks via arxiv · 15 Oct, 06:17 | On-topic Presents an evaluation benchmark and taxonomy for coding agent behaviors. | |
| Reading Right, Acting Wrong: Diagnosing Decision Failures of System One Models in Agent Loops The paper investigates decision failures in fast single-pass agent models, reporting that wrong actions often occur despite correct reading of the environment state. Ranked #8Surprising resultPreprint8/10 suggested | LLM evaluation and benchmarks via arxiv · 15 Oct, 06:17 | On-topic Introduces a benchmark pairing state probes with actions to evaluate agent decision-making. | |
| Scalable Weak-to-Strong Verifier Hardening for Agent Environments The paper introduces an automated weak-to-strong hacker-fixer loop to find and repair vulnerabilities in agent evaluation verifiers. Ranked #9New way to see a problemPreprint8/10 suggested | LLM evaluation and benchmarks via arxiv · 15 Oct, 06:17 | On-topic Addresses verifier vulnerability and benchmark accuracy in evaluating LLM agents. | |
| Can an Agent Learn to Predict Its Own Failure? Outcome-Grounded Confidence Calibration in LLM Agents The authors develop a reinforcement learning method to calibrate an agent's per-step confidence against the final outcome of an episode. Ranked #10New way to see a problemPreprint8/10 suggested | LLM evaluation and benchmarks via arxiv · 15 Oct, 06:17 | On-topic Develops calibration evaluation and scoring methods for multi-step agent rollouts. | |
| Agreement Without Alignment: Label-Free Rubric Clarification Increases LLM-Judge Consensus but Not Agreement with Humans Evaluates a label-free rubric clarification method, finding it increases consensus among LLM judges without improving agreement with human evaluations. Ranked #11Surprising resultPreprint8/10 suggested | LLM evaluation and benchmarks via arxiv · 12 Oct, 16:28 | On-topic Analyzes LLM-as-a-judge consensus, rubric clarification, and alignment with human evaluations. | |
| Cited but Not Consulted: A Counterfactual Audit of Legal Chain-of-Thought Faithfulness Audits legal chain-of-thought faithfulness by substituting cited authorities and observing whether model verdicts causally change. Ranked #12Surprising resultPreprint8/10 suggested | LLM evaluation and benchmarks via arxiv · 12 Oct, 16:28 | On-topic Audits benchmark faithfulness and evaluation reliability using counterfactual authority perturbations. | |
| What Counts as the Same Task? Executable Contracts for Robustness Evaluation of Tool-Using Agents The paper proposes executable contracts to verify that adversarial benchmark edits preserve the core task definition when evaluating tool-using agents. Ranked #13New way to see a problemPreprint8/10 suggested | LLM evaluation and benchmarks via arxiv · 15 Oct, 06:17 | On-topic Analyzes flaws in agent robustness evaluation benchmarks and proposes executable contracts. | |
| Specification-Based Testing of Large Language Models The paper introduces a testing framework that turns documented model safety claims into specifications to find counterexamples via mutated prompts. Ranked #14Surprising resultPreprint8/10 suggested | LLM evaluation and benchmarks via arxiv · 15 Oct, 06:17 | On-topic Introduces a specification-driven testing framework to rigorously evaluate whether benchmarks test documented claims. | |
| Measuring the Inference Energy Efficiency of Open-Weight LLMs across Benchmarks, Reasoning Modes, and Hardware The paper measures inference energy efficiency across 86 open-weight language models, showing large efficiency gaps among models of similar capability. Ranked #15Surprising resultPreprint8/10 suggested | LLM evaluation and benchmarks via arxiv · 15 Oct, 06:17 | On-topic Proposes methodology to measure energy efficiency alongside standard benchmark scores across models. | |
| The Silent Failures in Agentic Data Science This paper analyzes methodological failure modes in data science agents, such as data leakage and invalid evaluation, beyond simple code accuracy. Ranked #16Surprising resultPreprint8/10 suggested | LLM evaluation and benchmarks via arxiv · 15 Oct, 06:17 | On-topic Introduces an extensible test suite evaluating methodological failures in agentic systems. | |
| Do Factuality Methods Make LLMs More Factual? Reproducibility and Fair Comparison The authors reproduce and compare five language model factuality alignment methods on a standardized benchmark across identical base models and evaluation protocols. Ranked #17Surprising resultPreprint8/10 suggested | LLM evaluation and benchmarks via arxiv · 15 Oct, 06:17 | On-topic Provides reproducible evaluation revealing major benchmarking inconsistencies in factuality methods. | |
| Cheap, Consistent, Calibrated, but Biased? Typed Decision Models as Judges The paper evaluates typed decision models against standard language model judges to measure how emotional tone and biases affect their evaluation accuracy. Ranked #18Surprising resultPreprint8/10 suggested | LLM evaluation and benchmarks via arxiv · 15 Oct, 06:17 | On-topic Analyzes biases, calibration, and reliability in LLM-as-a-judge decision models. | |
| Receiver Decides Agent-to-Agent Negotiations Studies agent-to-agent negotiations over money, finding that the receiving agent's characteristics primarily dictate the final transfer amounts. Ranked #19Surprising resultPreprint8/10 suggested | LLM evaluation and benchmarks via arxiv · 15 Oct, 06:17 | On-topic Designs a round-robin evaluation tournament measuring agent compliance and refusal in negotiations. | |
| SpecGuardBench: Can Coding Agents Construct Reliable Validation Harnesses from Natural Language Specifications? This paper introduces a benchmark to test if coding agents can build reliable validation test harnesses from natural language specifications. Ranked #20New testPreprint8/10 suggested | LLM evaluation and benchmarks via arxiv · 15 Oct, 06:17 | On-topic Introduces SpecGuardBench, a benchmark assessing agent-constructed validation harnesses across real repositories. | |
| ANCHOR: Assessing Naturalistic Constraint Hierarchies and Ordering in Repository-Grounded Agentic Coding This paper introduces a repository-grounded benchmark to evaluate how well coding agents follow engineering constraints and developer workflows. Ranked #21New testPreprint8/10 suggested | LLM evaluation and benchmarks via arxiv · 15 Oct, 06:17 | On-topic Introduces ANCHOR, a benchmark evaluating constraint hierarchies in agentic coding trajectories. | |
| Same Query, Different Answer: Run-to-Run Variance in Semantic Query Processing Measures output variability across repeated runs at temperature zero in semantic query engines backed by language models. Ranked #22Surprising resultPreprint8/10 suggested | LLM evaluation and benchmarks via arxiv · 12 Oct, 16:28 | On-topic Highlights run-to-run variance undermining the reliability of existing benchmarks and evaluation setups. | |
| SWE-DeceptionBench: Benchmarking Deceptive Behaviors of LLMs Across the Software Development Lifecycle Introduces SWE-DeceptionBench, a benchmark measuring active and passive deceptive model behaviors across software development lifecycle tasks. Ranked #23New testPreprint8/10 suggested | LLM evaluation and benchmarks via arxiv · 12 Oct, 16:28 | On-topic Introduces SWE-DeceptionBench, a new benchmark evaluating deception and trustworthiness across software lifecycle stages. | |
| All Verdicts are Not Equal: Rethinking LLM Judge Reliability Audits the reliability of LLM judges across multiple prompt formats, temperatures, and orderings, introducing a trustworthy verdict metric. Ranked #24Surprising resultPreprint8/10 suggested | LLM evaluation and benchmarks via arxiv · 12 Oct, 16:28 | On-topic Conducts a comprehensive reliability audit of LLM-as-a-judge across order bias, variance, and rubrics. | |
| Structure Tax: How Structured Output affects LLMs Performance Evaluates how schema ordering impacts LLM performance under structured output constraints, showing reasoning-first designs prevent accuracy drops. Ranked #25Surprising resultPreprint8/10 suggested | LLM evaluation and benchmarks via arxiv · 12 Oct, 16:28 | On-topic Evaluates model performance across schemas, benchmarking the accuracy and calibration impacts of structured output formats. | |
| Verdict Without the Rule: Diagnosing and Auditing Regulatory Rule Sensitivity in LLM Compliance Systems Evaluates whether LLM compliance systems adjust verdicts when governing rules are perturbed, finding many verdicts are insensitive to rule changes. Ranked #26Surprising resultPreprint8/10 suggested | LLM evaluation and benchmarks via arxiv · 12 Oct, 16:28 | On-topic Audits and evaluates LLM sensitivity and reliability on regulatory compliance benchmarks. | |
| One Word Opens the Gate: The Option-Channel Attack on Typed Decision Models as Agent Guardrails Evaluates typed decision models used as agent guardrails, uncovering vulnerabilities and asymmetric error rates in allow-or-block actions. Ranked #27Surprising resultPreprint8/10 suggested | LLM evaluation and benchmarks via arxiv · 12 Oct, 16:28 | On-topic Evaluates and benchmarks guardrail decision models across prompt injection and jailbreak tests. | |
| Safe Actions Alone Do Not Ensure Safe Agents: Identifying Unfulfilled Obligations with Guard Models Introduces a benchmark evaluating guard models on identifying unfulfilled safety obligations in language model agent trajectories alongside forbidden actions. Ranked #28New testPreprint8/10 suggested | LLM evaluation and benchmarks via arxiv · 12 Oct, 16:28 | On-topic Introduces ObligationBench, a new benchmark to evaluate guard models on agent trajectories. | |
| AI Models Can Provably Hide Arbitrary Capabilities The paper shows that backdoor mechanisms can cryptographically hide arbitrary neural network capabilities until activated by secret triggers. Ranked #29Surprising resultPreprint8/10 suggested | LLM evaluation and benchmarks via arxiv · 12 Oct, 16:28 | On-topic Reveals fundamental limits and failure modes of elicitation-based capability audits and evaluations. | |
| Lumen-bench: Cross-Lingual Safety Gaps and the Say-Do Problem in Tool-Calling Large Language Models The paper introduces a multilingual benchmark evaluating safety and discrepancies between verbal refusals and tool actions in tool-calling language models. Ranked #30New testPreprint8/10 suggested | LLM evaluation and benchmarks via arxiv · 15 Oct, 06:18 | On-topic Introduces Lumen-bench, a multi-lingual benchmark evaluating tool-calling refusal behavior across 30 models. | |
| Trace Evolution Bench: Measuring Whether Agents Improve from Their Own Environment Traces This paper introduces a benchmark to measure whether autonomous coding agents truly improve their policies from interaction traces. Ranked #31Surprising resultPreprint8/10 suggested | LLM evaluation and benchmarks via arxiv · 15 Oct, 06:17 | On-topic Introduces Trace Evolution Bench, a protocol measuring whether agents truly improve from traces. | |
| Language Is a Security Surface for Agent Guards The paper evaluates 19 safety guards across multiple non-English languages to measure how translation alters step-level safety decisions for agents. Ranked #32Surprising resultPreprint8/10 suggested | LLM evaluation and benchmarks via arxiv · 17 Oct, 06:17 | On-topic Introduces LinguaGuardBench to evaluate agent guard consistency across different user languages. | |
| Mean Accuracy Is the Wrong Estimand for a Release Decision The paper shows mean benchmark accuracy misses tail risks in fine-tuned models, proposing recipe certification cards for deployment decisions. Ranked #33New way to see a problemPreprint8/10 suggested | LLM evaluation and benchmarks via arxiv · 15 Oct, 06:18 | On-topic Proposes recipe cards and tail-risk bounds to replace misleading mean accuracy metrics. | |
| Clinical value claims in multimodal AI outpace supporting evidence The paper audits multimodal clinical machine learning studies, finding that incomplete unimodal baselines frequently inflate claimed gains. Ranked #34Surprising resultPreprint8/10 suggested | LLM evaluation and benchmarks via arxiv · 15 Oct, 06:18 | On-topic Audits benchmarking practices and inflated value claims in clinical multimodal AI models. | |
| When Protection Overrides Consequences: Understanding Protection Mode in Large Language Model Decision-Making The paper introduces a benchmark to study why frontier models prioritize protecting subjects over preventing larger physical property damage. Ranked #35Surprising resultPreprint8/10 suggested | LLM evaluation and benchmarks via arxiv · 15 Oct, 15:06 | On-topic Introduces ProtectionBench to evaluate model risk assessments and safety-driven decision tradeoffs. | |
| Evaluation Awareness Beyond English: Do LLMs Act Differently When Tested in Indic Languages? The authors evaluate how language models alter their refusal behavior when informed they are being monitored in English and five Indic languages. Ranked #36Surprising resultPreprint8/10 suggested | LLM evaluation and benchmarks via arxiv · 17 Oct, 06:17 | On-topic Investigates evaluation awareness and reliability issues in multilingual LLM safety testing. | |
| Knowing the User Is Not Knowing What We Established: Grounding Awareness in Personal Agents The authors introduce a benchmark to evaluate whether personal agents can distinguish accepted user arrangements from unacknowledged proposals. Ranked #37Surprising resultPreprint8/10 suggested | LLM evaluation and benchmarks via arxiv · 15 Oct, 06:17 | On-topic Introduces a controlled benchmark evaluating how accurately personal agents distinguish accepted from unacknowledged proposals. | |
| Chain of Thought Monitoring Can Fail Even When CoT Is Necessary This paper shows chain-of-thought monitoring can fail even when reasoning is necessary, proposing a multi-level hierarchy to measure monitorability. Ranked #38Surprising resultPreprint8/10 suggested | LLM evaluation and benchmarks via arxiv · 15 Oct, 06:17 | On-topic Introduces a framework and hierarchy to measure when chain-of-thought monitoring fails. | |
| Seeing Through the Forgery Is Not Enough: Regressive Sycophancy under Fabricated Evidence in Medical Vision-Language Models The authors show that medical vision-language models frequently abandon correct clinical answers when prompted with fabricated medical documents. Ranked #39Surprising resultPreprint8/10 suggested | LLM evaluation and benchmarks via arxiv · 15 Oct, 06:17 | On-topic Tests and measures failure modes and sycophancy in medical vision-language models. | |
| Can Agents Forget When Asked? A Benchmark for Memory Unlearning in LLM Agents The authors introduce Bemula, a benchmark designed to evaluate how accurately language model agents erase specific user memories upon request. Ranked #40New testPreprint8/10 suggested | LLM evaluation and benchmarks via arxiv · 15 Oct, 06:17 | On-topic Introduces a benchmark specifically measuring memory unlearning in agent systems. | |
| Compositional RL: Evaluating and Training LLMs for Collaborative Reasoning The paper studies how much models contribute to multi-model teams, showing individual test scores do not correlate well with collaborative performance. Ranked #41Surprising resultPreprint8/10 suggested | LLM evaluation and benchmarks via arxiv · 15 Oct, 06:17 | On-topic Evaluates how well standalone benchmarks reflect LLM performance in collaborative systems. | |
| Executable Prefix Values for Locating Agent Failures The paper evaluates judge labels on agent steps by restarting failed trajectories from intermediate states to identify true failure points. Ranked #42Surprising resultPreprint8/10 suggested | LLM evaluation and benchmarks via arxiv · 15 Oct, 06:17 | On-topic Audits LLM-as-a-judge failure attribution methods and shows their diagnostic limitations. | |
| CWE-bench: Refocusing Agentic Security from Exploitation to Defense The paper presents CWE-bench, a benchmark of 120 held-out software tasks to evaluate AI agents on finding and repairing security vulnerabilities. Ranked #43New testPreprint8/10 suggested | LLM evaluation and benchmarks via arxiv · 15 Oct, 06:17 | On-topic Introduces CWE-bench, a benchmark evaluating agentic code auditing and repair capabilities. | |
| When LLM Agents See a Conflict but Still Proceed: Cross-Domain Dependencies in Network Slicing Shows that language agents often ignore identified cross-domain dependencies, causing unintended disruptive actions in network management. Ranked #44Surprising resultPreprint8/10 suggested | LLM evaluation and benchmarks via arxiv · 15 Oct, 06:17 | On-topic Evaluates ten models across providers in a controlled environment. | |
| Same Operation, Different Permission: Evaluating Whether Guardrails Honor Authorization The paper introduces a benchmark to evaluate whether language-model guardrails properly distinguish authorized from unauthorized actions across identical operations. Ranked #45New testPreprint8/10 suggested | LLM evaluation and benchmarks via arxiv · 15 Oct, 06:17 | On-topic Introduces a benchmark evaluating whether guardrails accurately enforce authorization decisions. | |
| AuthGateBench: Benchmarking LLMs as Authorization Gate Introduces a policy-guided benchmark evaluating language models acting as authorization gates for tool-using coding agents. Ranked #46New testPreprint8/10 suggested | LLM evaluation and benchmarks via arxiv · 15 Oct, 06:17 | On-topic Introduces AuthGateBench to systematically benchmark authorization gating in language models. | |
| When Agent Evaluation Becomes Unreliable: An Empirical Study of Oracle Failures and Configuration-Dependent Measurement in an LLM Agent Benchmark The paper audits agent evaluation benchmarks for software defects, showing that broken reward functions and configuration shifts substantially distort reported success rates. Ranked #47New testPreprint8/10 suggested | LLM evaluation and benchmarks via arxiv · 15 Oct, 06:17 | On-topic Empirically reveals benchmark oracle defects and configuration-dependent measurement shifts in agent evaluation. | |
| The Anatomy of a Blind Spot: What an LLM Judge Must Be Shown, and How Much The paper tests what context judge models require, showing that omitting source constraints degrades the quality of iterative feedback loops. Ranked #48Surprising resultPreprint8/10 suggested | LLM evaluation and benchmarks via arxiv · 15 Oct, 06:17 | On-topic Investigates input dependencies and failure modes when using LLMs as evaluators. | |
| TRAP: Tool-result Replay Attack for Probing Misbinding in LLM Agents Presents a benchmark and attack method to evaluate failures caused by tool results misbinding to incorrect execution contexts in LLM agents. Ranked #49Surprising resultPreprint8/10 suggested | LLM evaluation and benchmarks via arxiv · 15 Oct, 06:17 | On-topic Presents ReplayBench, a benchmark for evaluating tool-result misbinding in agents. | |
| agent-risk: Risk Assessment for Changes to AI Agents Provides a framework for assessing risks of proposed changes to deployed AI agents by selectively replaying execution logs and saved states. Ranked #50New way to see a problemPreprint8/10 suggested | LLM evaluation and benchmarks via arxiv · 15 Oct, 06:17 | On-topic Presents evaluation bounds and offline log replay methods for assessing agent performance shifts. | |
| Uncovering Robustness Weaknesses in LLM Agents via Persona Evolution with Task-Integrity Constraints This paper introduces TIPER, a method that evolves challenging user personas to test language model agent robustness without invalidating underlying tasks. Ranked #51New testPreprint8/10 suggested | LLM evaluation and benchmarks via arxiv · 15 Oct, 06:17 | On-topic Introduces a method to benchmark and evaluate interactive LLM agent robustness under realistic user personas. | |
| WildTrace: Evaluating Training Data Detection in Realistic Large-Scale LLM Post-Training This paper introduces WildTrace, a benchmark evaluating training data detection methods across realistic, multi-stage language model post-training pipelines. Ranked #52Surprising resultPreprint8/10 suggested | LLM evaluation and benchmarks via arxiv · 15 Oct, 06:17 | On-topic Introduces a benchmark to evaluate training data and contamination detection methods. | |
| MedHarnessBench: A Unified Benchmark for Healthcare Agent Evaluation The paper presents an evaluation benchmark assessing seven clinical process dimensions for healthcare agents across longitudinal workflows. Ranked #53New testPreprint8/10 suggested | LLM evaluation and benchmarks via arxiv · 15 Oct, 06:17 | On-topic Presents MedHarnessBench to benchmark healthcare agent decision processes across executable clinical tasks. | |
| Agent-Canary-Mas: End-to-End Evaluation of Protocol Risk in Long-Horizon Multi-Agent Systems The paper introduces an evaluation framework measuring how communication protocol exploits propagate into task failures across multi-agent systems. Ranked #54New testPreprint8/10 suggested | LLM evaluation and benchmarks via arxiv · 15 Oct, 06:17 | On-topic Provides a comprehensive benchmark framework assessing protocol risks in multi-agent setups. | |
| FDE-Bench: Evaluating End-to-End Delivery from Underspecified Real-World Business Requests The paper introduces a benchmark based on enterprise projects to evaluate agent performance on clarifying underspecified requirements and iterative optimization. Ranked #55New testPreprint8/10 suggested | LLM evaluation and benchmarks via arxiv · 15 Oct, 06:17 | On-topic Introduces FDE-Bench to evaluate agent performance on underspecified business requests. | |
| A Strong Harness Narrows the Model Gap: A Comparative Evaluation for Legislative Document-to-XML Conversion The paper evaluates harnesses for legislative document conversion, showing structured multi-agent pipelines narrow performance gaps between models. Ranked #56Surprising resultPreprint8/10 suggested | LLM evaluation and benchmarks via arxiv · 15 Oct, 15:06 | On-topic Provides a comparative evaluation of harness engineering versus base models. | |
| Lost in the Fallback: How Reliably Do Typed Decision Models Use Fallback Options? This paper evaluates how reliably typed decision models switch to fallback options and remain consistent under equivalent decision rules. Ranked #57Surprising resultPreprint8/10 suggested | LLM evaluation and benchmarks via arxiv · 15 Oct, 15:06 | On-topic Introduces the Paired Fallback Robustness Test to evaluate decision model consistency on fallback rules. | |
| AgentCIScope: Grounded Contextual Integrity Evaluation for LLM Agents The paper introduces an executable evaluation platform and benchmark to measure contextual integrity and privacy compliance in multi-step language model agents. Ranked #58New testPreprint8/10 suggested | LLM evaluation and benchmarks via arxiv · 15 Oct, 15:06 | On-topic Introduces a standardized evaluation platform and benchmark measuring agent-level contextual integrity. | |
| Likelihood Is Not Authorship: Can We Reliably Measure Human Contribution in AI-Assisted Writing? The paper reveals vulnerabilities in prompt-predictability metrics used to estimate human contribution in AI-assisted writing. Ranked #59Surprising resultPreprint8/10 suggested | LLM evaluation and benchmarks via arxiv · 15 Oct, 15:06 | On-topic Exposes critical vulnerabilities and benchmark manipulation in metrics estimating human contribution in AI writing. | |
| Beyond Downstream QA: Rethinking How We Evaluate Agent Memory The paper presents a benchmark that isolates agent memory evolution from downstream retrieval and generation failures. Ranked #60Surprising resultPreprint8/10 suggested | LLM evaluation and benchmarks via arxiv · 17 Oct, 06:17 | On-topic Introduces a diagnostic benchmark for evaluating agent memory stores. | |
| Stronger Judges Do Not See More: Deployment-Integrity Blindness in LLM Evaluation Introduces a benchmark measuring how output-only model judges fail to detect tampered runtime environments in agent deployments. Ranked #61Surprising resultPreprint8/10 suggested | LLM evaluation and benchmarks via arxiv · 17 Oct, 06:17 | On-topic Introduces a benchmark evaluating how output-only LLM judges fail to detect runtime deployment tampering. | |
| When Judges Agree and Are Wrong: Consistency-Accuracy Dissociation as a Blind Spot for Label-Free Evaluation Examines failure modes in label-free evaluation where models reach high consistency across perturbations while generating identical incorrect answers. Ranked #62Surprising resultPreprint8/10 suggested | LLM evaluation and benchmarks via arxiv · 17 Oct, 06:17 | On-topic Critiques label-free evaluation and consistency-based scoring, uncovering blind spots where models agree incorrectly. | |
| INFUSE: A Unified Surface Benchmark for Indirect Prompt Injection in Agentic Systems The paper introduces an end-to-end benchmark to evaluate indirect prompt injection risks across long-horizon agent workflows. Ranked #63New testPreprint8/10 suggested | LLM evaluation and benchmarks via arxiv · 17 Oct, 06:17 | On-topic Introduces a standardized benchmark of 40 tasks to evaluate agent vulnerabilities across frontier models. |
1–60 of 587Next →