TreasureX Researchcs.AISecurity AI Agents: Cost Matters More Than Just Success Rate
Paul Kassianik, Blaine Nelson, Yaron Singer · July 16, 2026
A new paper argues that evaluating AI security agents solely by their success rate misses the real-world cost of each reasoning step, tool call, and data query. Researchers tested offensive (red-team) and defensive (blue-team) agents on standard benchmarks, but instead of reporting only peak performance, they compared models at fixed cost levels. They found that offensive CTF tasks benefit from more compute—open-weight models can approach proprietary ones cost-effectively. But defensive SOC investigations don't scale the same way; success depends more on disciplined tool use and selective enrichment than raw reasoning budget. The authors propose cost-aware, SOC-native evaluations to show which models are practically useful today. This matters for investors: cost-efficient defensive agents could reduce operational security expenses, while offensive agents may become cheaper to deploy. The paper is available at https://arxiv.org/abs/2607.15263.
Educational content only, generated with AI assistance from public research abstracts. Not investment advice. Thank you to arXiv for use of its open access interoperability.