Benzi has released a comprehensive benchmark comparing its AI-native code harness to other popular code harnesses. The test set includes 24 real GitHub issues spanning 10 different programming languages. The benchmark evaluates performance using three key metrics: lines of code read, wall clock time, and cost per fix.
The difficulty of each bug is measured independently using Claude Code's turn count, ensuring a consistent scale across all harnesses. Lines read counts only actual file-read calls, excluding search operations like grep or shell commands. This metric helps illustrate how much source code each harness needs to process as bugs increase in complexity.
Wall clock time measurements include the total runtime without subtracting the initial per-repository index build, providing a raw view of execution time. Some harnesses did not complete all tests, resulting in missing data points for specific bugs.
Cost analysis is based on published per-token rates for the AI models used by each harness. The benchmark highlights that harnesses using the DeepSeek model are significantly cheaper—roughly an order of magnitude—than those using the Sonnet model. Additionally, the cost slope for Claude Code increases faster with bug difficulty compared to other harnesses.
Overall, the benchmark demonstrates that Benzi's AI-native harness resolves 78.2% of 500 real issues at under 10 cents per fix, indicating competitive efficiency and cost-effectiveness. These results provide valuable insights for developers and organizations evaluating AI-powered code intelligence tools, especially as bug complexity grows and cost considerations become critical.