Real-SWE is a recently introduced benchmark that measures the performance of advanced AI models on private, real-world enterprise codebases. Unlike traditional benchmarks that rely on synthetic or public datasets, Real-SWE uses tasks sourced directly from licensed production codebases of real companies. These tasks reflect the complexity and context engineers encounter in their daily work, including proprietary systems, business-critical operations, and company-specific coding patterns.
The benchmark emphasizes several key challenges for AI coding agents. These include navigating private codebases inaccessible on the public internet, handling changes with direct business impact such as billing and tax calculations, and adapting to unique company conventions. For example, one task involves fixing invoice billing to correctly apply taxes based on varying business rules and external tax authority services.
Real-SWE tasks require agents to work across multiple technologies and tools, including AWS emulators, Docker, Kubernetes, various databases, and programming languages like TypeScript, Go, Python, and Node.js. The benchmark also evaluates models in realistic development environments, using native harnesses that simulate how engineers interact with code and infrastructure.
The codebases selected for Real-SWE come from companies with significant user bases and complex production workloads, such as a fintech platform processing over 100,000 bank statements and enterprise AI sales platforms. This ensures that tasks are grounded in genuine business needs rather than artificial scenarios.
Analysis of initial results reveals that AI models often fail due to missed requirements and difficulties understanding company-specific coding patterns. The benchmark highlights that current frontier models are still far from meeting the standards expected in real-world enterprise engineering.
Real-SWE provides valuable insights into the limitations of AI coding agents when applied to proprietary, complex environments. It underscores the importance of evaluating AI tools on realistic tasks to better understand their practical capabilities and areas needing improvement.