Real-SWE is a recently introduced benchmark aimed at evaluating leading AI models on private, real-world enterprise codebases. Unlike synthetic or expert-designed tasks, Real-SWE uses actual production code licensed from companies, reflecting the complexity and context engineers face in their daily work. The benchmark focuses on tasks involving proprietary systems, business-critical operations such as billing and tax calculations, and company-specific coding conventions.
The benchmark’s tasks require AI agents to navigate private codebases that are not publicly accessible, making them out of distribution for most existing models. These tasks have direct business impact, such as ensuring correct billing or migrating customers, and often span multiple services and infrastructure components. Real-SWE also emphasizes the importance of understanding company-specific engineering patterns, which many current AI models struggle to handle.
To mirror real engineering environments, Real-SWE uses native testing harnesses and evaluates AI models in combination with these tools rather than in isolation. The benchmark includes a variety of technologies and services such as AWS emulators, Kubernetes, PostgreSQL, Node.js, and more, reflecting the diverse ecosystems found in enterprise settings.
Codebases selected for the benchmark come from companies with substantial user bases and production workloads, including a fintech platform processing over 100,000 bank statements and an enterprise AI sales platform. Tasks often require changes across multiple files and demand that AI agents understand existing business logic and maintain system stability.
Analysis of early results shows that AI models frequently fail due to missed requirements and difficulty triaging complex systems. Short rollout attempts under 10 minutes failed 71.4% of the time, indicating challenges even with limited changes. The benchmark also highlights that current AI models have not been trained on these private codebases, as 99% of enterprise tokens remain inaccessible to them.
Real-SWE offers a more realistic measure of AI coding capabilities in enterprise contexts, emphasizing economic viability and real-world impact. The benchmark’s findings suggest that while AI shows promise, significant gaps remain before these models can reliably perform software engineering tasks in complex, proprietary environments.