Large language models have dramatically improved the ability to generate correct code, but producing formally correct code is only part of the challenge. Code quality also depends on factors like unnecessary abstractions, duplication, and poor design choices, which can lead to what is often called "sloppiness." This issue becomes critical in large projects where millions of lines of code are added monthly, making it difficult for humans to maintain oversight.
Attempts to measure code sloppiness face significant hurdles. Automated evaluation by AI models themselves has proven unreliable, as models can inconsistently judge code quality and are sensitive to superficial changes like renaming. Human review remains the most reliable method but is not scalable for large datasets or continuous AI training.
Research has explored simpler quantitative metrics. One effective measure is tracking changes in lines of code (LOC), which can indicate unnecessary code growth. More sophisticated metrics introduced by the SlopCodeBench study include verbosity, which captures duplicated or overly verbose code, and erosion, which measures the concentration of code complexity in a few large functions.
Comparisons between AI-generated code and established human-written repositories reveal that AI code tends to be roughly twice as verbose and complex. This suggests that despite passing functional tests, AI-generated code often accumulates poor design choices over iterative development cycles.
The SlopCodeBench evaluation also simulates real-world coding by erasing model context between development rounds. This approach shows that even state-of-the-art models struggle to maintain code quality over time, with strict test pass rates dropping to zero.
These findings highlight why human intuition and taste remain essential in assessing code quality. Future research directions include examining function coupling, code churn, and cohesion to develop more nuanced evaluation methods. The ongoing challenge is to balance automated code generation with meaningful quality control to ensure maintainable and efficient software development.