PyPI Resolves File Hosting Errors Affecting Package Downloads in August 2026
3 min read
During August 2026, users of the Python Package Index (PyPI) experienced intermittent 502 and 503 errors when downloading files from files.pythonhosted.org. These errors caused failures in package installations over a period of about two weeks. The issue was traced to a single cache node within Fastly's content delivery network (CDN), which serves PyPI's file hosting.
The root cause involved two separate problems. First, a canary deployment within Fastly's network introduced a misconfiguration at one Seattle-based cache node. This mismatch between caching and routing configurations led to persistent 502 errors for traffic routed to that node. Fastly has since removed PyPI traffic from its canary testing to prevent similar incidents.
Second, the PyPI team discovered and fixed several longstanding bugs in their Fastly configuration related to origin fallback and HTTP range request handling. Specifically, the fallback mechanism intended to switch from Backblaze B2 storage to Amazon S3 when B2 failed was not always triggered correctly, resulting in additional 503 errors. Additionally, segmented caching used to optimize partial content requests did not fully support certain range request types, causing some clients to receive 501 errors.
PyPI's file hosting architecture uses Fastly as a CDN fronting three backends: Backblaze B2 for cost-effective egress, Amazon S3 Glacier Instant Retrieval as a fallback origin, and a third backend for non-package file requests. Files are immutable and cached with long expiration times to optimize delivery. When B2 is unavailable, Fastly should fall back to S3, but the bugs prevented this failover in some cases.
The timeline of events began on August 15, when Fastly identified the problem at a single cache node. User reports helped narrow down the issue, leading to configuration fixes and removal of the affected routing override by August 20. Further improvements to logging and request handling were deployed over the following days. The final patch addressing the canary deployment bug was applied on August 28, after which downloads returned to normal.
The incident highlighted the importance of detailed user reports and monitoring in diagnosing complex CDN-related issues. It also exposed subtle bugs in range request handling and fallback logic that had gone unnoticed until triggered by real-world traffic patterns.
Looking ahead, the Python Software Foundation is expanding its infrastructure engineering team to improve PyPI's reliability and responsiveness. The foundation also encourages users to enable caching in continuous integration workflows to reduce repeated downloads, which can mitigate the impact of transient outages.
This episode underscores the challenges of operating a large-scale, globally distributed package repository and the value of collaboration between service providers and the open-source community in maintaining critical infrastructure.