OpenAI has just shared impressive figures about Habitat, the internal storage platform serving the company’s entire system. The platform has grown 10x year-over-year, reaching over 20 million requests per second while still running on Python.
The figure of 20 million requests/second is not a simple API endpoint. This is a storage platform serving everything from storing model weights and managing training data to processing inference outputs. When a user sends a prompt to ChatGPT, the related data has likely passed through Habitat at least once.
What is surprising is that Habitat achieved this figure while still running on Python. Python is notoriously slow in I/O-heavy tasks, but OpenAI optimized the architecture to the point where it could handle workloads at an unprecedented scale. However, there is a physical limit: at 20 million req/s, Python starts to become a bottleneck.
That is why the team decided to rewrite it in Rust. The language provides near-native performance, safe memory management without a garbage collector, and is well-suited for systems needing to handle millions of concurrent connections. This is also an increasingly popular trend in the industry: using Python for prototyping and research, then rewriting in Rust or Go for production at large scale.
OpenAI is not the only company following this path. Discord rewrote its storage system from Go to Rust, Dropbox moved its metadata service from Python to Rust, and Cloudflare uses Rust for most of its edge infrastructure. But the scale of Habitat—20 million req/s before the rewrite—shows that the compute demands of AI inference are at a level that most traditional systems cannot keep up with.
The question now is how high Habitat will reach after the rewrite. If the 10x annual growth rate continues, a figure of 200 million req/s could appear in the coming year, at which point even Rust might require additional advanced optimization techniques.