Why ChatGPT's storage layer moved from Python to Rust
Published: · Sergey Zhadov
ChatGPT does not answer from one magic computer. Opening a conversation, checking settings or signing in can trigger many data lookups, so a slow storage layer quickly makes the whole product feel broken.
Scale starts with tiny data calls
Habitat supports products used by more than one billion people each week across almost 40 regions. A login or a new chat can require routing, authorization and multiple reads. A tiny delay multiplied across that traffic becomes a serious mess.
The platform grew from a Python library attached to one database into a distributed system. Demand expanded more than tenfold year over year for three years. At that point another cache and a hopeful shrug were not enough.
Why the library became a service
When storage logic lived in client libraries, a routing change had to reach dozens of services. A rollback could restore an old buggy client and undo the fix. A central service created one place for deployment, observability, permissions and limits.
This is boring engineering, which is exactly why it works. Teams no longer need their own copy of critical storage logic, and applications do not receive direct access to every backend resource.
Python hit tail latency before raw throughput
The Python service handled more than 20 million requests per second at its peak. The nastier problem was scheduling delay in asyncio, CPU-heavy work, background jobs and uneven connection pools. Average latency looked fine while rare slow calls ruined the user experience.
Engineers measured event-loop delay, limited concurrency per process, added jitter to background refreshes and tuned connection reuse. Without that work, an overloaded process could attract even more traffic and dig its own hole.
Rust was the final step, not magic
In the second quarter of 2026, two engineers used Codex and GPT-5.5 to rewrite the service in Rust. It now carries 95 percent of production traffic and is reported to use CPU six times and memory fifteen times more efficiently.
That does not mean every Python project should be rewritten tomorrow. First make the workload predictable, add useful telemetry and identify the actual bottleneck. Otherwise you only recreate the same architectural crap in a faster language.
