快速扩展在线存储以服务超过10亿ChatGPT用户

每个 OpenAI 产品都依赖于快速、可靠的数据访问，无论是有人登录、查看其 Codex 设置，还是在 ChatGPT 中开始新对话。这些操作中的每一个都可能需要许多单独的数据查找，产品才能做出响应。如果这些请求很慢，产品就会感觉慢。如果这些请求失败，产品就会完全停止工作。

Habitat 是我们构建的在线存储平台，使 OpenAI 产品能够快速、可靠地访问所需信息。Habitat 现在每秒处理超过 7000 万个请求，支持每周超过 10 亿人使用的产品，覆盖近 40 个地理区域。Habitat 最初在 2023 年 DevDay 上为支持 GPTs 而推出，起初只是一个连接到单一数据库的简单 Python 客户端库。如今，它是一个复杂的分布式系统，服务于超过 500 PB 的数据。

图 01 · 什么是 Habitat？

在线存储平台

Habitat 是我们构建的在线存储平台，使 OpenAI 产品能够快速、可靠地访问所需信息。

• 请求
• 响应
• 变更（CDC）

在这个规模上构建和运营基础设施绝非易事，但也并非特别具有挑战性。让我们的情况独一无二的是，我们不得不以史无前例的速度扩展，以支持惊人的用户增长和产品需求，同时还要构建一个成熟的平台。通常，系统工程师会按 10 倍规模进行构建，并希望它能维持几年，同时为下一个 10 倍做准备。而在我们的情况下，过去三年我们同比增长了超过 10 倍。因此，构建和运营 Habitat 是一系列战术决策和排序：在最低层级理解每个组件，以尽可能从现有技术栈中榨取更多价值，同时抵御存储和计算容量短缺，为基础性投资争取时间。

• 7000 万+

每秒请求数

• 10 亿+

每周人数

• 500 PB+

数据

随着 OpenAI 的发展，Habitat 也必须随之发展：首先变得足够可靠，以承载关键任务产品流量；然后足够快，以服务全球用户；最后，能够娴熟地大规模运营。本文是关于我们如何扩展在线存储的两部分系列文章中的第一篇。在本文中，我们将分享 Habitat 如何演变、我们为什么将其从库转变为服务，以及我们如何将一个用不常见服务栈语言——Python——编写的服务扩展为可靠的存储平台层。

在未来的文章中，我们将详细讨论我们如何实现大规模多租户可靠性、我们优化读取性能的分层策略，以及我们如何扩展与 Azure Cosmos DB 的合作，以可靠地应对前所未有的需求。

什么是 Habitat？

Habitat 始于一个简单的想法：产品工程师不应该需要思考数据库管理。Habitat 最初在 2023 年 DevDay 上为支持 GPTs 而推出，是一个与 ChatGPT 主服务器交互的小型 Python 库。它支持一小组操作，这些操作在底层映射到数据库应用程序 Azure Cosmos DB。

这个库的职责是为产品团队提供一种简单的方式来存储和检索数据，而无需掌握底层细节。Habitat 负责必要的工作：弄清楚涉及什么类型的数据、数据应该来自哪里（或去哪里）、请求是否被允许，等等。

产品工程师无需关心模式查找、路由、授权、加密、序列化、请求塑形和连接池。他们甚至不需要考虑数据来自哪里：Azure Cosmos DB、缓存，还是其他类型的存储。

图 02 · Habitat 服务

简化的 Habitat 请求流程

通过将存储逻辑解耦为独立服务，我们为部署、可观测性和平台增强建立了单一控制点。

• 请求
• 响应

这个 Python 库运行良好，Habitat 在 OpenAI 的产品工程师中迅速得到采用，尽管并没有任何协同一致的中央推动来让团队停止使用自助式 Postgres 和 Azure Cosmos DB。

随着产品需求演变，产品开发者甚至可以轻松地向共享库添加对客户端缓存、压缩或加密等功能的支持。

构建服务以更好地支持多个复杂产品

到 2025 年中期，Habitat 作为客户端实现已经达到了极限。随着 Habitat 层变得更加复杂，并且 OpenAI 的服务数量增加，向后兼容的协议变更已经变得不可行。

在一个实例中，我们希望将最关键的数据集迁移到一组区域分布的 Azure Cosmos DB 账户，以缩小任何单一区域故障的影响范围。进行这项更改需要向客户端引入额外的路由逻辑，并将其置于功能标志之后禁用，确保它推广到所有客户端，然后启用该功能标志。

协调数十个服务的部署，并与每个团队合作推广，花费了数天时间。在启用之前，我们意识到我们想引入一些影子流量，以确保分片逻辑正确。这又花了几天时间才推广完成。修复我们意识到不正确的问题？又花了几天。最终，我们准备好启用该标志，结果其中一个团队因无关原因将其服务回滚到之前有缺陷的客户端，导致我们如此努力想要避免的故障。

对客户端库的更改需要在数十个服务之间进行复杂协调，这一过程被证明越来越脆弱、低效，并且容易发生运营故障。为了减少未来部署中的这种运营扩散，我们决定将 Habitat 拆分为自己的服务。

通过将存储逻辑解耦为独立服务，我们为部署、可观测性和平台增强建立了单一控制点。我们不再管理碎片化的更新，而是可以集中实施改进，为每个 OpenAI 产品带来即时收益。

集中式服务还为我们提供了一个单一咽喉点，以提供最强的数据安全和隐私原语。Habitat 服务是我们能够集中执行访问控制策略、进行审计日志记录，并限制对 Azure Cosmos DB 等底层存储资源访问的地方。Habitat 在保护用户数据和防止来自外部、内部和代理行为者的未授权访问方面发挥着关键作用。

大规模启动 Python 服务

我们知道我们需要一个服务，但我们还不想从 Python 迁移出去，即使 Python 作为服务会带来额外开销。与本地库执行相比，将 Python 用于高吞吐量服务会增加网络延迟，并增加大量 CPU 和内存扩展成本。此外，我们认识到 Python 的低效在 100 倍规模下将不可接受，这使得最终重写几乎不可避免。

然而，我们将此视为技术债务的战略性引入。当时我们的首要目标并非成本或资源优化，而是解放产品开发者并实现平台稳定性。通过短期内接受Python服务的性能权衡，我们得以优先处理更为紧迫的挑战，建立核心API，并构建起强大的基础设施。

我们还做了一个有根据的赌注，即我们自身编码模型的快速进步将在未来简化技术路径。我们押注于当完全迁移出Python成为必要时，Codex和GPT将使这一迁移变得可行。这一赌注最终被证明是正确的。

以Python服务运行Habitat在性能上并非最优，但却是必要的选择。Python让我们能够迅速行动，但这并不意味着我们可以不顾后果，接受明显更差的延迟。当平均用户请求导致数百次数据库调用时，用户感受到的是最慢的那次数据库调用。我们发现，在如此规模下运行Python服务的主要挑战在于管理这些尾部延迟。

跟踪asyncio延迟

Asyncio有助于Python并发执行I/O密集型工作负载，但无法绕过Python的GIL提供CPU并行处理。除了I/O密集型的请求代理外，Habitat还承担了许多CPU密集型职责和后台任务：路由、压缩、加密、校验和、下游健康检查、请求影子测试和对冲。

由于我们的服务中有如此多的CPU密集型工作负载和后台任务，asyncio调度延迟很容易主导尾部请求延迟。在为初始服务发布进行调优之前，我们在p99及更高延迟的请求追踪中看到，尽管下游存储响应迅速，但请求经常因等待负责解析响应的协程重新调度而停滞。

图03 · 跟踪asyncio延迟

并发不等于CPU并行

Python asyncio允许并发处理请求，但同一时间只有一个请求能在CPU线程上执行。当有大量CPU工作要做时，这对请求延迟有很大影响。

CPU请求/响应处理 Python网络读写 等待Cosmos

低CPU工作

简短的Python步骤；I/O等待重叠

高CPU工作

长时间的Python步骤使已准备好的响应等待

示意时间 0.0 / 40 示意单位

对于OpenAI的Python服务，我们发现除了测量内存、CPU、网络和磁盘使用的标准利用率和饱和度指标外，监控asyncio循环及其繁忙程度，并据此进行调优也至关重要。

通过定期调度后台任务并记录预期与实际执行时间之间的差值，我们能够实时实证测量事件循环调度延迟。在高利用率下，有许多昂贵的任务，即使每个进程的并发请求数量适中，也足以产生显著的调度抖动，高达数百毫秒，在某些边缘情况下甚至可达数秒。

因此，我们采取的策略是让每个进程仅处理少量并发请求，而是大规模扩展Python工作进程的数量。

减少特性标志配置中的尾部延迟

在初始服务发布中，我们通过实时服务CPU剖析发现了一个导致高asyncio延迟（以及由此产生的高尾部延迟）的根本原因：通过Statsig（一种管理特性标志的工具，可用于运行A/B测试等）定期解析特性标志配置的JSON。

默认情况下，Statsig配置为每分钟无抖动地轮询刷新配置，且配置包含了每个服务的所有生产规则。在其他地方，我们做出了一项架构决策，即每个Pod运行最多8个Python进程，以提高CPU使用率并提供更低的延迟。两者结合意味着，每分钟每个Pod都会有那么一刻，其所有工作进程都会暂停处理正在进行的请求，转而花费CPU周期解析一个巨大的配置文件。

一旦CPU剖析帮助我们找到了问题的根本原因，修复就很简单了：部署更小的针对性配置，延长刷新间隔，并为这类后台任务添加一些抖动。

平衡负载和管理连接池

为了保持低asyncio延迟，跨服务器进程的良好请求负载均衡也至关重要；如果不进行调优，连接池可能会与此背道而驰。

使用客户端连接池时，一个执行许多并发请求的单一客户端进程可能只建立少量服务器连接，结果将其所有负载发送到少数几个进程。在调整负载均衡方式之前，我们的服务利用率差异很大，一些尾部进程处理的并发请求数量是平均值的5-10倍。

我们在一次偶然事件中发现了这一点：尽管停止了使部分服务过载的客户端，一部分进程在突发流量过后仍长时间处于降级状态。事实上，我们注意到这些进程经历了失控的退化，接收越来越多的请求，直到我们重启它们。一旦某个Pod过载，某些行为会将更多流量固定到过载的Pod上。这是一类我们一些队友从以往工作中熟知的故障：亚稳态故障⁠(在新窗口中打开)。

我们怀疑连接池是罪魁祸首，并通过限制最大连接重用持续时间来测试这一怀疑，这确实限制了退化，证实了我们的调查方向。进一步调查发现，Python的aiohttp TCPConnector默认采用LIFO连接重用：最近返回的连接被选用于下一个请求。这通常是一个合理的默认值：重用最近的连接可以让为处理突发流量而创建的额外连接空闲超时，减少了维护额外连接的开销。但在这种情况下，它为我们制造了一个亚稳态故障。在请求突发期间，对较慢的过载服务器的请求较晚将连接返回到池中，因此更频繁地被后续请求选中，逐渐将更多流量集中到已经挣扎的Pod上。将连接池修补为使用FIFO重用打破了这一反馈循环，甚至减少了我们的稳态请求方差。

图04A · 客户端连接池

LIFO将新工作送回慢进程

在一波请求之后，较慢的服务器最后将连接返回到池中。LIFO促使更多工作集中到那些相同的较慢服务器上。

初始突发流量到达 A、B 以及较慢的进程 C。

图 04B · 客户端连接池

FIFO 打破了连接复用的反馈循环

FIFO 在突发流量后维持更多活跃连接，但在所有服务器之间公平地平衡工作负载。

初始突发流量到达 A、B 以及较慢的进程 C。

如今，我们主要依赖 Istio 和 Envoy 在 OpenAI 基础设施中提供连接池和更好的服务器负载感知均衡策略，从而完全避免这个问题。

避免淹没下游资源

为降低 asyncio 延迟而进行调优，同时拥有如此多的 Python 进程，其副作用之一是：海量连接极易压垮下游依赖（即所谓的“惊群效应”）。

一次常规的每日部署——如果没有调优为慢速——可能因连接循环而导致显著的 CPU 抖动。或者，连接泄漏可能通过饱和 NAT 网关来瘫痪网络。这些问题对其他服务来说也并不罕见，但由于进程数量多了一个数量级，触发阈值被显著降低，常常会饱和那些客户端在稳定状态下仅凭纯吞吐量并不预期需要处理的网络相关资源。

我们还依赖 Envoy 来最大化连接扇入。我们用它把 Python 的 HTTP/1 连接升级为 HTTP/2，以利用多路复用，然后对这些连接进行池化并延长连接生命周期。Envoy 还为我们提供了一个集中位置来实现速率限制和熔断器，而这些如果在每个独立的 Python 进程中实现，效果会差得多。

图 05 · 连接扇入

相同的请求，更少的连接

连接池和 HTTP/2 连接多路复用有助于减少下游的连接负载。

请求 响应 空闲 keep-alive

为什么 Habitat 做得更少

我们能够将 Python 扩展到如此程度的一个原因，是 Habitat 受限的 API，它使请求成本保持可预测。Habitat 不允许客户端构造任意 SQL 查询——这些查询可能导致大表扫描或跨多表连接——而是暴露一个简单的 NoSQL API。缺少强大的 API 是 Habitat 设计中的一个明确权衡。

我们的目标是优化简单、可预测、恒定工作量的请求。根据我们的经验，这类系统在扩展上要容易得多，也更难出错或被误用。具有不可预测扇出的请求在运维上是危险的：它们会使隔离、负载均衡复杂化，并引入难以扩展的延迟悬崖，对服务及其客户端都是如此。

在我们迁移到 Habitat 和 Azure Cosmos DB 之前，OpenAI 的大部分在线数据存储在 Postgres 上。当时，很容易审查所有查询和模式变更，以确保它们在发布到生产环境之前行为良好，并且是针对已索引数据运行的。随着团队和产品的增长，这很快变得难以管理，并且成为频繁故障的原因——热路径上一个昂贵的新查询就能拖垮数据库。

这里的问题在于成本不平衡：编写 SQL 查询很便宜、很容易，但运行起来可能很昂贵、很困难。在 Habitat 中，我们避免了这一点，并使昂贵查询在客户端极其明显。不存在可能使 Habitat 过载的无界查询，而复杂连接和图遍历要求产品团队承担一部分繁重工作，这有助于整体上优化出更高效的设计。

Habitat 暴露一个 NoSQL API，其建模围绕客户端定义的对象和边类型，灵感来自 TAO⁠（在新窗口中打开）。客户端预定义对象和边以及它们彼此之间的关系，但不定义每种类型的内容。由此产生的关系类似于图，但 Habitat 本身除了查询特定对象的直接边之外，并不支持典型的图遍历查询。

我们对这个图进行分区，使每个对象及其对应的边共置于同一个存储级分区中，但我们不会在数据库层面协同努力，将对象与其边所指向的远程对象共置。结果是，该模型很容易分区以实现水平扩展，但图遍历效率低下，因为对象之间的任意一次跳转都可能需要从存储在不同区域的两个完全不同的 Azure Cosmos DB 账户中获取数据。

对于具有更复杂查询需求的客户端，我们确实通过 Rockset 提供 Habitat 的离线二级视图。我们使用变更数据捕获（CDC）将近乎实时地将在线存储中的变更流式传输到隔离的 Rockset 实例。每个客户端团队负责为自己的复杂查询需求扩展其 Rockset 实例。

这种 Rockset 配置给我们的客户端带来了额外摩擦，但我们认为在当前这个时刻这是正确的权衡：让简单查询成为默认选项，同时为需要复杂查询的人提供一个逃生通道。这种设计将我们的在线存储与读密集型的分析和搜索工作负载隔离开来。

从 Python 迁移到 Rust

将 Python 重写推迟一年，使我们能够在超高速增长期间专注于更紧迫、更有影响力的挑战。随着平台日趋成熟、增长持续加速，并且它已成为 OpenAI 按核心数计算的第二大服务（按我们的 Envoy 占用计算为第四大），终于到了超越 Python 的时候。在巅峰时期，Python 帮助我们每秒服务超过 2000 万次请求。

在 2026 年第二季度，仅用 2 名工程师、Codex 和 GPT‑5.5，我们就能够用 Rust 重写整个服务。这个新的 Rust 服务目前处理我们 95% 的生产请求；我们将在未来几周内完全弃用 Python。我们的数据显示，Rust 服务的 CPU 效率比 Python 版本高 6 倍，内存效率高 15 倍，并且平均延迟和尾部延迟都显著更低。我们计划在未来的博客中分享更多经验。

优化我们的数据库层：Azure Cosmos DB

Python——以及现在的 Rust——服务只是 Habitat 的一个方面。在本系列的第二部分中，我们将解释我们如何快速扩展在线存储以服务超过 10 亿 ChatGPT 用户，届时我们会讨论存储层，以及 Habitat 如何每秒服务超过 500 PB 的数据和超过 7000 万次请求。

Every OpenAI product depends on fast, reliable access to data, whether someone is logging in, checking their Codex settings, or starting a new conversation in ChatGPT. Each of those actions may require many separate data lookups before the product can respond. If those requests are slow, the product feels slow. If those requests fail, the product stops working entirely.

Habitat is the online storage platform we built so OpenAI products can quickly and reliably access needed information. Habitat now handles more than 70 million requests every second, supporting products used by over 1 billion people each week, across almost 40 geographic regions. Habitat first launched to support GPTs at DevDay 2023, starting as a simple Python client-side library connected to a single database. Today, it’s a complex distributed system that serves more than 500 petabytes of data.

Figure 01 · What is Habitat?

Online storage platform

Habitat is the online storage platform we built so OpenAI products can quickly and reliably access needed information.

• Request
• Response
• Changes (CDC)

Building and operating infrastructure at this scale is no easy feat, but also not particularly challenging. What made our situation unique is the unprecedented rate at which we’ve had to scale to support staggering user growth and product demand while simultaneously building out a mature platform. Often, system engineers build for 10x scale, and hope for it to hold for a few years while preparing for the next 10x. In our case, we've grown more than 10x year-over-year for the last three years. As a result, building and operating Habitat has been a series of tactical decisions and sequencing: understanding each component at the lowest level to squeeze as much juice out of our existing stack, while fending off storage and compute capacity crunches to buy time for foundational investments.

• 70M+

requests per second

• 1B+

people each week

• 500 PB+

data

As OpenAI grew, Habitat had to grow with it: first by becoming reliable enough for mission-critical product traffic, then fast enough for global users, and finally, to deftly operate at massive scale. This post is the first in a two-part series on how we scaled online storage. In this post, we’ll share how Habitat evolved, why we turned it from a library into a service, and how we stretched a service written in an uncommon serving stack language—Python—into a reliable storage platform layer.

In a future post, we’ll go into detail about how we made multi-tenancy reliability at scale, our layered strategy for optimizing read performance, and how we scaled our partnership with Azure Cosmos DB to reliably handle unprecedented demand.

What is Habitat?

Habitat started from a simple idea: product engineers shouldn’t need to think about database management. Habitat first launched to support GPTs at DevDay 2023 as a small Python library that interacted with ChatGPT’s main server. It supported a small set of operations that mapped under the hood to the database application, Azure Cosmos DB.

The library’s job was to give product teams a simple way to store and retrieve data without needing to master the underlying details. Habitat took care of the necessary work: figuring out what kind of data was involved, where it should come from (or go), whether the request was allowed, and so on.

Product engineers need not concern themselves with schema lookup, routing, authorization, encryption, serialization, request shaping, and connection pooling. They didn’t even need to consider where the data comes from: Azure Cosmos DB, caches, or other types of storage.

Figure 02 · Habitat service

Simplified Habitat request flow

By decoupling the storage logic into a standalone service, we established a single point of control for deployments, observability, and platform enhancements.

• Request
• Response

This Python library worked well and Habitat saw rapid adoption among product engineers at OpenAI, despite no concerted central push away from using self-serve Postgres and Azure Cosmos DB.

As product needs evolved, it was even easy for product developers to add to the shared library support for features like client-side caching, compression, or encryption.

Build a service to better support multiple, complex products

By the middle of 2025, Habitat had reached its limits as a client-side implementation. As the Habitat layer had grown more complex and OpenAI’s services count increased, backward-compatible protocol changes had become infeasible.

In one instance, we wanted to reduce the blast radius of any single region outage for our most critical data sets by migrating them to a set of regionally distributed Azure Cosmos DB accounts. Making this change required introducing extra routing logic into the client, disabled behind a feature flag, ensuring it rolled out to all clients, and then enabling the feature flag.

Coordinating deployments across dozens of services and working with each team to roll it out took days. Before enabling this, we realized we wanted to introduce some shadowing to ensure the sharding logic would be correct. That took another couple of days to roll out. A bug fix for something we realized was incorrect? Another couple of days. Eventually, we were ready to enable the flag, only for one of the teams to roll back their service for unrelated reasons to a previously buggy client, causing the outage we had worked so hard to avoid.

Changes to the client library necessitated complex coordination across dozens of services, a process that proved increasingly brittle, inefficient, and susceptible to operational failures. To reduce this operational fan out for our future deployments, we decided to pull Habitat into its own service.

By decoupling the storage logic into a standalone service, we established a single point of control for deployments, observability, and platform enhancements. Instead of managing fragmented updates, we could implement improvements centrally, providing immediate benefits to every OpenAI product.

A centralized service also gives us a single chokepoint to provide the strongest data security and privacy primitives. Habitat service is where we can centrally enforce access control policies, perform audit logging, and limit access to underlying storage resources like Azure Cosmos DB. Habitat plays a critical role in protecting user data and preventing unauthorized access from external, internal, and agent actors.

Launching a Python service at scale

We knew we needed a service, but we didn’t want to migrate off Python quite yet, even with Python’s additional overhead as a service. Using Python for a high-throughput service increased network latency and added substantial CPU and memory scaling costs compared to local library execution. Moreover, we recognized that the inefficiencies of Python would not be acceptable at 100x scale, making an eventual rewrite almost certain.

However, we viewed this as a strategic incursion of technical debt. Our primary objective then was not cost or resource optimization, but rather unblocking product developers and achieving platform stability. By accepting the performance trade-offs of a Python service in the short term, we were able to prioritize more immediate challenges, establish our core APIs, and build out a robust infrastructure.

We also made a calculated wager that the rapid advancement of our own coding models would simplify the technical path in the future. We bet that by the time a full migration off Python was required, Codex and GPT would make that migration achievable. That bet eventually proved correct.

Running Habitat as a Python service would be suboptimal, performance-wise, but a necessary choice. Python lets us move quickly, but it doesn’t mean we could throw caution to the wind and accept meaningfully worse latencies. When the average user request results in hundreds of database calls, the slowest database call is the one the user feels. We’ve found the main challenge in running a Python service at this scale is in managing these tail latencies.

Tracking the asyncio delay

Asyncio helps Python execute I/O-bound workloads concurrently, but does not help work around the Python GIL and provide CPU parallelism. In addition to I/O-heavy request proxying, Habitat handles many CPU-heavy responsibilities and background tasks: routing, compression, encryption, checksumming, downstream health checking, request shadowing, and hedging.

With so many CPU-heavy workloads and background tasks in our service, asyncio scheduling delay can easily dominate tail request latency. Before tuning for our initial service launch, we saw in traces for requests with p99 and higher latency that while downstream storage responded quickly, requests frequently stalled while waiting for the responsible coroutine to be rescheduled to parse the response.

Figure 03 · Tracking the asyncio delay

Concurrency is not CPU parallelism

Python asyncio allows concurrent request processing, but only a single request executes on the CPU thread at a time. This has high impact on request latencies when there's a lot of CPU work to be done.

CPU request/response processing Python network read/write Wait for Cosmos

Low CPU work

Brief Python steps; I/O waits overlap

High CPU work

Long Python steps keep ready responses waiting

Illustrative time 0.0 / 40 illustrative units

For Python services at OpenAI, we find that in addition to measuring standard utilization and saturation metrics on memory, CPU, network, and disk usage, it is critical to also monitor the asyncio loop and how busy it is, then tune accordingly.

By periodically scheduling background tasks and recording the delta between expected and actual execution time, we are able to empirically measure event loop scheduling delay in real time. At high utilization, with many expensive tasks, even modest numbers of concurrent requests per process are enough to produce significant scheduling jitter, up to hundreds of milliseconds and in some edge cases several seconds.

As a result, we resort to keeping each process serving only a small number of concurrent requests and instead massively scale out the number of Python worker processes.

Reducing a tail latency in our feature flag configurations

In our initial service launch, we discovered through live service CPU profiling one root cause of high asyncio delay (and resulting high tail latencies): periodic JSON parsing of our feature flag configurations via Statsig (a tool that manages feature flags, and can be used to run A/B tests and more).

By default, Statsig was configured to poll for refreshed configs every minute with no jitter, and the config included every production rule across every service. Elsewhere, an architectural decision was made to run up to 8 Python processes per pod to push higher CPU usage and provide lower latencies. Combined, this meant that every minute each pod would have some moment where all of its workers stalled processing in-flight requests and instead would spend their CPU cycles parsing a giant configuration file.

The fix was straightforward once CPU profiling helped us root cause the issue: deploy a smaller targeted config, lengthen the refresh interval, and add some jitter to background tasks like these.

Balancing loads and managing connection pools

In order to maintain low asyncio delay, it is also critical to maintain good load balancing of requests across server processes; connection pooling can end up being antithetical to this without tuning as well.

With client-side connection pooling, a single client process that does many concurrent requests might establish only a handful of server connections and as a result send all of its load to only a handful of processes. Prior to adjusting how we do load balancing, our service had a wide variance of utilization with some tail processes serving 5-10x the number of concurrent requests as the average.

We discovered this in a chance incident where, despite stopping the client that was overloading part of our service, a subset of processes remained degraded well past the bursty traffic. In fact, we noticed those processes experienced runaway degradation, receiving increasingly more requests until we restarted them. Once a pod became overloaded, some behavior was pinning more traffic onto the overloaded pod. This was a class of failures some of our teammates were well-acquainted with from prior work: metastable failure⁠(opens in a new window).

We suspected the connection pool was to blame and tested this suspicion by capping max connection reuse duration, which indeed limited the degradation and confirmed our investigation direction. Further investigation found that Python’s aiohttp TCPConnector defaults to LIFO connection reuse: the most recently returned connection is selected for the next request. This is normally a reasonable default: reusing recent connections allows the extra connections created to handle bursty traffic to idle timeout, reducing overhead to maintaining extra connections. In this case, it created a metastable failure for us. During a burst of requests, requests to slower overloaded servers returned connections to the pool later and were therefore selected more frequently by subsequent requests, gradually concentrating more traffic on the pods already struggling. Patching the connection pool to use FIFO reuse broke this feedback loop and even reduced our steady state request variance as well.

Figure 04A · Client-side connection pooling

LIFO sends new work back to the slow process

After a burst of requests, slower servers return connections to pool last. LIFO encourages more work to concentrate on those same slower servers.

An initial burst reaches A, B, and the slower process C.

Figure 04B · Client-side connection pooling

FIFO breaks the connection-reuse feedback loop

FIFO maintains more active connections after a burst, but balances workloads fairly across all servers.

An initial burst reaches A, B, and the slower process C.

Today, we mostly depend on Istio and Envoy to provide connection pooling and better server-load-aware balancing strategies throughout OpenAI infrastructure and avoid this problem altogether.

Avoiding flooding downstream resources

One side effect of tuning for low asyncio delay and having so many Python processes is that it becomes very easy to overwhelm downstream dependencies with the vast number of connections (known as a “thundering herd”).

A regular daily deployment—if not tuned to be slow—can cause significant CPU churn from connection cycling. Or a connection leak can take out the network by saturating the NAT gateway. These are not uncommon problems for other services too, but the threshold for triggering is lowered significantly by having an order of magnitude more processes, often saturating network related resources that clients are not expecting to need to handle in a steady state based on pure throughput alone.

We also rely on Envoy to maximize our connection fan-in. We use it to upgrade Python’s HTTP/1 connections to HTTP/2 to take advantage of multiplexing and then to pool those connections and extend connection lifetimes. Envoy also gives us a central place to implement rate limits and circuit breakers that would be less effective in each standalone Python process.

Figure 05 · Connection fan-in

The same requests, fewer connections

Connection pooling and HTTP/2 connection multiplexing help reduce connection load on downstreams.

Request Response Idle keep-alive

Why Habitat does less

One reason we could scale Python this far was Habitat’s constrained API, which keeps request cost predictable. Rather than allowing clients to construct arbitrary SQL queries that could result in large table scans or joins across many tables, Habitat exposes a simple NoSQL API. The lack of a powerful API is an explicit tradeoff in Habitat’s design.

We aim to optimize for simple, predictable, constant-work requests. In our experience, these systems are substantially easier to scale and difficult to get wrong or misuse. Requests with unpredictable fanout are operationally dangerous: they complicate isolation, load balancing, and introduce latency cliffs that are hard to scale for both the service and its clients.

Before we moved to Habitat and Azure Cosmos DB, most of OpenAI’s online data was stored on Postgres. At that time it was easy to review all query and schema changes to make sure they were well-behaved and operated against indexed data before shipping to production. As the team and products grew, this quickly became unmanageable and was a frequent cause of outages where a single expensive new query on a hot path took out the database.

The problem here is in cost imbalance: it is cheap and easy to write SQL queries that are expensive and hard to run. In Habitat, we avoid this and make expensive queries exceedingly obvious client-side. There are no unbounded queries that can overload Habitat and complex joins and graph traversals require product teams to do some of the heavy-lifting which helps overall optimize for more efficient designs.

Habitat exposes a NoSQL API modeled around client-defined object and edge types, inspired by TAO⁠(opens in a new window). Clients predefine objects and edges and how they relate to each other, but not the content of each type. The resulting relationships resemble a graph, but Habitat itself does not support typical graph traversal queries outside of querying direct edges of a particular object.

We partition this graph so that each object and its corresponding edges are colocated in a storage-level partition, but we make no concerted database-level effort to colocate objects and the remote objects to which their edges point. The result is that the model easily partitions for horizontal scalability, but graph traversals are inefficient since any particular hop between objects may require fetching from two entirely different Azure Cosmos DB accounts stored in different regions.

For clients with more complex querying needs, we do provide an offline secondary view of Habitat exposed via Rockset. We use change data capture (CDC) to stream changes from the online storage out to isolated Rockset instances in near-real-time. Each client team is responsible for scaling their own Rockset instance for their complex querying needs.

This Rockset provisioning introduces extra friction to our clients, but we think is the right tradeoff to make at this particular moment: making simple queries the default while providing an escape hatch for those who need complex queries. This design isolates our online storage from read-heavy analytical and search workloads.

Migrate from Python to Rust

Deferring a Python rewrite for a year allowed us to focus on more urgent and impactful challenges during our hypergrowth. With the platform maturing and our growth continuing to accelerate, and being the second largest service by core count at OpenAI (and fourth for our Envoy footprint), it was finally time to move past Python. At its peak, Python helped us serve more than 20 million requests every second.

In Q2 2026, with just 2 engineers, Codex, and GPT‑5.5, we were able to rewrite the entire service in Rust. This new Rust service is now handling 95% of our production requests; we’ll be deprecating Python entirely in the coming weeks. Our data shows the Rust service is 6x more CPU efficient and 15x more memory efficient than the Python version, with significantly lower average and tail latencies. We plan to share more learnings in a future blog.

Optimizing our database layer, Azure Cosmos DB

The Python—and now Rust—service is only one facet of Habitat. In part II of this series explaining how we rapidly scaled our online storage to serve over 1 billion ChatGPT users, we’ll talk about the storage layer and how Habitat serves more than 500 petabytes and over 70 million requests every second.