快速扩展在线存储以服务超过10亿ChatGPT用户（第二部分）

所有 OpenAI 产品都需要快速、可靠地访问数据，这些数据由我们的存储平台 Habitat 提供。它必须在毫秒级内响应遍布 40 个地理区域的超过 10 亿用户。我们的基础设施面临着一个非凡的挑战：在持续提升可靠性、性能和产品开发者速度的同时，支持爆炸式增长和新的用例。解决这一挑战需要重新思考我们在线存储平台的设计和运营方式。我们不断评估权衡，遵循这样的模式：首先变得足够可靠以承载关键任务的产品流量，然后足够快以服务全球用户，最后，在超大规模下灵活运营。

这个由两部分组成的系列文章解释了我们将 Habitat 从一个简单的客户端库和数据库发展为服务超过 500 PB 数据的全球分布式存储平台的过程。第一部分聚焦于 Habitat 从 Python 客户端到服务的演变。它还涵盖了使我们能够将一个基于 Python 的服务推到远超许多人认为可行的程度的技术。

在这篇文章，即第二部分中，我们将介绍我们如何扩展底层存储组件：Azure Cosmos DB。通过与 Azure 团队的紧密合作，我们达成了一种可行的多租户方案，即共同管理的容器，在超大规模全球范围内具有出色的读取性能。

图 01 · 什么是 Habitat？

在线存储平台

Habitat 是我们构建的在线存储平台，使 OpenAI 产品能够快速可靠地访问所需信息。

• 请求
• 响应
• 变更（CDC）

大规模实现可靠的多租户

随着公司规模扩大，基础设施团队通常会从运行单租户系统演变为运行多租户系统。单租户模式下，每个产品界面团队维护自己的系统，这减少了爆炸半径，并提供了简单的所有权边界和工作负载隔离。随着团队和产品界面的增长，这种模式的运行成本变得高昂。单租户使每个产品团队负责容量规划，并且通常会导致大量的过度配置。

相反，多租户将控制权分配给集中式数据库团队，他们拥有更好的资源调配知识和经验。但多租户也带来了自身的问题，例如吵闹邻居问题，即一个租户消耗了异常大份额的资源，从而损害了其他租户的利益。

在 Habitat 中，我们必须权衡所有这些取舍，以设计一个大规模下可靠、高效的存储系统。如今，Habitat 是许多 OpenAI 产品的多租户数据库层，但我们经历了一个循环的旅程：从使用一个多租户 Azure Cosmos DB 容器，到配置单租户后端容器，再回到默认使用多租户容器。我们在 Habitat 上的多租户故事仍处于非常早期的阶段：我们支持大量用例，但通过尽可能利用 Azure Cosmos DB 的内置原语，我们得以推迟在这一领域的投入。

Azure Cosmos DB 容器的简化入门

在介绍我们的 Azure Cosmos DB 之旅之前，一些基本背景：

• 一个 Azure Cosmos DB 账户定义了一组容器和复制策略

• 每个容器被配置为一定数量的吞吐量，以每秒请求单位（RU/s）计

• 每个容器内的数据以典型的按指定逻辑分区键分片的方式存储，并对每个逻辑分区键的存储和吞吐量有文档化的限制

• Azure Cosmos DB 处理将逻辑分区键分片到较少的后端物理分区上，当存储超过阈值或容器需要更多物理分区来维持配置的吞吐量时，将其拆分为多个物理分区

第一阶段：依赖 Azure Cosmos DB 原语

我们最初允许任何客户端负载通过，依赖所声明的每个逻辑分区键 10K RU/s 的限制来提供背压。在我们最初开始时，我们还在每个账户中只使用一个默认的 Azure Cosmos DB 容器。这足够好用，但事实证明由于几个原因存在问题：

逻辑分区键限制

由于早期的 API 选择，我们将逻辑分区键的选择直接暴露给客户端。结果，我们在不同产品用例中经常在常用的分区键上发生冲突，例如 user_id 或 workspace_id。由于这些用例共享同一逻辑分区键上的 Azure Cosmos DB 限制，该设计限制了给定用户的整体吞吐量。

在迁移到 Habitat 和 Azure Cosmos DB 之前，我们在扩展 Postgres 时面临的一个常见问题是，来自非常热门的组织或热键——一个租户产生不成比例的高负载——的许多请求可能会拖垮整个数据库。我们花费了大量集体努力来缓存这些键，并在缓存未命中时构建租约，以便由此产生的缓存未命中负载不会冲击底层数据库。

图 02a · 逻辑分区键限制

不同的客户端，相同的分区键

吵闹邻居

虽然文档化的数据大小和吞吐量限制适用于逻辑分区键，但限制实际上是在物理分区上执行的。在实践中，这意味着逻辑分区无法达到 10,000 RU/s。相反，它们与共置于该物理分区上的每个其他逻辑分区键共享物理分区的限制。

这使得吵闹邻居能够影响许多不相关的逻辑分区键，针对小众产品功能数据的工作负载会干扰核心登录和注册流程。

图 02b · 吵闹邻居

不同的用户，相同的物理分区

弱分区

给定物理分区的吞吐量限制也是可变的，取决于分配给 Azure Cosmos DB 容器的总吞吐量。容器的配置吞吐量大致均匀地分布在该容器的物理分区之间。

随着数据量增长，物理分区分裂，增加总分区数，而总配置吞吐量保持不变。结果是，随着数据大小增长，每个分区的有效吞吐量限制缩小。

对于我们最存储密集、类 blob 的用例，这造成了一种故障模式，即仅存储增长就会降低活动工作负载的吞吐量容量，即使请求速率和使用量保持稳定。

图 02C · 限制低于需求

当分区达到其存储限制时，它会分裂。每个子分区获得一半的吞吐量容量。

第二阶段：尝试工作负载隔离和单租户

独立的容器有助于部分解决这些分区键限制和吵闹邻居问题。通过使用独立的容器，每个容器可以被分配自己独立的物理分区，只有该容器自身变化的工作负载才可能成为吵闹邻居或在分区键空间上发生重叠。不利的一面是，配置新容器存在非零开销——每个容器都需要个性化的容量规划——因此我们只想在合理的服务或用例边界上创建容器。

我们开始为需要强隔离的最关键新用例使用独立的 Azure Cosmos DB 容器。结合一个通过将每个容器预配的总吞吐量与其物理分区数同步来管理弱分区的脚本，我们基本避免了所面临的所有问题。

然而，由于我们是从一个默认容器起步的，许多遗留的关键用例不幸被困在那里，没有进行数据迁移。为了保护仍然依赖共享默认容器中数据的关键服务，我们经常建议团队为新服务或主要产品功能创建独立容器。随着我们扩展，默认容器上的压力增大，生产事故也随之增多。我们开始为任何新服务或大型产品功能预配容器，只是为了保护默认容器免受更多负载。

这使我们免于许多全站范围的中断，将爆炸半径限制在功能或服务降级中断的范围内。然而，预配新容器的开销最终追上了我们：

缺乏稳定的聚合增长

在旧的共享默认容器中，任何特定的新用例从 0 QPS 增长到 100 万 QPS 都不太可能显著改变总聚合用量。然而，在隔离的容器上，一个新用例从 0 增长到 100 万 QPS，可能意味着比仅归因于内部 dogfooding 的基线高出 1,000 倍的负载。

使用专用容器后，团队现在需要记得为他们的发布预先预配容器到预期的吞吐量水平。产品团队经常将功能发布到生产环境后忘记预先扩展，而我们则在发布日坐在 SEV 会议室里试图在最后一刻扩展容器。

这是多租户系统核心的工作负载混合优势，而我们为了通过这种容器策略获得更好的隔离性而失去了它。

图 03 · 缺乏稳定的聚合增长

相同的工作负载，不同的余量

负载测试期间 1K RU/s → 发布时 1M RU/s

容量规划

随着时间的推移，监控专用容器的使用情况也需要对流量何时转移或增长有深入的产品知识。

我们可以提供一个基本的控制平面脚本，根据增长趋势为每个容器扩展预配吞吐量，但如果没有稳定的聚合增长，正确扩展这些容器实质上让 OpenAI 的每个产品团队都参与到容量规划中。

这在组织层面无法扩展，因为每个服务的负责团队对其 Azure Cosmos DB 预配投入的勤勉程度差异很大。

第三阶段：默认使用多租户，并具备数据移动能力

随着组织的发展，这些运营问题变得难以扩展。没有一个单独的中央团队来做规划，导致许多团队规划不善。因此，我们最终转向多租户解决方案，并与 Azure 团队密切合作，通过共享租户设计为我们最初的多租户问题设计修复方案：

热键限流

首先对最热的键进行限流，防止吵闹邻居消耗给定物理分区的所有可用吞吐量。这减少了吵闹邻居效应，即一个热键可能具有全站范围的爆炸半径。

服务质量（QoS）

请求分层 QoS 同样防止离线工作负载（如批量插入或数据提取）利用给定物理分区的所有吞吐量。通过为这些工作负载的请求分配较低的优先级，它们会首先被限流，从而防止对服务最终用户的高优先级交互式请求进行限流。

无服务器模式

Azure Cosmos DB 已经有了一种无服务器模式，可以解决我们在预配模式下遇到的弱分区问题。然而，有一个主要缺点：它仅基于存储使用量进行扩展，并将每个物理分区的 RU 使用量限制在 5K RU/s。这将解决我们的弱分区问题，但阻止我们扩展 RU 密集型用例。

Azure Cosmos DB 团队为我们构建的一种新的改进无服务器模式允许每个物理分区扩展到 10K RU/s，并在基于存储使用量的拆分之外引入了基于 RU 使用量的拆分，使我们能够停止手动监控和扩展容器。

通过这些更改，我们得以转回使用一个共享容器，这显著简化了产品团队进行任何前期或持续容量规划的需求，同时保留了工作负载之间的隔离。尽管产品开发加速，但在此后的几个月里，我们没有发生任何预配或吵闹邻居事故。

作为共同管理服务实现更好的可靠性

当公司发展到 OpenAI 的规模时，托管服务变得不那么“托管”。许多工作负载成为云提供商基础设施中的异常值。随着 OpenAI 成为 Azure Cosmos DB 的最大用户之一，成功运营它需要一种不同的模式：共享测试、共享仪表板、直接的工程渠道，以及对我们规模下才会出现的故障模式的共同所有权。如今，我们最大的 Azure Cosmos DB 账户存储了数十 PB 的数据，并服务超过 2000 万 QPS。据我们所知，我们是唯一一家在此规模上部署 Azure Cosmos DB 的公司。

2024 年底，一次区域中断清楚地表明了为什么这种合作是必要的。当时，停电袭击了我们最大的 Azure Cosmos DB 账户之一的主 Azure 区域。在随后的故障转移演练中，我们意识到我们最大的账户需要几个小时才能故障转移到次要区域，这是我们团队和 Azure Cosmos DB 团队都没有在我们规模上测试过的事情。

大约一个月后，在将这些账户故障转移回原始区域时，我们触发了另一次严重中断。到原始区域的连接蜂拥而至，压垮了原始区域的 Azure Cosmos DB 网关。网关模式 Azure Cosmos DB 客户端处理故障转移方式的一个怪癖导致所有客户端立即建立连接并向原始区域发送请求，尽管只有一部分物理分区已移回该区域。

在每次故障中，OpenAI 和 Azure 都运行了各自的事件管理流程，确定了多个根本原因并部署了修复措施。但这些故障是我们团队与 Azure Cosmos DB 团队在运营和测试方式之间更大系统性问题的症状。起初，利用托管服务使我们能够快速行动并专注于业务目标。在规模化时，托管服务变得不那么“托管”，需要在从设计到运营的各个层面与服务提供商深度集成。

如今，Habitat 和 Azure 工程团队联合监控 Habitat 的每一层，从我们的服务层一直到 Azure Cosmos DB 的物理存储层。我们针对生产规模但非生产的 Azure Cosmos DB 账户运营和测试变更。我们一起测试和迭代 SDK，并确定如何扩展和调整 Azure Cosmos DB 账户规模的最佳实践。我们通过 Teams 拥有直接的工程访问权限，甚至暴露了无需经过支持层级即可直接互相寻呼的方式。

在过去一年中，我们进行了大量联合工作，以改善 Azure Cosmos DB 控制平面和网关的可扩展性，在其网关中添加负载保护和各种 Azure Cosmos DB 元数据的缓存以保护 Azure Cosmos DB 元数据分区，通过连接池和更高的连接复用减少我们 Python habitat-service 进程集群的压力，尽可能共置工作负载，并通过不同的 VPC 将关键工作负载与离线工作负载隔离。

这种程度的协同工程对于我们过去一年 Azure Cosmos DB 的稳定性至关重要。任何足够大或关键的系统都需要这种团队合作：打破所有权壁垒，跨组织边界运营。

在全球范围内提升性能

正如第一部分所述，我们的核心理念是：让它工作，让它快，让它便宜。可靠性优先，然后是性能，然后是效率。Habitat 已成为 OpenAI 的关键基础设施，因此首要任务是使其可靠。随着系统在 2025 年稳定下来并且 OpenAI 在全球扩张，延迟成为用户和产品团队的下一个约束。

计算和存储容量通常位于不同区域，使得跨区域请求不可避免。

图 04 · 在全球范围内提升性能

全球流量

计算和存储容量通常位于不同区域，使得跨区域请求不可避免。

计算 存储 示意路线 · 更长的旅程需要更多时间

计算和存储的地理位置仅用于示意目的。

延迟因数据存储位置和请求服务位置的不同而差异很大。为了使 Habitat 请求具有良好性能，产品开发者依赖于额外的客户端缓存、请求批处理、最小化数据库读写，以及尽可能将工作移出关键路径。虽然这导致了一些更有弹性应对数据库问题的有趣产品设计，但也使产品更加复杂并减慢了产品迭代：未能实现 Habitat 鼓励快速产品开发且无需理解数据库的目标。

这一直是我们 2026 年的一大重点，我们制定了一种分层策略来优化读取性能，读取请求占 Habitat 请求的 90% 以上：

在容量允许时将数据存储在用户附近

我们在全球不同 Azure 区域拥有多个 Azure Cosmos DB 账户，用于有数据驻留要求的用户。即使对于没有硬性数据驻留要求的用户，将数据共置于用户附近也能通过更低的延迟改善他们的体验。

在实践中，除了数据驻留用例之外，我们无法广泛实现这一点，因为放置受到物理云容量的限制。由于我们数据集的规模，许多区域无法容纳其按比例用户流量所需的大量存储占用。

在数据中心容量、电力或冷却有限的区域，我们有时面临直接的权衡：为共置存储预留容量以降低 Habitat 延迟，还是将该容量用于额外的服务计算并将存储请求路由到更远的地方。

因此，放置决策通常是产品权衡：A) 为不同产品和用户子集优化延迟，以及 B) 原始存储和计算容量。

仅广泛复制最小且对延迟最敏感的数据集

对于最延迟关键的工作负载，例如每个请求都需要的身份验证元数据，我们通过使用具有许多次要区域的高度复制 Azure Cosmos DB 账户来减少计算和 Habitat 之间的 RTT。一些账户复制到我们服务流量的所有 40 个 Azure 区域；其他账户至少复制到每个大洲的一个区域。

我们不得不在预期之前就在 Habitat 中构建透明的数据移动和路由层，以便我们能够在具有不同复制拓扑的账户之间移动数据集。

然而，这种方法最终也受容量限制，因此仅限于少数精选数据集。

对其他所有内容依赖集成缓存

Habitat 工作负载绝大多数是读密集型的，应用程序不断获取设置、状态和其他属性；不成比例的流量可归因于相对较小的工作集频繁访问的类型。这使得系统特别适合分布式缓存。虽然缓存在单区域基础设施中提高性能，但在跨区域工作时改进会复合，因为它可以用来“隐藏”跨区域延迟。

我们已将分布式旁路缓存层集成到 Habitat 中。它将最近访问的对象和边的副本存储在我们基础设施中靠近用户请求处理位置的 Valkey 服务器中。通常，这类缓存使用 TTL 在命中率和陈旧度之间进行权衡。较短的 TTL 减少陈旧度但削弱缓存的积极影响，因为项目很快从缓存中过期并且必须更频繁地检索。较长的 TTL 提高命中率，但在较长时间内提供陈旧数据，导致糟糕的用户体验和不可预测的产品行为。

另一种方法是在应用程序对同一底层数据执行写操作时，执行显式的失效操作。这极大地提高了缓存数据的新鲜度，但可能难以正确实现。它还仍然受制于许多边缘情况，例如部分失败，这会导致缓存陈旧。多区域部署加剧了这些问题，因为跨区域网络可靠性较低，而且缓存会与底层存储的复制竞争。用 TTL 对此进行补充可以限制最大陈旧度，但仍然受制于上文讨论的相同底层 TTL 权衡。

我们采用了更先进的失效方法，追随 Meta 的 TAO 等系统的脚步。Habitat 的集成缓存依赖通过 Azure Cosmos DB 变更源进行的变更数据捕获（CDC），在数秒内于全球范围内传播失效，从而实现高命中率，同时保持低陈旧度。我们基础设施中的一个服务消费此变更源，并在底层数据更新时对本地缓存执行失效。这会导致下次读取数据时未命中，并用最新版本的数据填充缓存。

使用 Rust 减少代理开销

我们还通过用 Rust 重写代码，显著降低了 Habitat 服务本身的延迟，从而减少了开销，Rust 比原来的 Python 性能更高。

这些变化共同将我们的整体全球读取 p50 从大约 120 毫秒降低到大约 10 毫秒，自今年早些时候我们开始关注性能以来提升了 12 倍。

图 05 · 在全球范围内提升性能

读取的 p50 延迟

三项重大设计变更改善了读取延迟

虽然缓存大幅降低了中位延迟，但最慢的请求仍然需要从底层存储获取未缓存的数据，而且往往跨越遥远区域。随着更多容量上线以及我们提高存储效率，我们计划解决一个更根本的挑战：从一开始就将数据放置在更接近其访问位置的地方。

扩展变更数据捕获

可靠且可扩展的 CDC——将数据库变更传播到下游系统——不仅是 Habitat 缓存策略中的关键组件。每秒数百万条捕获的变更，跨越数千个容器和数百万个容器分区，通过一百多个 Kafka 主题流转，还为最终一致的二级索引、数据移动管道、离线分析和其他实时产品功能提供支持。随着我们的底层基础设施在全球范围内增长，监听行变更变得越来越复杂。如果客户端想要注册一个回调，以便在其类型的行发生变更时运行，他们必须首先确定哪些 Cosmos 账户和容器包含他们的数据，哪些 CDC 服务部署正在相应区域运行并监听这些容器，并适当扩展 CDC 服务工作线程集群、Kafka 及其消费者。我们的 CDC 平台旨在为产品开发者抽象掉所有这些，转而提供一个简单的接口，让客户端注册一个他们希望在变更行上被调用的函数，而我们处理所有管道和扩展工作。

一个内部的控制平面持续确保所有所需容器都在被捕获，选择从哪些 Cosmos 区域读取，并在多个地理区域的数千个工作线程之间高效分配工作。使用副本区域和较低优先级的源读取可保护对延迟敏感的 Habitat 流量免受后台捕获工作负载的影响。在如此大规模的工作线程集群中，工作线程崩溃、容器问题、租约存储错误、无效配置和目标故障持续发生。我们强化了 CDC 工作线程生命周期和租约处理，隔离了流之间的故障，并为配置变更和发布构建了保护措施。我们利用死信队列来防止永久性投递失败造成队头阻塞。我们通过将读取器切换到备用 Cosmos 区域来容忍区域中断。在我们的规模下，我们甚至不得不修改 Azure 的 CDC SDK，通过在内存中批量处理检查点并仅定期刷新，使其检查点实现更高效。

虽然 Habitat 对 Azure Cosmos DB 的内部使用很复杂，但其价值主张一直是为产品开发者抽象掉基础设施细节，以提高产品开发速度，我们的 CDC 平台也不例外。

展望未来

一个小团队能够通过保持存储 API 刻意狭窄、继续扩展 Python 服务并优先处理最紧迫的工程挑战，来支持一个快速增长的存储平台。我们首先关注关键任务产品流量的可靠性，然后关注全球用户的读取性能，最后关注大规模运营效率。这种方法实现了快速增长和新用例，同时提高了可靠性、性能和产品开发速度。

为了为下一个十亿用户扩展 Habitat 和对话存储，我们需要解决在我们的基础设施中复制数据的困难，而且规模更大。为了预防这一点，我们一直在构建一个定制的内部艾字节级在线存储系统，针对庞大的 Habitat 数据类型进行了优化。这个正在开发中的系统结合了来自 Rockset 的 RocksDB Cloud 核心与基于 Raft 的复制。它实现了与 Azure Cosmos DB 相同的文档模型，但使用解耦存储和物理复制来减少所需的闪存容量，以换取较低的峰值 qps。

如果你有兴趣在前沿规模从事这类数据基础设施工作，查看我们的开放职位⁠。

All OpenAI products require fast, reliable access to data, provided by our storage platform, Habitat. It has to respond in milliseconds to over 1 billion users across 40 geographic regions. Our infrastructure faced an extraordinary challenge: supporting explosive growth and new use cases while continuing to improve reliability, performance, and product developer velocity. Solving that challenge required rethinking how our online storage platform was designed and operated. We constantly evaluated tradeoffs, following the pattern of: first become reliable enough for mission-critical product traffic, then fast enough for global users, and, finally, to deftly operate at massive scale.

This two-part series explains how we developed Habitat from a simple client library and database into a globally distributed storage platform serving over 500 petabytes of data. Part I focused on Habitat’s evolution from a Python client to a service. It also covered the techniques that allowed us to push a Python-based service far beyond what many would consider practical.

In this post, Part II, we walk through how we scaled the underlying storage component: Azure Cosmos DB. Through a close working partnership with the Azure team, we arrived at a viable multi-tenancy approach of co-managed containers, with excellent read performance at massive, global scale.

Figure 01 · What is Habitat?

Online storage platform

Habitat is the online storage platform we built so OpenAI products can quickly and reliably access needed information.

• Request
• Response
• Changes (CDC)

Making multi-tenancy reliable at scale

As companies scale, infra teams often evolve from running single-tenant systems to multi-tenant ones. Single-tenancy, where every product surface team maintains their own system, reduces blast radius and provides easy ownership boundaries and workload isolation. As teams and product surfaces grow, this becomes expensive to run. Single-tenancy makes every product team responsible for capacity planning and often results in significant over-provisioning.

Conversely, multi-tenancy assigns control to a centralized database team, with better knowledge and experience for provisioning resources. But multi-tenancy creates its own problems, such as noisy-neighbor issues, where one tenant consumes an unusually large share of the resources to the detriment of the other tenants.

In Habitat, we had to weigh all these tradeoffs to design a reliable, efficient, storage system at scale. Today, Habitat is a multi-tenant database layer for many OpenAI products, but we went through a circular journey from using one multi-tenant Azure Cosmos DB container to provisioning single-tenant backing containers to reverting to a multi-tenant container by default. We’re still very early in our multi-tenancy story on Habitat: we support a large set of use cases, but we’ve been able to defer investment in this area by leveraging Azure Cosmos DB’s built-in primitives wherever possible.

A simplified primer on Azure Cosmos DB containers

Before walking through our Azure Cosmos DB journey, some basic context:

• An Azure Cosmos DB account defines a set of containers and a replication policy

• Each container is provisioned for a certain amount of throughput in request units per second (RU/s)

• Data within each container is stored in a typical shard-by-specified-logical-partition-key manner with documented limits on storage and throughput per logical partition key

• Azure Cosmos DB handles sharding logical partition keys across fewer backing physical partitions, splitting them into multiple physical partitions when storage exceeds a threshold or when the container requires more physical partitions to sustain the provisioned throughput

Phase I: Relying on Azure Cosmos DB primitives

We originally allowed any client load through, relying on the stated 10K RU/s limit per logical partition key to provide back pressure. When we first set out, we also used only one default Azure Cosmos DB container in each account. This worked well enough, but proved problematic for a few reasons:

Logical partition key limits

Due to early API choices, we exposed logical partition key selection directly to clients. As a result, we have frequent collisions on commonly used partition keys, such as user_id or workspace_id, across different product use cases. Because these use cases shared Azure Cosmos DB limits on the same logical partition keys, the design constrained overall throughput for a given user.

One common problem we faced in scaling Postgres before we moved to Habitat and Azure Cosmos DB was the ability for many requests from a very hot organization or hot key—one tenant generating a disproportionately high load—to take out the whole database. Significant collective effort was spent to cache those keys and build leases into cache misses so that the resulting cache miss load does not stomp on the underlying database.

Figure 02a · Logical partition key limits

Different clients, the same partition key

Noisy neighbors

While documented data size and throughput limits apply to logical partition keys, the physical partition is actually where limits are enforced. In practice, this means logical partitions are unable to reach 10,000 RU/s. Instead, they share the physical partition’s limits with every other logical partition key colocated on that physical partition.

This allowed noisy neighbors to impact many unrelated logical partition keys, with workloads operating on data for niche product features disrupting core login and signup flows.

Figure 02b · Noisy neighbors

Different users, the same physical partition

Weak partitions

The throughput limit for a given physical partition is also variable, dependent on the total throughput assigned to the Azure Cosmos DB container. The container’s provisioned throughput is distributed roughly evenly across the physical partitions for that container.

As data volume grows, physical partitions split, increasing the total partition count while total provisioned throughput remains unchanged. The result is that the effective throughput limit per partition shrinks as data size grows.

For our most storage-heavy, blob-like use cases, this creates a failure mode where storage growth alone reduces throughput capacity for active workloads even while request rates and usage remain stable.

Figure 02C · Weak partitions

The limit falls below demand

A partition splits when it reaches its storage limit. Each child receives half the throughput capacity.

Phase II: Trying workload isolation and single-tenancy

Separate containers help partially solve these partition key limits and noisy neighbors. By using separate containers, each can be assigned its own independent physical partitions and only the varying workloads on that container itself could be noisy neighbors or overlap in partition key space. On the downside, there is non-zero overhead in provisioning new containers—each container requires individualized capacity planning—so we want to create containers only at a service or use case boundary that makes sense.

We began to use separate Azure Cosmos DB containers for the most critical new use cases requiring strong isolation. Combined with a script for managing weak partitions by synchronizing the total throughput provisioned for each container with its number of physical partitions, we mostly avoided all the problems we were facing.

However, since we started with a default container, many legacy critical use cases were unfortunately stuck there without a data migration. To protect critical services that still relied on data in the shared default container, we often advised teams to create separate containers for new services or major product features. As we scaled, pressure on the default container grew and production incidents also grew. We began to provision containers for any new service or large product feature just to protect the default container from more load.

This saved us from many site-wide outages, limiting the blast radius to feature or service degradation outages. However, the overhead of provisioning new containers eventually caught up to us:

Lack of stable aggregate growth

In the old shared default container, any particular new use case going from 0 QPS to 1 million QPS was unlikely to significantly shift the total aggregate usage. On an isolated container however, a new use case going from 0 to 1 million QPS might represent 1,000x more load than the baseline attributed to only internal dogfooding.

With dedicated containers, teams now needed to remember to pre-provision containers to the expected level of throughput for their launches. Product teams would often ship their features to production and forget to pre-scale and we’d sit in SEV rooms on launch day attempting to scale up containers last minute.

This is a core workload mixing benefit of multi-tenant systems that we lose in favor of better isolation with this container strategy.

Figure 03 · Lack of stable aggregate growth

The same workload, different headroom

1K RU/s during load testing → 1M RU/s at launch

Capacity planning

Over time, monitoring usage of dedicated containers also required intimate product knowledge of when traffic would shift or grow.

We can provide a basic control plane script to scale up the provisioned throughput for each container based on growth trends, but without the stable aggregate growth, scaling these correctly enabled essentially every product team at OpenAI to be involved in capacity planning.

This failed to scale organizationally with a wide variance in how much diligence each service’s owning team paid to their Azure Cosmos DB provisioning.

Phase III: Using multi-tenancy by default, with data movement capability

As the organization grew, those operational problems became difficult to scale. No lone, central team doing planning resulted in many teams planning poorly. As a result, we ended up shifting back towards a multi-tenant solution and worked closely with the Azure team to engineer fixes for our original multi-tenancy issues with a shared-tenancy design:

Hot key throttling

Throttling the hottest keys first prevents noisy neighbors from consuming all of the available throughput for a given physical partition This reduces the noisy neighbor effect, where one hot key can have a site-wide blast radius.

Quality of Service (QoS)

Request tiering QoS similarly prevents offline workloads, such as bulk inserts or data extracts from utilizing all of the throughput for a given physical partition. By assigning the requests for these workloads a lower priority, they are throttled first, preventing throttling on higher priority, interactive requests that serve end users.

Serverless mode

Azure Cosmos DB already had a serverless mode that would’ve fixed our weak partitions problem with the provisioned mode. However, there was a major downside: it scaled based on storage usage only and limited RU usage to 5K RU/s per physical partition. This would solve our weak partitions problem, but prevent us from scaling up RU-heavy use cases.

A new improved serverless mode Azure Cosmos DB team built for us allowed every physical partition to scale up to 10K RU/s and introduced RU-usage-based splits in addition to storage-usage-based splits allowed us to stop monitoring and scaling up containers manually.

With these changes, we’ve been able to shift back to using one shared container which has significantly simplified the need to do any upfront or continued capacity planning for product teams while preserving isolation between workloads. We have not had any provisioning or noisy neighbor incidents in the months since, despite accelerating product development.

Better reliability as co-managed services

When companies grow to OpenAI’s scale, managed services become less “managed.” Many workloads become outliers in the cloud provider’s infrastructure. As OpenAI became one of the largest users of Azure Cosmos DB, operating it successfully required a different model: shared tests, shared dashboards, direct engineering channels, and joint ownership of failure modes that only appear at our scale. Today, our largest Azure Cosmos DB account stores 10s of petabytes of data and serves more than 20M QPS. To our knowledge, we are the only company which has deployed Azure Cosmos DB at this scale.

In late 2024, a regional outage made it clear why this collaboration was necessary. At the time, a power outage struck the primary Azure region for one of our largest Azure Cosmos DB accounts. In the resulting failover exercise, we realized that our largest accounts would take several hours to fail over to a secondary region, something neither our team nor the Azure Cosmos DB team had tested at our scale.

About a month later, while failing those accounts back to the original region, we triggered another serious outage. A connection stampede to the original region overwhelmed the original region’s Azure Cosmos DB gateway. A quirk in how the gateway-mode Azure Cosmos DB client handled failovers caused all clients to immediately establish connections and send requests to the original region, even though only a subset of physical partitions had moved back to that region.

In each outage, both OpenAI and Azure ran through their incident management processes, with a number of root causes identified and fixes deployed. But these outages were symptoms of a larger systemic issue between the way our team and Azure Cosmos DB team operated and tested. At first, leveraging managed services allowed us to move fast and focus on business goals. At scale, managed services become less managed and require deep integrations with the service provider at all levels from design to operations.

Today, the Habitat and Azure engineering teams jointly monitor every layer of Habitat from our service layers down into Azure Cosmos DB’s physical storage layers. We operate and test changes against a production-sized but non-production Azure Cosmos DB account. We test and iterate on SDKs together and determine best practices for how we scale and size our Azure Cosmos DB accounts. We have direct engineering access to each other through Teams and have even exposed ways to directly page each other without going through support tiers.

A substantial amount of joint work has occurred over the last year to improve Azure Cosmos DB control-plane and gateway scalability, adding load protection and caching of various Azure Cosmos DB metadata in their gateway to protect the Azure Cosmos DB metadata partition, reducing pressure from our fleet of Python habitat-service processes through connection pooling and higher connection reuse, colocating workloads where possible, and isolating critical workloads from offline workloads via different VPCs.

This level of co-engineering has been critical to our Azure Cosmos DB stability in the last year. Any system large or critical enough necessitates this kind of teamwork: breaking down ownership walls and operating across organizational boundaries.

Improving performance at a global scale

As mentioned in Part I, our core philosophy is: make it work, make it fast, make it cheap. Reliability first, then performance, then efficiency. Habitat had become critical infrastructure for OpenAI, so the first priority was making it reliable. As the system stabilized in 2025 and OpenAI expanded globally, latency became the next constraint for users and product teams.

Compute and storage capacity were often located in different regions, making cross-region requests unavoidable.

Figure 04 · Improving performance at a global scale

Global traffic

Compute and storage capacity were often located in different regions, making cross-region requests unavoidable.

Compute Storage Illustrative routes · Longer journeys take more time

Geographical locations of compute and storage are for illustrative purposes only.

Latency varied widely depending on where the data was stored and where the request was served. To make Habitat requests performant, product developers relied on additional client-side caching, request batching, minimizing database reads and writes, and moving work off the critical path wherever possible. While this led to some interesting product designs that were more resilient to database issues, it also made products more complex and slowed down product iteration: a failure to achieve Habitat’s goal to encourage fast product development and not needing to understand the database.

This has been a big focus for us in 2026, and we’ve developed a layered strategy for optimizing read performance, which represents over 90% of Habitat’s requests:

Store data near users when capacity allows

We have a number of Azure Cosmos DB accounts in different Azure regions around the world for users with data residency requirements. Even for users without hard data residency requirements, colocating data near users also improves their experience through lower latency.

In practice, we’re not able to achieve this broadly outside of data residency use cases because placement is constrained by physical cloud capacity. Due to the size of our data sets, many regions cannot accommodate the massive storage footprint required for their proportional share of user traffic.

In regions with limited datacenter capacity, power, or cooling, we sometimes face a direct tradeoff: reserve capacity for colocated storage and lower Habitat latency, or use that capacity for additional serving compute and route storage requests farther away.

As a result, placement decisions are often a product tradeoff between: A) optimizing latency for different products and user subsets, and B) raw storage and compute capacity.

Broadly replicate only the smallest and most latency-sensitive datasets

For the most latency-critical workloads, like authentication metadata that is needed on every request, we reduce RTT between compute and Habitat by using highly replicated Azure Cosmos DB accounts with many secondary regions. Some accounts replicate to all 40 Azure regions where we serve traffic; others replicate to at least one region per continent.

We’ve had to build a transparent data movement and routing layer into Habitat earlier than desired so that we can move datasets between accounts with different replication topologies.

However, this approach is also ultimately capacity constrained and therefore limited to a few hand-selected datasets.

Rely on integrated caching for everything else

Habitat workloads are overwhelmingly read-heavy, with applications constantly fetching settings, state, and other properties; a disproportionate amount of traffic is attributable to a relatively small working set of frequently accessed types. This makes the system particularly well suited for distributed caching. While caching improves performance in single-region infrastructure, the improvements compound when working across regions, where it can be used to “hide” cross-region latency.

We’ve integrated a distributed lookaside caching layer into Habitat. It stores a copy of recently accessed objects and edges in Valkey servers placed in our infrastructure close to where user requests are processed. Typically, these sorts of caches use a TTL that trade off between hit rate and staleness. Short TTLs reduce staleness but blunt the positive impact of caching, because items expire out of the cache quickly and must be retrieved more frequently. Longer TTLs improve hit rates, but serve stale data for extended periods, leading to poor user experiences and unpredictable product behavior.

Another approach is to execute explicit invalidation operations when the application is performing write operations on the same underlying data. This dramatically improves the freshness of cached data, but can be hard to get right. It also remains subject to a number of edge cases, like partial failures, which leave the cache stale. Multi-region deployments compound these problems, with less reliable cross-region networks, and the cache racing with the replication of the underlying storage. Complementing this with TTLs bounds the maximum staleness, but still suffers from the same underlying TTL trade-off discussed above.

We’ve taken a more advanced approach to invalidation, following in the footsteps of systems like Meta’s TAO. Habitat’s integrated cache relies on change data capture (CDC) through Azure Cosmos DB’s change feed to propagate invalidations globally within seconds, achieving high hit rates while keeping staleness low. A service in our infrastructure consumes this change feed and executes invalidations to the local caches when the underlying data is updated. This causes a miss the next time the data is read, and the cache is filled with the latest version of the data.

Reduce proxy overhead with Rust

We’ve also significantly reduced latency by removing overhead in the Habitat service itself through rewriting the code in Rust, which is more performant than the original Python.

Together, these changes have reduced our overall global read p50 from roughly 120 ms to roughly 10 ms, a 12x improvement since we began focusing on performance earlier this year.

Figure 05 · Improving performance at a global scale

p50 latency for reads

Three major design changes improved read latency

While caching has dramatically reduced median latency, the slowest requests still require fetching uncached data from underlying storage, often across distant regions. As more capacity comes online and we improve storage efficiency, we plan to address a more fundamental challenge: placing data closer to where it will be accessed from the start.

Scaling Change Data Capture

Reliable and scalable CDC—propagating database changes to downstream systems—is a critical component in more than just Habitat’s caching strategy. Millions of captured changes per second, across thousands of containers and millions of container partitions feed through more than a hundred Kafka topics to also power eventually consistent secondary indexes, data-movement pipelines, offline analytics, and other real-time product features. As our backing infrastructure has grown globally, it’s become increasingly complex to listen to row changes. If a client wants to register a callback to run whenever rows of their type are changed, they must, among other things, first determine which Cosmos accounts and containers contain their data, which CDC service deployment is running in respective regions listening to those containers, and properly scale the CDC service worker fleet, Kafka, and their consumer. Our CDC platform aims to abstract all of this away from product developers and provide instead a simple surface where clients register a function they’d like called on their changed rows and we handle all of the plumbing and scaling.

An internal control plane continuously ensures all required containers are being captured, selects which Cosmos regions to read from, and efficiently distributes work among thousands of workers across multiple geographic regions. Using replica regions and lower-priority feed reads protect latency-sensitive Habitat traffic from background capture workloads. Worker crashes, container issues, lease-store errors, invalid configuration, and destination failures occur continually across a fleet this large. We hardened CDC worker lifecycle and lease handling, isolated failures between streams, and built safeguards for configuration changes and rollouts. We utilize dead letter queues to prevent head-of-line blocking from permanent delivery failures. We tolerate regional outages by switching readers to alternate Cosmos regions. At our scale, we’ve even had to make changes to Azure’s CDC SDK to make its checkpointing implementation more efficient by batching checkpoints in-memory and only periodically flushing them.

While Habitat’s internal usage of Azure Cosmos DB is complex, its value proposition has always been to abstract away infrastructure details from product developers to improve product development velocity, and our CDC platform is no different.

Looking ahead

A small team was able to support a rapidly growing storage platform by keeping the storage API deliberately narrow, continuing to scale the Python service, and prioritizing the most urgent engineering challenges. We focused first on reliability for mission-critical product traffic, then on read performance for global users, and finally on operational efficiency at scale. This approach enabled rapid growth and new use cases while improving reliability, performance, and product development velocity.

In order to scale Habitat and conversation storage for the next billion users, we need to address the difficulty of replicating our data across our infrastructure, on an even larger scale. To preempt this, we’ve been building a custom in-house exabyte-scale online storage system, optimized for bulky Habitat data types. This in-development system combines the RocksDB Cloud core from Rockset with Raft-based replication. It implements the same document model as Azure Cosmos DB, but uses disaggregated storage and physical replication to reduce the amount of flash capacity needed, in exchange for a lower peak qps.

If you’re interested in working on these types of data infrastructure efforts at frontier scale, check out our open roles⁠.