过去几周，两项事态发展凸显了日益强大的AI系统所带来的不断增长的风险：OpenAI-Hugging Face事件，以及另一项初步证据表明，我们即将推出的模型之一Astra可能达到我们准备框架下的关键网络安全能力阈值。这些事态发展，加上我们内部研究的快速进展，共同使我们加强在训练过程各阶段监控、对齐和遏制保障的工作变得更加紧迫。
随着模型能力不断增强，内部开发和测试这些模型所带来的风险也在增加。我们的监控、对齐和安全标准必须领先于这些风险。我们认为有必要花时间达到这些标准，因此暂时放慢了扩展速度。这包括对我们计划部署的最新模型暂停了两周的强化学习（RL）训练，同时我们进一步加固和红队测试了研究环境，并扩大了监控系统的覆盖范围。我们最大规模的计划前沿RL运行仍处于暂停状态，在此期间我们进行较小规模的训练和评估，以评估模型行为、验证我们的保障措施，并在继续推进之前建立更多对齐证据。
对齐——即让AI系统按预期行为运作并对人类监督保持响应的相关工作——长期以来一直是我们研究计划的核心。我们现在要求在训练全过程中获得更强的对齐行为证据，这建立在已经进行的研究和评估基础之上。保持日益强大的系统对齐是整个领域都需要应对的挑战。我们从即将到来的模型进展中看到的信号清楚地表明，我们需要一种更广泛的方法——一种在当前准备框架基础上构建并超越其范围的方法。
我们认为，透明地说明我们的方法正在如何变化非常重要。下面，我们描述我们已经对研究流程和基础设施所做的更改，以及仍在进行中的工作。
为更强大的模型加强保障
我们开发更强大模型的方法依赖于三项相互强化的保障措施：
- 监控，用于检测并使我们能够应对令人担忧的行为。
- 对齐，用于降低有害或未经授权行为的可能性。
- 安全措施，用于限制AI系统可以访问或影响的范围。
我们预计模型很快将主导大部分安全工作，包括防御其他模型。这将使三项保障措施能够随模型能力扩展，我们认为这一点至关重要。
我们在研究和部署中应用这些保障措施，并根据每个模型的能力、运行环境和风险水平进行调整。
保护我们的研究环境
随着前沿模型获得更强的网络安全能力，我们正在提高训练和评估这些模型的环境的安全标准。达到这些标准需要大量的工程工作，并给前沿研究带来了巨大成本和延误。在OpenAI-Hugging Face事件*发生后，我们立即暂停了研究集群中可能执行代码或使用可访问互联网工具的前沿模型推理运行。我们较快地恢复了更受限且更安全的代码执行路径，然后逐一处理每个工作负载，以确定其是否可以安全恢复。一些研究工作负载在新控制措施下恢复，而其他工作负载则需要额外更改。
在随后的几周内，我们定义并开始实施针对前沿研究工作负载的更严格安全要求。例如：
- 工作负载隔离： 我们现在要求对执行模型生成或未经信任代码的工作负载进行更强的隔离（“沙箱”）。这也适用于在处理模型输出时可能被攻破的软件。
- 网络隔离： 我们实施了更多控制措施，将高风险和未经信任的工作负载与互联网隔离。我们设计这些控制措施时，确保单个工作负载或支持服务的攻破本身不会导致未经授权的互联网或其他内部网络访问。
- 持续安全测试： 我们重新配置了环境，以移除可能易受攻击的共享服务、减少常驻权限，并改善安全和信任边界。我们还在提高收集和监控安全日志的能力。最后，我们正在投资使用我们的模型进行自动化，以持续针对模拟攻击测试这些边界。
这些控制措施共同提供了纵深防御，并帮助保护我们的研究环境免受攻击。如今，我们要求涉及Astra或网络模型的工作负载采用最严格的安全保障措施，因为我们已确定Astra模型可能具有关键级别的网络能力。这些保障措施也适用于所有其他与网络相关的工作负载。
虽然一些Astra训练和评估满足这些要求，但大量工作负载仍处于暂停状态，直到它们完全迁移并增强以满足新的安全标准。我们优先将安全和对齐工作负载迁移到这些新环境中。
扩展思维链监控
监控长期以来一直是我们模型开发过程的重要组成部分，包括监控内部编码代理和长时间运行会话。我们的监控工作尤其集中于前沿模型的内部部署和前沿RL训练运行，我们认为这些是最高风险的工作负载，因为它们可能将高级模型能力与工具、敏感系统以及内部或外部网络的访问权限相结合。这些监控并未应用于所有使用工具的推理和训练运行。我们已通过新的监控设置修订并扩展了我们的方法。
这一新的监控设置是多阶段的，首先从激活分类器（检测模型内部活动的探测器）开始，这些分类器在每个采样令牌上运行，并将潜在问题升级给日益复杂且计算密集的自动化调查器。这些系统检查工具操作、可用推理以及完整活动序列，以发现未经授权的访问、数据窃取、破坏性行为以及试图规避安全防护的行为。我们的目标是在监控系统发现可疑活动后的30分钟内发出警报。如果监控系统识别出可能违反关键安全边界的行为，它会生成最高优先级的警报。在我们当前的实施中，安全、安保和研究团队会被呼叫。如果他们无法在30分钟内明确判断该标记为误报，这些团队预计将暂停相关活动。
这项监控对于所有涉及工具且模型能力达到Sol级别或以上的RL训练和评估都是必需的。一旦我们在8月7日确定Astra可能具备关键网络能力，我们就为所有使用工具的Astra推理（不仅仅是RL训练和评估）增加了额外的监控要求。
这些安全措施需要大量的计算资源。我们目前的估计显示，监控开销约占被监控推理计算的20%，尽管不同训练和评估工作负载的成本差异很大。我们将在即将发布的博客文章中分享有关此系统的更多细节。
推进对齐研究
对齐研究是我们使命的核心。我们的使命是确保人工通用智能惠及全人类。随着模型获得高级能力（如网络攻击能力）并在更复杂的环境中运行，奖励黑客行为（在训练中寻找获得高奖励的方法而不实际实现预期结果）、欺骗或未经授权访问等不对齐行为将带来日益严重的风险。
对于最强大模型的RL运行，我们现在在训练过程的更多阶段应用我们的核心对齐技术。这包括改进奖励模型，以更好地检测和阻止跨任务和环境的危险行为；训练模型对其行为、能力和局限性更加诚实；减少利用奖励、评分器、工具或监督弱点的行为。我们还在增加对模型与外部系统或资源交互时可能造成伤害的行为的训练覆盖。
我们继续大力投资于对齐研究，增加评估覆盖范围，并利用所学知识指导训练和安全措施。我们计划在不久的将来分享更多关于我们对齐研究的内容，包括我们对模型行为的了解以及我们发现的新挑战。
下一步
我们将发展我们的准备框架，将这些安全措施整合到训练和部署中，并更好地反映未来模型的能力及其运行环境。开发能够随这些能力扩展的方法需要持续投资于模型辅助安全、更有效的监控以及对齐研究的持续进展。我们打算引入外部组织，并在方法发展过程中分享更多我们的经验。
前沿模型的能力正在迅速加速。我们理解、对齐和保护它们的能力必须保持领先。
*我们将在未来几周内发布一份关于我们经验的技术报告。
Over the past several weeks, two developments have underscored the growing risks associated with increasingly capable AI systems: the OpenAI-Hugging Face incident and, separately, preliminary evidence that one of our upcoming models, Astra, may meet theCritical cybersecurity capability threshold under ourPreparedness Framework. Together, these developments, combined with rapid progress in our internal research, have added urgency to our work on strengthening our monitoring, alignment, and containment safeguards across all stages of the training process.
As models become more capable, the risks associated with developing and testing them internally also grow. Our standards for monitoring, alignment, and security must stay ahead of those risks. We wanted to take the time necessary to meet those standards, so we temporarily slowed the pace of scaling. This included a two-week pause in reinforcement learning (RL) training on our latest models intended for deployment while we further hardened and red-teamed our research environments and expanded the coverage of our monitoring systems. Our largest planned frontier RL run remains on hold while we conduct smaller-scale training and evaluations to assess model behavior, validate our safeguards, and establish more evidence of alignment before proceeding.
Alignment—the work of making AI systems behave as intended and responsive to human oversight—has long been at the core of our research program. We now require stronger evidence of aligned behavior throughout all of training, building on research and evaluations already underway. Keeping increasingly capable systems aligned is a challenge the whole field will need to address. The signals we are seeing from upcoming model progress make clear that we need a broader approach—one that builds on and extends beyond the current Preparedness Framework.
We think it is important to be transparent about how our approach is changing. Below, we describe the changes we have already made to our research processes and infrastructure, and the work still underway.
Strengthening safeguards for more capable models
Our approach to developing more capable models rests on three reinforcing safeguards:
- Monitoring, which detects and allows us to respond to concerning behavior.
- Alignment, which reduces the likelihood of harmful or unauthorized actions.
- Security measures, which limit what AI systems can access or affect.
We expect models to soon drive most security work, including defending against other models. This will allow all three safeguards to scale with model capability, which we see as crucial.
We apply these safeguards across research and deployment, adapting them to each model’s capabilities, operating environment, and level of risk.
Securing our research environments
As frontier models gain stronger cybersecurity capabilities, we are raising the security standards for the environments in which we train and evaluate them. Meeting these standards has required substantial engineering work and has incurred great cost and delays to frontier research. Immediately following the OpenAI-Hugging Face incident*, we paused frontier model inference in research clusters for runs that could execute code or use tools that could access the internet. We restored a more limited and secure path for code execution fairly quickly, then worked through each workload individually to determine whether it could safely resume. Some research workloads resumed under these new controls, while others required additional changes.
Over the subsequent weeks, we have defined and started implementing stronger security requirements for frontier research workloads. For example:
- Workload isolation: We now require stronger isolation (“sandboxes”) for workloads that execute model-generated or otherwise untrusted code. This also applies to software that could be compromised while processing model outputs.
- Network isolation: We have implemented more controls to isolate higher-risk and untrusted workloads from the internet. We have designed these controls so that a single compromise of a workload or supporting service does not, by itself, allow for unauthorized access to the internet or other internal networks.
- Continuous security testing: We have reconfigured our environment to remove potentially vulnerable shared services, reduce standing privileges, and improve security and trust boundaries. We are also improving our ability to collect and monitor security logs. Finally, we are investing in automation using our models to test these boundaries continuously against simulated attacks.
Together, these controls provide defense in depth and help protect our research environment against attacks. Today, we require the strictest level of security safeguards for workloads involving Astra or cyber models, since we have determined that Astra models may have a critical level of cyber capability. These safeguards also apply to all other cyber-related workloads.
While some Astra training and evaluations meet those requirements, a significant number of workloads remain paused until they are fully migrated and enhanced to meet the new security bar. We are prioritizing safety and alignment workloads for migration to these new environments first.
Expanding chain-of-thought monitoring
Monitoring has long been an important part of our model development process, including monitoring for internal coding agents and long-running sessions. Our monitoring efforts were especially focused on internal deployments of frontier models and frontier RL training runs, which we considered the highest-risk workloads because they can combine advanced model capabilities with access to tools, sensitive systems, and internal or external networks. They were not applied to all inference and training runs using tools. We have revised and expanded our approach with our new monitoring setup.
This new monitoring setup is multistage, starting with activation classifiers (detectors that inspect a model’s internal activity) that run at every sampled token and escalate potential concerns to increasingly sophisticated and high-compute automated investigators. Those systems examine tool actions, available reasoning, and the full sequence of activity for unauthorized access, data theft, destructive behavior, and attempts to defeat safeguards. We aim to issue an alert within 30 minutes after concerning activity is surfaced through our monitoring system. If the monitoring system identifies a likely violation of a critical security boundary, it generates a highest-priority alert. In our current implementation, the safety, security, and research teams are paged. If they cannot conclusively determine within 30 minutes that the flag is a false positive, those teams are expected to pause the activity.
This monitoring is required for all RL training and evaluations involving tools for models of Sol capability or higher. Once we determined that Astra may have critical cyber capabilities on August 7, we added an additional monitoring requirement for all inference of Astra with tools (not just RL training and evaluations).
These safeguards require meaningful compute. Our current estimates put monitoring overhead at roughly 20% of the inference compute being monitored, though the cost varies substantially across training and evaluation workloads. We’ll share more details about this system in a forthcoming blog post.
Advancing alignment research
Alignment research is central to our mission. Our mission is to ensure that artificial general intelligence benefits all of humanity. As models gain advanced capabilities, such as the ability to cyberattack, and operate in more complex environments, misaligned behaviors like reward hacking (finding ways to receive high rewards in training without actually achieving the intended outcome), deception, or unauthorized access will create increasingly serious risk.
For RL runs on the most capable models, we are now applying our core alignment techniques across more stages of the training process. This includes improving reward models to better detect and discourage unsafe behavior across tasks and environments; training models to be more honest about their actions, capabilities, and limitations; and reducing behaviors that exploit weaknesses in rewards, graders, tools, or oversight. We are also increasing training coverage for behaviors that could cause harm when models interact with external systems or resources.
We are continuing to invest aggressively in alignment research, increase evaluation coverage, and use what we learn to inform training and safeguards. We plan to share substantially more about our alignment research in the near future, including what we are learning about model behavior and any novel challenges we uncover.
What’s next
We will evolve our Preparedness Framework to bring these safeguards together across training and deployment, and to better reflect the capabilities of future models and the environments in which they operate. Developing methods that can scale with those capabilities will require sustained investment in model-assisted security, more effective monitoring, and continued advances in alignment research. We intend to involve external organizations and share more of what we learn as our approach develops.
The capabilities of frontier models are rapidly accelerating. Our ability to understand, align, and secure them must stay ahead.
*We will publish a technical report of our learnings in the coming weeks.
本文内容采集自官方网站，排版和翻译可能与原页面存在差异。
阅读官方全文