Microsoft CEO Satya Nadella published a discussion on the trust framework in the era of superintelligence. He stated that traditional software allows for tracing behavior back to specific code paths, whereas current superintelligent systems lack this mechanistic understanding and cannot attribute model behavior and outputs to specific training data or weight configurations. These systems are being deployed and accessing sensitive data, executing critical actions on behalf of users. He emphasized that responsibility for intelligent behavior cannot be outsourced to model providers, nor can superintelligence be accepted or rejected as a nested black box based solely on its suggestions and actions. Nadella proposed separating the supply of intelligence from the authorization of intelligence, surrounding nondeterministic models with deterministic system design, human control, and operational procedures, and establishing industry standards where existing standards are insufficient. He suggested managing cutting-edge models with closed-source and open-source weights as internal risks, not because they are inherently malicious, but because entities with sufficient capabilities that can access important systems may make mistakes or be compromised. Relevant practices include establishing identities, restricting permissions, recording activities, and setting isolation boundaries, with transparency in the model's reasoning chain as a non-negotiable starting point. He further stated that permission and action control must be external to the model, separating the model, orchestration layer, and action space. Principles include avoiding reliance on a single model for important outcomes or self-verification, ensuring key actions leave tamper-proof and human-readable evidence, continuously testing for failures, attacks, and edge cases, allowing organizations to independently decide access and action permissions, validating independently of the intelligence being verified, presuming models have been compromised and supporting mid-course pauses or shutdowns, and timely disclosure and sharing of lessons within the industry in case of failure or compromise. He concluded that the most trustworthy superintelligent systems are those that require the least trust in the model.
All Comments