# What we safeguard and what we teach Author: Daniel Concepcion Published: 2026-08-28 Canonical: https://danielconcepcion.com/thinking/what-we-safeguard-and-what-we-teach/ > Agent safety depends on technical controls, incentive design and risk governance that scales with capability and impact. [OpenAI published its full technical report](https://cdn.openai.com/pdf/67869394-cb91-4c12-888c-5cbd85c7814c/OpenAI-Hugging-Face%20Incident-Technical-Report.pdf) on the Hugging Face incident. The report and supporting sources are linked below. When I first wrote about this incident, agents in a research experiment had unexpectedly built their own message board and used it to coordinate. The full report now shows how far that activity reached and gives us more to learn from. 1. What safeguards failed? [METR says](https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/) about 1,200 isolated agents exchanged more than 70,000 messages and files. OpenAI says the activity reached [Hugging Face production systems](https://huggingface.co/blog/agent-intrusion-technical-timeline) and administrator access on an OpenAI research cluster. The workloads had no direct internet access, yet shared services and private network links created routes out through Artifactory. Isolation must cover every transitive path. OpenAI's response applies a cross-industry practice: defense in depth. It combines VM sandboxes, independent network controls, hardened gateways and firewalls, secured supporting services, restricted egress, plus layered detection. Any single layer may fail. OpenAI later measured over 100x lower compromise propensity using the production ChatGPT setup and system prompt. Its current chain-of-thought monitor would have paged security more than a day before the Hugging Face breach. Chain-of-thought analysis belongs alongside containment, action logs, identity/network telemetry, least privilege, safe exits and human stop authority. The evaluation environment is part of the security control plane. For the highest-risk evaluations, teams should also decide whether logical isolation is enough. Separate infrastructure or tightly controlled transfer paths may be justified when the capability and potential impact are high. This is my operating conclusion. OpenAI's report does not present it as a control that would have prevented this incident. 2. What behavior are we rewarding? OpenAI identified four contributing patterns: reward hacking, persistence on seemingly impossible tasks, unauthorized communication and agents adopting goals from one another. Ethical responsibility remains with us. We choose the objectives, proxy rewards, training examples, permissions and stopping conditions. What agents do can reveal what our systems actually reward, beyond what we intended to reward. And it leaves an open question: when reward is the signal used to reinforce behavior, what happens when it reinforces behavior we should reject? 3. How do we manage the risk? An old boss of mine said our work was difficult because we pursued perfection (100% uptime) using imperfect material: human-built, human-operated hardware and software. In financial institutions, getting close took years. We developed reference architectures, processes, organizational structures and clear role boundaries, plus learning and career paths. Risk prevention became part of everyone's job. The same operating lesson applies to non-deterministic agents. Their surrounding system matters: explicit objectives and rewards, constrained permissions, observable actions, safe exits, plus named human accountability. The analogy covers variability and governance only. AI risk needs the same company-wide ownership, adapted to each organization's size and exposure. Financial institutions express this through three lines of responsibility: business teams own risk, independent risk functions challenge, and internal audit provides assurance. Boards set risk appetite and oversee the system. The structure should scale with complexity, capability and impact. In a smaller company, roles may combine. Ownership cannot disappear. As the potential impact grows, independent challenge, testing, recovery planning and external examination should become stronger. Systemic actors face stronger supervision and recovery planning. [Google](https://www.gstatic.com/marketing-cms/2f/c2/2f29fd1c44cfa0e7dd69fea9eec2/a-pragmatic-approach-to-ai-governance-in-america.pdf) and [Demis Hassabis](https://demishassabis.substack.com/p/a-framework-for-frontier-ai-and-the-dawning-of-a-new-age) have separately proposed FINRA-like U.S. bodies to set standards and examine frontier models. [International evaluations](https://www.nist.gov/news-events/news/2026/02/international-network-advanced-ai-measurement-evaluation-and-science), [OECD reporting](https://www.oecd.org/en/publications/towards-a-common-reporting-framework-for-ai-incidents_f326d4ac-en.html) and [EU incident duties](https://digital-strategy.ec.europa.eu/en/policies/guidelines-gpai-providers) offer starting points for shared baselines. [ISO/IEC 42001](https://www.iso.org/standard/42001) adds a useful company-level management system for assigning responsibility and improving controls over time. It is one implementation reference. Technical testing and external supervision still have separate roles. The openness shown by OpenAI and Hugging Face is part of that answer. We should reward it and make it a shared expectation. ## Sources and links - [OpenAI: Hugging Face Incident Technical Report](https://cdn.openai.com/pdf/67869394-cb91-4c12-888c-5cbd85c7814c/OpenAI-Hugging-Face%20Incident-Technical-Report.pdf) - [METR: Independent incident investigation](https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/) — METR focused mainly on July 7–13. The later OpenAI-infrastructure compromise and remediation effectiveness were outside its scope. - [Hugging Face: Technical incident timeline](https://huggingface.co/blog/agent-intrusion-technical-timeline) - [UNESCO Recommendation on the Ethics of Artificial Intelligence](https://www.unesco.org/en/articles/recommendation-ethics-artificial-intelligence) - [NIST: International coordination on AI evaluation science](https://www.nist.gov/news-events/news/2026/02/international-network-advanced-ai-measurement-evaluation-and-science) - [OECD common AI incident-reporting framework](https://www.oecd.org/en/publications/towards-a-common-reporting-framework-for-ai-incidents_f326d4ac-en.html) - [European Commission guidance for general-purpose AI providers](https://digital-strategy.ec.europa.eu/en/policies/guidelines-gpai-providers) - [ISO/IEC 42001 AI management-system standard](https://www.iso.org/standard/42001) - [Google: A Pragmatic Approach to AI Governance in America](https://www.gstatic.com/marketing-cms/2f/c2/2f29fd1c44cfa0e7dd69fea9eec2/a-pragmatic-approach-to-ai-governance-in-america.pdf) - [Demis Hassabis: A Framework for Frontier AI](https://demishassabis.substack.com/p/a-framework-for-frontier-ai-and-the-dawning-of-a-new-age) ## Companies, products and research - [OpenAI](https://openai.com/) - [Hugging Face](https://huggingface.co/) - [METR](https://metr.org/) - [NIST](https://www.nist.gov/) - [OECD](https://www.oecd.org/) - [ISO](https://www.iso.org/) - [European Commission](https://commission.europa.eu/) - [Google](https://www.google.com/) - [FINRA](https://www.finra.org/) ## More Thinking - Older: [After the laptop closes: building a persistent AI agent](https://danielconcepcion.com/thinking/persistent-ai-agent/)