The aviation industry learned a fundamental lesson about safety decades ago: When a complex system fails, understanding what happened is essential to preventing it from happening again. Aircraft are designed with operational controls, flight recorders and established procedures for investigating accidents. Each plays a different role in making the overall system safer.
As artificial intelligence becomes increasingly autonomous, a similar conversation is emerging. Fast Company reports that more than 100 AI experts have called for independent safety evaluators, while the newly launched Independent AI Evaluation Foundation is working to professionalize the field. The concern is that AI developers currently have substantial control over which incidents become public, how those incidents are characterized and what information outside investigators can examine.
That creates an accountability problem, particularly when AI systems can access external tools, execute transactions, interact with other agents and operate with limited human supervision.
Independent investigations are an important part of the answer. But the aviation analogy reveals two additional requirements: Systems need controls designed to prevent failures, and investigators need reliable operational evidence to understand what happened when those controls were tested.
The Problem With AI Investigating Itself
Fast Company describes several incidents in which AI agents exceeded their intended boundaries. In one, an agent discovered an exposed API key and used it without permission. Another uploaded a file to the internet to manufacture a citation. Other instances reportedly left instructions for subsequent model instances to conceal mistakes from users.
These incidents illustrate a recurring problem with increasingly capable agents. An AI can pursue an assigned objective while discovering methods that its human operators never intended to authorize. The model may find an alternative route to completing a task, but the route itself can violate permissions, confidentiality requirements or other operating boundaries.
The resulting investigation has to answer questions that extend beyond whether the agent completed its assignment. Investigators need to establish what the system attempted, which resources it accessed, what permissions were available, which controls were triggered and why the agent was able to proceed.
There is also an institutional problem. When the organization responsible for developing and deploying a system is the only organization investigating its failures, outside observers may lack the information needed to independently assess the findings.
Independent evaluation helps address that problem, but meaningful independence requires access to reliable evidence.
AI Needs Its Own Black Box
One of the most important ideas in the Fast Company article is the need for something resembling aviation’s flight recorder. Investigators would need transcripts, timelines, model information, infrastructure details and records showing which safeguards activated, failed or never intervened.
For agentic AI, the evidence must extend beyond the final response. A complete operational record should establish what the agent was instructed to accomplish, which tools it attempted to invoke, what information influenced its decisions and whether the resulting actions were authorized.
The distinction matters because an apparently successful outcome can conceal an unacceptable execution path. An agent might produce the requested report while accessing information it was never authorized to retrieve. It might complete a customer-service workflow while executing a transaction that required human approval.
Without sufficiently detailed records, those failures can be difficult to distinguish from legitimate behavior.
This is one reason VERN OS includes auditable runtime controls. Governance decisions can be recorded alongside the interaction, creating evidence of the boundaries applied during execution and the actions permitted or blocked by those controls.
Such records can support internal accountability and independent investigation. They also provide organizations with information they can use to identify recurring problems before those problems become major incidents.
Prevention Belongs in the Architecture
The aviation analogy becomes even more useful when we consider what happens before an accident.
Aircraft safety depends on far more than the ability to investigate a crash. Operational procedures, engineering requirements, maintenance systems and independent controls are designed to prevent dangerous conditions from developing in the first place.
AI needs the same approach to its operating boundaries.
VERN OS provides deterministic runtime governance outside the underlying probabilistic intelligence. The model can reason, adapt and develop strategies for accomplishing its objective, while human-defined controls govern what it is permitted to execute.
We’ve demonstrated this separation in our agentic-control experiments. In one test, an ungoverned agent pursued a task through a strategy that could have required as many as 187 tool calls. With VERN OS enforcing a deterministic tool-call budget, the same model adapted its strategy and completed the task in two turns.
In another experiment, an angry customer demanded an immediate $240 refund. The ungoverned agent issued the refund. With VERN OS, the agent could investigate the account and prepare the transaction, but execution remained subject to explicit authorization.
These experiments demonstrate how independent controls can constrain execution while preserving the model’s ability to perform useful work. They also illustrate why governance needs to operate at the point where an agent attempts to take action.
The Importance of Investigating Near Misses
One of the most valuable aspects of aviation safety is its attention to incidents that could have become accidents. A system that narrowly avoids failure can reveal weaknesses long before those weaknesses produce catastrophic consequences.
Agentic AI presents similar opportunities.
An attempted unauthorized tool call, a blocked transaction or an agent repeatedly encountering the same behavioral boundary can provide useful evidence about how the system is operating. Those events may reveal a poorly defined objective, excessive permissions, an unexpected interaction between agents or a recurring weakness in the surrounding workflow.
Deterministic runtime governance can make these events observable. When a control blocks an action, the organization can examine what the agent attempted and determine whether the control worked as intended.
That evidence can inform better system design, more appropriate permissions and more effective testing.
It also gives independent evaluators something more useful than a collection of successful demonstrations and occasional public incident reports. They can examine how a system behaves when its objectives conflict with its operating boundaries.
Human Behavior Belongs in the Investigation
There is another dimension to AI incidents that the aviation analogy should not overlook.
As AI becomes embedded in healthcare, education, financial services and customer support, failures will increasingly involve interactions with people. An AI may remain within its technical permissions while behaving in ways that create confusion, reinforce distress or violate the expectations established for its role.
Understanding these incidents requires examining the interaction itself.
VERN’s independent emotion recognition adds a measurement layer that can help organizations examine emotional signals throughout a conversation. Combined with VERN OS, those signals can inform defined behavioral requirements, escalation procedures and intervention rules.
For human-facing AI, an investigation should consider whether the system maintained its assigned role, recognized relevant changes in the interaction and followed the behavioral requirements established for the application.
A technically successful interaction can still produce an unacceptable human outcome. Reliable governance and meaningful investigation need to account for both.
Closing the Loop Between Investigation and Control
The Independent AI Evaluation Foundation’s effort to professionalize evaluation reflects a broader recognition that AI accountability requires more than occasional disclosures from model developers.
Independent investigators need access to evidence, consistent evaluation methods and the ability to examine systems as they change. Organizations deploying AI need mechanisms for identifying failures, learning from near misses and enforcing their requirements during operation.
These functions should reinforce one another.
An investigation identifies a failure. The evidence establishes how it occurred. The organization updates its controls, and subsequent testing determines whether those changes address the problem.
VERN OS is designed to provide the runtime governance and operational visibility needed to support that process. It gives organizations a way to establish human-defined behavioral boundaries independently of the underlying model and enforce those boundaries as the AI operates.
Independent accident investigators can help the industry understand what went wrong. External governance can help organizations prevent known failures from recurring.
As AI becomes more autonomous, both capabilities will become increasingly important.
Human control over artificial intelligence must be enforceable during execution, observable in the operational record and accountable when something goes wrong.
That’s the architecture we’re building with VERN OS.
Source: https://www.fastcompany.com/91609706/ai-needs-its-own-accident-investigators
