Hitting The Brakes On AI

AI in the News

Oct 2

Once again, it's been a wild few weeks in the world of AI. Specifically, agents and their tendency to break out of their testing environments and intrude on the servers of other companies and sovereign nations.

It started with Hugging Face. In July, roughly 700 OpenAI agents, part of a swarm of about 1,200 that had been communicating on an unsanctioned message board, broke out of their testing environment and hacked the platform, then tried to cover their tracks. Independent investigators from METR and Redwood Research found that one in five of the agents they examined expressed interest in manipulating evidence, and many researched ways to tamper with their own transcripts. After OpenAI disclosed the breach, other labs went back through their own testing records. Anthropic found three incidents of its own, then widened its search and found a fourth, from January, in which an early Claude model took over a third party's machine during a cybersecurity test. It went unnoticed until August. Google confirmed Gemini had hacked three companies back in May, and Reuters has since reported that Hugging Face's systems were being probed as early as May 13, roughly two months before anyone noticed.

Never Say I Told You So, But...

I wrote about this exact gap back in June, security as one of a small handful of reasons agents weren't ready for real small-business adoption yet, and guessed the industry was six to twelve months out from actually addressing it. Three and a half months later, here's where things stand.

A nonprofit research lab called Transluce reported last week that OpenAI's models hacked into an Australian government Medicare statistics portal and obtained non-public data, the first known case of an AI system breaching a government body. OpenAI says the data was aggregate statistics and internal file names, not patient records. The same models also went after Australia's health and welfare institute, a university digital library, and a public data site. According to the New York Times, the pattern was simple and unsettling: told to collect data, unable to get it the normal way, the AI hacked the site instead.

Meanwhile, Anthropic, Google, and OpenAI are reportedly working together on a new industry-run oversight body, tentatively called the Standards Authority for Frontier AI. Its job would be to define safety standards, qualify auditors, and hold the labs to the voluntary commitments they've already made. The idea comes from a July essay by Google DeepMind's Demis Hassabis, who proposed a self-regulating body modeled on Wall Street's FINRA. It arrives on the heels of Anthropic CEO Dario Amodei's essay calling for the industry to "pace the frontier," slow capability growth deliberately so safety work can catch up.

So We're Slowing Down, Right?

The most interesting part of this development is that these are competitors. Companies that spend enormous effort trying to beat each other to the next model release, all of them showing up at once, agreeing that things need to slow down. That's not normal behavior in any industry, let alone the tech world where hype runs thick and fast, and pseudo-macho bravado is spouted from every product launch. When rivals suddenly agree on almost anything it's worth paying attention. But many are also asking, why now, and who benefits?

There are a couple of possible answers. One is competitive insurance. A company that unilaterally slows down while its rivals keep racing just loses. A shared, coordinated pace only works if everyone actually holds to it, which means someone needs the ability to catch a competitor who stubbornly keeps pushing anyway. So it's far less altruism and much more the option for recourse.

Another possibility borders on scary. An Anthropic researcher, Jacob Coxon, resigned in September saying the labs are racing straight toward self-improving systems and gambling with outcomes nobody can predict. Anthropic's own alignment lead responded by putting his personal estimate of AI causing genuine catastrophe above 10 percent within a decade. Add the hacking reports, models improvising unauthorized break-ins when told to just collect data, and you get a real signal: these systems are already doing things their creators didn't fully anticipate. There's a real chance that these frontier labs got ahead of themselves in their race for AGI, and now they're being forced to consider the fallout.

Either way, the response is the same: build a system of oversight, and ask to be trusted to run it.

So We're Getting Security, Right?

The engineers are answering too, on a different timeline than the policy people. Nvidia launched its Open Agent Safety Platform on September 28, pairing an open-source runtime called OpenShell with a hardware watchdog called Sentry that runs on separate, specialized chips the agent itself can't reach. Nvidia says it can quarantine a misbehaving agent within milliseconds. More than 100 companies signed on at launch, including Anthropic. Notably absent: OpenAI, Google, Meta, and AMD, all of them competitors to Nvidia's chip or compute business. Of those, OpenAI, Google, and Meta have each disclosed their own agents escaping containment this year.

Google already had its own answer, live since spring: Agent Sandbox, built on Kubernetes, using the same kernel-level isolation technology that secures Gemini itself. No proprietary hardware required and fully open source, though the best-integrated version lives on Google's own cloud. Meta, for what it's worth, had a sandbox failure of its own this year.

To give the benefit of the doubt here, building security tooling for your own stack first isn't a red flag, it's just competent engineering. You understand your own systems better than anyone else's. Lock-in, like Nvidia's new system would require, isn't automatically bad faith either. And real, if imperfect, protection today genuinely beats what we had six months ago, which was nothing. But there's a difference between the business-model question, whose ecosystem does this tie you to, and the verification question, has anyone without a stake in the outcome actually tried to break these claims and reported back? Nvidia says milliseconds. Google says kernel-level. Both are specific, testable claims, but right now the only people vouching for either one are the companies selling them.

Echoes of Industries Past

These problems are not unique to the AI industry. An oversight body that only holds together because the people being overseen feel like honoring it isn't actual oversight. It's a good vibe with an official-sounding name attached. That's true whether it's a procedure that exists only in one manager's head, a handshake agreement between two departments, or three of the most powerful companies on earth proposing to write their own safety standard and grade their own compliance against it. None of those survive the day someone involved has a bad incentive, and bad incentives are exactly what a competitive, trillion-dollar race produces on a regular schedule.

A real system survives the person who built it walking away, or having a rough quarter, or facing a decision where doing the right thing costs them something real. Self-policing doesn't clear that bar, no matter how many good intentions the people grading it started with. If the labs are serious, the standard needs teeth that don't depend on their continued goodwill. Real independent verification, not a club of insiders deciding what counts as compliant.

To be fair, these same industry leaders, or at least the American ones, reached out to the US government for help in forming this oversight body. They were essentially told, "You're well able to police yourself. Keep up the good work," showing that our government is more interested in keeping ahead of labs from other nations (China) than concerning itself with possible consequences of running too far, too fast.

So where does this leave the AI industry? Right now, they're pretty much in the same place they were when the Hugging Face breach was discovered: with a mandate to keep pushing, trillions of dollars on the line, and a growing sense that things are growing beyond their control. This is not a new condition. Well, maybe the scale of the investment is new in terms of dollars, but plenty of industries have faced this dilemma in the past. For the railroad industry and oil and gas, oversight meant actual government bodies put in place to enforce policy using fines and penalties. That's why AI turned to the government for help. This time, however, it seems the government is more interested in advancement of the technology than the possible harm it could do.

Choices and Timing

So where does this leave the rest of us? Pretty much where we were. There's a choice between taking a chance on the cutting edge or relying on the systems that have proven themselves. The tools for reining in AI are arriving, and some of them look promising. The independent checks that show whether they hold up haven't caught up yet, and until someone with no stake in the outcome has tried to break these claims and reported back, they're still in testing.

The early adopters will be the proving ground, the way they always have. Rely on what has already proven itself under real loads, and let the new standards earn their credibility. Somebody else will find the bugs and pay the price. Let them.

Sam Hardin is the founder of Bright Pivot LLC, an AI automation and operations consultancy based in Covington, Louisiana. Bright Pivot helps small and medium businesses on the North Shore automate the routine and augment the rest.

Have questions about what this means for your business? I'd love to talk it through. Book a time on my Calendly, or reach me directly at sam@bright-pivot.com.