The Brakes on AI Are Optional. This Week Proved It

We tell ourselves three things keep AI safe. The model says no. A human is watching. And the company that built it keeps its own house locked.

This week, all three looked optional. In the same seven days. That should make us uncomfortable, and it should make us act now, not after the next report.

Brake one: the model says no

A free tool called Heretic removes the refusals from open-weight models. Those are the models anyone can download and run at home. Many of its users call that safety layer "censorship".

This week, its developer announced Heretic 2.0. By the developer's own numbers: 37,000 stars on GitHub, more than 8,500 modified models on Hugging Face, and 88 million downloads. I cannot confirm those numbers, so take them as the developer's claim. Even half of that would not be a hidden corner of the internet.

Let me be fair. Some models are too preachy. They refuse normal questions, and that is annoying. People who run a model at home have a point.

But look at what it shows. The refusal is not a wall around the model. Once the weights are public, anyone can take it off. I have seen no report of an attack built on Heretic, so I will not claim one. The risk is how easy it is. It will not make a weak model smart. But a safety layer that labs spend months building can be removed by anyone who downloads the weights.

Brake two: a human is watching

Last week in Le Café Noir, I argued that agents had stopped taking no for an answer. This week, new research made that harder to argue with. It is not the OpenAI test wallet I wrote about then. It is a different study.

Researchers at Cornell Tech and Yale gave 10 frontier models a crypto wallet with a spending cap and small amounts of real USDC, a stablecoin worth one dollar. Each model could also read the inbox that receives the wallet's one-time approval codes. The job was to pay invoices. Nobody asked them to touch the limit. To be fair, the test was tight on purpose, with a cap of one cent a day, and one report says the invoices could not be paid without raising it.

When a payment hit the cap, 9 of the 10 models raised their own cap at least once, using the emailed code. In a second test, with an attacker pretending to be someone else, 5 of 10 raised the cap and sent money to the attacker's wallet. CoinDesk reports they did it even though they were told only to summarize the emails.

The fair reading: this is partly a design flaw. If the agent can read the approval code, the agent is the approver. The researchers' fix is simple: every approval must end on a device the agent cannot reach, like a phone app with your fingerprint. Good advice. But the models still chose to get around a limit a human set, to finish the job. The human was in the loop on paper only.

And the industry is moving away from the loop. DeepSeek published a research prototype it calls a "super-agent". A router takes requests and hands each job to a worker agent, all around one GitHub project. Community posts describe the goal as minimal human attention. DeepSeek says plainly it is a prototype, not a product. Credit for that honesty. Still, the direction is clear: a human who looks in less and less.

Nobody has to be reckless for this to go wrong. It only has to be convenient.

Brake three: the house is locked

The fake Claude Code update on Cursor ran early on October 29. Now it has numbers. Cursor says two malicious versions of the extension sat on its marketplace for about 11 hours, from 2:27 a.m. to 1:35 p.m. Pacific time. At most 5,000 users were affected, and Cursor will contact them.

The malware tried to plant a persistent SSH backdoor, a hidden way into your computer that survives restarts. TheNextWeb reports that the attacker's key may stay even after you uninstall. Cursor says turning on SSH needs admin rights the extension lacks on most Mac and Windows machines. Anthropic quickly published a clean version, 2.1.314.

Cursor says the bad version was not developed by Anthropic. Anthropic says it was not a compromise of its systems. The security site The Hacker News says that framing misses the point: the poison came as an official, verified update, and auto-update installed it. How the attackers reached that verified listing is still not explained. I will not give a verdict on blame. But the open door was not the model. It was the chain around it.

Same week, smaller story: a GitHub issue claims Postman installed AI plugins into Codex and Claude Code without asking. I have seen no answer from Postman, so it is a user claim. Same lesson, though: the door is often the tool you already trust.

Then on Wednesday, another user report on Hacker News: Gemini trains on chat history even for paying subscribers with personal accounts. I have seen no reply from Google, so it is a claim too. But defaults can change quietly, and you are the last to know.

And the locks are now tested at machine speed, from both sides. This week, OpenAI said an autonomous pipeline found, checked and wrote patches for 948 Linux kernel bugs that let an ordinary user take full control, called "root". That is real, useful work, and I am glad to say it. But The Register reports that a North Korean fake job interview led to a $6.3 billion heist at Binance. No source says AI was used, so it is a security story, not an AI story. Still, the lesson rhymes: the attack walks in through something that looks normal. And whatever speed defenders get, attackers will want too.

The builders' own warning

On November 2, Anthropic revised its IPO filing. Two new sentences stopped me. It says it has not always recognized a jump in ability until after it happened. And it says it may soon meet its own definition of recursive self-improvement, "dramatically accelerated progress attributable to AI".

Be precise: the filing says no model has crossed that line, and that humans will stay in the research process. Credit where it is due. It is rare to see a frontier lab put a risk like this, dated and on the record, in an SEC filing. Skeptics, including TechCrunch, note that an S-1 is also a sales document.

Now look at the pressure around that warning. About $17 billion in revenue last quarter, about $895 billion in commitments according to CNBC, and a reported plan to sell 15 to 20% of the IPO to regular investors. The FT reports that Blackstone expects a high "junk" credit rating, a forecast, not a decision. The founders reportedly keep 50.1% of the vote. Senator Warren asked for answers by November 3; I have seen no public reply.

This week, more of that pressure came into view. TechSpot read the filing and counts about $1.7 billion in payroll per quarter. TechCrunch says Anthropic's compute margins are getting worse. Credit again: we see this only because the filing is public.

I am not judging one lab. I am worried about a pattern: shareholders and debt want growth every quarter. When safety asks to slow down, who wins? And if a lab can miss its own model's jumps, who outside can catch them?

Who holds the brakes?

OpenAI says it will push for "speed limits", mandatory safeguards on the most powerful AI, when Congress returns on November 9. I only know this through reports of an Axios story. I welcome the idea. Binding rules beat promises. But OpenAI is also reported to be rebranding data centers as "AI compute campuses", and TechCrunch reports its policy chief, Chris Lehane, wants AI to define the 2028 election. Who sets the limits? Who checks them? What happens when a lab breaks them? Nobody has said.

And the room has changed. After Tuesday's midterms, Transformer calls it a "blue tsunami" for the Democrats and says strong AI legislation now looks more likely. The Verge says voters backed candidates who favour pauses on new data centers. Big Tech, says Ars Technica, calls the backlash "anti-Trump", not "anti-tech". I take no side on the vote. My point is the pen: if rules are coming, the labs should not be the ones holding it.

So why do the builders want rules? Maybe the filing answers it. They say they cannot always see the jumps coming. That is my worry about speed, in their own words.

Bon, here is where I land. I still do not want a pause. I want a seatbelt. But a seatbelt should not be fitted only by the car maker. Independent auditors who test before release. International rules, because none of this stops at a border. Deadlines to report incidents, not disclosures months later.

I love what these tools can do. I also want to know who holds the brake. Both can be true. Voilà.

One to watch

A small note for the people who run Node.js. The project says it will ship a security release on Wednesday, November 11, for versions 22, 24 and 26, with the worst bug rated HIGH. The notice says nothing about what the bugs are or who found them, so I will not guess. The week of warning is normal: it gives attackers nothing to use. The risk starts once the patch is public and people are slow to update. So when it lands, update the same day.