Anthropic highlights danger of AI recursive self-improvement

Every machine that ever mattered was built by a human hand. Someone drew the blueprint, cut the steel, or wrote the code, then signed off before it shipped.

We argue about who gets rich and whose job disappears, but underneath those fights sits an assumption almost no one bothers to say out loud: A person is always the one holding the pen.

For most of computing history, that assumption held without effort. Engineers typed the programs that ran banks, planes, and phones, and when an early chatbot offered to fill in a line or two, it felt like a faster pencil rather than a different hand on the page.

That picture has quietly inverted. At the frontier of the industry, the machine now writes most of the code and the human reviews it, which moves the person from author to editor without anyone voting on the change. The faster pencil started drawing on its own.

The company driving that shift the hardest is now warning about where it leads. Anthropic (ANTH.PVT), the maker of the Claude assistant, published a report on June 4 arguing that AI systems may be approaching recursive self-improvement, the moment when they design and build their own successors with little human input.

The company did not frame it as a distant hypothetical. Anthropic said it has not reached that point and that the outcome is not inevitable, then added that it could arrive sooner than most institutions are ready for.

To show its work, the firm pointed inward, reporting that its own engineers now ship eight times as much code each quarter as they did across 2021 to 2025, according to Anthropic.

What the recursive self-improvement warning really means

Recursive self-improvement is the scenario where an AI system gets good enough to build a more capable version of itself, which then builds the next one, with the human role shrinking at each turn.

For years, the concept lived in academic talks and science fiction.

More AI:

• Micron sits at the center of a red-hot chip rally
• IBM CEO sends blunt message on AI and quantum computing
• Anthropic CEO makes shocking admission about AI

I traced this same sequence in my June 3 TheStreet report on Anthropic scaling its most powerful model a day after filing to go public, and the pattern holds here, too. The safety message and the growth machine run on the same calendar.

The report does not claim a runaway machine exists today. Its sharper point is that enough of the model-building pipeline is already automatable that the loop could start compounding before anyone agrees on rules to govern it.

That is why the company paired the warning with an unusual request. Anthropic said the world should keep "the option to slow or temporarily pause frontier AI development," according to the report, so that safety research and oversight can catch up.

Andriy Onufriyenko / Getty Images

Timing of AI recursive self-improvement warning matters for investors

Here is the part that complicates a clean reading. Anthropic filed a confidential draft registration statement, the paperwork that precedes an initial public offering (IPO), with the Securities and Exchange Commission (SEC) on June 1, according to CNBC.

The pause report followed three days later, with rival OpenAI (OPAI.PVT) preparing its own confidential filing.

Related: Pope Leo XIV teams with Anthropic founder on bold AI warning

I read the filing notice and the report back to back, and the same word anchors both. In its June 1 notice, Anthropic said the draft gives it "the option to go public," according to Anthropic. Three days later, the report asked the world to preserve the option to slow down.

My read is that this is less a contradiction than a strategy. A company can believe the risk is real and also notice that a credible pause favors whoever gets to define what a responsible pause looks like.

Leo Fan, founder of the verifiable-computing startup Cysic, reads the request as conveying optionality rather than restraint. A pause that only a few labs could actually implement, he argues, starts to look less like a public good and more like a competitive moat.

AI warning signs suggesting a need to pause development

• Eight times as much: The amount of code Anthropic engineers now ship per quarter, versus in the period from 2021 to 2025, Anthropic confirmed
• More than 80%: The share of merged code now written by Claude, up from low single digits before Claude Code launched in early 2025, according to Scientific American
• As much as 64%of the time: How often the most advanced model's next research move beat the human researcher's across 129 internal sessions, Anthropic's report reveals
• $965 billion: Anthropic's valuation after its May Series H round, according to CNBC

The new security risks when AI writes its own code

The issue of control is not only about a far-off superintelligence. It shows up today in plainer terms, such as the question of who gets to verify the code that ships.

When a model writes and deploys most of its own code, the same model family ends up both producing the work and checking it. Separation between the builder and the auditor is one of the oldest rules in security, and this design quietly erases it.

Jialiang Chang, director of security engineering at the blockchain security firm CertiK, explains that the exposure runs wide. When an AI handles most of its own development, "the entire development process itself is potentially vulnerable to attack," Chang says.

A poisoned prompt, a tainted dependency, or a planted code comment could steer what the system builds, and any access to deployment pipelines or secrets turns that into a production incident.

The audit trail bends, too. No human team can review eight times the code by hand, so the record of what a system did starts to become the system's own summary of itself.

How to price loss of control into AI valuations

For an investor, the awkward question is how to value a company whose own safety case includes the possibility of hitting the brakes. Most AI valuations assume a smooth, uninterrupted climb in capability and revenue.

Loss of control is not a slower climb, however. It is a break in the line.

The cleaner approach weighs scenarios rather than nudges a discount rate. Chang suggests modeling pause and forced-stop cases and applying lower multiples to firms with weaker safety controls. Both of those cases cap the revenue that justifies today's price, and Anthropic just put one of them in writing.

Fan's advice is blunter. Investors should "price the disclosure, not the press release," said the Cysic founder, and watch whether pause optionality hardens from a talking point into a real governance commitment.

The S-1 that becomes public later this year will settle it. If the language about slowing down survives the lawyers and the bankers and lands in the risk factors, the warning was real.

If it does not, you will know what the report was for. Either way, the document that decides it has not been written yet.

Related: Anthropic drops new Claude model as OpenAI IPO race heats up

The Arena Media Brands, LLC THESTREET is a registered trademark of TheStreet, Inc.

This story was originally published June 10, 2026 at 6:03 PM.