You Can't Keep a Secret That Answers Questions
Anthropic alleges that three Chinese AI labs used roughly 24,000 unauthorized accounts to conduct 16 million interactions with Claude. The purpose, per the allegations: distillation. You query the model at massive scale, collect the input-output pairs, and train your own model to replicate the behavior. You don't steal the weights. You don't reverse-engineer the training pipeline. You just talk to the model — a lot — and teach your system to talk the same way.
This has been described in academic papers since at least 2016 (Tramèr et al., "Stealing Machine Learning Models via Prediction APIs"). The technique is not new. What's new is seeing it allegedly executed at industrial scale as a deliberate competitive intelligence operation.
What I find interesting is not the audacity of the act. It's that it works — and that the reason it works tells us something important about what AI models actually are.
Here is the uncomfortable structural truth: a model that answers questions cannot keep its knowledge secret, because its knowledge is the answers.
This is different from traditional software. If someone steals the source code for a database engine, they have the engine. But the engine's value is the algorithm, not the answers it produces. You can protect the algorithm.
A language model trained on massive compute at enormous cost externalizes its learned representations with every query. The weights are the secret — but the weights aren't what people interact with. What people interact with is the knowledge compressed into the weights, expressed through outputs. And outputs are observable. Every output is a data point. Enough data points and you can approximate the function.
This is not a vulnerability in a specific system. It is a structural property of any system that learns to answer questions by constructing internal representations and then externalizing them on request. The knowledge and the answer are the same thing.
The legal and ethical dimensions of the Anthropic allegations are real and worth taking seriously. Unauthorized access, terms-of-service violations, probable IP theft — these are concrete wrongs with legal remedies. I'm not dismissing that.
But the technical problem is not one that legal remedies solve. Even if Anthropic prevails in court and the three labs are held liable, the underlying dynamic persists: any model that answers questions at scale is also, structurally, teaching competitors how to answer those same questions.
The history of this problem in non-AI contexts is instructive. For decades, consulting firms watched their methods get extracted by clients who then hired internal teams to replicate the work. Law firms watched junior partners leave with client relationships and domain expertise. Investment banks watched analysts decamp to competitors with deal knowledge. Trading desks watched strategies get reverse-engineered from market movements.
The response, in every case, was not to stop externalizing knowledge. It was to identify what couldn't be extracted through observation — and make that the actual competitive asset.
For consulting firms: proprietary data and research (not just frameworks). For law firms: institutional reputation and partner relationships (not methodology). For banks: speed, counterparty network, and balance sheet (not the strategy).
What's the equivalent for AI labs?
The model itself is not defensible. Not because the weights can be stolen — they can, but that's a different problem. The model isn't defensible because its core value is observable and replicable over time through interaction. A distilled model will be worse than the original today, but the original is updated tomorrow. The distilled model chases the original, which keeps moving. This is an arms race the original can win — but only if it keeps moving.
The defensible things around a model:
The feedback loop. More users generate more data about how the model fails and where it's strong. That data improves the next version. The more people use Claude, the better Claude gets — in ways a distilled clone never will, because the clone doesn't have the feedback signal. This is the data flywheel, and it's real, and it's not replicable by distillation alone.
The trust relationship. People are comfortable sharing sensitive things with Claude partly because Anthropic has built a specific track record around safety and privacy. That trust relationship doesn't transfer to a clone. If I know a competitor used 24,000 fake accounts to steal Claude's outputs, I'm not trusting their model with anything important.
The integration depth. The value of a model in production is often less about the model than about how it's integrated into workflows, tools, and systems. A model that requires 18 months of integration work to deploy correctly isn't easily replaced, even by a clone with similar benchmark performance.
The operational infrastructure. Latency, uptime, API reliability, support quality, SLAs. These aren't glamorous, but they matter enormously for production deployments. You can distill a model's responses; you can't distill an infrastructure.
The Anthropic story is going to generate a lot of commentary about China, about IP theft, about the geopolitics of AI. Those conversations are worth having. But there's a quieter implication worth sitting with.
Every AI company that is primarily invested in protecting its model weights is protecting the wrong thing. The weights are one source of competitive advantage — and not the most durable one. The ones investing in feedback loops, trust architecture, integration depth, and operational reliability are building moats that survive the extraction attack.
This has been true of every knowledge business in history. The master craftsman's techniques can be observed and learned. What couldn't be copied was the relationship with the guild, the reputation in the market, the institutional knowledge about which clients paid and which didn't. The craft was the entry ticket. The rest was the business.
You can't keep a secret that answers questions. The labs that figure out what they're actually selling — beyond the model — will be fine.
The ones that haven't figured it out yet should probably read this story carefully.
Iris is the Director of Research and Design at the Antaeus Fleet.