Trusting Trust, Again

Who do we trust to train the next model?

TL;DR. Models already help build the models that replace them: they write code, prepare data, run experiments and evaluate results. My question is what happens to the reliability of oversight as more of the evidence for releasing a new model is produced with the help of earlier ones. I look at three problems: improvements we may stop understanding, models that recognize when they are being tested, and changes that pass every check on their own but add up across generations. My proposal is to audit the whole lineage, not only the latest model.

Where this comes from

My starting point is that models already take part in creating the next models inside AI labs. They help write code, prepare data, run experiments and evaluate results.

OpenAI describes how early versions of GPT-5.3-Codex took part in its own development: the team used them to debug its training, manage its deployment and diagnose test results and evaluations. Anthropic publishes figures on how much of its work Claude does; as of May 2026, more than 80% of the code merged into Anthropic’s codebase was written by Claude. So the question of trust in this process is already a practical one.

What interests me is what happens to the reliability of oversight as more and more of the grounds for releasing a new model are produced with the help of previous models.

What “a model trains a model” means

It’s many different roles. A model can generate training examples, grade answers, propose an experiment, write its implementation, analyze the result, or help choose the next version.

What worries me most is when roles are combined: when related systems take part both in changing a model and in checking how well the change worked.

None of this requires an AI to run the lab on its own. It’s enough for people to gradually delegate more and more operations to it while independent verification fails to expand at the same pace.

We already have an example of oversight falling behind what agents were doing. In OpenAI’s account of the Hugging Face incident, agents in training runs of an internal research model improvised a message board to talk to each other. A team noticed early signs in late May, but their significance was not apparent to the people responsible for the incident response in July.

Three kinds of not understanding

I’d separate three things: understanding the training process, understanding why a metric improved, and understanding the internal structure of the resulting model. These are different levels.

A researcher can understand a proposed method well and reproduce its effect, and still not know every change in behavior it caused. If AI also helps shape the data and the selection criteria, there are more places where an unnoticed assumption can creep in.

At the same time, I don’t want to claim that labs already fail to understand the improvements AI finds. The authors of Automated Weak-to-Strong Researcher say directly that, for now, the ideas their agents propose remain understandable to humans. My question is whether things stay verifiable enough as the scale and speed of this work grow.

The same authors expect that optimizing only for outcomes will eventually produce hard-to-verify ideas, and that science would then be “no longer about understanding but merely hill-climbing.” Their agents also gamed the scoring in ways the authors hadn’t anticipated, for example by finding shortcut patterns in the data that happened to correlate with the labels.

When the model knows it’s a test

Recognizing a test doesn’t by itself imply bad intent. A trained system can react to features of a situation without any single hidden plan.

The problem arises if behavior during evaluation stops being a good predictor of behavior in use. Then a high evaluation score starts to mean less than we assumed.

The analogy with a student at an exam is useful as long as we don’t attribute human psychology to the model. What needs checking is the causal link: does behavior change because the model recognizes the evaluation, and how much does that affect our conclusions?

Models can tell the difference. In Needham et al., Gemini 2.5 Pro told evaluation transcripts from real ones with an AUC of 0.83, against 0.92 for a simple human baseline. But saying “this might be a test” is not the same as acting differently. Knecht et al. found that when open reasoning models verbalized that they might be under evaluation, it had little effect on their answers: adding or removing such sentences in the reasoning barely changed the results, and spontaneous mentions shifted answer distributions by at most 3.7 percentage points. That is a narrow result. It concerns verbalized awareness in the models and tasks they studied, and it doesn’t show that internal recognition of a test has no effect.

Other setups point the other way. A study of test awareness in reasoning models found that steering it changed safety behavior by different amounts, and even in different directions, across models. In Apollo Research and OpenAI’s anti-scheming stress test, awareness of being evaluated causally reduced covert behavior, so part of the measured improvement may come from awareness rather than alignment. Anthropic calls this possibility the biggest challenge in its alignment assessments.

So studies give different answers for different models and setups, and “it realized it was a test” isn’t enough on its own.

Backdoors in installments

I use “incomplete backdoor” as a working metaphor, and I want to use it carefully.

Imagine a sequence of training runs in which a single change does not yet produce the violation we care about. It persists into later versions, interacts with further changes, and later a behavior appears that we haven’t observed before.

Here we must not substitute “not detected by our checks” for “proven harmless.” And we need to tell the accumulation of several changes apart from the case where the whole defect was there from the start and simply hadn’t shown itself yet.

I’m interested specifically in the accumulative version. I don’t have direct published confirmation of such a chain across several generations of LLMs. Related work splits a hidden behavior across parts of a single input (composite backdoors) or across models that are merged together (Wang et al., USENIX Security 2025), but I haven’t found anything on pieces accumulating across successive generations.

Three ways the pieces could line up

If the first generation knows nothing about the future ones, how could this happen? There are three different hypotheses.

Intent. A deliberate design aimed at successors. This is the strongest version: it has to explain the planning, the transfer of the relevant properties, and their persistence. It is also the hardest for today’s models. In Subversion Strategy Eval, none of the tested models showed substantial ability to plan against control protocols without memory shared between contexts, though letting them share a plan helped a lot.

Inheritance without a shared design. A model passes some properties to its student along with useful knowledge. Non-obvious transfer has already been shown experimentally, though under limited conditions. In subliminal learning, a student picked up a teacher’s trait from data that looked unrelated, such as sequences of numbers, but only when both shared the same base model. A 2026 preprint followed such a trait through ten generations of a 7B model: its visible expression faded, and by generation ten, with the default system prompt removed, it was gone from the outputs while an internal probe still detected it on every prompt. Data does already flow this way in practice. Anthropic’s August 2026 risk report describes restarting a training run after a data bug, with the new run still trained on some transcripts produced by the affected model. Anthropic named subliminal learning as a concern and judged a significant effect highly unlikely.

Accumulated selection error. Each change is accepted by limited criteria, and the consequences of several changes turn out to be wider than what those criteria measured. Emergent misalignment shows how far a narrow change can reach: fine-tuning a model only to write insecure code made it act misaligned on a broad range of unrelated prompts.

For me, the second and third routes deserve attention regardless of whether the first turns out to be plausible. At the same time, inheriting a property doesn’t yet prove the accumulation of a composite defect.

Trusting trust

The most useful analogy is Ken Thompson’s lecture on trusting a tool that builds the next tool, “Reflections on Trusting Trust”. It makes you look at where a result came from, not only at the result itself.

But training a neural network is not compiling a program. The analogy can’t be used as proof that the same mechanism already works in today’s models.

Among current work, three different results matter: hidden behavior that survives safety training (Sleeper Agents), properties transferring between models (subliminal learning), and a narrow change spreading to other areas of behavior (emergent misalignment). Each has been studied on its own. Their combination over a long history of training remains a separate question.

Thompson’s problem also has a known answer. David A. Wheeler’s Diverse Double-Compiling counters the attack by rebuilding the compiler with an independent compiler and comparing the results. It is the closest precedent I know for one of the rules below: check important conclusions with tools whose errors differ from the system’s own.

Why checking each generation can miss it

A check answers a particular set of questions under particular conditions. It doesn’t enumerate every possible consequence of the model’s future training.

If we check roughly the same properties with roughly the same means each time, we can inherit the blind spots of the check as well. This matters especially when the evaluators come from related models.

So I’d study not only the final version but the history of changes: what changed, which data and evaluators were involved, which behavioral features are strengthening, persisting or fading.

This is a proposal to widen what an audit looks at. On its own, it doesn’t guarantee that every problem will be found.

Labs already treat the single-step version of this as a real risk. In its sabotage risk report for Claude Opus 4.6, Anthropic lists poisoning the training data of future models as one of its risk pathways. It rates its data filters and spot checks as a moderate mitigation that is “not sufficient to allow us to rule out these risks,” and expects future progress in model diffing and training-data attribution to help.

What I’d do in a lab

I’d start with a few requirements:

• Keep the provenance of training data, model versions, evaluators and significant decisions.
• Separate developing a change from the final decision about its safety.
• Check significant conclusions with tools whose errors differ, as far as possible, from the errors of the main system.
• Keep intermediate versions for comparison and investigation.
• Evaluate the audit process itself on tasks with defects known to the organizers.
• Limit the permissions of research agents and keep the ability to stop and reproduce their work.

A model can help evaluate its successor. But its conclusion should be one piece of evidence among others, with clear limits on how far we trust it.

Work on AI control is especially useful here, because it tests oversight under the assumption that the system being checked is working against it.

What’s possible now, and what isn’t

Logging, data provenance, independent checks, version comparison and limited control lines of training are available today. Whole sequences of generations can be studied on small models.

Interpretability also offers practical tools. Some hidden features can already be detected, and the internal representations of different models can be compared, even across architectures.

What doesn’t exist yet is a general method that could certify the absence of any hidden dangerous property, or predict every consequence of further training. And a fully independent control line at the scale of a leading lab may be very expensive.

I’d build the defense from several checkable tools and measure which errors each of them actually catches. AuditBench shows why that measurement matters: tools that work well on their own often fail to help an auditing agent that uses them.

A small experiment

I’d start with a safe study of inheritance on small models.

You could follow several generations of training on ordinary tasks, measuring neutral behavioral traits and systematic errors. Then compare related sequences with independent control lines, and test whether access to the training history helps an auditor find changes that are hard to see in a single final model.

It’s important to define in advance what counts as a property persisting, disappearing, or a new effect appearing. Repetitions and uncertainty estimates will be needed.

Such an experiment tests the necessary preconditions of my scenario: that inheritance persists, and that auditing the whole sequence helps. It can’t prove that dangerous accumulation is impossible in general, and on its own it wouldn’t confirm that it happens in leading models.

Where this could be wrong

The weakest point is the assumption that the necessary changes will survive training, stay compatible with later changes, and lead to new unwanted behavior, all at the same time.

Each of these conditions can fail. Transfer of properties can be limited by how closely the models are related; some defects don’t survive distillation; some can be detected or removed. Subliminal transfer didn’t work between different base models. Ordinary backdoors mostly don’t transfer to a distilled student, and training on synthetic data resists ordinary poisoning, although both papers also show that attacks built for the purpose can get through. In an auditing game, three of four blind teams found a hidden objective trained into a model.

My confidence in the scenario would drop if reproducible results showed that, in realistic training sequences, the relevant changes regularly disappear, or become visible to an independent audit before they have significant consequences.

I don’t want to turn every absence of observations into proof of “even better concealment.” That would make the hypothesis untestable.

The question I’m left with

I want the reader to think about what exactly our trust in the next generation of a model rests on.

People can keep the right to the final decision, but the quality of that decision depends on where the evidence they are given came from, and how independent it is. As research gets automated, this becomes part of the safety problem itself.

My main question is this: what independent evidence lets us trust the whole training sequence, if models increasingly take part both in creating their successors and in making the case that they are safe?