The Tool That Vanished Underneath Our Skill

Here is the uncomfortable part. The skill was correct the day we shipped it. Every instruction matched a real tool. It passed review.

Then the platform moved. And “it worked when we shipped it” quietly stopped meaning “it works now.”

⏳ What is tool drift?

Tool drift is when the platform an AI agent runs on changes the tools available underneath a skill after that skill has shipped. The skill's instructions stay frozen, but a tool they depend on gets renamed, removed, or altered, so a once-correct call now fails.

It is not a bug you wrote. It is a bug that grew around code you never touched.

To make it concrete, here is the exact call our skill kept making. Toggle between the day it shipped, today, and the fix.

Same instruction, three worlds. On ship day it resolved. Today the middle box turns amber and the whole protocol halts. Nothing in the skill file changed.

💥 The break we almost shipped forever

The headline finding sat in a file called team-protocol.md. It opened with a rule marked “HARD RULE, no exceptions.”

The rule: any time the protocol spawns two or more agents, it must call TeamCreate first, spawn each agent with a team_name, message them through the team, and tear the team down at the end with TeamDelete.

That was a faithful description of how multi-agent work used to happen. The harness has since deleted that entire API. There is now a single implicit team, and you fan out by making parallel agent calls in one message.

Worse, a note in an old handover log already said, in plain text, “TeamCreate is NOT available in this harness.” A prior session had hit the wall and routed around it by hand. The skill itself was never patched. The knowledge existed; it just never made it back into the file that gives the orders.

🕳️ Logging into the void

The second real bug was quieter but just as revealing. The skill ends every run by logging the outcome — the score, the task type, the verdict — so the tool can learn from itself over time.

Three internal sources disagreed on when to log. One said log before the run is torn down. One said after. A third didn't mention logging at all.

The “after” path won by default. But tearing down the run clears its state first, so the logger read an empty run and wrote a receipt with no numbers on it.

The fix made one order canonical everywhere — log then tear down — and, as a belt-and-suspenders guard, taught the logger to recover the just-ended run if it ever finds the pointer already cleared. We verified both paths against the live runner: the recovered log now carries the real task type instead of “unknown.”

🔬 The full audit ledger

Tool drift was the headline, but an honest audit reads everything. Nine findings survived verification. Click any one to see where it lived and how it was fixed.

The privacy one is worth a sentence on its own. The logger sanitized the user's free-text notes before sending them to our server, but a second model-authored text field slipped past the filter unsanitized. It now runs through the same scrub. Full text stays only in the local log.

👻 Why drift is invisible

None of this shows up in the places teams usually look. That is what makes drift dangerous rather than merely annoying.

✅ How do you audit for tool drift?

You cannot diff against a spec you don't have, so you build the spec fresh each time from the platform as it is now. The checklist we ran:

1. Inventory every tool your skill names. Pull each tool and flag out of the instructions into a flat list.
2. Diff that list against the platform's current tool set. Anything named that no longer exists is drift. Anything “deprecated; ignored” is drift wearing a disguise.
3. Run the happy path end to end. Not unit tests — the actual flow, so a vanished tool has somewhere to fail loudly.
4. Treat “mandatory” steps as the highest risk. A rule the model can't skip around is a rule that detonates instead of degrading.
5. Find every place three sources should agree — docs, code, and runtime state — and confirm they still do.
6. Re-audit on every platform version bump, not just when you change your own code. The drift arrives while your files sit still.

📦 What shipped

A receipt, not a brag. This is the diff that closed all nine findings, straight from git.

The honest note: these are diff stats, not a scored build. Tool drift is caught by reading, not by a benchmark, and no scoring run happened here — so there is no composite to quote. The number that matters is the one finding that would have broken every multi-agent run, sitting under a rule marked “no exceptions,” invisible until someone read the skill against the world as it is now.