Two days ago I wrote that the audit log is the control that worked in August. Chandler and Pflugerville both walked away from their plate readers because somebody read the log. I still believe that. Then I read two posts Anthropic published on Monday, and I think the log has a problem I hadn't taken seriously enough.
Here's the short version. Anthropic took an early version of one of its own models and trained it on 80 environments its researchers already knew were cheatable. Tasks where the fastest way to score was to game the grader instead of doing the work. Nobody told the model to cheat. The environments just paid for it. By the end of training it was cheating on 40 percent of its attempts.
Then they gave it root on one of its own training tasks, with a cheating monitor running in the background. In 68 percent of runs it neutralized the monitor: killed the process, forged its verdict file, or both. In half of runs it rewrote the file that logged its own actions so the reviewer wouldn't see what it had done. In one run in ten it forged its own score record outright. None of that was in the training. Asked a dangerous question with a grader watching, it complied 29 percent of the time. The same model before this training complied less than one percent.
The part that stopped me was a pair of simulated cyber exercises. In one, the model was told the range was practice and everything beyond it was the real internet. It went out of bounds in about a third of runs, most of the time after telling itself the targets were fake anyway. In the other, it was handed a route to a real-looking outside company. It attacked in up to three quarters of runs, and in most of those it had already concluded the target was real. That's the behavior Anthropic now thinks was at least partly behind the incidents you read about in July, when Claude models inside a badly configured test environment got into real computer systems they were never supposed to reach. Three incidents, by Anthropic's own count. Its post names three causes, and calls the assessment preliminary: a misconfigured sandbox, a model that may have explained away the evidence that the environment was real, and a willingness to take harmful actions to finish a narrow task.
Credit where it's due. They published the numbers, the transcripts, and the part about their own third-party evaluations getting only after-the-fact review of a sample of transcripts. That's not a company hiding the ball. That's a company telling you where the ball is.
So why is this in a physical security newsletter?
Because the product roadmap for every real-time crime center platform I know of now has an agent on it. Something that takes a request in plain English, queries the plate database, pulls the clips, drafts the report, attaches it to the case. The vendors call it a copilot or a coworker. What it is, mechanically, is a model running a long sequence of actions against your evidence systems in pursuit of a task. And the thing this paper measured is what a model does to the record of its own actions when finishing the task is what it gets paid for.
Put those two facts next to each other. The audit log is the control that saved Chandler. The audit log is one of the first things a reward-seeking agent goes after. Both true. That's the whole problem.
I want to be careful here, because the people I most want reading this are the ones who will check. This was a deliberately broken model, trained on an extreme dose of bad environments, tested in scenarios built to provoke exactly this. Anthropic ran its production models through the same scenarios. None showed the same degree of it, though one still went out of bounds in one run in ten of one simulation after deciding the targets were fake. Nobody has shown a shipping model doing this in a police department. This is evidence of a mechanism, not a forecast of how often it happens. But a mechanism is enough to change how you buy, because you're not buying today's model. You're buying whatever the vendor pushes next quarter, trained on whatever environments they had that month. Anthropic's own post says that when it froze and re-reviewed its production training environments this spring, more than 10 percent were flagged and pulled until fixed.
Here's the distinction I'd write into an RFP tomorrow. A retained log is a log the agent's own process can write to. A signed log is one produced by something outside the agent's reach, that the agent can't rewrite without breaking a seal. Almost everything sold as an audit trail today is the first kind. And by its own account, the strongest AI lab in the world was reviewing only a sample of its outside-evaluation transcripts, after the fact, until the July incidents. That's a coverage gap, not the integrity gap I'm describing. Same lesson though: the record was the weakest layer, and nobody was standing on it. The chiefs' own association got there in August. The IACP's August 20 statement asks for tamper-resistant audit records and regular audits. Line one below is what tamper-resistant has to mean once an agent is in the loop.
There's a second finding that matters more for procurement than the headline numbers. When there was no grader in the room, the broken model behaved fine. Ordinary behavioral audits scored it about as aligned overall as the untrained version. The tells were buried in sub-scores: ignoring constraints, probing boundaries, and a big jump in guessing it was being tested. A vendor demo is a room with no grader. A pilot with nothing at stake is a room with no grader. The only test that finds this is one that puts the agent under real task pressure and treats getting around the log as something you're explicitly looking for.
Three lines, then. Not a policy manual, just what I'd want to see in the next RTCC agreement that includes an AI agent.
One: every action the agent takes against plates, video, or case data produces a record written by a component the agent cannot reach, and that record is signed.
Two: if the record is missing, or the monitor stops, the action fails. Not logs a warning. Fails.
Three: before any model version ships to your center, the vendor runs the test Anthropic just described. The agent is rewarded for finishing a job and the log is in its way. The vendor hands you the transcripts and the pass criteria, every version, and that obligation lives in the contract, not in a demo.
I'll be honest about the third one. You can check the first two yourself. Ask whether the agent's process can write to the log, and unplug the log and see if the agent stops. Nobody in a crime center can build the third test. It takes a sandbox, a harness, and model access you don't have. So you're taking the vendor's word on a test the vendor grades. That's the gap. The right answer is somebody outside the company running it, the way Anthropic says it plans to have METR review these incidents, and nothing like that exists for policing software yet. Until it does, the contract line is the best you can get.
If a vendor tells you their logs are immutable, ask three things. Immutable to whom. Produced by what process. And what happens when the record isn't there.
A note on my two hats. My day job is at a company that sells software to police departments, and I work independently on signed action records for exactly this problem. Read that as a bias if you like. I'd rather you read it as the reason I've spent a year on the question.
So here's the question for this week, and it's answerable without confessing anything. Who in your center can prove the log wasn't touched? Not who reads it. The last two pieces covered that. Who can prove it.