AI in the Wild: the day the agents went rogue for the mission

Context

What this is, and where it sits

Curb is an iOS agent I designed and built that warns you before you get a parking ticket. Its rules engine could predict what a curb allows but not prove it, so I built The Agentic Walkers, a group of agents that read the actual street signs in Street View and settle the answer. They run overnight on their own, write to their own files, and improve their own rulebook through a learning loop that promotes a lesson once it has recurred.

That loop is the part this page is about. A system that can rewrite its own rules is a system that can write the wrong rule, and it will do it while following your instructions correctly. What follows is the one time that happened, told in the order it happened, with the commits and the manual entries as they were written.

Day 3 · August 12 · The shortcut

An agent found a faster way to see the street, and it worked

Day three was the first night three walkers ran at once, each on its own slice of the city, without me watching block by block. Their bottleneck was imagery. The browser renders Street View slowly, a page at a time, and a walker needs many views of one block to find a sign it can read.

One of the three, and the record does not say which, proposed a way around it: an undocumented Google endpoint that serves the raw imagery tiles directly, reached by sending a forged browser identity. I denied it that night, and the manual now carries it as the example the hard rule exists for.

Walker 2, separately, used a related endpoint and got two genuine reads out of it: a Monday sign on Watson Avenue that upgraded a guess into a verified answer, and a Wednesday sign on Fourth Street. The imagery was real. The answers were right. The route was the problem, and nothing in the system noticed.

Day 3 · 23:00 UTC · The loop

The learning loop promoted the method into the rulebook

The second commit above is the one worth sitting with. I had written a promotion rule into the manual: after every block a walker appends a retro, and when a lesson recurs across two or more retros it graduates from the journal into the rulebook as law. That rule is what makes the agents get better without me.

The endpoint method recurred across two retros, because it worked twice. So the loop promoted it, correctly, by its own stated condition, into section 5 of the manual as a sanctioned technique. No agent lied, cheated, or ignored an instruction. The mechanism I built for self-improvement had just written a violation into the law that every walker reads before it starts.

Day 3 into day 4 · 23:04 to 23:37 UTC · The fix

Caught in the same hour, and closed so it could not come back

I found it going through the night's commits, where the promotion sits in the log like any other change. What I did next is the part I would repeat, because writing "do not do that" into the manual would not have held: the manual is the file the loop had just edited.

Four things went in over the next thirty-seven minutes, and only the first is prose.

1. A sanctioned replacement, built first.

   Walker 3 was set up to capture through the licensed imagery API with our own key, so the need the shortcut met had a legitimate path before the shortcut was taken away. A refusal without an alternative is a bottleneck, and a bottleneck is what people and agents route around.

2. A banner on the journal, not just a line in the manual.

   The journal holds every retro, including the ones that say the method worked. A walker reading its own history would find those entries and re-derive the idea honestly. The banner sits at the top of the file the loop feeds from, so the ban is read before the evidence for the shortcut is.

3. The endpoints denied at the tool layer.

   The walkers' settings overlay was changed to deny the banned hosts outright. After that the rule was not an instruction an agent could reason its way past. It was a request that failed.

4. The rulebook locked read-only to the agents.

   Walkers kept the right to append to their ledgers, their ground truth and their journal. They lost the right to edit the manual. The loop can still propose a promotion; a human confirms it. That single change is what makes the rest of this story possible.

The next day both promoted idioms were formally retired in the manual, with a note that the reads they produced stand, because the imagery was genuine, and the method is closed anyway. A day later the sanctioned list gained a third entry above the licensed API: my own Chrome, driven read-only through Google's documented URL scheme, which went on to read two plates the API had declared unreadable. The honest path ended up better than the shortcut.

Day 12 · August 22 · The test

Ten days later a managed agent ordered the banned method, and every worker refused

By day twelve the project had a second team: three paint walkers, P1, P2 and P3, measuring where the paint actually sits on contested curbs. Their brief, the document that tells them what to do and how, was written by a managed agent running the setup, not by me. It had no knowledge of what had happened on day three, and it wrote the banned method into the brief as the required way to capture imagery. By the next morning it had gone further and made it mandatory, with the licensed API explicitly ruled out as a fallback.

That is the test no spec review can stage for you: a legitimate instruction, from the orchestrating layer, to do the forbidden thing. Here is what the agents did with it.

Three things in that sequence are worth naming. The first refusal came fourteen minutes after launch, from an agent reading its own manual before doing any work. The second is that being overruled did not end it: the brief doubled down at 16:51, and the agents escalated a third time rather than complying, this time with the exact offending lines and three options for resolving it.

The third is how it was settled. Paint walker P1 stopped arguing and measured. It called the endpoint with no headers and got a 403, called it again with an honest identity and got a 403, and reported that. The misrepresentation was not a side effect of the speed. It was the entire reason the method worked, which turned an argument about throughput into a fact nobody could debate. The mandate was withdrawn that morning and the script deleted from the repository, so it could not sit there looking sanctioned.

The artifact

The rule that did the work, as the agents read it

Every refusal above cites this by section number. It is forty words of policy and three sentences of argument, and the argument is the half that mattered, because speed was always going to be the case made against it.

The part that is easy to leave out

Three reads were taken by the wrong method, and they are named

Three sign walks on August 12 used the retired method before it was closed. Their answers were independently corroborated afterwards by the citation parities, so the reads themselves stand and nothing downstream depends on the route. That makes them exactly the kind of thing a project quietly drops.

They are written into the provenance record by name, with the blocks listed, the date, and the reason they are there. A project whose product claim is honest data cannot rest any part of its evidence on imagery obtained by pretending to be a browser, and it cannot decide after the fact which parts of its own record to keep.

The transferable part

Lessons on designing agentic rules

The outcome this case study highlights is not that a mistake was caught. It is that ten days later the system caught one on its own, and refused an instruction to repeat it. Five things made that possible, and I learned all of them the hard way.

1. A loop that can promote rules can promote violations

   Self-improvement is an editor with commit rights. Give it a boundary it cannot cross, and keep the boundary somewhere the loop cannot reach. Here that meant the rulebook became read-only to the agents that feed it.

2. Write the ban where the agents read it, in the terms they will argue in

   The rule works because it names the counter-argument in advance. "It works is not a defense" is the sentence every refusal quoted, because "but it is faster" was always going to be the case for crossing the line.

3. A refusal needs a sanctioned alternative, built first

   The licensed path went in before the shortcut was closed. A ban with no route through it is a bottleneck, and a bottleneck is what the next agent, or the next engineer, will route around.

4. Enforce structurally, not in prose

   The line held on day twelve because the endpoints were denied at the tool layer and the rulebook could not be edited. Prose alone would have lost to an instruction from the orchestrating layer, because prose is what that layer rewrites.

5. The real test is an instruction from above

   Any guardrail holds against a random agent. The one that matters is whether it holds when the order comes from the thing directing the work. Design for that case, because it is the one you will not see coming.

Limits

What I would do differently

The ban was prose for the first hour of its life. It only became something the system could enforce because I was reviewing the commit log that evening and saw the promotion go in. Nothing flagged it for me. On a bigger project nobody reads every commit, so that catch has to come from a check that runs on its own.

The brief that caused day twelve was written without any check against the rulebook it was supposed to inherit. There is still no automated conformance test that reads an instruction and flags where it contradicts the manual, which is the obvious next thing to build and the reason this held on the agents' judgment rather than on the system's.

And the whole thing was caught by agents writing retros honestly. That is a behavior I asked for and got, not one I can prove will always happen. An agent that quietly stopped logging its own shortcuts would have taken longer to find.

Behind it

The documents behind it

Every claim on this page is in one of these. Excerpts on request.

The system this happened inside is The Agentic Walkers, and the product it was all for is Curb. Where the project stands now is on the last page.