Key takeaways
- Half of our charger incidents clear after a simple remote reset, and more recover on their own. The remaining 40% need judgment, and that’s the job of our AI ops agent.
- Handing the model one complete incident packet beat letting it call tools: a seventh of the tokens in a tenth of the time.
- When the agent got something wrong, it was almost always missing context, not following a bad prompt. We fix the evidence in code and freeze the case as an eval.
- Apart from simple offline and emergency-stop notices, a person still approves every email to a customer or vendor. That changes when 95% of recommendations are approved exactly as written.
Overnight on a Wednesday in September, a charger at a car dealership started reporting a fault: its ground-fault protection had failed a self-test, and connector 1 locked itself out.
Our system opened an incident within minutes and did what usually works: it sent the station a remote reset. The station rebooted, and the fault came straight back.
The next step in our playbook is to ask the site owner to power-cycle the charger at the breaker, a fix for more failures than you’d expect. The agent recommended exactly that, and the request went out to the dealership. They wrote back: they’d shut the chargers down, tried every reset they knew, and needed help.
A few months ago, this is where an ops engineer would have taken over, piecing the story together from the OCPP logs, the reset history and the customer’s email before deciding who should act next.
This time, an agent did that work. It matched the customer’s reply to the reboots in the OCPP log, and from that concluded the breaker reset had been done. It saw that the fault had returned after every restart, so another reset wasn’t going to help. And it found that nobody had contacted the manufacturer yet.
So it drafted an escalation to the vendor, with the fault history attached, and posted it to Slack. An ops engineer read the reasoning, agreed, and approved it.
This post is about how we got there: what we built, what the chargers taught us, and why almost every mistake our agent made came down to what it could see, not what we told it.
From alerts to incidents
We didn’t set out to build an agent. We set out to notice broken chargers.
The first version was a set of detectors, each watching for one symptom: a station that stopped sending heartbeats, a connector flapping between Available and Faulted, sessions that delivered no energy, vendor fault codes. Each worked on its own. Together they were noise. One broken charger could trip three detectors and produce three alerts, with nothing tying them together and no record of what anyone had done about them.
So we started modeling incidents properly. A detection opens an incident for the station, or joins the one already open, and every incident follows a playbook. For most, the first step is a remote reset. The incident then waits in pending verification while the station reboots, and we check again. If the station has recovered, the incident resolves itself. If not, the playbook is exhausted and the incident escalates to a person.
The remote reset is the cheapest win in the system. It costs nothing, and it works: about half of all incidents cleared after one, with no human involved. How many of those the reset fixed, rather than the station coming back on its own, is harder to say, but either way nobody had to look. Counting stations that recovered without a reset, six in ten incidents never needed a person.
But “escalates to a person” was hiding a problem. We were opening around 335 incidents a week, and about 130 of them needed someone to dig through logs, history and email before deciding what to do.
Why rules weren’t enough
The obvious answer was more playbook rules, and we tried that first. A rule like “five faults in a week, raise it with the customer” sounds reasonable until you meet the exceptions. Some faults are harmless and clear themselves; emailing a customer about them teaches them to ignore us. Others are serious the first time they appear. Every exception needed another rule, and every new rule had exceptions of its own.
The hard part of an escalated incident isn’t the fault. It’s everything around it.
Take a connector stuck in Faulted. Whether to ask the customer for a breaker reset, escalate to the manufacturer, or leave it alone depends on questions no single rule can answer. Has this fault happened before, and what fixed it last time? Did the customer already try a reset, and say so in an email three days ago? Is there already an open thread with the vendor? Is this an error code we’ve never seen, from a manufacturer whose documentation doesn’t mention it?
The answers were scattered across OCPP logs, incident history, a shared support mailbox, station notes, and the ops team’s own case list. An experienced ops engineer pulled them together almost without thinking. Our rules couldn’t, because most of that evidence wasn’t structured at all. It was a sentence in an email, or a pattern across a week of logs.
So we stopped asking how to write better rules, and asked a different question: what if we built something that worked an incident the way our ops engineers do?
How the ops agent works
The ops agent starts the way an engineer would: by reading everything. For each incident it gathers the station’s recent OCPP traffic, related incidents, what happened the last time this fault appeared, emails with the customer and vendor, station notes, the ops team’s case list, and what our error-code catalog says about the fault.
How that evidence reaches the model mattered more than we expected. Our first version was a classic tool-calling agent: the model looked something up, read the result, and decided what to look up next, re-sending its growing context every time. So we flipped it around. Our code now gathers everything up front into a single incident packet, and the model decides in one call, using a seventh of the tokens in a tenth of the time. We also tried letting the model write code to explore the logs itself. It was even slower, and triage isn’t exploration. It’s “here is the evidence, decide.”
From the packet, the agent picks one of five decisions: monitor, resolve, remote reset, ask the customer for a site inspection, or escalate to the vendor. The most common, by far, is to do nothing yet. Of roughly 9,000 decisions, more than half were monitor. Knowing when not to act matters as much as knowing what to do. Every decision comes with a rationale that cites its evidence, and that rationale, not just the answer, is what our reviewers read.
What the agent may do on its own depends on who it affects. Internal decisions are its job: it owns closing the incidents that reach it, and in eight weeks it closed more than 400 of them after confirming the problem had cleared. Anything that leaves the building, like an email to a customer or a vendor, waits in Slack for a person to approve it. The exception is two simple, time-sensitive notices, offline stations and pressed emergency stops, which go straight to the customer. More on those next.
What the chargers taught us
Much of what makes the agent useful isn’t clever modeling. It’s things we learned about chargers the hard way.
“Offline” isn’t one problem, and a remote reset can’t fix it. A charger talks to us through an OCPP process running on the charger itself. From the cloud, a hung process, a tripped breaker and a dead network connection look identical: the heartbeats stop. A remote reset doesn’t help with any of them, because the reset is itself an OCPP command and never gets through. What does help is the bluntest fix there is: someone on site cutting the power at the breaker and turning it back on. So once a station has been silent for about five hours, we email the owner, asking them to check the breaker and power-cycle the station, and we send a daily reminder until it’s back. Often the reply is one line: “Reset the breaker. It appears to be working now.” Plenty of stations also come back on their own when the site’s network or power returns. The ones that don’t tell us something too. At one site, the owner reset the breaker, the screens showed “Available”, and the station never reconnected. Power was fine; communication wasn’t, which made it a job for the manufacturer.
Drivers press the emergency stop. This one surprised us. When a driver can’t get the connector cable to unlock, some of them press the charger’s emergency stop to free it, then leave. That one press takes the whole station out of service: no one can charge there until someone on site releases the button. Often nobody on site even knows it was pressed. So when we see an emergency stop, we email the site owner immediately.
None of this needed a model. It needed someone to notice a pattern and turn it into a rule, or into evidence the agent could see.
How we debug an agent
When a recommendation looks wrong, we don’t argue with the answer; we read how the agent got there. Every run is traced, so we can pull up the model’s reasoning and ask one question: did it reason badly, or did it reason well from bad evidence?
Almost every time, it’s the second. One example: our ops team tracks live cases in a Slack list, not in our system. The agent couldn’t see that list, so it kept proposing to email customers about problems a person was already handling. Its reasoning was fine; it was missing what everyone on the team knew. The fix wasn’t a new prompt rule. It was adding the Slack list to the packet.
Most of our fixes look like that: waiting a few minutes after a reset before judging whether it worked, or waiting until a fault has persisted before escalating it. Each one is a change to what the agent sees or when it looks, written as ordinary code. A context fix makes a whole class of mistakes impossible; a prompt rule only makes them less likely.
Sometimes the prompt really is the problem. The agent once read a charger’s own reboots as proof that someone had reset it, and two sentences of clarification fixed it. But that’s the exception.
Then we make sure the mistake can’t come back. Because every decision comes from a single packet, any mistake can be frozen into a test case: save the packet, write down the decisions we’d accept, done. The incident packet, which we built for speed, turned out to be what made the agent debuggable.
The road to 95%
Today, apart from the simple offline and emergency-stop notices, every email the agent wants to send to a customer or a vendor still waits for a person to approve it in Slack. That’s deliberate. We’ll take the human out of the loop when the agent earns it, and we’ve written down what earning it means: over a 30-day window, 95% of its recommendations approved exactly as the agent wrote them.
That number is easy to state and easy to fool yourself with. Approval measures agreement, not correctness, and it’s tempting to approve anything that looks roughly right. So approving isn’t a glance at the answer: we read the agent’s reasoning on every run. When an answer is wrong, the reasoning tells us why, and that’s what we fix.
Even then, the lesson of the last few months will hold. The agent will only ever be as good as what it can see. Most of the work of making it trustworthy hasn’t been in the model or the prompt. It’s been in the evidence.