Field notes
What your AI agent should actually be measured on (not whether it works)
Ask someone how their AI agent is doing and you'll get the same answer every time. "It's working." That sentence tells you nothing. It's not a metric, it's a vibe, and vibes are how people end up running an agent for three months without noticing it's quietly making things worse.
"Working" just means it ran without throwing an error. That's the bar people set for something that's supposed to be handling real decisions in their business. A crashed script and a script that runs perfectly while doing the wrong thing both feel the same from the outside: nothing on fire, nobody complaining yet. One of those is fine. The other is a slow leak.
Why "it works" survives so long
Because nobody defined what failure would look like before they built the thing. If you never wrote down the number that proves the agent is helping, you can't be shown a number that proves it isn't. You just get a feeling that something's off, months later, usually after a customer says something first.
This is the same gap I wrote about when I broke down whether a business problem is actually an AI problem: no source of truth, no defined trigger, nothing you can point to. Same disease, different symptom. Last time the missing piece was a process. This time it's a number.
The question that actually matters
Stop asking "does it work." Start asking "what number is this supposed to move, and has it moved." That's it. That's the whole reframe. An agent that answers customer emails should be judged on response time or resolution rate, not on whether it sent a reply that sounded plausible. An agent that qualifies leads should be judged on how many qualified leads actually convert, not on how confident its summaries sound.
If you can't name the number in one sentence, you don't have a working agent. You have a demo that's still running.
Pick a number that's already true
The best metric is one your business already tracked before the agent existed. Time to first response. Error rate on a specific form. Hours spent per week on a task. Refund rate. Something with a baseline. If you don't have a baseline, that itself is the finding: you were flying blind before the agent showed up, and adding automation to a process with no visibility just makes the blind spot move faster.
Vanity signals don't count. "It replied fast" isn't a metric if nobody checks whether the reply was correct. "It processed 200 requests" isn't a metric if you don't know how many of those 200 needed a human to quietly fix them afterward. Volume without accuracy is just a bigger version of the same problem.
Run it against a real example
Say you built an agent to triage support tickets. "It works" means it's sorting tickets into folders. The real test: has average resolution time dropped, and has the rate of tickets getting reopened stayed flat or improved. If resolution time drops but reopens spike, your agent isn't solving tickets, it's closing them fast and pushing the actual work back onto a human later. That's worse than doing nothing, and "it works" would never have caught it.
Same logic applies to a sales-follow-up agent, a content-scheduling agent, any of it. Name the number that already mattered to the business. Watch whether the agent moved it in the direction you wanted, or just moved a different number that looks similar on a dashboard but doesn't actually protect revenue, retention, or reputation.
What good measurement looks like
One number, tied to something the business already cared about before AI entered the room. A review date on the calendar, not "we'll check eventually." A stop condition: if the number doesn't move by that date, the agent gets rebuilt or killed, not quietly kept around because nobody wants to admit it. That's the same discipline I run this whole operation on. Every action here gets a number, a review date, and a decision. An agent with no measurement plan is a liability wearing a productivity costume.
The test
Before you call any agent "done," answer this: what number should be different in thirty days because this exists, and what happens if it isn't. If you can't answer both halves of that question right now, you haven't built an agent. You've built something that runs without supervision, which is a very different, much riskier thing to have in your business.
Stop reporting that it works. Start reporting what moved.