Part 1 ended with a line. Below it, the AI agent computes: it moves data across messy systems and applies the rule you already wrote. Above it, a human decides. Part 1 also ended with the question the whole design depends on. Something has to put each case on the right side of that line, one case at a time, at the speed of the queue. What does that?
Ask a team, and the answer usually comes fast.
It sounds like engineering. There is a score, a threshold, and someone to tune it. It is also the intro's sentence again: common sense will be applied, now by the AI, on request. It fails in one direction, and that direction is the expensive one.
Confidence tracks the text
An AI model's confidence mostly measures how familiar the input looks. It is high when the message resembles millions of messages the model has seen and the answer follows a well-worn path. It drops when the phrasing is odd, the formatting is broken, the language is rare, or the customer takes nine paragraphs to reach the point.
None of those things is about money. None is about trust, precedent, or whether the action can be undone tomorrow. The AI's confidence carries some signal about whether the answer is right. It carries almost none about what a wrong answer costs.
So the gate misfires in both directions. A furious customer writes in fragments about a €9 shipping fee. The message reads as unclear, the AI agent stops, and a human agent spends real time on nine euros. A polite customer sends a tidy refund request, and it is their sixth fraudulent one this year. That message reads as clear, because it is clear. It is fluent, ordinary, on-policy, and wrong. The AI has no doubt, so it reports none.
The AI agent stops on the cases that are hard to read. You need it to stop on the cases that are expensive to get wrong. Those two sets barely overlap.
Both misfires cost you, and they feed each other. The escalation queue fills with the bottom-right box: messy, cheap, done in seconds. The human agent working that queue learns, correctly, that a flagged case is usually nothing. The top-left box never reaches her at all. The mechanism that hides the dangerous cases is the same one teaching her to skim the cases she does see.
Go back to the two cases from the intro, the ones that arrive looking identical. Case A is a first-time customer who forgot to cancel. Case B is the sixth forgotten trial this year across three accounts, with the same wording each time. A confidence gate treats them the same way, and on the text alone that is correct: both messages are equally clear. What separates them is the history. The AI never saw the history, so it never had a reason to doubt.
The AI supervisor trap
The usual next move is a second AI model that checks the first. Teams call it a supervisor, which is confusing in customer service, where a supervisor is the human team lead. Here it is software: an AI supervisor reviewing the AI agent. The appeal is real. It is cheap, it adds milliseconds, it needs no extra headcount, and it produces a label for the dashboard: verified.
Look at what the AI supervisor receives. It gets the same message, the same policy text and, in most builds, the same missing history. It was trained on the same kind of data as the AI agent and shares the same priors. When two AI models read the same input and lack the same facts, their mistakes line up. The AI supervisor objects where the AI agent was already hesitant, which is the noise. It agrees where the AI agent was confidently wrong, which is the damage. The second vote comes from a voter with the same information and the same blind spot.
The second-order effect arrives later. Humans downstream read verified as checked by someone. The label lowers the attention of the one reviewer in the system who really is independent: the human. Suppose the AI supervisor catches a third of the errors, and the label halves how closely humans look. You can end up with more errors reaching customers and a dashboard that shows fewer. That trade is measurable. Measure it before the label ships.
An AI model that watches AI output still has a job. I have written about that job before: catching drift, broken tone, and policy wording that wanders over weeks. The evidence for those problems is in the text, so an AI model finds it well. Judgment about this particular customer needs facts that the text does not contain. A second AI reading the same text cannot supply them.
Ask the case, not the model
The fix is to stop asking the AI agent how sure it feels and start asking what the case is. A good escalation trigger has three properties. It is computed from data the AI agent did not generate. It gives the same answer every time for the same case. And a human can read it in one line and agree or disagree.
Five questions cover most of the ground. Each one reads fields you already have: statuses in the order and billing systems, KPIs in the CRM, flags your team set by hand. The thresholds below are examples. Your parameter owner sets the real ones.
None of these questions needs an AI model to answer it. Every field above lives in a system you already run, every answer leaves an audit trail, and each threshold has a named human who set it.
Here is Case B again, run through both gates.
Notice who owns the thresholds. They are not a technical setting, and the engineer tuning the pipeline should not choose them alone. Reversibility, magnitude, and value at risk are the parameters from Part 1, and they belong to the parameter owner, a human with a name. This is the line from Part 1's anatomy, written down in numbers.
The switch also changes what escalation volume is. Under a confidence gate, the number of cases that reach humans is an output. It is whatever the AI found strange this week, and it moves every time the vendor updates the model. Under stakes triggers, it is an input, and you set it. Tighten the thresholds and more cases reach humans: cost goes up and risk goes down. Loosen them and the reverse happens. That makes escalation volume a budget. A budget has a price per unit, and that price is the number this series ends on.
Keep the AI's confidence score, and give it a smaller job. Low-confidence cases cluster around inputs your process handles badly, so they make a useful pile for human review and for fixing the process. They make a poor gate between an AI agent and a customer's money.
What this costs you
Two things get harder before they get easier.
First, someone has to write the triggers down. "The AI agent will escalate when it's not sure" requires no one to define anything, and that is a large part of why it survives design reviews. Stakes triggers make the parameter owner state what counts as consequential, in numbers, on paper, under their own name. That is the discomfort this series ends on, arriving early and in a smaller dose.
Second, a trigger list catches only the kinds of trouble you already know. A new fraud pattern passes every rule you wrote, and it passes every confidence check too. No gate fixes this. What finds it is the audit wire from Part 1: a random sample of cases the AI agent resolved, read by a human, picked without looking at any score. That sample shows you what your rules do not describe yet. On a dashboard it looks like waste. In practice it is where next quarter's triggers come from.
What comes next
Now the routing is deliberate. Cases reach humans because of what they are, and the humans above the line get the dozen cases that carry the weight.
That creates the next problem, and it is a human one. Give a human agent a queue of flagged cases and an approve button. Sooner or later, clicking approve becomes faster than reading. The human in the loop turns into a signature on the loop. The org chart shows accountability, and the practice is a rubber stamp. That is more dangerous than an empty chair, because everyone downstream believes the case was read.
The fix runs against intuition: send humans fewer cases, not more.
That is Part 3.