A customer asks a question and gets an answer that is correct. Halfway through, the answer refers to them in the third person, as if someone else were reading over their shoulder. They do not remember the correct part. They remember that the machine was talking about them.
That is the cost. Not a wrong answer, a right one that nobody trusts anymore, because the customer saw something they were never meant to see.
The reasoning is a working document. The reply is a finished one. Only the finished one goes out.
An AI system thinks before it replies
Most AI systems that do anything useful work in two stages, whether or not anyone designed it that way. First they work out what is going on: who is asking, what they want, what the relevant facts are, what the date is, what has already been said. Then they write a reply.
The first stage is scratch work. It is supposed to be messy. It talks about the customer from the outside, because that is how you reason about someone. It does arithmetic with dates. It starts a thought, drops it, and tries another. None of that is a flaw. It is how the system gets to a good answer.
The problem starts when the two stages are not kept apart, and some of the scratch work ends up in the reply.
What a leak looks like
You will recognise it once you have seen it. A few common shapes:
- The third person. "The user wants to know whether they can change the booking." Written to the person who asked.
- Visible arithmetic. "Today is the 6th, so the deadline of the 20th is 14 days away." The customer asked when the deadline was, not how it was worked out.
- Half-finished reasoning. A sentence that weighs two options and never says which one it picked, followed by an answer that picks one.
- Instructions repeated back. A line that sounds like it was written for the system rather than for a person, because it was.
Each of these can sit next to a perfectly correct answer. That is what makes them easy to miss in testing. If you only check whether the answer is right, a leak passes every time.
Why a right answer stops being trusted
People judge a reply on two things at once: whether it is correct, and whether it feels like it was written for them. Most of the time the second one is invisible, because it is working.
A leak breaks it in one line. The customer sees the machinery. They realise they are being described, not addressed, and the reply stops feeling like a conversation and starts feeling like a log file they were accidentally sent.
After that, the correct part does not get the credit it deserves. A reader who has just seen the system talk about them in the third person has a fair reason to wonder what else it got wrong. Usually the answer is nothing. It does not matter. The trust was spent on the way the reply looked, not on what it said.
Why another instruction does not fix it
The obvious first fix is to add a line to the prompt: never refer to the customer in the third person, never show your working. It helps. It does not hold.
An instruction is guidance. It works most of the time, and "most of the time" is not a standard you can promise anyone. The leak also tends to come back when something else changes: a new prompt, a different model version, a change to how the system is graded or rewarded. You fix it once, and a month later an unrelated change brings it back, and nothing tells you.
There is a quieter problem too. Telling a model in detail what not to write puts that thing in front of it. Anyone who has written "do not mention pricing" and then watched pricing appear knows the shape of this already.
Treat them as two documents
The fix that holds is a change of view rather than a better sentence. Stop thinking of the system as producing one thing. It produces two:
- The working notes. For the system, and for whoever is debugging it. Messy is fine. Third person is fine. Keep them, because they are how you find out why an answer came out the way it did.
- The reply. For the customer. It speaks to them directly, states the answer, and contains nothing they were not meant to read.
Once they are two documents, you can check the one that goes out, separately, after it is written and before anyone sees it.
That check is not another line in the prompt. It sits outside the system that wrote the reply, and it asks narrow questions with yes or no answers. Does the reply refer to the reader in the third person? Does it show working that was not asked for? Does it contain anything addressed to the system rather than the person? If the answer to any of them is yes, the reply does not go out as it is. It gets rewritten or held back.
A worked example
Here is the shape of the change, using a made-up question about a booking.
| Leaked reply | Checked reply | |
|---|---|---|
| Who it addresses | "The user is asking whether they can move the booking." | "Yes, you can move it." |
| The date | "Today is the 6th, the cutoff is the 20th, so 14 days remain." | "You have until the 20th." |
| The decision | "Could suggest the later slot, or the earlier one. The earlier one." | "The earlier slot is the better fit." |
| Is it correct? | Yes | Yes |
| Would you trust it? | Less | Yes |
Both columns contain the same facts and the same recommendation. Only one of them reads like it was written for the person who asked. The check is what decides which column the customer gets.
What a working check needs
You do not need anything sophisticated. A useful check has three properties.
It runs on every reply, not on a sample. Leaks are rare enough that sampling misses them and common enough that customers see them.
It is narrow. It is not grading the reply for quality or tone in general. It is looking for a short list of specific things that should never reach a customer, and nothing else.
It is allowed to stop a reply. If the check can only flag and never block, it is a report, not a check. The system needs a sensible next step when a reply fails: rewrite it once with the notes removed, or hand it to a person.
What to take away
- Separate the reasoning from the reply. Keep the working notes for debugging, and never let them be the thing the customer reads.
- Check the reply that goes out, as its own step, outside the system that wrote it. Do not rely on one more instruction in the prompt.
- Re-run that check whenever the prompt, the model, or the grading changes, because that is exactly when leaks come back.
A correct answer that reads like someone's notes still costs you the customer's confidence. Deciding what the customer is allowed to see is a design decision, and it stays yours. The AI does the busywork. You still make the calls.