You are being asked to approve something you cannot inspect. Not because anyone is hiding it, but because understanding it properly would take months you do not have, in a discipline that is not yours.
The standard advice is to find someone technical to translate. That is useful when you can get it, and it quietly concedes the wrong point: that the decision belongs to whoever understands the internals.
You will never be able to evaluate the technology. You can always evaluate the account someone gives of it, and that turns out to be most of what you needed.
Why the demo tells you so little
A demo is a performance of the system working. It is built from the paths that work, run on data that behaves, by someone who knows exactly where not to click. Nobody is being dishonest. You are simply watching the best case and being invited to generalize from it.
The problem is that the best case is not where your money goes. Your money goes into the other days, the ones with the malformed file, the customer who does something nobody anticipated, the week the source system changes format without telling anyone. None of that is in the demo, because none of it demos well.
So watching more demos does not help. Asking better questions does.
The three questions
These work regardless of the technology, because they are not about the technology. They are about whether the person in front of you has run the thing in a real environment or only sold it.
What does it do when it is wrong?
Every system is wrong sometimes. The question is what happens next. Does it stop and tell someone, keep going quietly, or retry until it succeeds at doing the wrong thing? Someone who has operated the system answers immediately, usually with an example, because they have watched it happen. Someone who has only sold it hears this as a question about accuracy and tells you how accurate it is.
Those are different answers to different questions, and the substitution is the signal.
What does it need from us every week to keep working?
Almost nothing runs unattended forever. Someone reviews exceptions, updates a list, checks that yesterday's run completed, re-approves something. That is real work and it lands on a real person on your team.
A good answer is specific and slightly unflattering: about two hours a week, usually the ops manager, mostly clearing items it was not confident about. A bad answer is that it is fully automated. Fully automated means either the work is invisible to the person describing it, or it has been quietly assigned to you without anyone saying so.
What will it not do?
Ask for the edges. Which cases does it hand back, which formats does it refuse, which decisions does it decline to make.
People who have run a system in production have a ready list, because every item on it cost them something to discover. People who have not will tell you it handles everything, which is never true of anything, and tells you exactly how much operating experience is in the room.
Reading the answers
You are not grading technical depth. You are checking whether the account has the texture of something real.
| What you hear | What it suggests |
|---|---|
| A specific example, unprompted | They have watched it run and fail |
| A number with a caveat attached | Measured rather than estimated |
| "It depends, here is what it depends on" | Understands the real variability |
| "It handles that automatically" | Has not met the exception yet |
| A reframe to a question you did not ask | The real answer is unflattering |
| Confidence with no edges | Selling, not operating |
None of this requires you to understand the system. It requires you to listen for whether the person describing it has been surprised by it, because everyone who has actually run something has been.
The one thing to insist on
Before you commit, ask to see it handle a case you choose. Not a case they prepared: one of your own, ideally a slightly awkward one, from your actual work.
The reaction is more informative than the result. Enthusiasm and a scheduled time means they have done it before. Reluctance, or a long explanation of why your example is not representative, tells you the demo and the product are further apart than they looked.
If it struggles with your case and they tell you plainly why, that is not a failure. That is the first honest data you have received, and it is worth more than a flawless run on their data.
What to take away
- Stop trying to evaluate the technology. Evaluate the account of it, which is something you are already qualified to judge.
- Ask what it does when wrong, what it needs weekly, and what it will not do. Vagueness on any of the three is the answer.
- Insist on seeing one case that you chose, and watch the reaction as closely as the result.
Being non-technical is not the disadvantage it looks like here. You spend your working life judging whether someone knows what they are talking about, and that is the skill this decision actually needs. The AI does the busywork, you still make the calls.