MEASUREMENT
How we know the answers are right.
Every AI vendor in sport will tell you their answers are accurate. Fewer will show you how they check. This is the method we use to score Ask Axon, what the last run came back with, and where the method falls short.
There is no right answer to check against.
WHY THIS IS HARD
Ask Axon works differently to a model that returns text. You ask a question in plain English, the agent finds the relevant schema, writes SQL, runs it against your warehouse, and builds an answer out of the rows that come back.
That's the challenge when trying to measure it. There's no reference answer to compare against, there are several valid ways to write the same query, and the truth sits in your database at the moment you asked rather than today.
So the usual measures don't apply. Exact match, text similarity, benchmark leaderboards, none of them work when the answer depends on live data and there's no gold copy to check it against.
What we can measure is whether the answer holds together against its own evidence.
Whether the agent understood the question, whether the SQL was sound, whether the rows it got back actually contained the answer, and whether the final wording matches those rows.
That last one is the hallucination check. If the numbers in an answer aren't supported by the data the agent itself pulled, we score it wrong, including in the cases where it happens to be true.
WHAT WE SCORE
Four dimensions, weighted.
Every evaluated question gets scored on four things.
The judgement is a model. The arithmetic is code.
HOW THE SCORE IS BUILT
Scores come from a frontier language model acting as a judge, working only from what was actually captured: the question, every tool call, every SQL statement and its status, the rows returned at the time, and the final answer.
The judge runs on a fixed, versioned prompt and it's told to score against that captured evidence only, never against its own knowledge of sport.
It has no idea who won the game. All it can see is what came back from the query.
Its output is schema-constrained, so it can't return prose. It has to commit to a number on every dimension, every time, with a short written justification citing specifics.
The arithmetic that turns those judgements into a score is then plain, auditable code rather than the model's own maths.
On top of that sit hard caps that no judge explanation can override.
The last cap is important. Agents retry constantly, a statement times out and gets reformulated, a query comes back empty and the agent has another go. Treating every stumble as a failure would punish behaviour that's completely normal, so a turn gets judged on where it ended rather than the bumps on the way.
THE NUMBERS
What the last run found.
Seven turns scored 100. The lowest scored 25, where a cap fired because every SQL execution in that turn had failed.
Roughly one in twelve answers came back poor.
If a vendor's accuracy comes back at 100%, they either haven't measured it properly or they're not showing you all of it.
Every score can be retraced.
CHECK OUR WORKING
Each score ships with its full arithmetic: the four dimension values, the weights applied, what each one contributed, the score before any cap, and every cap or note that fired. That sits alongside the SQL that ran and the rows that came back.
If you want the whole specification, including the alternative methods we considered and why we didn't use them, it's published in full: the technical reference.
SEE IT ON YOUR DATA
Bring the questions you don't trust yet.
Thirty minutes, your warehouse, your questions. Ask the hard ones.