How does Axon Perform measure whether Ask Axon's answers are accurate?

Every evaluated turn is scored on four dimensions: whether the intent was understood (weighted 30%), the quality of the SQL written (25%), whether the retrieved rows could actually answer the question (25%), and whether the final answer faithfully reflects that data (20%). Each is scored 0 to 10 by a language model acting as a judge, working only from the captured evidence, then combined by auditable code into a score out of 100. Scores of 80 and above are good, 50 to 79 partial, below 50 poor.

Why can't AI answers on live data be checked against a correct answer?

An agentic data assistant writes and runs SQL against a live database, so there is no fixed reference answer to compare against. There are many valid ways to write the same query, and the underlying data changes daily. Conventional metrics such as exact match or text similarity do not apply. Axon Perform instead measures faithfulness: whether the answer is supported by the evidence the agent actually retrieved. An answer whose figures are unsupported by its own retrieved rows is wrong regardless of whether it happens to be true.

What were Ask Axon's most recent accuracy results?

In a run of 20 July 2026 covering 53 production turns across both hosting regions, 40 turns scored good (80 or above), 9 scored partial (50 to 79), and 4 scored poor (below 50). Seven turns scored 100. The lowest score was 25, capped because every SQL execution in that turn failed. Roughly one in twelve answers came back poor.

What are the limitations of this accuracy measurement?

The method measures the answer against its own retrieved evidence, not against real-world truth. If the underlying database is wrong or stale, a faithful answer scores well while being factually incorrect. Scores come from a single judge model on a single pass, so they carry judge noise. The judge sees a sample of rows, by default the first 20 per statement. On surfaces where the client's own AI tool composes the final wording, Axon can verify the retrieved data was sufficient but not that a correct answer was written. Human expert calibration has not yet been carried out, so no comparative claims against other systems are made.

Can an individual accuracy score be audited?

Yes. Every score is published with its full arithmetic: the four dimension values, the weights applied, each dimension's contribution, the score before any cap, and every cap or note applied. This appears alongside the SQL that ran and the rows returned at execution time, so a score can be retraced end to end without access to Axon Perform's code.

MEASUREMENT

How we know the answers are right.

This is the method we use to score Ask Axon AI when you connect to your data from your AI tool.

There is no right answer to check against.

WHY THIS IS HARD

Ask Axon works differently to a model that returns text. You ask a question in plain English, the agent finds the relevant schema, writes SQL, runs it against your warehouse, and builds an answer out of the rows that come back.

That's the challenge when trying to measure it. There's no reference answer to compare against, there are several valid ways to write the same query, and the truth sits in your database at the moment you asked rather than today.

So the usual measures don't apply. Exact match, text similarity, benchmark leaderboards, none of them work when the answer depends on live data and there's no gold copy to check it against.

What we can measure is whether the answer holds together against its own evidence.

Whether the agent understood the question, whether the SQL was sound, whether the rows it got back actually contained the answer, and whether the final wording matches those rows.

That last one is the hallucination check. If the numbers in an answer aren't supported by the data the agent itself pulled, we score it wrong, including in the cases where it happens to be true.

WHAT WE SCORE

Four dimensions, weighted.

Every evaluated question gets scored on four things.

Intent understood

30%

Did the tool and SQL choices address what was actually asked?

This one carries the heaviest weight because everything after it depends on getting it right. Well-written SQL pointed at the wrong question doesn't help anybody.

SQL quality

25%

Right tables, right filters, right ordering?

It scores full marks when no SQL was needed and none was run, so questions that don't require a query aren't penalised for it.

Result relevance

25%

Could the question be fully answered from the rows that came back?

Full marks needs the set to be complete, so if seven tries were scored, all seven scorers come back rather than five of them.

Answer faithfulness

20%

Do the numbers and names in the answer match the data retrieved?

This is the direct check for fabrication. Each dimension is scored 0 to 10, then weighted into a single score out of 100.

Deterministic caps
What happened	Capped at
Every SQL execution in the turn failed	25
The query chain ended on an unrecovered failure	70
No tools and no SQL run at all	40
Errors that occurred but were recovered	No penalty

Deterministic caps

What happened

Capped at

Every SQL execution in the turn failed

The query chain ended on an unrecovered failure

No tools and no SQL run at all

Errors that occurred but were recovered

No penalty

How we know the answers are right.

There is no right answer to check against.

Four dimensions, weighted.

The judgement is a model. The arithmetic is code.

What the last run found.

Every score can be retraced.

Bring the questions you don't trust yet.