Ask Axon Answer-Quality Evaluation: Technical Reference

1. Why conventional accuracy measurement does not apply

Ask Axon is an agentic data assistant rather than a model returning text. A user asks a question in natural language, the agent discovers the relevant schema, writes and executes SQL against the client's warehouse, and composes an answer from the rows returned.

Three properties make conventional evaluation unusable:

• No reference answer exists. There is no gold text to compare output against.
• Many valid formulations. The same question can be answered correctly by materially different SQL.
• Ground truth is live and moving. The correct answer depends on the state of the client's database at the moment the question was asked, which differs from its state tomorrow.

Reference-based metrics (exact match, BLEU, ROUGE, embedding similarity) therefore cannot be applied. There is no reference, and the data changes daily.

What can be measured rigorously is the internal consistency of the captured turn: whether the agent understood the question, whether it translated that intent into sound SQL, whether the data it retrieved contained what was needed, and whether the final answer faithfully reflects that data. This is the faithfulness-first framing used across the LLM evaluation literature, equivalent to groundedness metrics in retrieval-augmented generation. An answer unsupported by its own retrieved evidence is wrong by construction, regardless of whether it happens to match the real world.

2. Evidence available to the judge

Every production turn is captured by the telemetry event stream. For each evaluated turn the judge receives a single JSON document containing:

Point-in-time snapshots

The rows the agent actually saw are stored, so an evaluation remains reproducible after the underlying data changes. Re-querying the database later would evaluate a different world state than the one the agent answered from.

Full trajectory, not just outcome

Agentic systems recover from errors. A statement times out and the agent retries; a query returns nothing and the agent reformulates. Judging only the final artefact would miss whether the process was sound, while treating every stumble as a failure would penalise normal exploration. The judge receives the whole trajectory and is instructed to judge the turn by where it ended.

3. The rubric: four weighted dimensions

Each dimension is scored 0 to 10. They are ordered to mirror the four places an agentic data pipeline can fail, in causal order, so a low score localises where the turn went wrong rather than only that it did.

Intent understood (30%)

The largest weight, because it is the root of the causal chain. Perfectly executed SQL against the wrong question is worth nothing, and every downstream dimension is conditioned on this one. It is also the dimension users feel most directly, since "it didn't understand me" is the dominant failure complaint for natural-language interfaces.

SQL quality (25%)

Measures the translation step. Scored 10 when no SQL was needed and none was run, for example a schema-discovery question, so non-SQL turns are not artificially penalised. Judged on plausibility and targeting rather than execution-equivalence against a gold query, because no gold queries are available.

Result relevance (25%)

10 means everything needed is present: the right entities, the right match or period, and complete sets. If seven tries were scored, all seven scorers are required, not five. 5 means material gaps. 0 means the rows cannot answer the question.

Answer faithfulness (20%)

The direct hallucination check. A fluent answer whose figures are unsupported by its own retrieved rows is scored poor even where it is plausible. This dimension is null, rather than zero, when no answer text was captured. See section 6.

Rationale for the weights

The weights encode two judgements. First, causal priority: intent errors invalidate everything downstream, hence the largest single weight. Second, retrieval quality in aggregate (SQL quality plus result relevance, totalling 50%) is weighted equal to understanding plus expression (30% plus 20%), reflecting that the core promise of the product is correct data retrieval.

These are configuration, not discoveries. They are stated openly, they are visible in every report's per-turn breakdown, and section 8 describes how they would be recalibrated against human labels. Sensitivity is bounded: with four subscores on a shared 0 to 10 scale, plausible weight changes of plus or minus 0.05 move the composite by at most a few points and cannot reorder a good turn below a poor one.

4. The judge

Scores are produced by a frontier language model acting as an evaluation judge, with three engineering controls addressing the method's known weaknesses.

1. A fixed, versioned system prompt pinning the evaluation frame: evidence-only judging, faithfulness first, guardrail blocks treated as the system working rather than a failure, and transient recovered errors not penalised.
2. Structured outputs. JSON-schema-constrained decoding. The judge cannot return free text; it must emit a schema-valid verdict with integer subscores 0 to 10, a categorical verdict of good, partial or poor, and a short explanation citing specifics. This removes parsing failures and forces a commitment on every dimension on every turn.
3. A deterministic scoring layer. The model provides graded judgements; the arithmetic that turns them into a score is plain, auditable code. No part of the final number is left to the model's arithmetic.

Every stored result records the judge model, so scores from different judges are never silently mixed. Each published report states the judge used, and a re-judge mode exists to re-score the same captured evidence with a newer judge.

The judge is explicitly instructed to score only against the captured evidence and never against its own world knowledge of sport results. It does not know the outcome of the fixture; it sees only what the query returned.

Why LLM-as-judge

The alternatives (section 7) either require ground-truth labels that do not exist, human effort that does not scale to continuous production monitoring, or surface metrics that say nothing about answer quality. LLM judges have documented agreement with human raters on graded quality tasks where the rubric is explicit, the output is constrained, and the evidence is fully contained in the prompt. The known failure modes of self-preference, verbosity bias and position bias are mitigated by design: the judge evaluates a trajectory of SQL and data rather than competing prose answers, and never compares two candidates side by side.

5. From subscores to a score out of 100

Summed only over dimensions actually observed. When all four are present the denominator is 1.0 and this is a plain weighted mean. When faithfulness is null the remaining weights renormalise to 0.375, 0.3125 and 0.3125. Subscores are clamped to the range 0 to 10 before use.

A compensatory model is used deliberately, because partial credit is meaningful: a turn that understood intent and wrote good SQL but retrieved an incomplete set is genuinely better than one that failed at every stage. Where compensation would mislead, caps are applied instead of adjusting weights.

Deterministic caps

After the weighted total, plain code enforces outcome floors that no judge explanation can override.

Caps are asymmetric by design. Deterministic rules can reliably detect hard failure states from event statuses but cannot detect subtle quality, so they only ever push scores down. They exist to address the main residual risk of LLM judging, which is occasional charitable drift on clearly failed turns.

Score bands

The judge independently emits a categorical verdict on the same scale, which acts as a cross-check on the numeric composite.

Auditability

Every score ships with its complete arithmetic: per-dimension values, nominal and effective weights, per-dimension contributions, the pre-cap raw score, and every cap or note applied. Alongside the SQL executed and the captured result rows, a reader can retrace any score end to end without access to Axon Perform's code.

6. Current results

Evaluation run of 20 July 2026. 53 production turns across both hosting regions, judged by Claude Fable 5.

Seven turns scored 100. The lowest score was 25, capped because every SQL execution in that turn failed. Roughly one in twelve answers came back poor.

These figures describe a single run judged by a single model. They are Axon Perform's own reported results and are not independently audited. No comparative claim against any other system is made or implied, for the reason given in section 8.

7. Stated limitations

A methodology that hides its limits is not one. The following are known and unresolved.

7.1 No external ground truth

The method measures internal consistency of the turn, not real-world correctness. If the database itself is wrong or stale, a faithful answer scores well while being factually wrong about the world. This is by design, since data correctness is a data-engineering concern with its own controls, but these scores should not be read as fact-checked against reality.

7.2 Single judge, single pass

Scores carry judge-model noise, typically around one point on a subscore for borderline turns. Ensembling and repeat-sampling are not currently used. The judge model is pinned and recorded, and structured decoding with an explicit rubric keeps variance materially lower than free-text judging.

7.3 Row sampling

The judge sees the first 20 rows per statement by default. A completeness failure hidden beyond that sample horizon in a large result set could be missed. Row counts are always shown in full, which bounds this risk.

7.4 Answerability is an upper bound

On surfaces where the client's own AI tool composes the final answer, that text is never transmitted to Axon Perform and cannot be scored. What can be verified is that the retrieved rows were sufficient to support a correct answer. Such turns renormalise the weights rather than assign a fabricated faithfulness score.

7.5 Subscores are ordinal

A 10-point rubric scale is ordinal, not interval. Treating it as interval in a weighted mean is a standard pragmatic approximation, identical in kind to grading rubrics and Likert aggregation. Band thresholds, not raw point differences, are the intended unit of interpretation.

7.6 Question relay on some surfaces

Where the raw user message is not exposed, questions are best-effort relays, and one captured turn can bundle several user questions. Turns are split per question segment before judging to keep the unit of evaluation clean.

7.7 No human calibration yet

Judge-to-human agreement has not been measured. Until it has, no comparative claims about other systems are made.

8. Alternative methods considered

9. Validation and governance roadmap

1. Human calibration set. Periodically double-score a stratified sample, by band, surface and client, with a human expert, and report judge-to-human agreement using Spearman's rho on scores and Cohen's kappa on bands. Target: kappa of 0.7 or above before any comparative claim is published.
2. Judge-model change control. Any judge upgrade re-scores a frozen anchor set of past turns first, and systematic drift is documented before adoption.
3. Weight review. Rubric weights are revisited against accumulated human labels rather than adjusted ad hoc.
4. Versioning. Rubric, prompt, weights and caps are versioned together, and every published report states the judge model and generation time.

10. Common questions