Evaluating agentic RAG without ranking metrics

Precision@k needs documents. With a deterministic simulator as ground truth, accuracy becomes error against simulation across benchmark queries

Precision@k measures whether the right documents came back. It is the standard way to evaluate retrieval and it is meaningless here, because nothing this system returns is a document. The output is a number that did not exist before the query ran. Either it is right or it is not, and finding out means having something to check it against.

The problem

Retrieval evaluation assumes a corpus with a ground truth relevance judgement per item. You retrieve, you compare against the judgement, you report the overlap. Every standard metric is built on that shape (Manning et al., 2008).

A decision support system breaks the shape in two places. There is no corpus, so there is nothing to judge relevant. And the answer is computed rather than retrieved, so correctness is a property of the computation, not of the selection.

The usual fallback for generative systems is human judgement or a model grading another model's output. Neither works well when the answer is a dollar figure. A grader cannot tell you whether $27.04 per tonne is correct. It can only tell you whether the sentence around it reads plausibly, which is the wrong question.

What replaces it

The ground truth is a simulator.

FRED's answers are approximations of what the underlying simulation would produce. That simulation is deterministic: the same inputs give the same outputs every time, with no sampling and no variance. So for any query, the correct answer can be computed exactly by running the reference system with matching parameters, and the comparison is exact rather than statistical.

This is the good case, and it is worth being explicit about why. Error against a deterministic oracle is bias, not noise. There is no run-to-run variation to average out. Every discrepancy is a real difference in behaviour that can, in principle, be traced to a specific cause.

Six metrics fall out of that.

The last three are not accuracy metrics and they matter as much as the first three. A system that is exactly right and takes four minutes has failed at the thing it exists to do.

The benchmark

55 queries across five locations, three conversion technologies, seven treatments and six query types. Ground truth comes from 11 reference simulation runs matching the treatments the agent selected at each location.

That last clause is the awkward one. The agent chooses a treatment autonomously from the phrasing of the question, so the reference run has to be configured after the fact to match what the agent picked. This is legitimate, since the comparison is of cost given a treatment rather than of treatment selection itself, but it means the benchmark does not test whether the agent chose well. Only whether it costed correctly once it chose.

Two baselines.

Naive RAG. Documentation embedded in a vector store, passages retrieved to answer siting questions. It handles qualitative questions fine and cannot return a number, a supply curve, or a coordinate. The failure is structural rather than a tuning problem, which is exactly what makes it a useful baseline: it establishes that the architecture is doing something the standard approach cannot do at all.

Calling the simulator at query time. No prediction layers, the agent invokes the simulator directly. A single facility analysis needs tens of thousands of sequential calls. This baseline does not produce a slow answer. It produces no answer inside any reasonable timeout, which is the point: precomputation is a prerequisite, not an optimisation.

What the numbers say

Demand estimation, MAPE 0.1%. Across all five locations and three technologies. This is a formula replication check rather than a model accuracy check, and it passing means the technoeconomic conversion is faithful. If it had failed, everything downstream would be measuring the wrong thing.

Harvest cost, 12 to 41% below the reference. Larger than the surrogate's own error, and not caused by it. The reference implementation selects clusters by proximity; this one selects by cost rank across a wider area. They are answering slightly different questions, and the cheaper answer is the correct one for the question asked. Reporting this as accuracy would be wrong in both directions, so it is reported as an algorithmic difference and explained.

Transport cost, and this is the real finding. Accuracy depends on procurement radius, severely.

The zone model assigns one circuity factor per terrain zone. Long hauls leave the local road network and run on highways, which are much straighter than the zone average, so applying the zone factor overestimates distance and therefore cost. A practical threshold emerges at about 30 km, and the honest way to report a query beyond it is with the transport component flagged as approximate.

Completion, 96.4%. 53 of 55. Both failures were the agent requesting clarification instead of assuming. That is the preferred failure mode and still a failed turn.

Latency. Cost-only queries 18.2 seconds on average, fire-aware queries 22.4, the difference being the sweep across the cost and fire weighting. Multi-year projections take 72.3 seconds to begin streaming.

The ablation, and what is wrong with it

Three components removed in turn.

Two of the three rows are estimated rather than measured, marked above. Only session continuity was measured directly, from timed parameter-refinement turns, giving the 2.0× speedup.

The radius estimator row is inferred from over-retrieval and under-retrieval overhead. The autonomous inference row is inferred from how many benchmark queries depended on implicit signals, not from rerunning all 55 with explicit parameters.

This is a real weakness and it is worth naming rather than burying in a footnote. An estimated ablation tells you the direction of a component's contribution and not its size. The fix is mechanical, just 55 reruns with a modified prompt, and it has not been done.

Where this approach does not transfer

It needs a deterministic oracle. Everything here rests on the reference simulator being exactly reproducible. Evaluating against a stochastic simulator would need distributions rather than point comparisons, and every metric above would change shape.

It does not test the agent's judgement. Ground truth is configured to match what the agent chose. If the agent consistently picked a poor treatment and then costed it perfectly, the benchmark would report success. Evaluating the choice itself needs a different design and probably a domain expert.

Coverage is partial. Validation covers three technologies with two treatments. Extending to all eleven treatments, and to multi-year projections for the other two technologies, remains open.

55 queries is small. Large enough to surface the transport threshold, which was the most useful thing the evaluation found. Not large enough for confidence intervals on a 96.4% completion rate, where two failures either way moves the number by four points.

More on how to read the numbers the system reports at /docs/reading-answers.

References

1. Manning, C.D., Raghavan, P. and Schütze, H. (2008). Introduction to Information Retrieval. Cambridge University Press.

2. Lewis, P. et al. (2020). Retrieval-augmented generation for knowledge-intensive NLP tasks. Advances in Neural Information Processing Systems, 33, 9459–9474.

3. Fight, R.D., Hartsough, B.R. and Noordijk, P. (2006). Users Guide for FRCS: Fuel Reduction Cost Simulator Software. USDA Forest Service, Pacific Northwest Research Station.