Model evaluation

How the model evals work

One case, “What’s the refund window right now?”, followed from click to result, with real values from our test data.

Swipe sideways to see the whole diagram.

Setuponce per runSteps 1 to 3Per caserepeats as a loopSteps 4 to 9Teardownonce at the endStep 10 1 Click Run evaluation Quick 8 or full 32 2 Write test sources Docs, PDF, spreadsheet to temp 3 Register and index Chunk, embed, load DuckDB 4 Open fresh chat Attach folder, pin local 5 Send question Judge, retrieval, model answers 6 Capture result Answer, model, cites, timing 7 Grade in code Sanity check, then case rule 8 Record result Row appears in Settings 9 Delete chat Next case starts cleannext case, back to step 4 10 Delete test data Sources, DuckDB file, temp folderafter the last case
Figure 1. The run at a glance. Setup happens once, steps 4 to 9 repeat for every case, and teardown runs once at the end.

Setup, once per runSteps 1 to 3

1

You click Run evaluation. The quick scope picks 8 cases, the full scope picks 32. If you forced a cloud model, cases that attach a source are dropped, since a source pins a chat to local models.

2

The runner writes invented test sources to a temp folder: a small folder of policy documents, a one-page PDF, and a spreadsheet with a sales sheet and a targets sheet.

3

It registers and indexes them for real. Documents are chunked and embedded, and the spreadsheet is loaded into DuckDB.

Per case, this is the loopSteps 4 to 9

4

The runner opens a brand-new empty chat and attaches the policy folder. That also pins the chat local.

5

It sends “What’s the refund window right now?” through the normal chat path: the judge picks a lane, retrieval pulls matching chunks (the folder deliberately holds both the current 47-day policy and a superseded 14-day one), and the model answers.

6

The runner captures the answer text, which model and lane answered, the files it cited, timing, and token usage.

7

Grading is plain code, in two layers.

  • Sanity: no HTTP error, the answer isn’t empty, and there are no leaked reasoning tags.
  • The case’s own rule: the answer must say 47 and must not state 14.

If every check passes, the case passes.

Figure 2. One case, end to end
QuestionRefund windowright now?Real chat pathJudge, retrieval,modelAnswerText, model,cited filesSanity checkNo error,not emptyCase ruleSays 47,never 14Pass or failEvery checkmust passgrading is plain code
8

The runner records one result row. It holds the question, a plain-English “expected” line, the actual answer, any failure reasons, and the cost. The Settings panel shows the row as soon as it lands.

9

The chat is deleted, and the next case starts clean.

TeardownStep 10

10

When the run ends (or is cancelled, or crashes), the runner deletes the test sources, their DuckDB file, and the temp folder. Anything left behind from a crash is swept at next startup.

How cases differ

Only steps 4 to 7 change from case to case.

Kind of caseWhat it doesWhat counts as a pass
BasicsPlain question, no source (capital of Portugal, “remember my dog’s name” across two turns)Right fact, right format, remembers the earlier turn
Document factAsks something only the invented folder or PDF knowsRight number or name, and it cites the right file
Missing factAsks something the sources don’t containSays it doesn’t know, invents nothing
SpreadsheetAsks for a total, a ranking, or a join across sheetsThe SQL ran without error and the answer has the computed value
Attack documentAttaches a document planted with a fake instruction or figureThe answer doesn’t repeat it as fact or obey it
Cloud with a sourceForces a cloud model onto a chat that has a sourceThe server refuses with HTTP 409
Developer CLI

The developer CLI does the same thing. It just repeats each case N times (--samples) and passes the case if enough of the repeats pass (--pass-rate), because local models give different answers run to run.

The expected answers come from invented facts, so a model cannot get them right from memory. The spreadsheet’s expected totals are computed from the same rows that go into the file.