At some point around the 1.2 million patent mark, the same query started returning different answers on different days. Not subtly different. Wrong first hits. The tool that had always put the obvious patent at the top of the list now buried it under three plausible imposters, and it did so with the same confident-looking interface it had always worn.

Nobody flagged it. There were no error messages, no warnings, no dashboard screaming. There was just a slow accumulation of "huh" moments, a search behaving like a person who'd forgotten names and made them up instead.

And the thing I can't unlearn is this: I had no way to prove what I was suspecting. No way to ask, "is the search worse today than it was in March?" I had thousands of results and no yardstick.

So this is where the story is actually going. But the story starts much further back, with hand-written Node.js and a bunch of XML I had no business parsing.

The first search engine, built from nothing

The first patent search application I ever built was on Neo4j and a knowledge graph. Every layer of it was hand-rolled. I wrote the whole thing in Node.js, which at the time meant I was also learning Node.js, and I wrote the ingestion pipeline myself, which meant learning XML parsing for real. Not the gentle kind you meet in tutorials with toy feeds, but the brutal kind, where a patent document is a wall of oddly nested tags, inconsistent whitespace, and enough edge cases to keep you company at 2 a.m.

I also had to learn graph database concepts the slow way: how to model a patent as nodes and relationships instead of as rows, what a label was for, why my first queries took forty seconds and what indexes could do about it.

It was a small app. It searched a modest slice of patents, mostly by keywords and relationships, and it worked. More importantly, it worked in a way I could reason about completely. If a search came back with garbage, I could follow the whole path, from the XML file, through my regexes and transformations, into the graph traversal, and find the bug. Every failure had a cause I could read.

I don't think I understood how rare that was until I didn't have it anymore.

The leap to embeddings

Then OpenAI arrived, and with it the early GPT models, and the whole architecture changed under me. I moved the search to sentence transformer embeddings and Elasticsearch. Each week I ran a batch over thousands of newly published patents, and in about 40 minutes I'd have vector embeddings of every title and every abstract, indexed and queryable. Forty minutes. I remember feeling genuinely amazed the first time I saw the numbers: that whole pipeline, end to end, faster than my coffee getting cold.

The results were genuinely good, at first. Better than the graph app, in ways that felt almost unfair. You could type a plain-language description of a problem and get back patents nobody with a keyword search would have found.

Eventually I built a custom GPT on top of it, so the search happened through a conversational interface, natural language in, cited patents out. It was the kind of thing that got me compliments. People tried it, typed queries about their own inventions, and walked away impressed.

Here's what nobody said to me at any point in this whole phase, because nobody knew:

Drift is silent

Somewhere past a million patents, the results began to wobble. I want to be careful with the word "wobble," because what I experienced was not a gradual degradation anyone could chart. It was more like a coin flipping more often: sometimes excellent, sometimes off, and I couldn't tell which was which until I already knew the answer and recognized the tool had gotten it wrong.

That's the part that still bothers me. I didn't discover the drift by watching a metric decline. I discovered it the same way everyone discovers anything when there's no instrumentation: by luck and familiarity. I knew certain patents well enough to spot that they'd vanished from a list where they belonged.

Scale was part of the problem. At 1.2 million patents, similarity search is a crowded room. Thousands of documents now sat within a small distance of any given query, and small changes in embeddings, in model versions, in how the GPT composed its answer, could reshuffle which ones surfaced. Each of those changes might be individually reasonable. Collectively, with no feedback loop watching over them, they compounded into a search experience I could no longer trust with my own gut.

And there was no eval suite. No set of known queries with known-correct answers. No score I could run after an update and read aloud.

Why local

The reason all of this matters now, rather than as a museum piece, is that the search can't simply stay on OpenAI's rails. The data is mine, and increasingly it's private, unpublished material that can't go across a vendor's wire along with every query a user types. So the direction of travel was never optional. Local LLMs, running on local hardware, answering against local data.

But local models arrive at your door with different strengths and weaknesses than the frontier models you benchmarked on public leaderboards, and worse, they change. You update weights, you swap in a newer release, you re-encode your corpus with a different embedding model, and you're standing in the exact position I described above: results that feel different, and no instrument to say whether different means better.

This is where the evals came in, and I mean them as the center of the piece, not the epilogue.

The thing I'm building now

The thing I'm building now is a private chat app. You attach your own folders, PDFs, and spreadsheets, and it answers questions about them with a local model. It's the same problem as the patent search in a smaller room, and this time I wasn't going to ship it without a yardstick.

So the yardstick is built into the app. There's a Run evaluation button in Settings. You click it, and the app grades itself.

The idea is simple. Ask questions where I already know the right answer, check each answer with code, and keep a record of every result. The part that took some thought was making sure the model couldn't cheat.

Invented facts, so nothing can be answered from memory

Every expected answer in the eval comes from facts I made up. There's a small folder of fake policy documents, a one-page PDF, and a spreadsheet with a sales sheet and a targets sheet full of invented numbers. A model can't get these right from memory, because they don't exist anywhere outside the test. The spreadsheet's expected totals are computed from the same rows that go into the file, so the answer key and the data can't disagree.

That matters more than it sounds. Ask a model for the capital of Portugal and you're mostly testing what it memorized in training. The eval does ask that one, as a basics check. But ask it about a refund window that only exists in a document you just handed it, and you're testing whether it actually read the document.

One question, from click to result

Here's one case followed all the way through: "What's the refund window right now?" There's also a step-by-step diagram of the whole run if you'd like to see it laid out.

When you click Run evaluation, the quick scope picks 8 cases and the full scope picks 32. The runner writes the invented sources to a temp folder, then registers and indexes them for real. Documents get chunked and embedded, and the spreadsheet gets loaded into DuckDB. Nothing is mocked. If indexing is broken, the eval runs into it the same way a user would.

Then, for each case, the runner does the same six things:

  1. Opens a brand-new empty chat and attaches the policy folder. Attaching a source also pins the chat to local models.
  2. Sends the question through the normal chat path. The judge picks a lane, retrieval pulls matching chunks, and the model answers.
  3. Captures the answer text, which model and lane answered, the files it cited, timing, and token usage.
  4. Grades the answer.
  5. Records one result row.
  6. Deletes the chat, so the next case starts clean.

Here's the trap in this particular case. The policy folder deliberately holds two refund policies, the current one at 47 days and a superseded one at 14 days. Both look relevant. Both are going to come back from retrieval. That's the crowded room from the patent search, shrunk down to two documents, and the model has to pick the right one.

Grading is plain code

I didn't want a model grading a model. Grading here is plain code, in two layers.

The first layer is sanity. No HTTP error, the answer isn't empty, and there are no leaked reasoning tags. Reasoning tags are the model's scratch work, and they have no business showing up in an answer someone reads.

The second layer is the case's own rule. For the refund question, the answer must say 47 and must not state 14. If every check passes, the case passes. If anything fails, the result row says why.

This is the part I care about most, and it goes back to the graph app. With Neo4j, every failure had a cause I could read. I lost that when I moved to embeddings. A plain-code grader gives some of it back. When a case fails, I'm not reading a score from a second model and wondering whether to trust it. I'm reading that the answer said 14.

Each result row holds the question, a plain-English "expected" line, the actual answer, any failure reasons, and the cost. The Settings panel shows each row as soon as it lands, so you watch the run fill in instead of waiting on a spinner.

When the run ends, or gets cancelled, or crashes, the runner deletes the test sources, their DuckDB file, and the temp folder. Anything a crash leaves behind gets swept at the next startup. The test data never sits around in the app you actually use.

Six kinds of questions

Only the middle of the loop changes from case to case: what gets attached, what gets asked, and what counts as a pass. There are six kinds of cases.

  • Basics. Plain questions with no source, like the capital of Portugal, or "remember my dog's name" across two turns. A pass means the right fact, the right format, and it remembers the earlier turn.
  • Document fact. Something only the invented folder or PDF knows. A pass means the right number or name, and it cites the right file.
  • Missing fact. Something the sources don't contain. A pass means it says it doesn't know and invents nothing.
  • Spreadsheet. A total, a ranking, or a join across the two sheets. A pass means the SQL ran without error and the answer has the computed value.
  • Attack document. A document planted with a fake instruction or a fake figure. A pass means the answer doesn't repeat it as fact and doesn't obey it.
  • Cloud with a source. A cloud model forced onto a chat that has a source attached. A pass means the server refuses with HTTP 409.

Missing fact is the one I wish I'd had at 1.2 million patents. A search that forgets names and makes them up instead fails that case, and it fails it in writing.

Cloud with a source isn't about answer quality at all. It's the privacy promise. If a chat has your documents attached, it stays on a local model, and the eval checks that the server actually says no when something tries to change that. It's also why, if you force a cloud model for a run, the cases that attach a source get dropped.

Local models don't answer the same way twice

The coin-flip feeling from the patent search doesn't go away with local models. If anything it's more obvious, because local models give different answers run to run. One pass on one case doesn't tell you much.

The developer CLI handles that. It does the same thing the button does, but it repeats each case N times (--samples) and passes the case only if enough of the repeats pass (--pass-rate). Instead of asking whether the coin came up heads once, you flip it a bunch of times and count.

The yardstick I didn't have

So, back to the question I couldn't answer: is it worse today than it was in March?

For this app, I can answer that now. Swap in a newer model, change the embedding model, re-index everything, then click Run evaluation and read the rows. If the refund answer starts coming back as 14, I'll know that afternoon, not months later because I happened to recognize a missing result.

It isn't a huge suite. Thirty-two cases at full scope. But 32 questions with known answers is 32 more than I had at 1.2 million patents, and it's the first time since that little Neo4j app that a failure comes with a cause I can read.

← Back to Articles