Screening Listings with a Recall Eval
LLM screening of a live auction catalogue, built around a recall eval. Known good deals reaching the shortlist rose from 8 to 14 of 23, to all 23.
Problem
A live auction catalogue of about a thousand items needs screening for good deals. An earlier run silently left out deals that were worth seeing, and nothing measured how many.
Constraints
- Every item must get an explicit score, with none silently lost
- Cost is recorded per run, so a better score has to be weighed against its price
- The shortlist feeds a slower, more expensive photo-based second stage
Overview
This is one part of a larger private tool: the part that screens a live auction catalogue and decides which items deserve a closer look. I describe only the screening and the eval here.
The baseline asked a model to pick promising items from batches of 80 listings. Then I found deals it had left out. So the work became about measuring that instead of guessing: freeze a real day, write down the deals that were missed, and count how many each change catches. I set the goals and review the results. Coding agents wrote most of the code, including the eval script.
How the screening works
Stage one scores every listing, not only the ones that look interesting. Listings go to the model in batches of 60 and each row comes back with a score. If rows are missing from a response, they are asked for again, up to twice. The first pass puts similar items next to each other. A second pass shuffles them with a seeded order so each item has different neighbours, and it only re-screens items that scored under the threshold.
Code, not the model, then picks the shortlist, using rules: a score threshold, a saving rule, and a cap on the shortlist size. Stage two rates each shortlisted item with its own photos at a higher reasoning effort and gives a verdict.
The analysis module is pure, with the model call passed in, so the eval can run the real screening code against a saved catalogue.
The eval
The fixture is the 1,044 live items of one day, plus gold lists recorded with it: 23 deals that a production run had left out, one trap (a cheap-looking item that should be rated avoid), 57 items an earlier run rated great or good, and a subset of 35 where the saving is large. Each run prints how many of each list reached the shortlist, the shortlist size, the cost and the time. Saved results can be re-scored with new rules at no model cost.
| Version | Deals caught of 23 | Cost per run |
|---|---|---|
| Baseline: batches of 80, a cap of 12 per batch, one pass | 14, 14 and 8 across three runs | about $0.04 |
| Score every row, one pass | 23, 23 and 23 at a low threshold, but a shortlist of about 570 items | about $0.11 |
| Two passes with the shuffle | 23 of 23 in three runs | about $0.20, 175 to 186 s |
| Chosen settings: threshold 5, cap 350, saving rule | 23 of 23 in both runs, trap caught | about $0.20 |
Two results shaped the choice. Lower reasoning effort cost about three cents less per run but missed a deal once, so I kept the higher setting. And the shortlist is a cost trade: catching every deal means sending more items to the expensive second stage.
Limits
This is one saved day and a handful of runs, so these are not statistics. The gold lists are small. Of the 35 large-saving items, 32 reach the shortlist at the chosen settings, below the 34 I set as the goal. The three that miss are the same borderline items in every run, and no rule I tried lets them in. I would rather publish that than round it up.
Stack
- Runtime: Node 22, no npm dependencies
- AI: Anthropic Messages API, Haiku (claude-haiku-5-5)
- Tests and eval: node:test, a frozen fixture and a recall script