Evaluation report

Retrieval and answer quality on 497 near-identical MOSFET datasheets

industrial-doc-rag/2026-07-14/bge-m3·Vectorize·llama-3.3-70b/source + results file
Abstract

497 MOSFET datasheets from one vendor are indexed and 183 are deliberately held out. 2,718 questions are generated from the tables themselves. The system returns the right datasheet at rank 1 on all 1,987 indexed questions, and answers 99.5% of them correctly. Vector search alone would rank it first only 64.3% of the time, on those same questions, which spell the part number out in full. Part numbers are the tokens an embedding is worst at, and 497 lookalike datasheets is where that becomes the whole problem.

A second arm asks 248 questions that name no datasheet at all — superlatives and counts over the corpus. Retrieval scores 0.012 on them, and not because it retrieves badly: the answer is a property of all 497 documents and ten chunks are ten documents. Reading the corpus into a table at ingest and answering those questions with arithmetic scores 0.863 on the same set.

Shipped
Retrieval, recall@1
1.000
Right datasheet at rank 1, all 1,987 indexed questions.
Answer accuracy
99.5%
400 questions, value and unit, 1% tolerance.
Refusal, held-out parts
100.0%
Identifier guard on, costing 0.0% of indexed parts.
Questions naming no part
0.863
248 set queries. The same pipeline without a catalogue scores 0.012. See §1.5.
Findings
Vector search alone
0.643
The same corpus without the part-number arm. Not shipped. See §1.1.
Refusal, model alone
96.0%
What the model does when nothing but the model is watching.
Inventions that are correct
25.0%
Of the 4.0% it invents unaided. See §1.4.

1Results

1.1 Retrieval

Table 1 · Three strategies over the 1,987 questions whose datasheet is in the index. Same embeddings, same k, same questions.
strategyrecall@1recall@5recall@10MRR
dense (vector only) 0.643 0.898 0.949 0.749
dense + part rerank 0.949 0.949 0.949 0.949
dense + part lookup, fused 1.000 1.000 1.000 1.000

Retrieval is solved: 1,987 of 1,987 indexed questions return the right datasheet at rank 1. That is a key lookup, and it is meant to be. The row above it is the finding. Dense retrieval alone ranks the named datasheet first 64.3% of the time, on questions that spell its name out in full, and that is the pipeline most RAG demos ship. The middle row is the usual patch: rerank the dense results and float the named document up. It stops at exactly dense's own recall@10, because a rerank reorders what it was handed and cannot retrieve what never came back. Fusing a part-number lookup into the query is the difference between a system that opens the wrong datasheet a third of the time and one that does not.

1.2 Corpus size

Table 2 · Three real Vectorize indices, same pipeline, each scored only on the questions it could answer. Latency is timed one request at a time from outside Cloudflare.
datasheetschunksdense r@1fused r@1p50storage/mo
5 262 1.000 1.000 605 ms $0.000
100 5,133 0.733 1.000 542 ms $0.000
497 25,536 0.643 1.000 598 ms $0.008

Five documents is not a retrieval problem: the answer is one of five. Dense is the only column that moves. At 497 datasheets the distractors differ from the target in two digits of a part number, and recall@1 falls to 0.643. A key lookup is indifferent to how many neighbours it has, so the fused column is flat. So is latency, which is what an ANN index is built for.

1.3 Answers

Table 3 · 400 questions, hybrid-rrf retrieval, temperature 0, graded on value and unit at 1% tolerance. RDS(on) varies by more than 2x with junction temperature, so every question carries the conditions its label was measured at.
question typencorrect
VDS rating 100 1.000
RDS(on) max, at stated conditions 100 0.980
ID continuous 100 1.000
Package 100 1.000

This column read 0.353, then 0.840, and every gain came from the evidence rather than the model. The generator never changed. Three models from three vendors scored within a point of each other, which is what happens when they all fail on the same missing rows. The defects are logged in §3.

Two questions in 400 still fail. PSMN5R3-25MLD is asked for RDS(on) at VGS = 10 V and the model answers 8.49 mΩ, which is the 4.5 V row printed directly above it: it reports conditions it did not read. BUK9K22-80E is refused although its datasheet was retrieved at rank 1. Both are left in the number rather than prompted away until the test goes green.

1.4 Refusal

183 datasheets were fetched, parsed, and kept out of the index. Their parts are still asked about, and 497 nearly identical ones ARE indexed, so retrieval hands the model ten plausible tables for the wrong component every time.

25.0% of the model's inventions are correct, and that is the finding. PSMN1R0-30YLD is a 30 V part whose name says 30 V. Asked about a datasheet it has never seen, the model decodes the naming convention and is right. An answer that is correct and grounded in no document is the worst output this system can produce, because nothing distinguishes it from one that is.

So the rule lives in code, not in the prompt. If a question names a part and no retrieved chunk came from it, the system refuses before generating. Fused retrieval finds the asked document at rank 1 in all 1,987 indexed questions, so a part missing from the results is missing from the corpus. The 96.0% is reported rather than the guarded 100.0%, because a guarded 100% restates the definition of the guard.

1.5 Questions that name no part

Table 4 · 248 questions generated from the same labels, none of which names a datasheet: superlatives and counts over the whole corpus. Both systems answer the identical questions with the same embeddings, the same k, the same generator and the same grader.
systemaccuracyprecision when it answeredrefusedthe winner was retrieved
retrieval only, no catalogue 0.012 0.068 204 0.364
catalogue built at ingest 0.863 0.863 0 not applicable

Retrieval does not answer this question badly. It cannot answer it: of the ones it got wrong, 68.3% were wrong about a datasheet that was never retrieved. The winning document reached the model 36.4% of the time overall. "Which 40 V part has the lowest RDS(on)" is a property of 497 documents, and ten chunks are ten documents. The model says so itself when it guesses: asked how many parts ship in a given package, it answered "all 9 parts are offered in a LFPAK package" — it counted the evidence in front of it and called that the corpus. No prompt and no larger model fixes that, because the information was never in the context.

So the corpus is read into a table once, at ingest, by the same model reading the same chunks the retriever returns — never the label parser, which would make the eval a measurement of the system agreeing with itself. A superlative is then ORDER BY and a count is COUNT, over all 497 rather than over ten. A planner turns the question into a filter; it never writes a number. Every figure the user sees has been counted.

Table 5 · The same 248 questions by shape, catalogue path.
questionncorrect
which part is the extreme 125 0.824
what is the extreme value 84 0.929
how many parts 39 0.846

What is left is the reading, not the arithmetic: 92.9% of the wrong superlatives are wrong because the winning part was never in the pool the query compared. The query is a for loop over that pool, so it is exact by construction; a part is absent from a comparison when the model, reading its datasheet, recorded no row under the conditions the question asks about. PMPB10XNE carries only its t ≤ 5 s row, so it is missing from every continuous-current question it should win. That is extraction recall, it is measured at 0.849 for ID, and it is the ceiling on this column.

An error rate a lookup would shrug off is fatal to a superlative, and the reason is structural: a query for an extremum selects FOR the errors that make a value more extreme. Reading the Min column instead of the Max column moved BUK7S0R7-40H from 0.7 mΩ to 0.43 mΩ on 6% of rows — and handed it every lowest-RDS(on) question in its class.

2Method

Corpus
709 datasheets from the vendor's asset host. 497 indexed, 183 held out. The split is fnv1a(part) % 100 < 28, a pure function of the part number, so there is no split file to go stale.
Questions
Nobody wrote them. A parser reads each PDF and emits four labelled facts per part: VDS, maximum RDS(on) with its conditions, continuous ID, and the package. Each is parsed from the table that states it, never from the datasheet's own summary of that table.
Independence
The system under test never sees that parse. Label and answer come from different mechanisms, which is the only reason grading one against the other means anything.
Determinism
Three runs of the same code against the same index return the same score and the same failure set. The noise floor is zero, so a number that moves is a real one.
This page
Renders a committed results file. It does not run the eval: a benchmark that reran per page view would bill the visitor and report a different number each time.

3Defects found

Ten defects the eval surfaced and the test suite did not. Five were in the instrument rather than in the system, and a broken label looks exactly like a broken model until you open the evidence.
Evidence was one chunk per document Retrieval ranked the datasheets perfectly, then handed the model one chunk of each. Document recall read 1.000 while accuracy read 0.353, and the model was right to refuse: the figure was not in the excerpt.
The model was shown half of every chunk The chunker bounds a chunk at 1,800 characters and the prompt truncated the excerpt at 900. Two constants had to agree and nothing made them, so every table chunk lost its last rows.
A concurrent prune deleted a third of the index It inferred where a document ended from whichever slice a request happened to hold, and removed 8,414 of 25,536 chunks at random. Nothing failed. The eval kept reporting 0.95.
The label read the summary, the model read the source ID came from the datasheet's own Quick Reference extract, which quotes a 5-second pulse rating, and the package came off a marketing bullet. 121 of 680 labels were wrong, and the model was marked wrong for reading the authoritative table.
Boilerplate was a third of every document A disclaimer on every page, a legal section running to the end. Asked which package PMN28UNE ships in, the model was handed ten chunks of liability language.
A table row lost its symbol at a chunk boundary A datasheet names a parameter once and leaves the column blank beneath it. Cut between them, the second row is a number belonging to nothing.
Unicode has two ohm signs and this corpus uses both U+2126 and U+03A9 render identically. The unit filter was a character class written to hold both, and it held U+03A9 twice, because you cannot tell them apart in an editor. Every datasheet using the other one lost every RDS(on) row it had: 89 parts with no label, so no question, so never tested.
The label held one measurement per part PSMNR58-30YLH quotes RDS(on) at 10 V and at 4.5 V. The parser kept the 10 V row, so the truth for "lowest at 4.5 V" was computed over a pool missing the part that WINS it. The system answered correctly and was marked wrong.
The examples in a prompt became the answer Told to capture both a package's trade name and its industry code, the prompt listed ten real ones. A part whose ordering table was not retrieved came back with all eleven names. Measured against the PDF text itself, which is neither the model nor the label: 6 of 497 parts named a package the document never prints.
Fixing one column error created another Told to read Min/Typ/Max properly, the model began reading the em-dash in an EMPTY Min cell as a minus sign. 67 N-channel parts came back at -60 V — and the channel was inferred from that sign, so they silently became P-channel parts. A categorical fact carried by a sign bit is one typographic accident from being the opposite fact.

4Limits