We measured our AI pipeline’s error rate

Across 173 vulnerabilities where we held independent NVD ground truth, our AI-assisted classification pipeline attached materially wrong data to 0 of them — a detected error rate of 0.0%.

A deterministic verification layer now catches and suppresses every one of these before it reaches a page. We publish the number because a claim about accuracy is only worth as much as the measurement behind it.

What this number is — and isn’t

The measured error is a wrong-CVE join: our AI-assisted editorial step (which drafts a product label, severity and remediation for each CVE) occasionally stamped that row onto the wrong CVE ID — e.g. describing a Linux-kernel CVE as an IP-camera flaw. The 0.0% is the rate of that specific mistake in the editorial classification layer.

It is not an error rate for the authoritative facts on the site. CVSS scores, exploitation status, affected products and CVE descriptions are taken directly from NVD and the CISA KEV catalog, never generated — and are gated separately. The verification layer described here is exactly what keeps the editorial layer from overriding those official sources.

A separate measurement: severity calibration

Don’t confuse this 0.0% with the override figures in our quarterly reports. This page measures wrong-CVE joins — whether a row was attached to the wrong CVE id. A different check measures severity calibration: our pipeline drafts a provisional severity, and the evidence layer re-scores it against the data before anything publishes.

That layer overrules the draft often and by design — it lowered the AI’s proposed rating 65 of 84 times in our first 90 days and 19 of 31 times in Q2 2026, and never once raised a rating the data didn’t support. A high override rate there is the safety layer working as intended, not a published error — the two numbers measure different failure modes. (The authoritative CVSS you see on a CVE always comes straight from NVD; it is never drafted.)

How we measure it (reproducible)

For every CVE where we hold both an editorial classification and NVD ground truth (173 of them), a deterministic check flags a conflict when the editorial identity contradicts NVD in one of two ways:

• 0 cases — the editorial CVSS differs from NVD by ≥1.5 and the editorial product shares no term with NVD’s product or description.
• 0 cases — NVD has not yet scored the CVE, but its description names something the editorial product plainly contradicts.

The rule is tuned for precision (zero false positives on 173 checked rows in validation), so 0.0% is a lower bound — it counts only errors provable against NVD, not every error that may exist. Method and code: editorialConflictsWithNvd in our open page-builder.

The 0 corrected classifications

Every flagged CVE, with the wrong editorial label our pipeline drafted (left) and the NVD ground truth it was reconciled against (right). All are now suppressed on-site in favour of the NVD record.