We measured our AI pipeline’s error rate

Across 200 vulnerabilities where we held independent NVD ground truth, our AI-assisted classification pipeline attached materially wrong data to 27 of them — a detected error rate of 13.5%.

A deterministic verification layer now catches and suppresses every one of these before it reaches a page. We publish the number because a claim about accuracy is only worth as much as the measurement behind it.

What this number is — and isn’t

The measured error is a wrong-CVE join: our AI-assisted editorial step (which drafts a product label, severity and remediation for each CVE) occasionally stamped that row onto the wrong CVE ID — e.g. describing a Linux-kernel CVE as an IP-camera flaw. The 13.5% is the rate of that specific mistake in the editorial classification layer.

It is not an error rate for the authoritative facts on the site. CVSS scores, exploitation status, affected products and CVE descriptions are taken directly from NVD and the CISA KEV catalog, never generated — and are gated separately. The verification layer described here is exactly what keeps the editorial layer from overriding those official sources.

How we measure it (reproducible)

For every CVE where we hold both an editorial classification and NVD ground truth (200 of them), a deterministic check flags a conflict when the editorial identity contradicts NVD in one of two ways:

• 24 cases — the editorial CVSS differs from NVD by ≥1.5 and the editorial product shares no term with NVD’s product or description.
• 3 cases — NVD has not yet scored the CVE, but its description names something the editorial product plainly contradicts.

The rule is tuned for precision (zero false positives on 200 checked rows in validation), so 13.5% is a lower bound — it counts only errors provable against NVD, not every error that may exist. Method and code: editorialConflictsWithNvd in our open page-builder.

The 27 corrected classifications

Every flagged CVE, with the wrong editorial label our pipeline drafted (left) and the NVD ground truth it was reconciled against (right). All are now suppressed on-site in favour of the NVD record.