defend.network publishes AI-assisted threat intelligence. This page documents – mechanism by mechanism – how that output is verified, what gets blocked before publication, what we don’t verify, and what has broken in the past. Written for skeptical security professionals.
Large language models confidently fabricate. In most domains that is an annoyance; in vulnerability intelligence it is dangerous. A fabricated CVE ID sends an engineer hunting for a patch that doesn’t exist. A wrong severity buries an actively exploited flaw below routine advisories. An invented “fix available” claim closes a ticket that should have stayed open. A hallucinated attribution can defame a named organization.
Most AI-generated security content ships none of its evidence. Our position is the opposite: an automated publication is only trustworthy if its claims are mechanically checkable, and the checks run before publication, not after a complaint. The pipeline below treats the AI draft as untrusted input. Structured facts – CVE existence, CVSS scores, exploitation status – are replaced with data from authoritative sources or removed. Narrative claims that can’t be traced to a source are stripped, genericized, or visibly flagged. When too much fails verification, the briefing does not publish at all.
Every daily briefing goes through the same sequence. Nothing in steps 3–8 trusts the draft from step 2.
Ten security feeds are pulled live at run time: The Hacker News, BleepingComputer, Krebs on Security, Dark Reading, The Record, SecurityWeek, CISA Advisories, Schneier on Security, HackRead, and Sophos News. The most recent items from each (up to five per feed, thirty articles total) become the day’s evidence corpus. Every later verification step checks claims against this exact text.
Before any AI writing happens, CVE IDs, vendor/product names, and threat-actor names are extracted from the corpus with pattern matching, and each entity is indexed by how many independent sources mentioned it. The entity matchers are deliberately tuned to avoid false positives: failing to mark something as corroborated is acceptable; falsely claiming corroboration is not. An AI language model then drafts the briefing with this corroboration index in its instructions – multi-source stories must lead, and single-source claims must stay attributed (“BleepingComputer reports…”) rather than asserted as fact. A continuity index of the last seven briefings prevents re-reporting ongoing campaigns as new.
Every CVE the draft mentions must appear character-for-character in the source corpus. A CVE that the AI “remembered” or extrapolated is removed from the structured data, and any occurrence in the prose is replaced with a visible [CVE redacted: unverified] marker rather than silently deleted. The same source-grounding pass covers other high-risk specifics:
[N] citations may only reference those declared sources – fabricated citation indices are stripped. Paragraphs with no verifiable source get a visible “no source verified” chip.Each surviving CVE is looked up in the NIST National Vulnerability Database. When NVD has the record, its CVSS score, vendor/product data, and references override whatever the AI wrote – canonical data wins, always. Each CVE lands in one of three states: published (NVD holds an authoritative CVSS), reserved (record exists, analysis pending), or pending (not yet in NVD – normal for day-one disclosures). Published records are cached up to seven days; reserved ones are re-checked within a day, and a separate daily re-enrichment job upgrades briefings in place when NVD catches up.
The full CISA Known Exploited Vulnerabilities catalog is fetched (cached no longer than 24 hours) and every CVE is checked against it. Exploitation status follows a strict ladder, most-certain first: KEV-confirmed > source-reported (exploitation language in the immediate vicinity of the CVE in source text, not in KEV yet) > PoC published (public exploit code, no attacks reported) > none. The AI cannot set this field; it is derived entirely from KEV data and source text. EPSS exploitation-probability scores from FIRST.org are attached as a third triage axis, with “high” meaning a score of at least 0.5 or the 95th percentile.
AI models over-rate severity, because security news always sounds dramatic. So the day’s severity is recomputed from structured data. Each CVE gets a score from its strongest evidence tier:
Bounded supplements are added: +2 if a briefing CVE was added to KEV within the last 24 hours (capped), +1 for a reserved/pending CVE (possible novel disclosure), +1 when “no patch”/“unpatched”/“zero-day” language sits near a CVE in source text, +1 for a million-record-scale breach. The total maps to the day’s rating: ≥ 8 critical · 5–7 high · 3–4 medium · otherwise low. When at least one CVE has real structured data, this computed rating overrides the AI in both directions. When no CVE data exists to score, the AI’s judgment is kept but capped at medium – a critical or high day requires structured evidence, full stop.
A second, independently prompted AI pass plays fact-checker: it reads only the source corpus and the finished draft, and lists every sentence whose claims no source supports. Its findings act on a hard threshold: eight or more unsupported claims and publication is blocked outright – the run returns an error for operator review and nothing is committed. Between one and seven findings, each flagged sentence is annotated in the published page with a visible warning marker reading “this claim could not be independently verified against source articles”. The verifier flags; it is never allowed to silently rewrite.
Before anything is committed, a set of mechanical guards runs:
[unverified], [CVE redacted…], TODO/FIXME) are stripped from every reader-facing summary field – titles, excerpts, threat-level text – so pipeline internals never reach search snippets or the RSS feed.Each briefing then ships with an on-page “Editorial integrity” note disclosing the actual counts – CVEs verified, awaiting enrichment, KEV-confirmed, and anything stripped or genericized before publication – and a per-run health record (verifier flags, redaction counts, citation density, severity adjustments) is logged so drift is visible over time.
After deployment, an independent script fetches the production site and asserts the invariants directly on live pages: the homepage links resolve to permanent CVE pages; the current week’s vulnerability report contains no card that claims “Active Exploit” while also saying “No exploitation reported”; CVE pages declare themselves as their own canonical URL; the briefings hub contains no leaked [unverified] markers; the RSS feed parses as valid XML; the sitemap lists CVE pages; and the legacy-URL redirect map returns the correct 301 targets. The pipeline also re-reads its own commits at the end of every run and self-heals the homepage if the day’s briefing failed to land there.
Machine-verified against authoritative data: CVE existence and state (NVD), CVSS scores and affected products (NVD), exploitation status (CISA KEV plus source text), exploitation probability (EPSS), federal patch deadlines (KEV), and the day’s calibrated severity. These are facts the pipeline checks or replaces; the AI cannot override them.
Not independently fact-checked: the narrative prose itself. The analysis paragraphs are AI synthesis of the source articles – the pipeline verifies that their specifics (IDs, names, numbers) trace back to sources and runs a skeptical second pass over every sentence, but no human reads a briefing before it publishes, and a second AI pass is a strong filter, not a guarantee. That is exactly why every briefing carries inline citations to the originating articles and labels each block with a verification tier (below): you can see which statements rest on canonical data and which rest on source reporting, and click through to judge the source yourself.
Failure classes we shipped, caught, and structurally fixed. A verification pipeline earns trust by what it does after it fails. Three real ones:
Vulnerability cards once showed an “Active Exploit” badge directly above “No exploitation reported” chips – the badge came from the AI draft while the chips came from verified state.Fix: the status field is now mechanically derived from the verified exploitation state, a publish-time assertion refuses to build a report containing a contradiction, and the post-deploy smoke test re-checks every live card.
NVD’s machine-readable product lists often put bystander vendors first (e.g. appliance vendors listed on Linux-kernel CVEs), so taking the first entry mislabeled what a CVE actually affects.Fix: CISA KEV’s human-curated vendor/product now wins when available; otherwise vendors are ranked by whether they are named in the CVE’s own description and reference links before any list is truncated.
Internal redaction markers (“[unverified]”) once leaked into reader-facing excerpts – honest internally, garbage in a search snippet.Fix: a publish-time lint strips placeholder tokens from every summary field, the clause containing a redaction is dropped entirely (deleting only the token would quietly assert the unverified claim), and the smoke test checks the live briefings hub for leaks.
Honesty about edges is part of the methodology:
Upstream dependence. Verification is only as good as NVD and CISA KEV. If NVD is slow to analyze a CVE, we can confirm the ID is real but not its score; if a vulnerability is exploited but not yet in KEV, our highest-confidence exploitation signal lags reality. Below the KEV tier, “in the wild” rests on the wording of source reporting, not our own telemetry.
Enrichment lag. Fresh CVEs commonly spend one to three days “partially verified” while NVD analyzes them. A daily re-enrichment job upgrades published pages in place when canonical data arrives; a CVE still unresolved after 45 days is relabeled “unverified – not found in NVD” rather than left pending forever.
AI prose is not fact-checked journalism. No human reviews briefings before publication, and the skeptical second pass is itself an AI. If that second pass fails entirely (an API error), the briefing publishes with a logged warning rather than being held. Source-grounding checks match text verbatim – they confirm a claim’s specifics appear in a source, not that the source itself is correct.
Coverage and corpus. Ten feeds is broad but not exhaustive; a story none of them covers, we miss. The day’s severity reflects the single worst verified story, not your environment’s exposure. And this is a young publication with a corpus measured in months – the archive depth of decade-old outlets isn’t there yet.
Not operational threat intel. Briefings are awareness and prioritization digests. They do not include IOC feeds, detection rules, or ATT&CK mappings. Always verify remediation against the official vendor advisory before acting in production.
Every briefing labels its content with these markers. They mean exactly this, as implemented:
Verified facts (green zone) – the CVE is published in NVD with an authoritative CVSS score. Vendor, product, score, and references on these rows come from NVD, cross-checked against CISA KEV – not from the AI.
Partially verified (amber zone) – the CVE ID is real and appears verbatim in source articles, but NVD hasn’t finished enrichment (reserved or pending). The row shows “awaiting NVD” instead of an inferred score, and auto-upgrades when canonical data arrives. If no NVD record ever materializes after 45 days, it moves to a gray “unverified” zone instead.
AI analysis (purple zone) – synthesized narrative. Each threat paragraph lists its sources, individual claims carry clickable [N] citations to the originating article, and sentences the second-pass verifier couldn’t substantiate carry an inline ⚠ marker. Verify before acting.
Exploitation chips follow the same ladder everywhere on the site: “In the wild” means active exploitation is reported (by CISA or by sources); “In CISA KEV” marks the high-confidence subset confirmed by CISA (KEV entries show both chips); “PoC published” means public exploit code with no reported attacks; and “No exploitation reported” means exactly that. Triage statuses (Active Exploit / PoC Available / Under Review / No Known Exploit) and patch-urgency actions are derived from these verified states – a KEV-listed CVE always reads “Patch immediately”.
More on who we are and why this exists on the About page. See the pipeline’s output on the daily briefings, the live vulnerability dashboard, and the weekly report archive. Found an error? contact@defend.network – corrections are part of the method.
Get the verified daily briefing in your inbox. Free for security professionals.