Is AI slop in vulnerability reporting a real problem, or is it overblown? Yes: it is a real problem, and the pressure it exposes sits in absorption, because your queue cannot read everything it receives. Today, writing an automated vulnerability report costs next to nothing, while evaluating one still consumes expensive engineering hours. That widening gap is the asymmetry behind the noise landing in your triage queue. To navigate this flood, you have to separate raw candidates from validated findings, accepted vulnerabilities, and shipped patches. That distinction answers three questions in turn: what the slop is, whether the capability behind it is real, and what to demand from anyone selling you findings.
What is “AI slop” in vulnerability reporting, and how is it degrading triage for everyone?
“AI slop” is maintainer shorthand for verbose, low-effort vulnerability submissions generated at near-zero marginal cost that fail under verification. The FFmpeg project dubbed this flood “CVE slop”. These reports present four markers: no working proof of concept, duplicate findings of known issues, non-security bugs dressed up as severe flaws, and confident prose substituted for evidence. Reviewing a single submission still demands 30 to 60 minutes of senior engineering focus, so overall signal degrades even as genuine vulnerabilities surface.
This trend sits right beside legitimate AI-assisted vulnerability discovery. The exact same models power both, yet the output differs based on verification discipline. Slop submissions share familiar tells: reproduction steps that fail immediately, version mismatches, non-existent API calls, fabricated stack traces, and routine platform behaviours framed as security compromises. Because generation tools face no penalty for exaggeration, most automated submissions arrive claiming high or critical severity.
That dynamic creates an intake crisis. Linus Torvalds described the Linux security list as “almost entirely unmanageable” because different researchers running identical tools repeatedly surface the same issues. When curl received reports every 18 hours, volunteer maintainers spent hours debunking fabricated write-ups. The noise became severe enough that Google paused its open-source bug bounty programme, demonstrating what happens when maintainers absorb the first wave of reports and why disclosure is outrunning patching. Both were volume problems at the intake door.
Structural gating is the proven response. Teams that manage the volume rely on automated intake filters. Platforms like HackerOne and Bugcrowd enforce reporter identity validation, historical reputation scoring, and submission throttling. The underlying principles are straightforward: require a working reproduction before an engineer reads anything, deduplicate incoming claims against GitHub Security Advisories (GHSA) and internal trackers, route submissions by code component rather than reported severity, and publish an explicit intake policy that discourages automated farming.
Everything above assumes the queue is full of fakes. The strongest number in the research says most of them are not, which means the irritation you just recognised is pointing at the wrong culprit.
Is AI actually making open source less safe, or is it just finding bugs that were always there?
Mostly it is finding bugs that were always there. Whether this makes open source less secure comes down to a rate comparison: does validated discovery outrun downstream remediation capacity? The models do uncover real flaws. In an independent assessment of 1,752 findings, 90.6% proved to be genuine bugs, and 62.4% represented high or critical severity flaws.
These discoveries represent defects that sat dormant in codebases for years, sometimes decades. AI scanners identified a 16-year-old FFmpeg flaw that had survived five million automated test runs, alongside a 27-year-old OpenBSD kernel bug. Mozilla’s work on Firefox 150 is the existence proof that structured intake changes the outcome: using Claude Mythos Preview within a dedicated testing harness, Mozilla surfaced 271 security flaws, with 180 classified at high severity, and integrated fixes directly into release cycles.
The conversion funnel
Discovery volume collides directly with maintenance capacity. Over six months, Anthropic recorded 29,000 candidate vulnerabilities across open-source software. That pool yielded 6,157 formal reports sent to maintainers, leading to 584 CVE or GHSA identifiers and 516 upstream patches. (A separate Project Glasswing effort logged 1,596 disclosures across 281 projects, resulting in 97 patches). Reaching 516 patches from 29,000 raw candidates represents a conversion rate below two percent. Your team is on the receiving end of this funnel, and the ratio is what lands on their desks.
Remediation capacity remains the binding constraint. Maintainers want secure code, but lack the hours to review hundreds of unrequested pull requests. Anthropic directed $1.5 million to the Apache Software Foundation to fund triage and patch engineering. Without targeted funding, a tragedy of the commons unfolds: genuine vulnerabilities sit unaddressed in crowded backlogs because maintainers cannot isolate them from ambient noise. Understanding why severity scores misorder risk helps teams prioritise, but the flood of reports breaks the patch queue when remediation capacity cannot keep up.
If candidates are nearly free and patches are not, every number a vendor quotes you is a stage number. You are about to be asked to buy one.
What should a CTO check before trusting a vendor’s AI-assisted vulnerability findings?
A vendor’s finding count tells you very little. What actually separates tools is their false-positive rate on your codebase, backed by working reproductions and verified patch workflows. High candidate numbers usually indicate that a vendor is offloading triage labour onto your engineering team. Before signing off on an automated vulnerability scanning tool, ask these six questions, worded so you can send them as-is:
- What validation layer sits between raw model output and the findings delivered to our dashboard?
- What is your verified false-positive rate on codebases with our language profile, compared to standard synthetic benchmarks?
- Does every reported finding include a self-contained, reproducible proof of concept that runs in an isolated environment?
- How does your system deduplicate findings against public CVE and GHSA databases and our existing issue backlog?
- Who authors, tests, and verifies candidate patches before they reach our developers or upstream maintainers?
- What explicit policies govern retractions, false alerts, and disclosure windows when your scanner makes an error?
Scrutinise vendor benchmarks with care. Independent researchers at Trail of Bits demonstrated that published claims, such as 1Password’s automated patching benchmark, did not survive independent replication. Effective scanning architectures rely on adversarial validation and cross-model checking, where a secondary model actively attempts to disprove findings generated by the primary scanner. Early single-pass systems mirrored the reporter’s own framing in roughly 15% of analyses, which makes independent cross-checking necessary. When evaluating vendor metrics, ask which stage their number describes: a candidate, a finding that was reproduced, a vulnerability someone accepted, or a patch that shipped. Insist on validating a vendor’s findings before you buy.
AI models do identify vulnerabilities, so the only question worth asking about a finding is whether the evidence arrived with it: the affected version, a script that runs, and a demonstrated impact, or whether resolving it means repeating the discovery work. The gates you establish at intake decide whether AI tools strengthen your security posture or overwhelm your engineering team, exposing the broader remediation problem across software development.
Frequently Asked Questions
How can I tell if an AI-generated vulnerability report is real before I spend an hour reproducing it?
Look for evidence that travelled with the claim, not fluency. A real report names the affected version or commit, gives reproduction steps that run as written, and demonstrates impact rather than asserting it. Red flags include references to functions that do not exist, fabricated stack traces, a reflex “critical” severity, and a symptom sold as an impact. If none of that is present, treat it as a candidate, not a finding.
Is it true that most AI-generated vulnerability reports are hallucinated?
Mostly no, and that is the uncomfortable part. In the strongest evidence available, 90.6% of 1,752 independently assessed findings were judged real, and 62.4% were rated high or critical. The queue feels fake because volume and evidence-free framing make it read that way, not because the underlying bugs are invented. Assume the findings are real and the packaging is the problem.
What is the difference between a candidate vulnerability and a validated finding?
A candidate is anything a scanner emits. A validated finding has been reproduced, deduplicated against known CVEs and GHSAs, and shown to have genuine security impact. An accepted vulnerability has an owner and an agreed fix, and a shipped patch closes it. Ask which stage any number describes before comparing it with anything, because candidate counts and patch counts differ by orders of magnitude.
What happens if my team simply ignores AI-generated vulnerability reports?
You will lose real findings along with the noise. High-quality reports get deprioritised when they arrive in the same queue as evidence-free ones, so blanket dismissal turns an absorption problem into missed vulnerabilities. The better move is to publish an intake policy and reject on missing evidence rather than on suspected origin, so legitimate reporters self-select and your senior engineers read fewer reports.
Do AI vulnerability scanners find bugs that human researchers have missed?
Yes, and the examples are old rather than exotic. A 16-year-old FFmpeg flaw survived more than five million automated test executions before AI-assisted discovery surfaced it, and a 27-year-old OpenBSD flaw was found the same way. That is the strongest argument that the capability is genuine. The open question is not detection but how many of those candidates convert into shipped patches.
Why do so many AI-generated reports claim critical severity?
Because nothing in the generation process rewards caution. A system with no responsibility for remediation, no reputation to protect and no visibility of your backlog pays no price for overstating severity, so “critical” becomes the default output. Treat asserted severity as a claim awaiting demonstration. Route by affected component instead, and let your own assessment decide what a senior engineer opens first.
Can I filter AI slop just by requiring a proof of concept?
Partly, and less than you would hope. Requiring a reproducible proof of concept removes a large share of low-effort reports, but a generated PoC can look plausible while referencing versions, functions or configurations that do not exist, so the gate still needs a human check that it actually runs. Pair the requirement with deduplication and reporter reputation rather than relying on any single filter.
How long does triaging one vulnerability report actually take, and why does that matter?
A serious read costs roughly 30 to 60 minutes of senior engineer time, and that figure has not moved. Generation cost fell to near zero while adjudication stayed fixed, which is the whole asymmetry behind the flood. It also means volume is the wrong metric to optimise: cutting the number of reports your team reads is worth more than any improvement in scanner throughput.
What should a small open source project do when it cannot afford a triage process?
Publish an intake policy and let it do the filtering. State what a report must contain, require a reproducible proof of concept before a human reads it, deduplicate against GHSA and your own tracker, and set per-reporter rate limits. HackerOne and Bugcrowd already run reputation systems worth copying the principles from. The policy does most of the work, because most noise will not bother to comply.
What makes a vulnerability report complete?
A complete report lets someone else reproduce the problem without asking you a question. That means the affected version or commit, the exact configuration, runnable reproduction steps, observed versus expected behaviour, and a demonstrated security impact rather than an asserted one. Length and confident narrative add nothing. If the reader has to redo the discovery work to act on the report, it is not complete.
Is a vendor’s published false-positive rate a reliable number to compare against?
Only if it was measured on code like yours rather than on a benchmark the vendor chose. Ask what the validation layer is, how many findings a second model or an independent human attempted to disprove, and whether the rate covers shipped findings or raw output. Trail of Bits’ critique of 1Password’s AI patching benchmark is the reference case for why vendor benchmarks need independent replication.
