When the wire transfer clears on a deepfake voice fraud, most people think the damage is done. It’s not. That’s when three more problems start: legal exposure, regulatory notification obligations, and the very real chance your insurer denies the claim.
The FBI Internet Crime Complaint Centre recorded 21,442 BEC complaints totalling $2.7 billion in losses in 2024. Regulatory notification obligations under GDPR, HIPAA, and PCI DSS can be triggered completely independently of the financial loss — and cyber insurance claims are getting denied when organisations can’t show they had documented verification controls in place.
This article is about what happens after prevention fails. If you want the full scope of AI social engineering risk, the pillar guide covers that. Here we’re mapping the legal, regulatory, and insurance consequences you need to understand now.
It’s three problems at once, not one.
Deepfake voice fraud creates exposure on three fronts simultaneously: the financial loss itself (usually irreversible), regulatory notification obligations if data was accessed, and insurance coverage risk if your controls were undocumented.
Wire transfers executed on fraudulent instructions are classified as authorised push payment fraud. Your organisation authorised the transfer, which dramatically limits what the bank owes you. In the US and Australia, there is no automatic compensation mechanism for corporate entities. That money is gone.
But here’s what most organisations miss in the first hours: if the attacker got in via a credential reset or MFA bypass, you now have a data breach sitting alongside the fraud. Data access — not financial loss — is what triggers GDPR, HIPAA, and PCI DSS obligations. A lot of organisations don’t realise this until it’s too late to respond properly.
And then there’s the personal liability question. D&O (Directors and Officers) liability can attach personally to executives who failed to mandate verification controls. The criminal conviction of Uber’s former CISO and the SEC action against the SolarWinds CISO make this concrete — personal liability for security governance failures is real, not theoretical.
Put it together and you see why the documented losses are only part of the story. A $7.3M wire fraud loss is bad. That same loss plus a GDPR enforcement action plus an insurance claim denial is materially worse. The board needs to understand all three fronts exist, not just the one that shows up in the bank statement.
Your cyber policy probably covers less voice fraud than you think — and the coverage you do have may come with conditions attached.
Most cyber insurance policies treat social engineering fraud separately, under a rider or extension with much lower sub-limits. We’re talking $100,000–$250,000 typically. You might have a $5 million cyber policy and only $250,000 of social engineering fraud coverage. And that $250,000 can be denied if you didn’t document your verification controls.
Coalition began covering deepfake-enabled wire fraud in 2024 and expanded in 2025 to cover AI-generated audio and video. In February 2026, Upcover added Coalition’s Deepfake Response Endorsement to eligible Australian policies — the first time deepfake-specific cover reached the local SMB market in any serious way. These policies come with conditions attached.
The key condition is the verification clause. This is a policy requirement that your organisation confirmed changes to payment instructions via a previously known contact method before making payment. If you paid a fraudulent invoice without a documented callback procedure, your insurer has grounds to deny the claim.
So here’s what you need to do. Request your policy’s social engineering fraud extension wording. Find out exactly what verification conditions are required. Document your compliance before an incident occurs. Building a verification protocol is exactly what insurance verification clauses are asking for — one process satisfies both requirements.
The regulatory trigger is data access. Not financial loss.
A failed fraud attempt that gave the attacker access to customer records triggers notification obligations. A successful $500,000 wire transfer with no data access may not. That distinction matters enormously for how you respond in the first hours.
The help desk credential-reset vector is where this gets serious: voice impersonation leads to MFA reset, which leads to credential access, which leads to a data breach, which starts the regulatory clocks running. This is the chain that connects voice fraud to regulatory exposure, and it’s not theoretical — it’s how these attacks play out.
GDPR. The 72-hour breach notification window starts at awareness. Not at confirmation. Not after a full forensic investigation. It starts when you know there’s been a breach. Penalties can reach 4% of global annual turnover, and GDPR applies to any organisation handling EU personal data regardless of where you’re headquartered. The operational tension is real: 72 hours isn’t enough time to complete a forensic investigation, which means you may be required to notify before you fully understand the scope.
HIPAA. The notification window is 60 days for breaches of Protected Health Information. HIPAA applies to business associates too — any tech company that stores, processes, or transmits PHI on behalf of a healthcare entity. C-suite executives face civil fines up to $1.5 million and criminal penalties up to 10 years’ imprisonment.
PCI DSS. When voice social engineering compromises payment systems or cardholder data, PCI DSS obligations may apply. Non-compliance post-breach can mean fines, higher transaction fees, and loss of card acceptance rights.
The NY DFS Part 500 regulation (23 NYCRR 500) sets an explicit standard of care — mandatory MFA, documented access controls. Courts outside New York may reference it as a benchmark even for organisations that don’t technically fall under its jurisdiction. Get qualified legal counsel involved before a crisis, not during one.
These aren’t projections. These are investigated, documented incidents. Understanding the attacker economics behind these cases explains why the volume keeps rising even as awareness grows.
Gatehouse Dock Condominium Association (Florida). Nearly $500,000 lost to a BEC scheme — money contributed by residents for essential building repairs. This is a reminder that SMBs and non-corporate entities aren’t exempt from significant losses.
H2-Pharma (Alabama). More than $7.3 million lost in a BEC attack — money intended for cancer treatments and children’s allergy drugs. Both H2-Pharma and Gatehouse Dock are co-plaintiffs in Microsoft’s civil action against RedVDS infrastructure, and both losses are largely unrecovered.
Arup, Hong Kong (January 2024). $25.6 million (HK$200 million) transferred in a single day. Every other participant in the video conference was a real-time AI-generated deepfake, including the CFO and multiple colleagues. None of the funds have been recovered.
FBI IC3 2024 aggregate. $16.6 billion in total cybercrime losses. 21,442 BEC complaints with $2.7 billion in losses. BEC is the single highest-loss cybercrime category — by a wide margin.
Now look at those numbers against a typical insurance sub-limit. A $250,000 social engineering fraud sub-limit covers less than half of the Gatehouse Dock loss, 3.4% of H2-Pharma, and less than 1% of Arup. That’s the gap you’re working with.
Speed is the variable you can still control. The FBI IC3 has a process for exactly this: the Financial Fraud Kill Chain, coordinated through the IC3 Recovery Asset Team. In 2024, the RAT achieved a 66% success rate, freezing $469.1 million in domestic fraudulent funds. That’s real money recovered — when the process is followed quickly enough.
Step one: file an IC3 report via ic3.gov immediately. This activates the RAT and creates the official record you’ll need for insurance claims and legal proceedings. One documented case: a $956,342 BEC wire reported two days after transfer — the RAT froze the account and returned $955,060. Speed matters more than completeness. The report can be updated later, but the recovery window cannot be extended.
Step two: contact the sending bank’s fraud division in parallel. Not after filing the IC3 report. At the same time. Parallel action is what determines how much is recoverable.
For financial sector entities, FinCEN Alert FIN-2024-Alert004 applies — include the key term “FIN-2024-DEEPFAKEFRAUD” in SAR filings. And note that the FBI’s out-of-band verification recommendations are the same controls that satisfy your insurer’s verification clauses. One documented callback protocol covers both requirements.
The first 24 hours determine your recovery options, your insurance eligibility, and your regulatory compliance. The sequence is not arbitrary.
Hour 0–2: Preserve evidence. Before anything else. Call logs, transaction records, email trails, any recorded audio. Do not delete, overwrite, or forward potentially compromised communications. This is the one thing you cannot recover if you get it wrong.
Hour 0–4: File IC3 and contact the bank — in parallel. File an IC3 report via ic3.gov and contact the sending bank’s fraud division at the same time. Hours, not days.
Hour 1–4: Contact legal counsel. Get a lawyer who specialises in cyber incident response. They’ll advise on regulatory notification obligations based on what data the attacker may have accessed.
Hour 2–6: Contact your cyber insurer before any public communication. Most policies require insurer pre-notification as a condition of coverage. A premature public statement can jeopardise the claim.
Hour 4–12: Conduct initial scope assessment. Did the attacker access data, or was this purely financial fraud? If data access is possible, assume the GDPR 72-hour clock is already running. Operate on the most conservative assumption until you know otherwise.
Hour 12–24: Brief the board. Document what happened, what’s been done, what exposure exists, and what decisions need to be made. This is the governance record that matters for D&O liability.
The defensive controls that reduce your legal exposure — verified callback protocols, documented out-of-band verification, dual authorisation — address all three fronts simultaneously. They satisfy your insurer’s requirements and demonstrate reasonable governance to a regulator or court. If you’re reading this after an incident, the absence of those controls is the exposure.
This article covers the consequences of a failure. For the broader AI fraud threat landscape — how these attacks work, who the targets are, and what the full scope of risk looks like — the pillar guide covers all of that.
Under certain D&O liability scenarios, yes. If a court or regulator finds that the absence of documented verification controls constituted a governance failure, personal liability can attach to executives who had authority to mandate those controls. The sophistication of the fraud is not a mitigating factor. Worth noting: 38% of CISOs are not covered by their company’s D&O policy. Check whether your role is an “insured” under the policy terms before you need to find out the hard way.
Most cyber policies treat social engineering fraud separately, under a rider or extension with sub-limits commonly at $100,000–$250,000. The main cyber policy limit probably does not apply to voice fraud wire transfer losses. Social engineering fraud often falls into a gap between cyber insurance (breach response) and crime insurance (fraud losses) — most organisations don’t know which policy responds until they file a claim. Find out now.
It’s a policy condition requiring you to have confirmed payment instruction changes via a previously known contact method before making payment. It’s the primary mechanism by which insurers deny social engineering fraud claims. Without a documented callback procedure, the clause may void your claim entirely.
If the attacker gained access to personal data via credential compromise, notification obligations are likely triggered. Under GDPR, the 72-hour clock starts at awareness of the breach — not at confirmation. Get qualified legal counsel involved immediately to assess your specific obligations by jurisdiction.
File immediately. The IC3 Recovery Asset Team achieved a 66% success rate in 2024. Recovery rates decline significantly as funds move through intermediary accounts. File even if you don’t have the full picture yet — the report can be updated, but the recovery window cannot be extended.
Yes, if the company is a business associate of a healthcare entity — storing, processing, or transmitting Protected Health Information on behalf of a covered entity. A cloned voice convincing a help desk to reset credentials that grant PHI access triggers HIPAA notification obligations regardless of your company’s primary industry.
Not for a serious incident. Documented losses range from $500,000 (Gatehouse Dock) to $25.6 million (Arup). A $250,000 sub-limit covers less than half of the smallest documented case. Review your sub-limits against realistic incident scenarios — not against the sub-limit that looked reasonable when you bought the policy.
It’s an FBI-managed inter-bank rapid recall process coordinated through the IC3 Recovery Asset Team. When a victim files an IC3 report, the RAT contacts the receiving bank to freeze and recall the fraudulent transfer. It achieved a 66% success rate in 2024, freezing $469.1 million in domestic fraudulent funds. File the IC3 report first and contact your bank simultaneously — both within hours of discovery.
This article provides general information about legal, regulatory, and insurance considerations related to AI-enabled fraud. It does not constitute legal advice. Consult qualified legal counsel for advice specific to your organisation’s circumstances and jurisdiction.
Building an AI Voice Fraud Defence Stack Without a Dedicated Security TeamVoice cloning now requires as little as 3–30 seconds of audio to produce an indistinguishable replica of any executive’s voice. The FBI reported $2.77 billion in Business Email Compromise losses in 2024. In documented cases from 2024 and 2025, finance employees authorised millions in wire transfers after what appeared to be legitimate video conferences — every face and every voice AI-generated.
Security awareness training cannot protect staff against a voice that sounds exactly like their CEO. As covered in why training alone cannot protect your finance team, NCC Group‘s researchers concluded it “would just not be reasonable to expect the victims to detect the subterfuge” during live vishing exercises using AI voice cloning against real organisations.
What works — for organisations of any size — is process. This article gives you a concrete defence stack you can deploy without a dedicated security team. It’s part of the broader picture of AI-enabled social engineering threats facing your organisation. These are process controls that break the attacker’s ability to exploit urgency and authority impersonation, regardless of how convincing the voice clone is. And every single one of them can be set up by a generalist.
When AI can produce a voice indistinguishable from the real person, the defence has to shift. You stop asking “can I tell this is fake?” and start asking “does this request follow our verified process?”
Voice cloning defeats the human ear. NCC Group trained a working model using minutes of publicly available voice samples on consumer hardware, and that clone deceived victims in practical security assessments. Research puts deepfake voice detection accuracy as low as 38.2% — under test conditions, not during a time-pressured call from someone who sounds exactly like the CEO.
Enterprise-grade detection tools do exist. But they require dedicated security staff, significant budget, and enterprise infrastructure. That’s not realistic for a company of 50–500 without a security team.
The attacker’s real weapons are urgency and authority impersonation. A cloned CEO voice saying “I need this wire done now, we will lose the deal” creates exactly the right conditions for fraud. Attackers script calls to make hesitation feel like insubordination.
Process controls neutralise both levers. Out-of-band verification, dual authorisation, and code word protocols don’t try to identify whether a voice is synthetic — they verify the request through a channel or process the attacker cannot control. None require specialist tooling.
Attackers don’t restrict targeting by company size. The average BEC wire transfer request reached $128,980 in Q4 2024 — organisations of all sizes are viable targets if their executives have public audio available, and most do. The complete AI-enabled social engineering threat landscape covers why commodity attack infrastructure makes company size irrelevant to attacker targeting decisions.
Out-of-band verification means confirming any phone or video request involving financial actions or credential resets through a pre-established, independently verified secondary channel. Callback verification — hanging up and calling back on a known number — is one type of this broader principle. It also covers any separately established channel: a corporate directory number, a secure portal, or in-person confirmation.
The channel must be separate from the channel on which the request arrived. Never a callback to the number that called you — caller ID can be spoofed.
Why does this defeat voice cloning? Because you’re verifying the channel, not the voice. A perfect voice clone cannot simultaneously control the requester’s pre-registered mobile number or appear in person.
Implementation steps:
Establish a verified contact directory. Mobile numbers and secondary contact methods for everyone authorised to request financial actions. Verified in person — not by email.
Distribute via a secure internal channel. A locked intranet page, laminated copy, or secured messaging channel. Not email.
Define the policy trigger. Any phone or video request involving wire transfers, payment changes, vendor banking detail modifications, or credential resets requires out-of-band verification before action.
Hang up and verify independently. Contact the requester via a different channel from the verified directory. Confirm the specific request details. Proceed only after independent confirmation.
Define the exception procedure. If the requester cannot be reached within 30 minutes, escalate to a second authoriser. Never proceed based solely on the original call.
A legitimate requester will accept a callback without objection. An attacker will resist it — and that resistance is itself a signal.
Dual authorisation means two separate individuals independently verify and approve high-value or unusual transactions before anything gets executed. Neither person can rely on the other’s verification — the second approver has to independently confirm the request through their own out-of-band channel.
The principle is straightforward: a single fraudulent call can compromise one person. Dual authorisation means the attacker has to simultaneously deceive two people through two separate verification processes. This is the four-eyes principle, and it’s standard in financial controls for exactly this reason.
Implementation steps:
Define the threshold and triggers. A reasonable SMB starting point: any transfer above $10,000, any new payee, any change to existing vendor banking details, any deviation from standing payment orders.
Assign approval pairs. No single person can authorise a payment above the threshold alone. Document the pairs and their pre-authorised alternates.
Require independent verification from each approver. The second approver must confirm independently — not by asking the first approver “did you verify this?”
Document every triggered transaction. Timestamp, both approvers’ names, verification method, and exceptions. This is your audit trail for any subsequent dispute or investigation.
Define exception handling. Pre-authorised alternates step in when primary approvers are unavailable. Never reduce to single-person approval under time pressure.
Dual authorisation adds 15–30 minutes to a transaction approval. That’s deliberate. FACC Aerospace lost $58 million to CEO impersonation fraud, and the board fired both the CEO and CFO for inadequate controls. A 30-minute process delay is not comparable to that outcome.
A code word protocol is a pre-established, rotating challenge phrase shared privately between specific individuals — CEO and CFO, executive and finance lead — that has to be provided before any unusual financial action gets authorised.
A voice clone cannot defeat it. A clone replicates the sound of a voice, not the content of a private agreement. The attacker doesn’t know the code word because they weren’t part of the private process that created it. No external party can access knowledge established in a private conversation.
This concept is familiar in personal safety contexts — Kaspersky recommends code words for family emergency verification. Formalising it as a business control is less common, and that gap matters.
Implementation steps:
Establish code words in person or via a verified secure channel. Never by email or phone. If it can be intercepted, it doesn’t qualify.
Assign code word pairs per critical relationship. CEO–CFO, CEO–Finance Lead, CTO–Help Desk Lead. Each pair has its own code word known only to those two individuals.
Rotate quarterly. Rotate immediately if any party suspects compromise, or if a staff member who knew the code word leaves.
Define the challenge procedure. When a financial request arrives via phone or video, the recipient asks for the current code word. No code word means no action.
Integrate into onboarding. New executives and finance staff establish their code word pairs in person during role setup.
The protocol requires no software and no vendor — just a private conversation that an attacker with every public recording of the CEO’s voice still cannot replicate.
Help desks are targets for voice fraud because they hold the keys to user accounts. An MFA reset or credential change triggered by a cloned voice can compromise email, file shares, and every application linked to single sign-on. NCC Group’s vishing assessments successfully performed password resets and email address changes using real-time AI voice cloning against real organisations. This is not theoretical.
The checklist below is concrete enough to print, post at workstations, and adopt as internal policy without modification.
IT Help Desk Verification Checklist
[ ] Require a unique case number or callback PIN before any action. Any credential reset, MFA enrolment, or privilege change must reference an existing ticket. If the caller cannot provide one, create a new case and verify through a secondary channel first.
[ ] Never call back the number that called you. Always use the corporate directory number. Caller ID can be spoofed.
[ ] For MFA enrolment or credential changes: require in-person or verified video with a liveness challenge. Ask the caller to perform a spontaneous physical action. This defeats pre-recorded deepfakes — though the value of liveness challenges will evolve as the technology does.
[ ] Enforce two-person approval for any privilege escalation. No single operator can grant elevated access based on a phone call alone.
[ ] Log every verification request. Timestamp, claimed identity, verified identity, action requested, verification method, outcome.
[ ] Treat urgency as a red flag, not a reason to proceed. A legitimate executive will accept verification. An attacker will resist it.
The hardware complement: Phishing-resistant MFA (FIDO2/passkey) eliminates an entire class of vishing attack. FIDO2 keys bind cryptographically to specific domains — they cannot be verbally shared or socially engineered. CISA designates FIDO/WebAuthn as the gold standard. Deploy to finance staff, IT administrators, and C-suite first. FIDO2 keys such as YubiKey cost $25–$50. Passkeys are increasingly built into existing devices — Windows Hello, Apple Touch ID, and Face ID qualify where supported.
A mandatory cooling-off period imposes a minimum waiting period before executing unusual high-value transfers — even after apparent verification. The recommended duration: 30 minutes minimum for transfers above your defined threshold, two hours for new payees or changed banking details.
Urgency is the attacker’s lever. A mandatory delay removes the ability to exploit time pressure regardless of how convincing the impersonation is. The Arup case makes this concrete: the finance employee initially suspected fraud but was convinced by what appeared to be confirmation on the video call. A mandatory hold would have created a window for that suspicion to become action — and for out-of-band re-verification to happen. If re-confirmation cannot be obtained within the window, the transaction does not proceed.
Implementation steps:
Define “unusual” in your payment approval policy. Remove judgement calls. An unusual transaction is: any transfer above threshold, any new payee, any change to vendor banking details, any request accompanied by urgency language.
Set the cooling-off duration. Thirty minutes minimum above threshold. Two hours for new payees or changed banking details.
Document and communicate the requirement. Written policy — so it can be enforced and compliance demonstrated in an investigation.
Conduct out-of-band re-verification during the cooling-off period. Not optional even if the original verification appeared successful.
No exceptions for urgency claims. Urgency language triggers the longer cooling-off period, not a shorter one. Document this explicitly so staff can enforce it.
An OSINT (Open Source Intelligence) attack surface audit is about inventorying which executives have public audio and video that could serve as voice clone training data. Think of it as the defensive version of the attacker’s reconnaissance.
Attackers collect voice training data from public sources: conference talks, podcast recordings, LinkedIn videos, earnings calls. NCC Group documented obtaining samples “often low quality or including background noise” that were still sufficient after standard audio processing. An executive with two or three public recordings has provided far more training material than an attacker needs.
Implementation steps:
List all executives and senior staff with financial or operational authority. CEO, CFO, CTO, Finance Director, IT Director.
Search publicly available audio and video for each person. Their name plus: “podcast,” “interview,” “conference,” “YouTube,” “LinkedIn,” “earnings call,” “webinar.”
Categorise exposure level. High: multiple long-form audio sources. Medium: one or two appearances. Low: minimal or no public audio.
Assign higher-verification protocols to high-exposure individuals. Require code word verification for all financial requests, not just unusual ones.
Consider reducing future exposure where practical. Not all content needs to remain publicly indexed indefinitely.
Repeat quarterly, or when executives take on new public-facing roles. OSINT exposure mapping tools such as Brightside AI can assist, though a manual search is sufficient for most organisations without a dedicated security team.
The detailed legal, insurance, and regulatory response is covered in what you are liable for without a verification protocol. Here are the immediate actions in the first 60 minutes.
Financial recovery first. Contact the receiving bank immediately and request a wire recall. Notify your own bank’s fraud department at the same time. The FBI emphasises contacting financial institutions within 48 hours for the best chance of recovery.
Internal notification. Alert senior leadership, IT (the attacker may still have access if voice fraud compromised credentials), and legal counsel. Don’t communicate about the incident via the compromised channel — use out-of-band communication for all incident discussion.
Evidence preservation. Preserve everything: call recordings, emails, chat logs, transaction records, access logs. Document the timeline — when the request arrived, what verification steps were followed or skipped, when the fraud was discovered.
Regulatory obligations. GDPR, HIPAA, and PCI DSS may impose notification timelines from the point of discovery — engage legal counsel in the first 60 minutes. The legal and insurance stakes if these controls are absent are covered in the next article. The controls above form a process-first defence against AI-enabled social engineering threats facing your organisation.
As little as 3–30 seconds of clear audio is sufficient for current tools. NCC Group trained a usable model with minutes of publicly available samples, and quality improves with more material. Any executive with public video or podcast appearances has already provided more than enough.
Yes. In practical assessments using real-time AI voice cloning, NCC Group found that detection “would just not be reasonable to expect.” Research shows deepfake voice detection accuracy as low as 38.2% under test conditions — in a live, time-pressured call, performance is likely worse. Process controls are more reliable than human detection.
Out-of-band verification is the broader principle: confirming a request through any pre-established, independently verified secondary channel. Callback verification — hanging up and calling back on a known number — is one specific type of out-of-band verification.
Quarterly is the recommended baseline. Rotate immediately if any party suspects compromise or if a staff member who knew the code word leaves.
A common SMB starting point: any transfer above $10,000, any new payee, or any change to existing vendor banking details. The average BEC wire transfer request reached $128,980 in Q4 2024 — thresholds below this protect against the typical attack range.
No. It applies to unusual transactions: above threshold, involving new payees, changed banking details, or accompanied by urgency language. Regular standing orders to established payees are exempt.
FIDO2 keys such as YubiKey typically cost $25–$50. Passkeys are increasingly built into existing devices — Windows Hello, Apple Touch ID, and Face ID qualify where supported. Deploy to finance staff, IT administrators, and C-suite first.
Use concrete losses: $2.77 billion in BEC losses in 2024, the FACC Aerospace $58 million loss that led to both the CEO and CFO being fired. The primary investment is process change, not budget. Compare the cost of a 30-minute delay against an unrecoverable wire transfer.
The same controls apply. Out-of-band verification, dual authorisation, and code words verify the request through a channel the attacker cannot control — audio or video. For video calls, add a liveness challenge: ask the caller to perform a spontaneous physical action.
Yes, provided the number was verified independently — in person or via the corporate directory — and is not the number that initiated the suspicious call.
Some policies exclude losses where verification protocols were not followed. Implementing these controls strengthens both your defence and your insurance position. See the legal and insurance stakes if these controls are absent for detail.
A blanket ban is impractical. Conduct the OSINT audit, assign higher-verification protocols to high-exposure executives, and ensure all staff follow the checklist regardless of who calls.
Why Security Awareness Training Is Not Enough to Stop AI Voice FraudHere’s the number your security training budget can’t answer: humans detect deepfake audio approximately 48% of the time. Worse than a coin flip. For deepfake video it drops to around 24.5%. An arxiv systematic review examined more than twelve empirical studies and found the same failure pattern across different populations, languages, and voice cloning systems. A NIH-indexed Monash University review confirmed it — “human judgement of deepfake audio is not always reliable.”
The standard response to AI voice fraud is to schedule more security awareness training. On the surface, that seems reasonable. Training has improved employee behaviour against email phishing, and phishing is the category most organisations have put money into. But AI voice fraud is a different class of attack. It exploits a perceptual limitation, not a knowledge gap. You can train employees to understand what vishing is. You cannot train their auditory cortex to detect synthesis artefacts below the human perceptual threshold.
Vishing attacks surged 442% between the first and second halves of 2024 (CrowdStrike 2025 Global Threat Report). The ShinyHunters/Scattered Spider campaign compromised more than 760 organisations — including Google, Cisco, Wynn Resorts, and Harvard University — through IT helpdesk impersonation calls. Deepfake-enabled fraud losses are projected to reach $40 billion by 2027. For the full AI voice fraud threat picture, the economics are very much in the attacker’s favour.
This article looks at the empirical limits of human detection, evaluates the technical controls available, and identifies where process controls need to pick up the slack when both fail.
No — not reliably. The research is consistent even when conditions favour the listener. Under controlled laboratory settings with participants primed to listen for fakes, detection accuracy reaches 60–73%. Watson et al. (2021) found an average detection accuracy of 42%. Barnekow et al. (2021) found participants correctly identified a cloned voice in only 37% of cases. Frank et al. (2024) found rates of 50–60%, described as “slightly above chance.”
The lab numbers are already bad. Real-world conditions make them worse.
A real vishing call gives the target a ringing phone in the middle of a workday, a caller claiming to be a senior executive or IT support, and a request loaded with urgency and authority. The arxiv study found meaningful degradation in detection accuracy under divided attention. Add time pressure and the social cost of being wrong about a legitimate caller, and whatever marginal advantage a trained employee had disappears completely.
For deepfake video it’s even worse. Human detection accuracy sits at approximately 24.5% (Keepnet Labs 2026). A 2025 iProov study found only 0.1% of participants correctly identified all fake and real media presented to them.
The accessibility of the attack makes it worse again. Modern voice cloning requires as little as three seconds of audio to generate a replica with 85% voice match accuracy. Executive voices aren’t hard to come by — earnings calls, conference presentations, podcast appearances, and LinkedIn video are all free training data for any attacker who bothers to look. For more on why cloned voices are so convincing, the technical mechanisms explain the perceptual reality.
An employee who completes every training module, scores 100% on the vishing quiz, and genuinely understands how voice fraud works still has roughly even odds of detecting a high-quality voice clone under real call conditions. Training addresses what they know. It does not change what their ears can perceive.
Training fails against AI voice fraud because it addresses a knowledge deficit while the attack exploits a perceptual limitation. These are different problems that require different solutions.
This isn’t a blanket attack on security awareness training. SAT genuinely works against email phishing. When the threat cues are visual, cognitive, and verifiable — check the sender domain, hover the link, look for urgency and grammatical anomalies — training improves detection. The employee can pause, examine the email, run through a checklist, and make a considered decision.
Voice fraud removes every one of those checkpoints. The call happens in real time. There’s no link to hover, no domain to inspect, no opportunity to pause and verify while the caller waits. The employee must decide under social pressure, in the moment, based on whether the voice sounds genuine. That’s an auditory perceptual task, not a cognitive knowledge task.
The residual risk numbers confirm it. Organisations running regular vishing simulation programmes find approximately 33% of trained employees still disclosing sensitive information under pressure despite strong warnings (Keepnet Labs). That’s not a training failure in the conventional sense — it’s a ceiling that further training doesn’t lower.
To be clear: training is not useless. It has real value as part of a defence stack — it teaches employees to follow verification protocols, escalate suspicious requests, and stay sceptical. The problem is treating it as the primary defence when the underlying detection capability is demonstrably below the threshold required.
Current detection technologies — liveness detection, behavioural biometrics, and AI voice analysis — each address a specific attack surface. None provides comprehensive real-time protection against AI voice fraud on its own.
Liveness detection requires real-time physical actions — turn your head, hold up three fingers, blink — to confirm a live human rather than a synthetic video feed. It works well against pre-recorded deepfake video. It’s less effective against real-time AI-generated feeds, which are improving fast. And it doesn’t apply to voice-only calls at all.
Behavioural biometrics analyses micro-patterns of user behaviour — keystroke dynamics, scroll speed, device handling — to distinguish genuine users from synthetic fraud. The accuracy figures are impressive: 98.7% against synthetic identity fraud (Innovify/BIIA data). The catch is that this applies to digital session analysis. It won’t tell you the caller claiming to be your CFO is synthetic.
AI voice detection tools are an emerging category. Commercial tools claiming 96–98% accuracy in laboratory conditions drop to 50–65% in real-world deployments. Research by CSIRO found that leading tools collapsed below 50% accuracy when confronted with deepfakes produced by systems they hadn’t been trained on. The adversarial adaptation cycle is ongoing.
Detection technology adds a complementary layer, not a replacement for process controls. The gap between detecting synthetic account activity at 98.7% and detecting a live synthetic voice call in real time remains substantial — and unsolved at commercial scale.
Hardware-key MFA (FIDO2/passkeys) resists voice social engineering structurally, not probabilistically. SMS-based MFA fails because it was never designed for an adversary who can hold a convincing real-time conversation.
In a vishing call, the attacker already has valid credentials. The only barrier left is the MFA step. They trigger an MFA prompt, then instruct the target to read the SMS code aloud to “verify your identity.” The target reads the code. Credential compromised. No malware required, no technical sophistication — just a convincing voice and a cooperative target.
MFA fatigue (prompt bombing) is the push-notification equivalent. The attacker floods the target with approval requests until one gets approved out of frustration. The Uber breach of September 2022 followed this exact pattern. MFA fatigue accounts for 14% of security incidents in the 2025 Verizon DBIR — a rising SMB threat because push MFA is the default for many cloud services.
FIDO2 hardware keys eliminate these attack vectors at the cryptographic layer. The private key never leaves the hardware authenticator. Authentication challenges are bound to a specific domain — a fake login page cannot receive a valid authentication response because the domain binding fails automatically. There’s nothing to read aloud, no push notification to approve, no code to share.
Google’s deployment of FIDO security keys across 85,000+ employees produced zero successful phishing attacks. Microsoft has extended phishing-resistant MFA to 92% of its workforce. CISA designates FIDO2/WebAuthn as one of only two approved phishing-resistant authentication implementations.
The SMB cost concern is manageable. Hardware security keys cost $25–50 per unit. For a 200-person company, you’re not doing full coverage straight away — you’re protecting the 20–30 highest-risk users: executives, finance team, IT administrators, and helpdesk staff. That targeted deployment costs $500–1,500 and eliminates the highest-value attack surface. One important operational note: legacy authentication fallbacks need to go once hardware keys are deployed. If SMS codes remain as a backup option, attackers will use them.
Vishing simulations test whether employees follow verification protocols under pressure. That’s valuable — and it’s also the full extent of what they can achieve. Simulations can’t test whether employees are better at detecting synthetic voices, because that’s not a trainable skill at the accuracy levels the threat requires.
Well-run simulation programmes can achieve up to 90% attack recognition rates as measured by protocol adherence (Keepnet Labs). But that same data shows 33% of trained employees still disclosing information despite strong warnings — a floor that repeated simulation doesn’t lower. The detection gap persists because the problem is perceptual, not procedural.
There’s also a negative-return risk in detection-focused training. Research found that deepfake detection training improved accuracy by around 20%, but participants also became more anxious and less confident — measurable psychological cost without a corresponding improvement in practical outcomes. Some participants overestimated their detection capability, creating false confidence that actually weakened overall verification practices.
The right framing for simulation programmes is as process-gap identification tools. Run them quarterly, targeting finance and helpdesk teams specifically — these are the people handling the highest-value requests. Measure protocol adherence: did the employee follow the callback procedure? Did they escalate the request? Use results to fix process gaps, not to assess whether employees can spot a synthetic voice.
Process controls — specifically out-of-band verification, dual-authorisation procedures, and structured escalation protocols — are the compensating layer where human and technical detection both have documented failure modes.
Callback verification is the highest-leverage single vishing prevention control, identified consistently by security researchers including Vectra. Any sensitive request received by phone — wire transfer, credential reset, access change — must be independently verified by hanging up and calling back on a pre-registered, separately confirmed number. The mechanism is channel disruption: the attacker controls the incoming call; callback verification transfers control to the target. An attacker impersonating the CEO cannot receive calls on the CEO’s pre-registered number. The attack breaks at the channel layer without requiring the target to detect anything.
The Arup case shows exactly why this matters. A finance worker was tricked into wiring $25 million in a deepfake video conference call while actively attempting to verify the request. But verification happened within the channel the attacker controlled. Out-of-band verification would have broken the attack regardless of how convincing the deepfake was.
Dual-authorisation for financial transfers eliminates the single point of failure that CEO fraud exploits. Wire transfer instructions above a defined threshold require two separate authorisations from different individuals. The attacker who successfully impersonates the CEO — bypassing detection, bypassing training — still cannot authorise a transfer unilaterally.
The complete defence stack: security awareness training (verification protocols, escalation behaviour) + technical controls (behavioural biometrics for session-layer fraud, phishing-resistant MFA for credential attacks) + process controls (callback verification, dual authorisation). No single component is sufficient. Together, they are resilient because no single failure cascades to a catastrophic outcome.
The threat is scaling. Vishing surged 442%. Financial services organisations face average losses of $603,000 per deepfake incident. The default response — train your people harder — is empirically insufficient against an attack that defeats human detection at the perceptual level. This is one piece of the wider AI social engineering landscape — from attack mechanics through to legal exposure — that demands a layered response. Building that process layer without a dedicated security team is what we look at next.
Not reliably. Multiple studies find human detection of deepfake audio ranges from 37–73% in controlled laboratory settings. Under real-world conditions — divided attention, authority pressure, urgency — detection rates fall further. A 2025 iProov study found only 0.1% of participants correctly identified all fake and real media. Training improves awareness of vishing as a threat category but does not meaningfully improve auditory discrimination.
Three seconds of clear audio is sufficient to create a voice clone with an 85% voice match to the original speaker. Higher-quality clones require only 10–30 seconds. Sources of executive audio include quarterly earnings calls, conference presentations, podcast appearances, LinkedIn videos, and media interviews — all publicly accessible.
MFA fatigue, also called prompt bombing or push bombing, is an attack where an adversary with valid credentials repeatedly triggers push-notification MFA approval requests until the target approves one out of frustration. No technical bypass is required — only persistence. The attack often includes a vishing call impersonating IT support: “Approve the MFA notification to stop the alerts.” MFA fatigue accounts for 14% of security incidents per the 2025 Verizon DBIR.
No. Training has genuine value as part of a layered defence — it teaches employees to follow verification protocols, escalate suspicious requests, and maintain scepticism under pressure. The argument is that training alone is insufficient as the primary defence because it cannot overcome the perceptual limitation that places human detection accuracy below reliable thresholds. Thirty-three percent of trained employees still disclose sensitive information under vishing pressure despite strong warnings — a floor that further training does not lower.
SMS MFA sends a one-time code that can be read aloud during a vishing call, intercepted via SIM swapping, or captured through SS7 protocol exploitation. FIDO2 uses asymmetric cryptography bound to a specific domain and hardware device: the private key never leaves the authenticator, authentication fails automatically on phishing sites, and there is no code to read aloud. CISA designates FIDO2/WebAuthn as one of only two approved phishing-resistant authentication implementations.
Behavioural biometrics achieves 98.7% accuracy against synthetic identity fraud using four-modal fusion analysis. This high accuracy applies to digital session analysis — detecting automated or synthetic account activity. Its applicability to real-time voice call fraud detection is limited because the analysis operates at the device interaction layer, not the voice layer. It will not detect a synthetic caller in a live conversation.
Callback verification is a procedural control requiring that any sensitive request received by phone be independently verified by hanging up and calling back on a pre-registered, separately confirmed number. It breaks the attacker’s control of the communication channel — the mechanism that makes vishing impersonation viable. Security researchers identify it as the single highest-leverage vishing prevention control because it works regardless of how convincing the attacker is.
Quarterly is the recommended cadence, targeting finance and helpdesk teams as the employees handling the highest-value requests. Measure protocol adherence — did the employee follow the callback procedure, escalate the request, resist authority pressure? — rather than detection accuracy. Use results to fix process gaps, not to grade employees on their ability to detect synthetic voices.
The campaign compromised more than 760 organisations through IT helpdesk impersonation vishing calls targeting SSO credentials. Confirmed victims included Google, Cisco, Wynn Resorts, and Harvard University. Operators were paid $500–$1,000 per successful call using pre-written scripts. It demonstrated the scale at which vishing can be industrialised — compromising organisations with established security programmes.
Hardware security keys cost $25–50 per unit. For a 200-person company, the initial deployment covers 20–30 high-risk users — executives, finance team members, IT administrators, and helpdesk staff. This targeted deployment costs $500–1,500 and addresses the highest-value attack surface. Full workforce deployment can follow on a longer timeline as budget allows.
Why AI-Enabled Fraud Is Accelerating — The Economics Behind the ThreatA complete AI fraud operation now costs an attacker under $60 a month. Less than most SaaS subscriptions. You’re looking at $5 for a synthetic identity kit, $24/month for anonymous virtual machine infrastructure through a service like RedVDS, and about $30/month for a Dark LLM that writes the scripts. That’s your attacker’s stack.
On the other side of that ledger: the FBI’s Internet Crime Complaint Centre recorded $16.6 billion in total cybercrime losses in 2024, with business email compromise alone accounting for $2.7 billion from 21,442 complaints. This article lays out both sides of that equation — what attackers spend and what organisations lose — so you can understand the economic logic driving the acceleration. This article is part of our comprehensive guide to the full AI social engineering threat landscape, which covers how these attacks actually work in practice across every stage of the threat.
Costs collapsed below the threshold where running many parallel fraud operations stopped making sense and started being obviously rational.
Group-IB documented a 371% surge in dark web forum posts featuring AI keywords since 2019, with a tenfold increase in replies. AI isn’t an occasional exploit anymore. It’s embedded as core criminal infrastructure.
Three things happened at roughly the same time. Dark LLMs appeared — criminal-purpose language models with no safety guardrails, priced at around $30/month. Synthetic identity kits dropped to $5 on the dark web. And anonymous VM infrastructure became commodity-priced through services like RedVDS at $24/month. Three separate cost collapses, all landing together.
What previously required technical skill and significant investment now requires a credit card and a few subscriptions. The volume data makes this concrete — multi-step fraud attacks rose 180% year-over-year (BIIA), deepfake attacks surged 880% in 2024 (Pindrop), and identity fraud attempts using deepfakes surged 3,000% in 2023 (Deloitte).
The attacker cost stack has three layers, each available independently on the dark web.
Identity layer — $5. Synthetic identity kits sell for approximately $5 per package on dark web forums. That gets you a generated face image, a cloned voice sample, fabricated supporting credentials, and a fake employment history. Everything needed to construct a fraudulent identity for KYC bypass, new account fraud, credit applications, or social engineering with a credible false persona.
Scripting layer — ~$30/month. Dark LLMs like WormGPT and FraudGPT are criminal-purpose language models trained without safety guardrails. They generate personalised phishing scripts and social engineering pretexts at scale. Group-IB identifies at least three active vendors with over 1,000 active subscribers, with subscriptions ranging from $30 to $200 per month. These aren’t rough tools thrown together — they get updates, support requests, and feature iterations, just like legitimate SaaS products.
Infrastructure layer — $24/month. RedVDS gave criminals access to disposable virtual computers for as little as $24/month, making fraud operations cheap, scalable, and hard to trace. In one month, 2,600 distinct RedVDS virtual machines sent an average of one million phishing messages per day to Microsoft customers alone. Microsoft’s Digital Crimes Unit and Europol took RedVDS offline in January 2026 after linking it to approximately $40 million in U.S. fraud losses since March 2025.
Total: under $60/month for a complete AI fraud capability. The criminal marketplace follows the same competitive dynamics as legitimate SaaS — vendor competition is driving further price compression at every layer. That dynamic has a name.
Cybercrime-as-a-Service (CaaS) is a criminal market model that mirrors B2B SaaS. Attack capabilities are packaged into subscription services, sold by specialised vendors across different layers, with transparent pricing and service-level guarantees. Group-IB notes that CaaS vendors mimic aspects of legitimate SaaS businesses — pricing tiers, subscription models, customer service support, the lot.
Think about how a legitimate company assembles its SaaS stack — cloud hosting, analytics, authentication, CRM — from a bunch of specialised vendors. An attacker does exactly the same: one vendor for identity kits, one for Dark LLM scripting, one for infrastructure. Each vendor specialises. That division of labour reduces the barrier to entry at every layer and makes the overall ecosystem much harder to dismantle.
Deepfakes-as-a-Service (DaaS) is a growing sub-market of CaaS — AI-generated video and audio impersonation tools available on subscription, with tiered pricing and product update cycles. Group-IB documented $347 million in verified deepfake fraud losses in a single quarter, which tells you how fast this sub-market is scaling.
Here is the thing that matters most if you’re thinking about defensive strategy: disrupting a single vendor does not collapse the ecosystem. Microsoft’s RedVDS takedown was their 35th civil action targeting cybercrime infrastructure. New services keep emerging because the underlying market incentives are intact. Remove one supplier in a competitive market and you create an opening for others — and state-sponsored actors using the same infrastructure makes this even harder to contain.
The aggregate figures actually understate the problem because most losses go unreported.
FBI IC3 2024 data: total cybercrime losses hit $16.6 billion, with business email compromise accounting for $2.7 billion from 21,442 complaints. Group-IB documented $347 million in deepfake fraud losses in a single quarter. RedVDS alone drove approximately $40 million in U.S. losses since March 2025 — from one infrastructure provider charging $24/month.
In January 2024, a multinational company’s employee in Hong Kong authorised 15 wire transfers totalling HKD 200 million — roughly USD $25.6 million — after a video conference where every other participant, including the company’s CFO, turned out to be an AI-generated deepfake. The Hong Kong Police Force confirmed the incident. No arrests, and the funds remain unrecovered.
1 in 10 adults encountered AI voice cloning scams, and 77% of voice scam victims reported financial losses (BIIA). The FTC documented $12.5 billion in U.S. consumer fraud losses in 2024 — a 25% increase despite fraud report volumes staying flat. Scams are getting more effective, not more numerous.
From Microsoft’s RedVDS case files: H2-Pharma, an Alabama-based pharmaceutical company, lost more than $7.3 million through RedVDS-enabled BEC — money intended for cancer treatments and children’s allergy medications. These aren’t enterprise targets with big security teams. They look like your clients, or your suppliers.
Run the numbers the way you’d evaluate any business unit.
The attacker’s operating cost: $720/year (12 × $60/month). The Hong Kong case yielded $25.6 million from a single operation. Even conservatively — if 1 in 100 BEC attempts succeeds at a $125,000 average yield — the annual return on a $720 investment beats any legitimate business benchmark by a wide margin.
The efficiency multiplier makes it worse. AI-generated spear phishing achieves a 54% click-through rate compared to 12% for human-crafted attempts, according to CrowdStrike data. That’s a 4.5× efficiency gain. AI tools reduce the cost of attacks while simultaneously making them more effective. At $60/month per operation, running 10 simultaneous campaigns costs $600/month. The rational strategy is to run many moderately targeted attacks in parallel and let the hit rate do the work.
For defenders, the asymmetry is the point. Your organisation needs to invest in training, detection tooling, verification protocols, and incident response capabilities. 91% of enterprises are planning to increase spending on voice fraud prevention over the next 12 months (Modulate, January 2026). That defensive investment costs orders of magnitude more than the $720/year attacker operating cost it’s designed to counter. The asymmetry doesn’t resolve on its own — why these low-cost attacks achieve such high hit rates is a function of detection failure at the human level.
The Group-IB trend line is an accelerating curve. The 371% increase in dark web AI mentions since 2019 has its steepest acceleration in 2024–2025. Pindrop tracked a 1,210% surge in deepfake attacks by December 2025. Deloitte projects generative AI fraud losses will climb from $12.3 billion in 2023 to $40 billion by 2027.
The next phase is agentic AI fraud — fully automated, machine-to-machine fraud chains where AI agents execute attacks end-to-end without any human operator involved. Experian identifies agentic AI fraud as the top emerging fraud threat for 2026. Once that transition happens, the last remaining constraint in the attacker cost stack — the human operator’s time — disappears. The scaling limit shifts from operator availability to computational capacity. And computational capacity is cheap.
Commoditised tools also reduce the skill barrier, which brings new entrants into criminal markets who couldn’t previously participate. More attackers. More parallel campaigns. At $60/month each.
For your organisation, the strategic question is how to calibrate defensive investment to the new economic reality — and what the cost-benefit case for specific defensive controls looks like when run against an attacker operating cost of $60/month. Attack volume will keep increasing regardless of individual disruption actions like the RedVDS takedown, because the underlying economic incentives are structural.
For the broader context of AI-enabled fraud, including how these attacks translate into specific attack vectors and who is running them, the AI-enabled social engineering threat landscape overview covers the full picture across every dimension of this threat.
A Dark LLM is a language model with safety guardrails deliberately removed, sold on dark web platforms for criminal use. Unlike commercial models that refuse to generate phishing content or social engineering scripts, Dark LLMs like WormGPT and FraudGPT are purpose-built for exactly those tasks. Group-IB identifies at least three active vendors with over 1,000 active subscribers; subscriptions range from $30 to $200 per month.
Approximately $5. A synthetic identity kit combines stolen PII with AI-generated facial images, voice samples, and fabricated employment histories — everything needed to pass KYC verification, open fraudulent accounts, or run social engineering with a credible false persona.
RedVDS was a virtual desktop infrastructure provider that sold anonymous VM instances at $24/month. Microsoft’s Digital Crimes Unit and Europol disrupted it in January 2026 after linking it to approximately $40 million in U.S. fraud losses since March 2025. In one month, 2,600 RedVDS VMs sent an average of one million phishing messages per day to Microsoft customers alone.
Voice cloning is the higher-frequency attack vector — cheaper to run, requires less compute than real-time video deepfakes, and works over a standard phone call. Pindrop documented an 880% surge in deepfake attacks in 2024, with voice-based attacks comprising the majority, and a 1,300% year-on-year increase in deepfake voice calls.
Group-IB documented $347 million in deepfake fraud losses in a single quarter. The Hong Kong Arup case in January 2024 resulted in $25.6 million lost in a single incident involving a deepfake video conference. RedVDS infrastructure enabled approximately $40 million in U.S. losses since March 2025. FBI IC3 reported $2.7 billion in BEC losses for 2024, a growing proportion of which involves AI-generated voice and content.
Both are Dark LLMs sold on the dark web for criminal use. WormGPT launched in June 2023; FraudGPT followed days later, with subscriptions ranging from $90 to $200 per month depending on the tier. Multiple competing products — WormGPT, FraudGPT, DarkBERT, DarkBARD — tell you that the criminal AI market has matured to the point of active vendor competition, which only drives prices down further.
AI-generated spear phishing achieves a 54% click-through rate compared to 12% for human-crafted attempts, according to CrowdStrike data — a 4.5× efficiency gain. AI tools reduce the cost of attacks while simultaneously making them more effective.
CaaS means attacking your organisation no longer requires a skilled hacker — it requires a subscription. SMBs handle high-value transactions while typically lacking dedicated security teams — an attractive combination for attackers operating at $60/month.
Individual disruptions slow but don’t stop AI-enabled fraud. Microsoft’s RedVDS action was their 35th civil action targeting cybercrime infrastructure; new services emerge because the market incentives remain intact.
Deepfakes-as-a-Service is a sub-market of Cybercrime-as-a-Service that provides AI-generated video and audio impersonation tools on a subscription basis, with tiered pricing, customer support channels, and product update cycles. Group-IB documented $347 million in DaaS-related fraud losses in a single quarter — evidence of how quickly this sub-market has scaled from niche capability to mainstream criminal infrastructure.
Synthetic identity kits combine stolen PII with AI-generated photos, voice samples, and fabricated employment histories to pass KYC verification at financial institutions, open fraudulent accounts, and establish social engineering backstories — all for $5 per kit.
State Actors and Cybercriminals Are Now Using the Same AI Fraud InfrastructureThe AI tools that nation-states develop to attack high-value targets become commercially available to criminal operators within months. The capability ceiling for financially motivated attacks on ordinary organisations is now set by nation-state innovation — not criminal ingenuity alone.
DPRK, Iranian, Russian, and Chinese state-sponsored groups are documented using the same $30/month Dark LLMs, the same deepfake infrastructure, and the same disposable virtual machine platforms as independent cybercriminals.
This article maps the documented threat actors, the shared infrastructure they use, and what that convergence means for your threat model. Every claim references a named intelligence source: Google’s Threat Intelligence Group (GTIG), CrowdStrike‘s 2025 Global Threat Report, Group-IB, and Microsoft’s Digital Crimes Unit. For the broader picture of the AI-enabled social engineering threat landscape, our pillar guide covers the full context.
At least four nation-states — DPRK, Iran, Russia, and China — have state-sponsored groups with documented, active use of generative AI in offensive operations.
Google’s GTIG has published direct evidence of APT42 (Iran / IRGC), APT28/FROZENLAKE (Russia / GRU), APT41 (China / PRC), and UNC1069/MASAN (DPRK) using AI platforms including Gemini and the Hugging Face API. CrowdStrike’s 2025 Global Threat Report documents FAMOUS CHOLLIMA (DPRK) conducting corporate infiltration operations using GenAI for persona creation and live job interview assistance.
None of these groups are building bespoke AI systems. They are using commercially available or openly hosted models. The same Gemini that APT42 uses to craft phishing lures is available to anyone with a Gmail account. The same Hugging Face API that APT28 uses to dynamically generate Windows commands is public infrastructure.
That is the convergence mechanism. The same AI tools used to clone voices and generate synthetic identities are accessible to state actors and criminals at identical price points.
North Korea operates at least two documented threat groups using AI offensively — FAMOUS CHOLLIMA targeting corporate hiring pipelines, and UNC1069/MASAN conducting cryptocurrency theft.
FAMOUS CHOLLIMA operatives create AI-generated LinkedIn profiles with believable employment histories and AI-generated profile images, then use GenAI to produce plausible technical answers during live job interviews. The goal is legitimate employment at private technology companies, providing long-term insider access. CrowdStrike responded to 304 FAMOUS CHOLLIMA incidents in 2024, with nearly 40% classified as insider threat activity. These are documented incidents at private technology companies — not government agencies or defence contractors.
UNC1069/MASAN uses Gemini for cryptocurrency research and reconnaissance, then distributes the BIGMACHO backdoor via deepfake video lures impersonating known figures in the cryptocurrency industry. Victims are directed to download what is presented as a “Zoom SDK” link.
Any organisation with active engineering recruitment is within FAMOUS CHOLLIMA’s operational scope. Their methods — AI-generated personas, GenAI interview assistance — are already mirrored in commercial tools. Synthetic identity kits, including AI-generated faces and voices, are available on dark web markets for approximately $5 each, with sales continuing to rise through 2025 (Group-IB).
Iran and Russia represent two distinct patterns: systematic intelligence collection and surveillance targeting on one hand, architecturally novel malware on the other.
APT42 (Iran / IRGC) uses Google’s Gemini for phishing lure creation, translation of specialised vocabulary, target reconnaissance on think tanks and political organisations, and research into Israeli defence matters. APT42 also attempted to build a “Data Processing Agent” — a tool that converts natural language queries into SQL to track individuals by phone number, travel patterns, or shared attributes. That is AI-assisted mass surveillance, built on a publicly available LLM.
Iran is not running a single experimental programme. TEMP.Zagros (tracked as MUDDYCOAST), a separate Iranian group, used Gemini to develop custom malware including web shells and a C2 server — and inadvertently exposed hard-coded C2 domains and encryption keys to Gemini in the process, which GTIG used to disrupt the campaign.
APT28/FROZENLAKE (Russia / GRU) introduced an architectural shift that matters for detection. GTIG identified APT28 deploying PROMPTSTEAL against Ukrainian targets — the first documented case of LLM-querying malware in live operations. CERT-UA independently corroborated the finding, tracking it as LAMEHUG.
PROMPTSTEAL masquerades as an image generation programme. Instead of hardcoded exfiltration commands, it queries the Qwen2.5-Coder-32B-Instruct model via the Hugging Face API at runtime to dynamically generate Windows commands — collecting system information, process lists, network configuration, Active Directory data, and Office documents before exfiltration.
GTIG calls this “just-in-time AI” — malware that generates its own instructions dynamically rather than executing a fixed payload. No fixed payload means static signature detection has nothing to match.
APT41 (China / PRC) used Gemini throughout August 2025 for C2 framework development, code obfuscation, and infrastructure reconnaissance (Google GTIG). APT41 sought assistance with C++ and Golang code for a C2 framework called OSSTUN, used prompts related to obfuscation libraries to harden tooling against detection, and used open forums to lure victims toward exploit-hosting infrastructure.
The distinguishing feature of Chinese group AI use is breadth. A separate China-nexus actor observed by GTIG used Gemini across every phase — reconnaissance, phishing, lateral movement, C2 configuration, and exfiltration — treating it as a general-purpose force multiplier rather than a specialist tool.
Cybercrime-as-a-service (CaaS) is the commercial layer where AI attack infrastructure becomes available to any buyer with a dark web connection and a small monthly budget. The price points are specific: Dark LLMs for approximately $30 per month, disposable virtual machines from RedVDS for $24 per month (before its disruption), and synthetic identity kits for approximately $5 each.
Group-IB reports AI mentions on dark web forums are up 371% since 2019. Dark LLMs — purpose-built underground services operated behind Tor with no safety guardrails — have more than 1,000 active users. These are not jailbroken chatbots. They generate phishing content, malware code, and fraud scripts without restriction.
RedVDS, disrupted by Microsoft and Europol in January 2026, is the clearest case study. It provided subscribers with disposable Windows virtual machines leaving minimal forensic trace, combined with AI face-swapping and voice cloning for fraud. Since March 2025, RedVDS-enabled activity drove approximately $40 million in fraud losses in the United States alone. In a single month, more than 2,600 distinct VMs sent an average of 1 million phishing messages per day to Microsoft customers alone.
RedVDS is not an edge case. Affected sectors include construction, manufacturing, healthcare, logistics, education, and legal services. H2-Pharma, an Alabama-based pharmaceutical company, lost more than $7.3 million through a RedVDS-enabled BEC scheme.
For a deeper look at the economics of attack infrastructure and why these price points matter, that analysis is in our companion article.
The capability commoditisation cycle runs in one direction. State actors develop techniques against high-value targets, those techniques prove effective, and within months they appear on dark web markets as subscription services.
PROMPTSTEAL’s just-in-time AI architecture is reproducible using open-source models and public APIs. Qwen2.5-Coder-32B-Instruct is on Hugging Face. The Hugging Face API is public. The only missing ingredient is the targeting and access method — which CaaS supplies.
FAMOUS CHOLLIMA’s AI-assisted persona creation is already mirrored in the $5 synthetic identity kits on the same markets where Dark LLMs are sold. That commoditisation cycle is already complete.
The tools DPRK, Iran, Russia, and China develop to attack high-value targets become commercially available to criminals targeting ordinary organisations within months. That is what this article maps — and the full picture of AI-driven fraud that SMB leadership needs to understand is in our comprehensive guide to AI-enabled social engineering threats.
The convergence requires three specific updates to your threat model.
Hiring pipelines are now an attack surface. FAMOUS CHOLLIMA’s AI-assisted corporate infiltration is documented, not theoretical — CrowdStrike responded to 304 incidents in 2024. Any organisation with active engineering recruitment needs identity verification that goes beyond a polished LinkedIn profile. Synthetic identity kits at $5 each make this scalable, not targeted.
Email and voice channels must account for AI-generated content. RedVDS provided AI face-swapping, voice cloning, and multimedia email thread generation at consumer price points. Deepfake fraud drove $347 million in verified losses in a single quarter (Group-IB). The entry cost for a complete AI fraud operation is under $100 per month.
Endpoint detection has a gap. PROMPTSTEAL’s just-in-time AI architecture bypasses static signature detection because no fixed payload exists. Most EDR tools identify known malware patterns. Dynamically generated runtime commands do not match that model. APT28 pioneered this architecture; it will enter criminal toolkits through the same commoditisation cycle every previous state innovation has followed.
You don’t need to build state-actor-grade defences. You need to recognise that the baseline sophistication of criminal attacks has permanently shifted upwards and update your threat model accordingly. The tools nation-states use today are available to the criminals targeting your organisation tomorrow.
For a more complete picture of the full AI-driven fraud landscape and why commodity criminal infrastructure is now nation-state tested, the companion articles cover every angle of this threat.
Not directly. The risk is capability commoditisation — state actors develop AI tools for high-value targets, those tools become commercially available to criminals within months. The same $30/month Dark LLMs and $5 synthetic identity kits are used by both.
PROMPTSTEAL is malware deployed by APT28/FROZENLAKE (Russian military intelligence) against Ukrainian targets. It queries the Qwen2.5-Coder-32B-Instruct model via the Hugging Face API at runtime to dynamically generate Windows commands — the first documented case of LLM-querying malware in live operations. Static signature detection cannot catch it because no fixed payload exists to match. CERT-UA independently corroborated the finding as LAMEHUG.
Under $100 per month. Dark LLMs run approximately $30 per month. Synthetic identity kits are around $5 each. RedVDS provided disposable virtual machines for $24 per month before its January 2026 disruption. Price points sourced from Group-IB’s 2026 whitepaper and Microsoft’s Digital Crimes Unit.
A Dark LLM is a purpose-built underground AI service sold on dark web markets, operated behind Tor with no safety guardrails. Unlike ChatGPT or Gemini, these are built specifically to generate phishing content, malware code, and fraud scripts. Group-IB reports more than 1,000 active users across multiple vendors.
FAMOUS CHOLLIMA operatives created AI-generated LinkedIn profiles and used GenAI to produce plausible technical answers during live job interviews — obtaining legitimate employment at private technology companies. CrowdStrike responded to 304 incidents in 2024, with nearly 40% involving insider threat activity.
RedVDS was a cybercrime-as-a-service platform disrupted by Microsoft and Europol in January 2026. It provided disposable Windows VMs for $24 per month combined with AI face-swapping and voice cloning for fraud. It drove approximately $40 million in US losses and operated more than 2,600 VMs sending around 1 million phishing messages per day. Its disruption shows the scale of commercial AI fraud infrastructure — and that similar platforms will emerge to replace it.
This is an active gap. PROMPTSTEAL queries an external LLM at runtime to generate commands dynamically, which bypasses static signature detection. Most EDR tools identify known malware patterns, not dynamically generated instructions. The defensive toolchain has not yet adapted to this architectural shift.
APT42 (Iran / IRGC) used Gemini to craft phishing lures, translate specialised content, and conduct target reconnaissance on think tanks and political organisations. They also attempted to build a “Data Processing Agent” converting natural language queries to SQL for tracking individuals by phone number, travel history, and shared attributes.
CaaS is a business model where criminal tools and infrastructure are sold on subscription, mirroring legitimate SaaS markets. AI has expanded CaaS capabilities significantly — Dark LLMs for content generation, synthetic identity kits for persona creation, AI-enhanced VMs for automated fraud campaigns. Group-IB reports AI mentions on dark web forums are up 371% since 2019.
BIGMACHO is a backdoor deployed by UNC1069/MASAN (DPRK) via deepfake video lures impersonating known cryptocurrency industry figures. Victims are directed to download a malicious “Zoom SDK” link. The operation targets cryptocurrency organisations for financial theft.
At the infrastructure level, increasingly not. Both use the same Dark LLMs, the same synthetic identity kits, and the same disposable VM platforms. The difference is in intent (intelligence vs. financial gain) and persistence. But the tools and techniques the defending organisation faces are converging.
The Business Functions Most at Risk From AI Voice Phishing AttacksAI voice cloning has turned vishing from a low-quality phone scam into an attack vector that neither voice recognition nor caller ID checks can reliably stop. Finance teams, IT help desks, and executive functions each face their own attack chains, their own mechanics, and their own outcomes.
CrowdStrike documented a 442% surge in vishing attacks in H2 2024. The FBI’s IC3 recorded $2.77 billion in Business Email Compromise losses across 21,442 complaints in 2024. The three highest-value documented AI fraud cases — Arup in Hong Kong ($25.6M), a Singapore multinational ($499K), and a UK energy company ($243K) — all targeted finance functions using voice or video impersonation.
This is part of the broader threat landscape of AI-enabled social engineering that’s reshaping enterprise security. If you want to understand how voice cloning actually works, read that first. This article maps each high-risk business function to its specific attack chain and what attackers need to pull it off.
Vishing (voice phishing) is a social engineering attack over the phone where an attacker impersonates someone trusted to extract information, authorise transactions, or get into your systems.
A phone call is more dangerous than an email because it happens in real time. Email training teaches people to pause, check, and verify. A voice call removes that buffer entirely. The attacker adapts as the conversation goes, escalates urgency, and exploits the reflex to comply when someone in authority is on the line.
Two traditional verification heuristics have both been invalidated at the same time:
Vishing is growing faster than smishing because there’s a live attacker adapting in real time, not a static message. Cisco Talos reported vishing accounted for over 60% of phishing-related IR engagements in Q1 2025. The FBI issued a specific warning in May 2025 about AI voice messages combined with smishing as a multi-stage attack vector.
Vishing also operates outside the channels most security tools monitor. Email and SMS phishing leave digital artefacts. A vishing call leaves a phone record — and whatever the human on the other end decides to do next.
Finance teams are the primary target because they have direct authority to execute wire transfers. One successful vishing call can route millions into a criminal account — no malware, no network intrusion required.
The finance function attack chain:
Urgency culture creates a structural vulnerability. Finance operations run on deadlines — month-end close, deal closings, regulatory reporting. A call about a confidential acquisition requiring an immediate transfer doesn’t sound unusual in that environment. Pretexting — fabricating a scenario to manipulate a target — accounts for 27% of all social engineering breaches per the Verizon DBIR. Voice cloning makes the fabricated scenario audibly indistinguishable from a legitimate request.
Finance staff are also reluctant to push back on the CFO or CEO. Attackers design pretexts that make hesitation feel like insubordination. Wire-transfer BEC scams increased 33% in Q2 2025 versus Q1.
Three cases establish the scale. Each targeted finance. Each used voice or video impersonation. Each succeeded.
Arup, Hong Kong, 2024 — $25.6M–$39M
A finance employee was invited to a video conference that appeared to include the CFO and several colleagues. Every participant was synthetic. The employee received instructions to execute wire transfers across 15 transactions totalling HK$200 million. Some reports put total losses at $39 million. It’s the largest documented AI voice and video fraud case on record.
Singapore, March 2025 — $499,000
A finance director joined a Zoom call with what appeared to be senior leadership. Every face and voice was AI-generated from publicly available media. The finance director authorised a $499,000 transfer.
UK energy company, 2019 — $243,000
The earliest major documented AI voice fraud case. The UK subsidiary CEO received a call matching the German parent company CEO’s voice — correct accent, speech patterns, mannerisms — and transferred $243,000 to a Hungarian supplier. The voice had been cloned from publicly available conference recordings. This case established that AI voice fraud was operationally viable as early as 2019.
The pattern across all three is the same: finance targeted, urgency or confidentiality as pretext, senior executive impersonated, wire transfer executed before verification could happen. Traditional methods failed — including video calls. Research explains why: humans correctly identify high-quality deepfake video only 24.5% of the time, and deepfake voice detection accuracy is as low as 38.2%.
The IT help desk is the most underappreciated attack surface in enterprise security. A single successful MFA reset call gives the attacker access to every SSO-connected application in the organisation: email, CRM, file storage, HR systems, ERP. The lot.
The MFA reset attack chain:
This is not theoretical. Scattered Spider and ShinyHunters ran professionalised help desk vishing campaigns targeting 760+ organisations, charging $500–$1,000 per call through a vishing-as-a-service model. Confirmed victims include Google, Cisco, Wynn Resorts (800,000+ employee records), CarGurus (12.5 million records), and Harvard University.
What the attacker needs: a voice sample (publicly available), the help desk number (on the website), and knowledge of which account to target. SMBs at 50–500 employees are particularly exposed — formal phone-based verification protocols are rare at this scale, and voice recognition is often the only control.
The threat actors driving these campaigns represent a professionalised criminal supply chain purpose-built for help desk exploitation.
BEC payment diversion is different from CEO fraud direct-instruction attacks. Instead of fabricating a transaction from scratch, the attacker intercepts a real one already in progress.
The BEC payment diversion attack chain:
Voice cloning upgrades email-only BEC by adding vocal authentication at the critical moment. The attacker controls both channels — compromised email and cloned voice — creating a multi-channel attack that standard controls struggle to detect.
The Verizon DBIR found a median BEC loss of $50,000 per incident. Pretexting incidents have nearly doubled to over 50% of all social engineering incidents. BEC and social engineering fraud represent roughly half of all cyber insurance claims over the past five years.
Pig butchering (sha zhu pan) is a long-duration investment fraud where an attacker builds a relationship with a target over days or weeks, convinces them to invest in a fraudulent platform, then disappears with the funds.
Traditional pig butchering is labour-intensive — one scammer, a handful of targets, weeks of personalised contact. AI changes that entirely. Synthetic identity kits — voice profile, AI-generated photographs, fabricated social media history — can be assembled for as little as $5 per persona. Bots sustain relationships around the clock without a human operator anywhere in the picture.
The FTC opened more than 65,000 romance scam cases last year with $3 billion in reported losses. AI-enabled fraud losses are projected to reach $40 billion by 2027 (Deloitte). Pig butchering has historically targeted individuals, but executives are increasingly targeted via investment pretexts from apparently known contacts — and deepfake technology means a video call is no longer proof of identity.
Honestly? You can’t reliably tell from the call itself. Current-generation voice cloning is indistinguishable in most conditions, and caller ID spoofing makes the number appear legitimate.
The answer is out-of-band verification — a communication channel the attacker cannot control. Hang up and call back on a number you look up independently in the corporate directory, not one provided during the call. For wire transfers and MFA resets, the verification protocols required go beyond a quick callback. Defensive controls for each of these attack vectors and how to build a verification protocol without a dedicated security team are covered separately.
No amount of training eliminates this risk entirely: 33% of trained employees still disclose information during simulated vishing tests. Procedural controls are the next layer down.
The landscape of AI-enabled social engineering is accelerating across every attack surface. Mapping each business function to its specific attack chain is the starting point for defences that hold.
Phishing uses email to deliver fraudulent messages; vishing uses voice calls; smishing uses SMS. Vishing is more dangerous per incident because real-time conversation lets the attacker adapt to responses, create immediate pressure, and exploit voice recognition trust. AI voice cloning makes vishing nearly undetectable.
Current voice cloning tools require as little as 3–30 seconds of clean audio from earnings calls, conference presentations, YouTube videos, podcasts, or LinkedIn audio posts. No hacking required.
The Arup case in Hong Kong (2024) — $25.6M–$39M in losses from a deepfake video conference where a finance employee interacted with synthetic recreations of the CFO and multiple colleagues. Every participant was AI-generated.
Yes. Current-generation tools support real-time synthesis — the attacker speaks naturally, AI converts their voice to the target’s with minimal latency. NCC Group demonstrated this using a consumer-grade GPU and minutes of publicly available audio.
Vishing-as-a-service (VaaS) is a commercialised attack model where groups like Scattered Spider sell vishing call execution for $500–$1,000 per call using pre-written scripts targeting IT help desks. It’s the ransomware-as-a-service model applied to voice fraud — it lowers the skill barrier and increases volume.
SMBs (50–500 employees) often lack formal identity verification protocols for phone-based requests. The average cost of a single successful voice attack is $5,000–$25,000, with 20% of organisations experiencing $25,000–$100,000 per incident (Modulate survey). SMBs are targeted nearly four times more often than large enterprises per Verizon DBIR 2025.
Contact your IT security and finance teams immediately. Freeze any transfers in progress. Document the call details — time, displayed number, what was requested. Report to FBI IC3 (ic3.gov). Do not call back the number that contacted you.
Pretexting is fabricating a scenario to manipulate a target — “I’m travelling and lost my phone.” It accounts for 27% of social engineering breaches per Verizon DBIR, and has nearly doubled to over 50% of all social engineering incidents. Voice cloning makes the fabricated scenario audibly indistinguishable from a legitimate request.
Yes. Vishing is classified as T1566.004 (Phishing: Voice) within the Initial Access tactic. T1598.004 covers reconnaissance-phase vishing. Both are used in enterprise threat modelling frameworks.
Caller ID spoofing uses VoIP infrastructure to display a trusted internal number or an executive’s registered mobile. Combined with voice cloning, both the number and the voice match what the target expects — eliminating both primary verification heuristics at the same time.
Deepfake video calls are more resource-intensive but achieve higher-value outcomes — the Arup case ($25.6M) and Singapore case ($499K) both used video. Voice-only vishing is far more scalable and accounts for the majority of attacks by volume. More than $200 million was lost to deepfake scams in Q1 2025 alone (SecurityWeek).
How AI Voice Cloning and Deepfake Technology Actually WorksIn January 2024, an employee at Arup in Hong Kong joined a video call with what appeared to be his CFO and several senior colleagues. He authorised 15 wire transfers. The total: $25.6 million USD. Every person on that call was a deepfake.
Most coverage of this topic stops at “AI can clone voices now.” That’s not useful. It doesn’t tell you how the pipeline works — from a 30-second clip pulled off a podcast to a live fraudulent call that bypasses every intuitive check a victim has. Without understanding the mechanics, you can’t make an honest assessment of the risk.
Voice cloning is one layer. Deepfake video, caller ID spoofing, and Dark LLMs scripting the conversation combine into an attack package that requires no specialist skills. This article walks through each layer — including the detection accuracy data that explains why human vigilance alone doesn’t hold up.
For the broader strategic context, read what every business needs to know about AI-enabled social engineering threats.
AI voice cloning is a machine learning technique. It analyses the acoustic characteristics of a person’s speech — pitch, cadence, tone, rhythm, filler word patterns — and uses that to generate synthetic speech that sounds like them. Feed the model a clean audio sample, give it text input, and it outputs speech in the target’s voice.
The architecture works by converting raw audio into spectrograms — visual representations of sound frequencies over time. An encoder-decoder model (or a diffusion-based system) learns the unique vocal fingerprint in those spectrograms. NCC Group’s research describes this as disentangling linguistic content (what is being said) from identity content (who is saying it). The model holds the “who” constant while generating whatever “what” the attacker needs.
Three distinct capabilities get lumped under “voice cloning,” and it’s worth separating them:
For live vishing calls, real-time voice conversion is the variant that matters — the attacker can respond, adapt, and improvise. Cloud infrastructure is what changed the threat landscape. You don’t need to understand the ML. You need an audio sample and a subscription.
Far less than most people assume. McAfee research found that 3 seconds of audio produces an 85% voice match. ThreatLocker puts high-fidelity cloning at 30 seconds. NCC Group trained a convincing clone from just a few minutes of publicly available samples. These aren’t contradictions — they reflect different quality thresholds on a sliding scale.
Where does that audio come from? No hacking required. PurpleSec’s analysis of the Arup attack spells out the sources: LinkedIn profile videos, conference presentations, earnings calls, interview recordings, corporate marketing videos. For any executive who has done a podcast, presented at a conference, or posted a video — that material already exists and is accessible to anyone with a browser.
Audio quality matters more than duration. Clean, isolated speech is more valuable than hours of recording in a crowded room. The Biden deepfake robocall used to disrupt the 2024 New Hampshire primary cost $1 and took less than 20 minutes.
This is the piece most coverage skips. Here is the actual pipeline, stage by stage.
Stage 1 — Source Audio Collection. The attacker identifies targets — typically financial decision-makers and the executives who’d plausibly contact them — and harvests audio from public sources. No credentials required.
Stage 2 — Audio Isolation and Preprocessing. Raw audio is cleaned using noise-gating, equalisation, and background removal. NCC Group’s team partly automated this with ML-based speaker identification. Output: isolated, clean speech in the target’s voice.
Stage 3 — Neural Network Training. The cleaned audio is converted to spectrograms. A self-supervised model extracts the voice embedding — a mathematical fingerprint of the target’s vocal identity — and fine-tunes a pre-existing model against it. With cloud GPU instances, this step drops from hours to minutes.
Stage 4 — Synthetic Speech Generation. In text-to-speech mode, the attacker types a script and generates audio in the target’s voice. In real-time speech-to-speech mode, the attacker speaks and the output stream transforms their voice live.
Stage 5 — Deployment in Live Attack. The cloned voice is routed through a virtual audio device into the attack channel — a phone call via VoIP, or directly into Microsoft Teams or Google Meet. Caller ID spoofing displays the target’s real number on the victim’s screen. The victim sees the correct number, hears the familiar voice, and has no instinctive reason to doubt either signal.
The entire pipeline can be run by a single person with no ML background. Understanding how cheap the toolchain has become is worth its own read.
Yes. The Arup attack proved it operationally.
Attackers collect executive video footage from public sources, then train face-swap and lip-sync models. Tools like DeepFaceLab are open source. PurpleSec notes that realistic deepfakes can be generated in approximately 45 minutes. GANs or diffusion models synthesise facial movements; lip-sync models match mouth movements to the cloned audio; a virtual camera driver injects the synthetic feed directly into Zoom or Teams.
What made the Arup attack effective was the multi-participant setup. The meeting featured the CFO and multiple senior colleagues — creating false consensus, establishing authority through numbers, and building urgency through a “confidential” framing. No software vulnerability was exploited. The attackers leveraged AI video and audio to impersonate trusted individuals across all 15 transactions before the fraud was discovered.
Deepfake video attacks are more resource-intensive than voice-only vishing but deliver higher-value outcomes. For how these attacks target specific business functions, that’s the logical next step.
Dark LLMs are purpose-built criminal language models — not jailbroken mainstream tools, but systems built from the ground up with no safety training. They run behind Tor, ignore safety rules by design, and are sold as subscription services.
WormGPT launched in June 2023, built on GPT-J-6B, promoted as “a ChatGPT alternative for blackhat.” FraudGPT followed days later, sold by “CanadianKingpin12.” The same vendor advertised DarkBERT, DarkBARD, and DarkGPT. Outpost24 documented pricing: WormGPT from $90/month, FraudGPT at $90–$700 depending on term.
Group-IB data via The Register shows dark web AI mentions are up 371% since 2019. And the distinction from jailbroken mainstream models matters. Jailbreaking ChatGPT requires ongoing prompt engineering against actively improving safety filters. Dark LLMs generate malicious content reliably by design — phishing emails, pretext scripts, malware code, synthetic persona backstories. In a voice cloning attack, the Dark LLM is the scripting layer: contextually convincing cover stories in any language, no English skills required.
Voice cloning impersonates a real person you already trust. A synthetic persona is a fabricated identity — someone who never existed — built to establish a new trust relationship from scratch.
The components: an AI-generated face, a synthetic voice, a fabricated employment history, and a matching social media presence. Group-IB documented that complete synthetic identity kits sell for approximately $5 on dark web marketplaces. A cloned executive voice only works if the victim already trusts that executive. A synthetic persona fabricates a “new colleague” or “vendor rep” — the trust is built through the interaction itself.
BIIA’s 2026 data shows synthetic identities were used in 21% of first-party frauds detected in 2025, and deepfake files grew from roughly 500,000 in 2023 to 8 million in 2025. This is mainstream fraud infrastructure. And the reason it spreads so fast is that humans are reliably poor at catching it.
The numbers do the talking. Humans detect deepfake audio at approximately 48% accuracy — worse than a coin flip. For deepfake video, DeepStrike puts human accuracy at 24.5% — roughly one in four. A 2025 iProov study found that only 0.1% of participants correctly identified all fake and real media they were shown.
The confidence gap compounds this. Approximately 60% of people believe they can spot a deepfake. Actual performance is near random. That overestimation is what keeps people trusting instinct over procedure.
Why is detection so hard? Generative models — GANs and diffusion models — are trained specifically to minimise detectable artefacts. Open-source deepfake detectors can see accuracy fall by as much as 50% against new in-the-wild deepfakes not in their training data. There’s also the Liar’s Dividend: as deepfake awareness spreads, authentic audio and video can be dismissed as synthetic. Real evidence gets denied as AI-generated.
These detection rates mean human vigilance cannot be the primary defence. The Arup fraud revealed the gap: no mandatory out-of-band verification protocol was in place. That’s the shift — from training people to spot fakes, to designing procedures that remove the detection requirement entirely. For the full AI-enabled social engineering threat landscape, the picture is bigger than any single attack type.
Yes. A single podcast appearance provides more than enough source audio. Cloud-based voice cloning tools require as little as 3–30 seconds of clear speech. Any executive who has spoken publicly — podcast, conference, earnings call, LinkedIn video — has already provided sufficient material.
McAfee research found that 3 seconds produces an 85% voice match. Higher fidelity requires 20–30 seconds. Either threshold is met by most publicly available recordings of business leaders.
Yes. The 2024 Arup attack demonstrated a multi-person deepfake conference where the CFO and multiple executives were all synthetic. The fraud totalled $25.6 million and was discovered only through routine post-transaction follow-up.
Perceptually indistinguishable from real speech in most cases. Humans detect deepfake audio at approximately 48% accuracy. Modern voice cloning tools produce natural-sounding output with realistic cadence and filler words, and phone audio compression hides minor artefacts.
NCC Group completed their proof-of-concept on a consumer laptop GPU. With cloud APIs renting GPU compute by the hour, training compresses further. A single person with no ML background can run the full pipeline in well under a day.
A voice clone replicates a real person’s voice. A synthetic persona is a fabricated identity that never existed — AI-generated face, synthetic voice, backstory. Voice cloning exploits existing trust; synthetic personas create new fake people to build fresh trust from scratch.
No. Jailbreaking requires ongoing prompt engineering against safety filters. Dark LLMs like WormGPT and FraudGPT are purpose-built with no safety training, reliably generating malicious content by design. Criminal SaaS at $90–$110/month depending on the tool.
It simultaneously eliminates both intuitive checks: “Is this their number?” and “Does this sound like them?” Both answers appear to be yes, which removes the victim’s instinctive reason to pause.
The full toolchain is cheap. Cloud voice cloning access, a Dark LLM subscription, caller ID spoofing, VoIP infrastructure. Synthetic identity kits sell for approximately $5. The Biden deepfake robocall cost $1 and took less than 20 minutes. A complete attack package runs well under $100.
The second-order effect of deepfake proliferation: authentic audio and video can be dismissed as AI-generated. Real evidence gets denied as synthetic. It erodes trust in all digital communications — not just the communications that are actually fake.
What Is C2PA and How Does Content Provenance Infrastructure WorkDeepfake incidents surged from 500,000 in 2023 to over 8 million in 2025. C2PA — the Coalition for Content Provenance and Authenticity — is the open standard the industry converged on to address this through cryptographic provenance. Over 6,000 organisations have joined the Content Authenticity Initiative, hardware manufacturers ship C2PA-enabled devices, and platforms display Content Credentials. But the trust layer is still being completed, and that matters for your adoption decisions. This page maps the terrain.
C2PA is an open standard that cryptographically binds provenance metadata to media files. As AI-generated media proliferates, there is no reliable way to verify whether an image or video is what it claims to be. C2PA creates a signed, tamper-evident record — a Content Credential — that travels with the file. It demonstrates a signer made certain claims, not that those claims are accurate. How C2PA content credentials work and what they cannot prove has the technical detail.
A Content Credential is a digitally signed data structure embedded inside a media file. A camera, AI platform, or editing tool hashes the file and signs the package using an X.509 certificate — the same model that underpins HTTPS. Any verifier can check that signature and confirm whether the content changed since signing. C2PA bundles everything needed for verification inside the file, so it works offline.
C2PA was founded in 2021 by Adobe, Microsoft, BBC, Intel, Arm, Truepic, and Sony under the Linux Foundation. CAI (Content Authenticity Initiative) is the adoption community; C2PA is the standards body. The spec is royalty-free and core tooling is open source under MIT licence — no vendor lock-in. The adoption landscape is in which hardware and platforms have adopted C2PA in 2026.
No. C2PA depends on Certificate Authorities on its Trust List — content signed by an unrecognised CA shows as “unknown source.” Few CAs are listed, certificates cost ~$289/year from DigiCert, and there is no Let’s Encrypt equivalent. Nikon added C2PA to the Z6 III, discovered a signing vulnerability, and had to revoke all issued certificates — invalidating every credential those cameras had produced. Where the trust layer works and where it breaks in 2026 covers the full picture.
Signing outpaces verification — that is the defining tension. Leica shipped the first C2PA camera in 2023, and Samsung Galaxy S25 and Google Pixel 10 now sign natively, bringing credential creation to mainstream consumer hardware. LinkedIn, TikTok, and Cloudflare support or preserve credentials at scale. But most distribution intermediaries still strip embedded metadata, so signed content often arrives at viewers without its credential attached. Which hardware and platforms have adopted C2PA in 2026 maps the full landscape.
Adoption was already growing, but regulation has fixed the timeline. EU AI Act Article 50 enforcement begins August 2026, requiring machine-readable disclosure on AI-generated content. California SB 942 took effect January 2026. The EU Code of Practice specifies multi-layer marking that maps directly to the Durable Content Credentials architecture. If your organisation produces AI-generated content for public distribution, the compliance clock is already running. How EU AI Act and global regulations make C2PA urgent in 2026 covers what applies to your business.
Most platforms strip embedded metadata during processing, removing C2PA manifests before viewers see them — a byproduct of standard image and video transcoding pipelines, not deliberate suppression. Durable Content Credentials address this by combining the manifest with invisible watermarking (survives processing) and content fingerprinting (enables credential recovery from a repository even after stripping). This three-pillar architecture maps to the EU multi-layer marking requirement. How durable credentials make provenance survive metadata stripping explains the mechanism.
C2PA and watermarking are complementary, not competing. C2PA manifests provide rich structured provenance — who signed, when, with what tool — but are fragile across distribution pipelines. Invisible watermarking survives processing but carries only a lookup identifier. Durable Content Credentials combine both. AI deepfake detection takes a different approach entirely — identifying anomalies in content rather than asserting positive origin. The two methods address different parts of the problem and are most effective in combination. See how C2PA content credentials work and how durable credentials survive metadata stripping.
Credentials can carry identity assertions — creator name, organisation, GPS location. Every signed asset becomes a data point linking identity to time and place. The World Privacy Forum warns that absent credentials may become a negative trust signal, encouraging participation even when disclosure is unwanted. Redaction exists in the spec but is optional. If your creators include people whose identity should not be disclosed, the privacy and identity risks of C2PA identity assertions covers what to manage.
Understanding the Standard — How C2PA Content Credentials Work: manifest structure, signing, verification. The C2PA Trust Layer in 2026: infrastructure gaps and certificate barriers.
Adoption and Compliance — C2PA Adoption in 2026: hardware, platforms, verification reality. Regulations Making C2PA Urgent: EU AI Act, California SB 942, compliance frameworks.
Building and Risks — Durable Content Credentials: surviving metadata stripping. C2PA Pipeline Architecture: SDKs, certificates, cloud patterns. Privacy Risks: identity assertions, surveillance risk.
C2PA confirms a device signed a file, but cannot verify the camera was pointed at what the caption claims. This is a permanent limitation of provenance systems — a credential establishes that a claim was made, not that it reflects reality.
No. Detection tools analyse content for anomalies. C2PA creates a provenance record at capture so viewers can see chain of custody. Provenance shows origin; detection flags content lacking a trail. Neither alone is sufficient.
A valid credential confirms a recognised signer made specific assertions at a specific time. It does not verify those assertions are accurate or that the signer acted in good faith. C2PA is a chain-of-custody record, not a truth-verification system.
No. C2PA uses standard X.509 PKI — the same certificate model as HTTPS — not blockchain. It reuses existing infrastructure, works offline, and needs no ledger access. If you have evaluated blockchain provenance tools, C2PA’s trust model is meaningfully different.
The official tool at contentcredentials.org accepts uploaded images and video. LinkedIn and TikTok display credentials on supported content. A missing credential does not mean content is fake — it may have been stripped during distribution.
The open-source SDK (c2pa-rs) and CLI (c2patool) are at github.com/contentauth, with a Node.js wrapper. Cloud pipeline patterns are in architecture patterns for building C2PA signing into a cloud pipeline.
C2PA is real infrastructure with real gaps. The standard works, the trust layer is incomplete, and regulatory deadlines are fixed. Pick the section that matches your question.
C2PA Identity Assertions and the Privacy Risks of Content CredentialsC2PA and content provenance infrastructure is built to prove where content came from and tie it to its creator. Identity linkage isn’t a side effect — it’s the whole point. And the privacy implications depend almost entirely on the implementation choices your organisation makes before you deploy a signing pipeline.
When your Claim Generator signs a media file, it embeds assertions into a manifest: a signing certificate identifying the tool or device, optional CAWG extensions linking verified social media profiles and government-issued identity to the creator, and device metadata like camera serial numbers — all cryptographically bound and publicly readable. That’s a lot of personal data to be leaving in files that might end up anywhere.
The World Privacy Forum‘s 2025 analysis of C2PA identified specific privacy gaps in the conformance programme and trust model that haven’t been resolved. This article maps out the risk surface, looks at what controls are actually available, and gives you design recommendations for keeping your implementation’s exposure to a minimum.
The claim generator’s signing identity is always present as an X.509 certificate. You can’t remove it without invalidating the manifest. On top of that, the manifest can include device metadata: camera serial numbers, GPS coordinates, and software identifiers.
Tim Bray did a live examination of a Leica M11 image using c2patool and found 29 EXIF values, including the camera’s body serial number. His reaction: “I could see that as a useful provenance claim. Or as a potentially lethal privacy risk.” He’s right on both counts — and none of that required CAWG at all.
The optional CAWG identity assertion layer goes further still. When a creator attaches a CAWG identity assertion, it links verified identities from multiple providers: social media profiles (cawg.social_media), government document verification (cawg.document_verification), organisational affiliations (cawg.affiliation), website ownership (cawg.web_site), and crypto wallet addresses (cawg.crypto_wallet). Examining his own Content Credential, Bray found a chained attestation: “Adobe says that LinkedIn says that Clear says that the government ID of the person who posted this says that he’s named Timothy Bray.”
None of this is encrypted. C2PA metadata is publicly verifiable — anyone who receives the media file can read all embedded assertions using c2patool or any conformant reader. The specification lists “Privacy” as a design goal. But that guidance is non-binding.
CAWG extends core C2PA with optional assertions allowing creators to attach verified identity to signed content. The CAWG Identity Assertion Specification 1.2 was DIF Ratified on 15 December 2025.
CAWG identity assertions work through an identity claims aggregator — an actor that chains attestations from multiple identity providers into a single assertion record. Adobe’s Connected Identities platform is the primary production implementation you’ll encounter.
The privacy concern comes down to pipeline defaults. Once your organisation implements CAWG identity assertions, every piece of content signed through that pipeline exports the creator’s identity: social profiles, document verification records, and organisational affiliations embedded directly in the file. The opt-in model means creators theoretically choose what to embed — but in practice, tool configuration determines what data is included, and most creators never inspect what their tools are actually embedding.
This is why the architecture of your signing pipeline needs to be locked in before you start embedding credentials, not after.
Yes. If a creator signs multiple files with the same CAWG identity assertion, that verified identity appears in every single file. Anyone collecting those files can correlate the identity across platforms without the creator’s ongoing knowledge or consent. That’s a structural feature of the provenance infrastructure. It’s not an edge case.
The aggregation problem makes it worse. A single Content Credential reveals limited information. At scale — verified identity, location metadata, device identifiers, and publication timestamps across many files — you’ve got a serious surveillance surface. And C2PA metadata is designed to be machine-readable and automatically ingested.
Once identity data is embedded and the file is out there, it cannot be unlinked. There is no mechanism to retroactively remove identity from copies already in circulation. C2PA’s own Harms Modelling acknowledges this: loss of control over personal information and enforced suppression of speech are recognised as possible outcomes of the system.
Social media platforms currently strip metadata — including C2PA — partly to protect user privacy. C2PA adoption pressure creates real tension between that default and the provenance verification goal. Regulatory frameworks that intersect with C2PA privacy obligations add another layer that implementers frequently underestimate.
The World Privacy Forum (WPF) published “Privacy, Identity and Trust in C2PA,” which is the most detailed independent privacy analysis of the C2PA ecosystem available. It’s worth reading if you’re planning any kind of implementation.
The WPF’s core finding reframes what C2PA actually does: “C2PA is widely misunderstood: it doesn’t detect deepfakes or flag potential copyright infringement. Instead, it’s quietly laying down a new technical layer of media infrastructure — one that generates vast amounts of shareable data about creators and can link to commercial, government, or even biometric identity systems.”
The WPF found the conformance programme validates structural compliance but does not assess whether implementations actually minimise identity data exposure or provide creator consent mechanisms that meet privacy obligations. They used C2PA’s own Harms Modelling as evidence of acknowledged but unmitigated risk: the specification team recognised that content credentials could enable political targeting and chilling effects on journalism — but the conformance programme doesn’t operationalise those acknowledged risks.
The burden sits with implementers: “The burden to figure it out isn’t on consumers — it’s on businesses and organisations to think carefully about how they implement C2PA, with appropriate risk assessments.”
Conformance programme gaps that affect privacy posture are covered in the trust layer analysis if you want the full picture.
CAWG defines the training-and-data-mining assertion (cawg.training-mining) with four sub-assertions: cawg.ai_generative_training, cawg.ai_training, cawg.ai_inference, and cawg.data_mining — each letting a creator signal allowed, notAllowed, or constrained for that specific use.
The fundamental limitation is this: AI training consent in C2PA is a signal, not a legally enforceable right. There is no technical enforcement layer preventing an AI training pipeline from ignoring the assertion entirely. The false assurance risk is real — creators who embed notAllowed may genuinely believe they’ve protected their rights when all they’ve done is state a preference.
C2PA Section 6.8 defines a redaction mechanism. Assertions can be removed from a manifest when an asset is used as an ingredient.
Redaction is not deletion. The specification requires that a record of what was removed be added to the claim — the assertion label stays visible, so the type of information removed is still readable. The claim generator’s X.509 signing certificate cannot be redacted. That baseline identity exposure is irreducible.
Redaction requires a deliberate pipeline stage, not a manual afterthought. And the timing limitation is absolute: redaction only works on copies you control. Files already distributed retain the original embedded data.
Default to no CAWG identity assertions. Most provenance verification use cases work just fine with the claim generator’s signing identity alone. Add CAWG assertions only when your use case explicitly requires named creator identity.
Audit what your claim generator embeds before you deploy. Use c2patool to inspect the assertions your signing pipeline produces. Many tools embed more metadata than their documentation suggests — device serial numbers, GPS coordinates, and software identifiers can appear without any CAWG configuration at all.
Implement selective redaction as an explicit pipeline stage. If identity assertions are required for internal verification, design the pipeline to redact them before external distribution.
Evaluate identity claims aggregation multiplicatively. Each identity provider — social media, government document verification, biometric verification — adds data that cannot be unlinked once embedded. The combination is more identifiable than any single provider.
Understand the first-mile trust limitation. HackerFactor demonstrated complete identity forgery using c2patool, creating a file where “every C2PA validation tool says the signatures look correct.” Signing-stage data minimisation matters more than downstream validation.
Map your regulatory obligations before deployment. Embedded identity data may trigger GDPR data minimisation or CCPA disclosure obligations that C2PA’s voluntary framework does not address. A privacy impact assessment should precede deployment, not follow it.
Implementation design choices that reduce the privacy surface at the architecture level provide additional context. And if you’re still getting up to speed on the technical side, the foundational C2PA and content provenance infrastructure overview covers the context for the risk surface described here.
A Content Credential can contain the signing certificate identity (X.509), device metadata (camera serial numbers, GPS coordinates), software identifiers, and optionally CAWG identity assertions linking verified social media profiles, government document verification, organisational affiliations, and website ownership to the creator. What ends up in there depends on the claim generator’s configuration and whether CAWG extensions are implemented.
Yes, in principle. A creator who signs multiple files with the same CAWG identity assertion embeds that verified identity in every file. Any party who collects those files can build a picture of the creator’s publishing activity across platforms — without their knowledge or consent.
The claim generator identity is the X.509 certificate that signs the manifest — it identifies the tool or device, not the human creator. A CAWG identity assertion is an optional, explicit declaration of the human creator’s identity, potentially including verified social media accounts, government ID, and organisational affiliations.
No. C2PA metadata is publicly verifiable. Anyone who receives the media file can read all embedded assertions using c2patool. There is no encryption or access control layer on C2PA assertions.
You can use C2PA’s redaction mechanism (Section 6.8) to remove identity assertions before you distribute the file. What stays visible is that something was removed. And once a file is out of your hands, redaction on your end has no effect on copies already in circulation.
No. The WPF identifies specific privacy gaps in the conformance programme and recommends strengthening protections — mandatory data minimisation and explicit consent frameworks for identity embedding. It’s not a verdict against C2PA, it’s a caution about implementation.
No. CAWG training-and-data-mining assertions are signals, not enforcement mechanisms. Compliance depends entirely on whether downstream platforms and AI companies choose to honour the signal.
A mechanism that chains attestations from multiple identity providers into a single identity assertion — social media verification, government ID check, and biometric verification, all embedded as one record. Adobe’s Connected Identities platform is the primary production implementation.
Yes. The C2PA trust model validates that a claim was signed, not that the identity is truthful. HackerFactor documented a complete forgery where every C2PA validation tool confirmed the signatures as correct. What you embed at the signing stage matters more than downstream validation.
Yes, but at a cost. Stripping removes the embedded identity and provenance data from that copy. The trade-off is this: stripping also removes the provenance record. You can protect privacy or preserve the authenticity chain — often not both.
C2PA’s harms model is the specification team’s internal framework acknowledging that content credentials could enable civil liberties threats — surveillance, political targeting, location tracking, chilling effects on journalism. The World Privacy Forum argues the conformance programme does not adequately operationalise these acknowledged risks.
No. The claim generator’s signing identity alone is enough for most provenance verification use cases. CAWG identity assertions attach verified personal information to every file through the pipeline — only add them when your use case genuinely requires named creator identity, and only after you’ve mapped the privacy implications.