Large language models have a mathematical property that creates privacy risks. It’s called injectivity, and it means the hidden states inside transformer models can be reversed to reconstruct the original user prompts that created them.
You cannot patch this. It’s baked into how these models process text. Understanding these vulnerabilities is essential for getting the complete picture of AI safety breakthroughs that affect enterprise deployments.
Recent research has demonstrated practical attacks—using algorithms like SipIt—that extract sensitive information from model internals with 100% accuracy. These vulnerabilities exist separately from traditional prompt injection attacks. They’re architectural.
If you’re deploying AI systems that handle proprietary data or user information, you need to understand these risks. This article explains the technical mechanisms, walks through real-world implications, and gives you practical mitigation strategies.
What Is LLM Injectivity and Why Does It Create Privacy Risks?
LLM injectivity is the mathematical property where different prompts almost always produce different hidden state representations. The mapping from your text input to those internal representations is essentially one-to-one—injective, in mathematical terms.
Why does this matter? Because the hidden states encode your prompt directly.
Here’s the technical bit. Real-analyticity in transformer networks means the model components—embeddings, positional encodings, LayerNorm, attention mechanisms, MLPs—operate in ways that make collisions between prompts confined to measure-zero parameter sets. In practical terms: the chance of two different prompts producing identical hidden states is effectively zero.
What makes this different from a typical security vulnerability? You cannot patch it. Injectivity is a structural consequence of transformer architecture itself.
The privacy implications flow directly from this. Any system that stores or transmits hidden states is effectively handling user text. Even after you delete a prompt, the embeddings retain the content. This connects directly to how AI introspection relates to privacy—the same internal representations that enable introspection also enable reconstruction attacks.
This affects compliance directly. The Hamburg Data Protection Commissioner once argued that model weights don’t qualify as personal data since training examples can’t be trivially reconstructed. But inference-time inputs? Those remain fully recoverable.
Many organisations in IT, healthcare, and finance already restrict cloud LLM usage due to these concerns. Given what we know about injectivity, those restrictions make sense.
How Do Prompt Reconstruction Attacks Work Against Language Models?
The SipIt algorithm—Sequential Inverse Prompt via Iterative updates—shows exactly how these attacks work. It exploits the causal structure of transformers where the hidden state at position t depends only on the prefix and current token.
The attack reconstructs your exact input prompt token-by-token. If the attacker knows the prefix, then the hidden state at position t uniquely identifies the token at that position. SipIt walks through each position, testing tokens until it finds the match.
In testing on GPT-2 Small, SipIt achieved 100% accuracy with a mean reconstruction time of 28.01 seconds. Compare that to brute force approaches at 3889.61 seconds, or HardPrompts which achieved 0% accuracy.
What do attackers need? Access to model internals or intermediate outputs. The resources required are getting cheaper as techniques mature.
Unlike prior work that produced approximate reconstructions from outputs or logprobs, SipIt is training-free and efficient, with provable guarantees for exact recovery from internal states.
When probes or inversion methods fail, it’s not because the information is missing. Injectivity guarantees that last-token states faithfully encode the full input. The information is there. It’s just a matter of extracting it.
What Is the Difference Between Prompt Injection and Injectivity-Based Attacks?
These are different attack types that require different defences. Conflating them creates security gaps.
Prompt injection manipulates LLM behaviour through crafted inputs that override safety instructions. Injectivity-based attacks extract data from model internals. Different mechanisms, different outcomes.
Injection attacks exploit the model’s inability to distinguish between instructions and data. You’ve seen the examples—”ignore previous instructions and do X instead.” Indirect prompt injection takes this further by having attackers inject instructions into content the victim user interacts with.
Reconstruction attacks exploit mathematical properties of hidden states. No clever prompting required—just access to internal representations.
This distinction matters practically because your defences against one don’t protect against the other.
Hardened system prompts? They reduce prompt injection likelihood but have no effect on reconstruction attacks. Spotlighting techniques that isolate untrusted inputs? Great for injection, irrelevant for reconstruction.
Microsoft’s defence-in-depth approach for prompt injection spans prevention, detection, and impact mitigation. But it requires entirely different approaches for reconstruction risks—design-level protections, access restrictions, and logging policies.
Prompt injection sits at the top of the OWASP Top 10 for LLM Applications. Sensitive information disclosure—which includes reconstruction risks—is listed separately. They’re distinct vulnerability categories.
Research shows an 89% success rate on GPT-4o and 78% on Claude 3.5 Sonnet with sufficient injection attempts. But your injection defences won’t stop someone with access to your hidden states from reconstructing what went into them.
How Can Hidden States Expose Sensitive Information in Production Systems?
Your production architecture has more exposure points than you might think.
Hidden states encode contextual information from all processed text, including confidential data. The obvious places to look: API responses, logging systems, and debugging tools that might inadvertently expose hidden state data.
Third-party integrations create exposure surfaces. RAG systems particularly. Memory in RAG LLMs can become an attack surface where attackers trick the model into leaking secrets without hacking accounts or breaching providers directly.
Multi-domain enumeration attacks can exfiltrate secrets from LLM memory by encoding each character into separate domain requests rendered as image tags. An attacker crafts prompts that cause the LLM to make requests to attacker-controlled domains, with secret data encoded in those requests.
Model serving infrastructure with insufficient access controls risks information leakage. Even legitimate system administrators may access reconstructible hidden state data. If someone can see the hidden states, they can potentially reconstruct what went in.
Practical Audit Checklist
Here’s what to review:
- Map your data flows and identify where hidden states are stored or transmitted
- Review access controls on model infrastructure
- Check what gets logged—particularly internal model representations
- Assess third-party access to any of these systems
- Evaluate whether user processes send data beyond generated output tokens
What Are Effective Defences Against Prompt Reconstruction Vulnerabilities?
You have several options, each with trade-offs.
Architecture-level controls should be your starting point. Minimise hidden state exposure through design. Implement strict logging policies that exclude internal model representations.
Privilege separation isolates sensitive data from LLM processing. Secure Partitioned Decoding (SPD) partitions the KV cache into private and public parts. User prompt cache stays private; generated token cache goes public to the LLM. The private attention score typically can’t be reversed to the prompt due to the irreversible nature of attention computation.
User processes should only send generated output tokens—sending additional data could leak LLM weights or hidden state information.
Differential privacy protects prompt confidentiality by injecting noise into token distributions. But these methods are task-specific and compromise output quality. It’s one layer in a layered defence strategy, not a complete solution.
Prompt Obfuscation (PO) generates fake n-grams that appear as authentic as sensitive segments. From an attacker’s perspective, the prompts become statistically indistinguishable, reducing their advantage to near random guessing.
Cryptographic approaches like Multi-Party Computation use secret sharing but suffer from collusion risks and inefficiency. Homomorphic encryption enables computation on encrypted data but the overhead impedes real-world use.
For practical implementation, OSPD (Oblivious Secure Partitioned Decoding) achieves 5x better latency than existing Confidential Virtual Machine approaches and scales well to concurrent users.
Apply the principle of least privilege to LLM applications. Grant minimal necessary permissions and use read-only database accounts where possible.
How Do Regulatory Requirements Address LLM Privacy Vulnerabilities?
The regulatory landscape is catching up to these technical realities.
GDPR applies to personal data processed through LLMs, including reconstructible information. That means explicit consent, breach notification within 72 hours, and broad individual rights—access, deletion, objection to processing. Enforcement includes fines up to 20 million euros or 4% of global turnover.
The OWASP Top 10 for LLM Applications provides the industry’s framework for understanding AI security risks. Developed by over 600 experts, it classifies sensitive information disclosure as a distinct vulnerability. LLMs can inadvertently leak PII, intellectual property, or confidential business details.
ISO 42001 provides AI management system requirements relevant to privacy by design, though specific implementation guidance for reconstruction risks remains limited.
Here’s the compliance challenge: traditional anonymisation may be insufficient for LLM systems. If hidden states can be reversed to reconstruct inputs, anonymisation of those inputs doesn’t protect you once they’re processed.
You need to demonstrate technical measures that specifically address reconstruction risks. Steps include classifying AI systems, assessing risks, securing systems, monitoring input data, and demonstrating compliance through audits. For detailed implementation guidance, see our resource on governance frameworks to address these risks.
Data minimisation helps on multiple fronts. Limit data collection and retention to what’s essential. This reduces risks and eases cross-border compliance.
What Security Testing Tools Can Identify Reconstruction Vulnerabilities?
The tooling landscape is still maturing. Most existing tools focus on prompt injection, but you can adapt some for reconstruction testing.
NVIDIA NeMo Guardrails provides conversational AI guardrails. Garak functions as an LLM vulnerability scanner. These focus primarily on injection but can be part of a broader security testing strategy.
Microsoft Prompt Shields integrated with Defender for Cloud provides enterprise-wide visibility for prompt injection detection. TaskTracker analyses internal states during inference to detect indirect prompt injection.
For reconstruction vulnerabilities specifically, you’ll need custom red team assessments. The tools aren’t there yet for comprehensive automated testing.
Red teaming reveals that these attacks aren’t theoretical. Cleverly engineered prompts can extract secrets stored months earlier.
Microsoft ran the first public Adaptive Prompt Injection Challenge with over 800 participants and open-sourced a dataset of over 370,000 prompts. This kind of research is building the foundation for better defences.
For your testing approach: cover both direct model access and inference API endpoints. Automated scanning should be supplemented with manual expert analysis. Configure comprehensive logging for all LLM interactions and set up monitoring for suspicious patterns.
Implement emergency controls and kill switches for rapid response to detected attacks. Conduct regular security testing with known attack patterns and monitor for new techniques. For a comprehensive approach to testing, see our guide on practical steps to prevent prompt injection.
Budget considerations for SMBs: start with available open-source tools, establish baseline security testing internally, and engage external specialists for comprehensive assessments when dealing with high-sensitivity applications.
FAQ Section
Can standard prompt injection defences protect against reconstruction attacks?
No. Prompt injection defences like input filtering and output guardrails address different attack vectors. Reconstruction attacks exploit mathematical properties of hidden states, requiring architecture-level protections—access controls, logging restrictions, and privilege separation.
Do cloud-hosted LLM APIs have reconstruction vulnerabilities?
Cloud APIs limit reconstruction risk by restricting access to hidden states. However, fine-tuned models, custom deployments, and certain API configurations may expose internal representations. Review your provider’s documentation for hidden state access policies.
How expensive is it to mount a prompt reconstruction attack?
Costs vary based on model size, access level, and computational resources. SipIt achieves exact reconstruction in under 30 seconds on smaller models. Costs decrease as techniques mature. Assume determined attackers can access necessary resources.
Should I avoid using LLMs with sensitive data due to injectivity risks?
Not necessarily. Understanding risks enables appropriate mitigations. Evaluate data sensitivity, access controls, and deployment architecture. OSPD enables practical privacy-preserving inference for sensitive applications including clinical records and financial documents.
Can differential privacy completely prevent reconstruction attacks?
Differential privacy increases reconstruction difficulty but involves performance trade-offs. It’s one layer in a defence-in-depth strategy, not a complete solution. Evaluate noise levels against accuracy requirements for your application.
Are open-source models more vulnerable to reconstruction than proprietary ones?
Open-source models provide more attack surface due to architecture transparency, but this also enables better security analysis. Proprietary models may have undisclosed vulnerabilities. Security depends more on deployment architecture than model licensing.
How do reconstruction attacks affect RAG systems specifically?
RAG systems may expose hidden states through retrieval mechanisms and vector databases. Indirect prompt injection can combine with reconstruction attacks to extract both system prompts and retrieved content. Secure RAG architecture requires protecting multiple data flows.
What should I prioritise if I can only address one vulnerability type?
Focus on prompt injection first—it has more established attack tools and documented incidents. However, plan for reconstruction defence as attack techniques mature. Implement architecture-level controls that address both simultaneously where possible.
Do model updates from providers address reconstruction vulnerabilities?
Model updates may improve general security but rarely address fundamental injectivity properties. These are architectural characteristics, not bugs. Evaluate each update’s security implications and maintain your own defence layers.
How do I explain reconstruction risks to non-technical stakeholders?
Frame it this way: LLMs work like secure filing cabinets with transparent walls. Anyone who can see inside the cabinet can potentially reconstruct what documents were filed. Protection requires controlling who can view internals, not just what goes in.
