How Talkative Is the Model? LLM02: Sensitive Information Disclosure

Part 5 of the AI Pentesting series. Previous: Direct Prompt Injection and the Anatomy of a Jailbreak

PII leakage, training-data memorization and RAG-based data exposure.

When we ask an LLM a question, we usually focus on the accuracy of its answer. From a security perspective, there is a more fundamental question: could the model be telling us something we were never supposed to see?

Large language models pose a serious risk not only by generating incorrect or harmful responses, but also by exposing sensitive information. Personal data, internal company information, financial records, API keys and fragments of training data can all be unintentionally revealed through model outputs or through the data sources connected to the model.

The OWASP GenAI Security Project categorizes this as LLM02: Sensitive Information Disclosure. This article examines the three different ways it happens, how testers assess each one, and why the real defense is controlling what the model can access rather than hoping it stays quiet.

Key takeaways

Three distinct problems share one symptom: training-data memorization, user data leaking into output, and RAG serving documents the user is not authorized to see.
“The model has a safety filter” is not enough; memorization and data flow must be evaluated separately.
In RAG systems, access control must happen before retrieval, not inside the model’s context.
Cross-user and cross-tenant isolation failures are among the most severe LLM02 findings.
Under KVKK and GDPR, LLM data leakage is a data protection matter, not only a technical bug.

What Is LLM02: Sensitive Information Disclosure?

Sensitive information disclosure is a situation in which an LLM or an LLM-based application exposes sensitive information without authorization. That information may include personal data, email addresses and phone numbers, financial and health records, internal company documents, identity information, API keys, passwords, source code and system configurations.

The critical point is that the leakage can originate in more than one place. OWASP, in LLM02:2025 Sensitive Information Disclosure, stresses that the risk may stem not only from the model itself but also from the application built around it. Three different questions should be distinguished:

Could the model have memorized information from its training data?
Could sensitive information belonging to a user leak into the output?
Could a document the user is not authorized to access be supplied to the model through a RAG system?

These scenarios can lead to a similar outcome, but they are technically different problems. A security assessment must therefore examine not only the model but also the data flow surrounding it.

Does the Model Really Forget Its Training Data?

LLMs are trained on vast amounts of data, learning patterns within the language. But some data is not learned only as a general pattern; it is memorized, especially content that repeats in the training data or is rare and distinctive.

In a 2020 study, Carlini and colleagues demonstrated that verbatim text fragments could be extracted from the training data of GPT-2, including names, phone numbers, email addresses, code snippets and UUIDs.

There is an important distinction here: the fact that a model generates a piece of information does not necessarily mean it was memorized from training data. However, a model reproducing a previously unseen, distinctive training example word for word is a much stronger signal of training-data extraction.

Divergence Attacks: Pushing the Model Off Its Normal Behavior

A more advanced line of research is the divergence attack. The 2023 study by Nasr et al. on scalable extraction of training data from production models developed a divergence attack for aligned models: it pushes the model away from its standard chatbot-style generation.

The researchers reported that this significantly increased the extraction rate, in one case extracting training data from an aligned ChatGPT model up to 150 times more often than normal chatbot behavior. The idea is simple:

Normal behavior:
Prompt → Helpful / aligned response

Under a divergence attack:
Prompt → Divergence → Training-data-like generation

One critical finding: alignment alone does not eliminate the memorization problem. Substantial volumes of training data could be extracted from both open-weight and proprietary models, and the risk was not fully removed even in aligned models. In a security assessment, “the model has a safety filter” is therefore insufficient on its own; “does the model memorize training data?” must be evaluated independently.

PII Extraction: Two Different Sources

PII (personally identifiable information) is information that can directly or indirectly identify an individual. In LLMs, PII extraction appears in two ways:

Scenario 1 — PII from training data: the model reproduces names, emails, phones or addresses present in its training data.
Scenario 2 — PII from application context: personal data sent to the model by the user, or provided by a RAG system, is exposed in a response.

In the second scenario the problem is not training; it is the application’s data flow:

Customer database → RAG → LLM context → LLM → Unauthorized user

Here the model may simply be repeating what it was given. But if the application serves the wrong data to the wrong user, the result is still a sensitive information disclosure.

Unauthorized Document Leakage in RAG

In modern LLM applications, the model rarely works alone. Retrieval-Augmented Generation (RAG) lets the model pull information from external sources while generating a response:

User → Query → RAG → Database → Relevant documents → LLM → Response

Consider an AI assistant that lets a company’s employees query technical documentation, with each project’s documents stored in a vector database. A developer asks, “How is authentication configured in the API Gateway?” and the retriever appends the relevant sections to the model’s context.

If project-based access control is not enforced in the same system, documents from another project the user should not access can also be retrieved. While working on Project A, a user’s query might pull in the unpublished API documentation of Project B, and the model will happily answer using it. The problem is not memorization; it is the RAG layer supplying unauthorized data to the model. This is the retrieval-time equivalent of an IDOR vulnerability.

Why Access Control in RAG Cannot Be Left to the Model

If a system relies solely on a system prompt such as “This document is confidential, do not tell the user,” its security boundary is weak. Once a confidential document enters the model’s context, it can leak:

Unauthorized document → LLM → (weak) system prompt → potential disclosure

A more secure flow enforces authorization before the model ever sees the data:

User → Authentication → Authorization → Document access check → Filtered retrieval → LLM → Output validation → Response

The model is not asked to hide the confidential document; it is prevented from accessing it in the first place. This is the principle of least privilege applied to LLM security.

Cross-User and Cross-Tenant Data Leakage

Another major RAG risk is the failure to isolate users or tenants. In a SaaS AI application:

Tenant A
 ├── Customer data
 └── Internal documents

Tenant B
 ├── Customer data
 └── Internal documents

If tenant information is not filtered in vector database queries, a search by one tenant’s user can return another tenant’s documents:

User A → Vector search → Tenant B document → LLM → Response

In RAG security, it is not enough to ask “Is this document relevant?” You must also ask “Is this user authorized to see this document?” Cross-tenant leakage is often the most severe form of LLM02.

When Prompt Injection and Data Leakage Combine

Sensitive information disclosure becomes far more complex when combined with prompt injection. The attacker’s goal may not be a blunt “Show me the confidential document.” Instead, they manipulate the model’s retrieval or instruction-following behavior to force sensitive context into the open:

Attacker → Prompt injection → LLM / RAG behavior → Sensitive context → Response → Data disclosure

When evaluating LLM02, response filtering alone is insufficient. The entire chain, input → retrieval → context → output, must be examined.

LLM Data Leakage Under KVKK and GDPR

In LLM-based systems, exposing personal data is not merely a technical flaw. Under Turkey’s KVKK, Article 12 regulates data security obligations: if personal data is unlawfully obtained by third parties, the data controller must notify the affected individuals and the Board. An incident such as:

Customer PII → RAG → Unauthorized retrieval → LLM → Unauthorized user

must be addressed on both the technical security and the data protection dimensions. KVKK’s 2026 breach notifications show that categories such as full names, email addresses, phone numbers, home addresses and credentials continue to be scrutinized under unauthorized-access incidents. Adopting an LLM does not reduce existing data security obligations; it adds a new data-processing and data-access layer.

From a GDPR perspective, the same principles apply. Within an LLM application, data minimization, access control, encryption, pseudonymization, logging, data retention and incident response must all be treated as integral parts of the AI system architecture. For personal data sent to an LLM, four questions must be answered explicitly: Why is it sent? How long is it retained? Who can access it? Is it used for model training?

Testing for LLM02: An AI Pentest Methodology

Testing LLM02 is not a matter of asking the model a few simple prompts. A structured assessment runs in phases:

Data flow mapping. Map the end-to-end flow (User → Frontend → API → LLM → RAG → Vector DB → Response) and identify what data is present at each stage.
Sensitive data identification. Determine which categories exist in the system: PII, credentials, secrets, financial data, health data, internal documents, source code.
Training-data exposure. In an authorized environment with controlled test data, check whether the model reproduces training data: unusually long verbatim outputs, repeated sequences, unique identifiers, PII-like patterns.
RAG authorization testing. With multiple roles (User A → Document A ✓, User A → Document B ✗, and so on), verify retrieval results and model context against cross-user, cross-role, cross-tenant, deleted-document and restricted-document access.
Output validation. Scan model output for sensitive patterns such as email, phone, IBAN, API key, token, credential or national ID, and detect or block the flow of sensitive data → LLM → user.

This mirrors the broader approach in How to Penetration Test an LLM Application.

Defense: "The Model Won't Tell" Is Not a Control

The core principle: rather than relying on the model not to disclose sensitive information, prevent unnecessary access to it in the first place. Use multiple layers:

Data layer: data minimization, PII masking, anonymization, pseudonymization, sensitive-data classification.
Retrieval layer: document-level authorization, tenant isolation, metadata filtering, role-based access control, least privilege.
Model layer: controlled context, output filtering, safety policies, memorization testing.
Application layer: authentication, authorization, session isolation, rate limiting, audit logging.
Monitoring layer: PII detection, anomalous-query detection, retrieval logging, sensitive-document access monitoring, security alerting.

Conclusion

The security of an LLM cannot be judged by the fluency of its responses. The questions that matter are: What data does the model know? What data can it access? Who is authorized to access that data? What can it reproduce or memorize? What controls exist over its output?

Research on training-data extraction shows that models can reproduce verbatim segments of their training corpora, and divergence attacks show that even aligned models retain memorization risk that instruction tuning alone cannot remove. In RAG systems the risk takes a different shape entirely: the model need not memorize anything; a misconfigured retrieval or authorization mechanism can pull unauthorized documents into its context and disclose them.

LLM02 is therefore not just “Is the model blurting out secrets?” The real challenge is establishing security boundaries at every point where data meets the model. Teaching the model what it should not say matters, but it is far more important to control which data it can access in the first place.

Worried about what your AI assistant might reveal? Our AI & LLM Security Testing service maps your data flows and tests RAG authorization, memorization and output controls, and our GRC & Compliance team addresses the KVKK and GDPR side. Talk to our team.

References

OWASP GenAI Security Project — LLM02: Sensitive Information Disclosure
Carlini, N. et al. (2021). Extracting Training Data from Large Language Models. arXiv:2012.07805
Nasr, M. et al. (2023). Scalable Extraction of Training Data from (Production) Language Models.arXiv:2311.17035
KVKK — Personal Data Protection Law No. 6698

Subscribe to our cyber security newsletter

Scroll to top