Sensitive Data Exposure
Sensitive data exposure in AI occurs when a model reveals information it should not: training data it memorized, system prompt instructions it was told to keep confidential, or data from documents it processed on behalf of a user. The model does not apply confidentiality judgment. It generates the most statistically plausible response to the input it receives. If that response includes sensitive content, the model produces it.
This is categorically different from a conventional data breach. No database was compromised. No access control was bypassed. The model was asked a question, and it answered with information that should never have been available through the answer.
How Sensitive Data Exposure Happens in AI Systems
Training data memorization is the most structurally significant source. Models trained on large corpora memorize fragments of their training data. Researchers have demonstrated that targeted queries can extract verbatim text from training sets, including personal records, internal documents, and proprietary content, all of which were included without adequate filtering. The exposure is not a deployment misconfiguration. It is a consequence of how the model learned.
System prompt extraction occurs when a model is caused to reveal the instructions it was given by the developer. System prompts frequently contain business logic, confidentiality instructions, tool configurations, and internal context that the operator intended to keep private. Adversarial prompting can cause a model to reproduce these instructions directly or paraphrase them in ways that expose their content.
Context window leakage occurs in multi-turn and agentic systems, where sensitive content accumulates over a session. Retrieved documents, prior conversation turns, tool outputs, and API responses all share the context window. A later prompt injection or a direct elicitation attempt can surface content from earlier in the session that the user was never intended to access.
Retrieval-augmented generation (RAG) exposure applies to systems that retrieve documents at inference time to ground model responses. If the retrieval layer returns sensitive documents and the model is manipulated through prompt injection to reproduce their contents, the RAG system becomes the delivery mechanism for data it was designed to protect.
