Model behaviour
LLM data leaks: prompts, outputs and training data
An LLM can reveal sensitive data in what it says, not only in what is stolen from it. OWASP calls this sensitive information disclosure and gives concrete controls.
Last reviewed against legislation.gov.uk, National Cyber Security Centre and OWASP GenAI Security Project. Sources
The risk
What OWASP says
OWASP's LLM02:2025 covers the risk that an LLM, especially one built into an application, exposes sensitive data through its output [4]. The data can include personal information, financial and health records, confidential business data, credentials and details of the model itself [4]. Its examples are personal data leakage, exposure of proprietary algorithms including through training-data inversion, and disclosure of sensitive business data [4].
Three routes
How it happens
Training data
Data that went into training can come back out. The NCSC notes that large language models are trained on vast amounts of data, often scraped from the internet, so filtering is impractical [3].
Prompts and context
Anything placed in a prompt, or retrieved into it from your documents, can appear in the answer. If the model can see a file, assume a user may be able to get it to say what is in it.
Manipulated inputs
The NCSC describes prompt injection as one of the most widely reported weaknesses in language models: crafted inputs that make a model reveal confidential information or trigger harmful actions [3].
What OWASP advises
Prevention
OWASP's strategies include sanitising sensitive content before it enters training data, strict input validation, least privilege for what the model can reach, privacy techniques and user education [4]. It also says that restrictions in a system prompt may be bypassed by prompt injection, so they should not be the only safeguard [4].
Do not rely on the system prompt
Telling the model not to reveal certain data is not a control. Remove the data from the model's reach instead [4].
Practical steps
For an LLM you build or buy
-
1
Decide which data classes must never reach the model. -
2
Give the application only the data the user is entitled to see. -
3
Log inputs and outputs, with the same access controls as other sensitive logs. -
4
Ask the supplier whether your data is used for training and whether you can opt out, which OWASP lists as good practice [4]. -
5
Test with prompts designed to extract data before release.
See also supplier security questions.
Keep reading
Read next
Reporting
When an AI data breach must be reported under UK GDPR, how the 72 hour clock works, and what to record whether or not you report.
Read the guide
Examples
Three public AI data exposure cases, a library bug, staff uploads and an open database, with what each shows UK organisations about cause and response.
Read the guide
Prevention
Practical controls to stop data leaking through AI tools and pipelines, drawn from NCSC guidance, OWASP and IBM research, in order of priority.
Read the guideHow CyPro can help
Need a hand working through this?
Finding out whether an incident reaches you is one job. Checking your own systems, tightening access and briefing the board are several more. CyPro's responders and consultants can take any of them on.
Incident response
If you may already be compromised, our responders contain it, preserve the evidence, find out what was touched and get you running again safely.
More on cypro.co.uk
24/7 security monitoring
Our SOC analysts watch your systems day and night for the signs of an attack in progress: unexpected processes, new outbound connections, logins that do not fit.
More on cypro.co.uk
Fractional CISO for the board
A senior security lead who advises the board on what to fix first, what to accept and what to tell customers, and owns the decision record.
More on cypro.co.uk
Penetration testing
A focused test of your systems and the controls around them, so you know they hold before an attacker finds out for you.
More on cypro.co.uk
Hardening and segmentation
Our consultants put the controls in place that limit the damage when something is missed: firewall rules, network isolation and tighter admin access.
More on cypro.co.uk
Vulnerability scanning
Find every internet-facing system you run, including the forgotten ones, and see which are still on a vulnerable build.
See the scanning service
Talk to a specialist
Worried this affects you?
If data has leaked through an AI tool or an exposed AI database, the first hours decide how much is lost and what you must report. An incident response team helps you contain it, work out what was exposed and meet your notification duties.