Lately, there are tons of explainers on how to use LLMs, or position pieces on why they are great or terrible. This article is none of those. This is for data scientists and machine learning experts who already know LLMs, but need to discuss privacy concerns more broadly.
I’m your privacy engineering friend, and I’ll give you some language for discussing privacy issues around LLM with anyone. You’ll be able to explain the issues in non-technical terms with lawyers, CEOs, and any part of your company. It can even help with conversations outside of work, if friends and relatives start asking about what could go wrong with models. This article will help you translate technical ML terms into clear privacy implications.
A Simple Dichotomy of Privacy Concerns from LLM
For simplicity, let’s think of privacy concerns from LLM in two main categories: models share correct information about people, and models share incorrect information about people.
Models share correct information about people:
As a machine learning person, you may already be familiar with memorization and membership inference attacks. Both of these attacks allow models to share information about someone that they may not want to be shared, or that wasn’t intended by the model to be learned or shared.
- Membership inference attack: If you can figure out that someone is part of a training set, you might be able to infer sensitive information. One example is if the dataset is based on HIV-positive patients. Alternatively, membership in the training set could be a proxy for other sensitive information. For example, race can be inferred from zip codes in the USA. Malicious attackers can do nasty things with sensitive information like health status or race. Additionally, revealing what categories people fall into can still lead to bias.
- Memorization: Training data extraction and data extraction are both types of attacks that can reveal information about people. This sensitive information could include sensitive employment information, identifying information like full names and addresses, and contact information like phone numbers.
Even when these attacks don’t lead to concrete financial or physical harm to someone, there are a number of privacy concerns. Attacks such as doxing can cause individuals real emotional harm, even when stalkers don’t show up at the door and just threaten to. In general, sharing correct information about people through these attacks takes away any control the person has about how their data is used. People don’t like that; it feels creepy and leads to a lack of trust.
The legal concerns are real, too. In these attacks, data is shared beyond the original purpose for which it was collected. Purpose limitation is a tenant of the EU data regulation GDPR.
Models share incorrect information about people
Privacy harms go beyond leaking true identifying information. LLMs can create hallucinations, in which they generate false information that wasn’t in the original data. It might be tempting to dismiss hallucinations as unrelated to privacy or harmless. However, if your model is intended to generate authoritative results, false information can cause real harm to people. If you’ve ever had someone spread false rumors about you, you understand how painful this can be.
In privacy lingo, the harm to people caused by hallucinations would be called “misrepresentation”, or “reputational harm.” The act of disseminating false information is also called “distortion.” Basically, it’s misinformation about people, and they can’t fix or correct the data. Even without LLMs, bad data can lead to legal fines. Consider simple bad data entry, in which an insurance company was fed incorrect information about someone and then refused to insure them, leading to a lawsuit.
Even when the information is not malicious and it’s just simply wrong, it can do harm. For example, if an LLM claimed that I won the Nobel Prize in math, you might find it suspicious that I’m blogging about privacy and not making better use of my time-solving theorems (regardless that there is no Nobel Prize for math). Therefore, it could cause harm to my reputation as a privacy engineer.
Due to these potential harms of incorrect data, privacy regulations like GDPR explicitly allow people to correct data about themselves. Even if the hallucination does not lead to financial or physical harm, it takes away people’s ability to control what is believed about them. I’ve said it before, and I’ll say it again: people want control over their data.
There is debate about whether “hallucination” is even a good word to use for LLMs. Some have proposed “misinformation”, or “bullshitting” as more accurate terms. From a privacy perspective, LLM hallucinations about people are simply incorrect data about people that they can’t control.
Summary
In summary, when LLMs leak correct or incorrect information about people, it can lead to privacy harm. The harms can be measurable, or they can be simply painful for the individual. In the latter case, it can make someone feel creeped out or have less trust in the model. From a GDPR perspective, data use should be limited to a specific purpose, and people should have the right to correct their data. Therefore, LLM-builders should aim to prevent the leakage of personal information, whether or not it is correct.
My sources and where to learn more:
Membership attack example using HIV-positive patients https://dcai.csail.mit.edu/lectures/data-privacy-security/
Race can be inferred from zip codes: https://cdt.org/insights/how-does-the-internet-know-your-race/)
Sensitive employment information revealed in translate tool: https://slator.com/translate-com-exposes-highly-sensitive-information-massive-privacy-breach/
Survey of Hallucinations in Natural Language Generation: Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y., Dai, W., Madotto, A., & Fung, P. (2022). Survey of Hallucination in Natural Language Generation. ArXiv. https://doi.org/10.1145/3571730
GDPR and the right to correct data:https://commission.europa.eu/law/law-topic/data-protection/reform/rights-citizens/my-rights/my-data-incorrect-can-i-correct-it_en#example
Concerns about the term Hallunciation in LLMs: https://betterprogramming.pub/large-language-models-dont-hallucinate-b9bdfa202edf
Privacy attacks on LLMs with research links: https://www.private-ai.com/2022/06/17/the-privacy-risk-of-language-models/
Reputational Harms and LLMs: https://www.theprivacywhisperer.com/p/ai-hallucinations-and-privacy-a-reputational
Distortion as a Privacy Harm: Solove’s “Taxonomy of Privacy Harms” is a foundational law article that describes distortion as well as other general privacy harms https://scholarship.law.upenn.edu/penn_law_review/vol154/iss3/1/