Preventing Generation of Verbatim Memorization in Language Models Gives a False Sense of Privacy
Abstract
Studying data memorization in neural language models helps us understand the risks (e.g., to privacy or copyright) associated with models regurgitating training data, and aids in the evaluation of potential countermeasures. Many prior works -- and some recently deployed defenses -- focus on "verbatim memorization", defined as a model generation that exactly matches a substring from the training set. We argue that verbatim memorization definitions are too restrictive and fail to capture more subtle forms of memorization. Specifically, we design and implement an efficient defense based on Bloom filters that perfectly prevents all verbatim memorization. And yet, we demonstrate that this "perfect" filter does not prevent the leakage of training data. Indeed, it is easily circumvented by plausible and minimally modified "style-transfer" prompts -- and in some cases even the non-modified original prompts -- to extract memorized information. For example, instructing the model to output ALL-CAPITAL texts bypasses memorization checks based on verbatim matching. We conclude by discussing potential alternative definitions and why defining memorization is a difficult yet crucial open question for neural language models. Show more
Permanent link
https://doi.org/10.3929/ethz-b-000642948Publication status
publishedExternal links
Book title
Proceedings of the 16th International Natural Language Generation ConferencePages / Article No.
Publisher
Association for Computational LinguisticsEvent
Subject
Machine Learning; PrivacyOrganisational unit
09764 - Tramèr, Florian / Tramèr, Florian
More
Show all metadata
ETH Bibliography
yes
Altmetrics