Threat Models for Memorization: Privacy, Copyright, and Everything In-Between
From Google TechTalks · 56:15
Show by Google TechTalks
A Google TechTalk, 2025-03-05, presented by Michael AerniPrivacy in ML Seminar. ABSTRACT: Machine learning models often memorize parts of their training data, leading to potential privacy and copyright violations. While extensive research explores attacks and defenses against such risks, the importance of selecting an appropriate threat model is frequently overlooked.Uploaded 26-01-26This talk emphasizes why choosing a suitable threat model is critical through two recent research examples. The first example focuses on heuristic privacy defenses, which forgo strong guarantees (such as differential privacy) to achieve high utility and reasonable privacy on natural data. Without worst-case guarantees, the assumed threat model becomes crucial. Yet, in recent work [1], we reveal that many evaluations of heuristic privacy defenses report strong privacy in fully average-case settings—but the defense completely violates the privacy of some samples even in realistic settings.In the second part of this talk, the focus shifts to memorization in large language models (LLMs) and its implications for privacy and copyright. Current research mostly focuses on either worst-case data extraction or coarse notions such as linguistic novelty and creativity. Our recent study [2] explores a middle ground: how much training data do LLMs reproduce for natural tasks and benign prompts? We find that current conversational models may output text that on average consists of up to 15% short snippets found verbatim on the internet—and close to 100% in the worst case.These examples underscore that adopting overly optimistic or inappropriate threat models can lead to a false sense of security. Hence, properly auditing and mitigating these threats requires moving beyond purely benign assumptions.[1] Michael Aerni, Jie Zhang, and Florian Tramèr. Evaluations of Machine Learning Privacy Defenses are Misleading. CCS 2024.[2] Michael Aerni, Javier Rando, Edoardo Debenedetti, Nicholas Carlini, Daphne Ippolito, Florian Tramèr. Measuring Non-Adversarial Reproduction of Training Data in Large Language Models. ICLR 2025.
More episodes of Google TechTalks
More episodes
- Technology Assisted Reading Acquisition: Children Acquiring Literacy Naturally
- sex on the internet, the realities of porn, sexual privacy,
- Privacy Auditing of Large Language Models
- The Limits and Possibilities of One Run Auditing
- PhotoTechEDU Day 29: Photographing VR Panoramas
- Privacy Ripple Effects from Adding or Removing Personal Information in Language Model Training
- Inside VMware Fusion
- Java Posse Episode 100 (Live Special)
- Cascading Adversarial Bias from Injection to Distillation in Language Models
- Practical Common Lisp
PodSearch is a free, independent podcast search engine. Read about PodSearch, the privacy policy or contact us.