Kavli Affiliate: Wei Gao
| First 5 Authors: Kai Huang, Wei Gao, , ,
| Summary:
Text-to-image diffusion models can be fine-tuned in custom domains to adapt
to specific user preferences, but such unconstrained adaptability has also been
utilized for illegal purposes, such as forging public figures’ portraits and
duplicating copyrighted artworks. Most existing work focuses on detecting the
illegally generated contents, but cannot prevent or mitigate illegal
adaptations of diffusion models. Other schemes of model unlearning and
reinitialization, similarly, cannot prevent users from relearning the knowledge
of illegal model adaptation with custom data. In this paper, we present
FreezeAsGuard, a new technique that addresses these limitations and enables
irreversible mitigation of illegal adaptations of diffusion models. The basic
approach is that the model publisher selectively freezes tensors in pre-trained
diffusion models that are critical to illegal model adaptations, to mitigate
the fine-tuned model’s representation power in illegal domains but minimize the
impact on legal model adaptations in other domains. Such tensor freezing can be
enforced via APIs provided by the model publisher for fine-tuning, can motivate
users’ adoption due to its computational savings. Experiment results with
datasets in multiple domains show that FreezeAsGuard provides stronger power in
mitigating illegal model adaptations of generating fake public figures’
portraits, while having the minimum impact on model adaptation in other legal
domains. The source code is available at:
https://github.com/pittisl/FreezeAsGuard/
| Search Query: ArXiv Query: search_query=au:”Wei Gao”&id_list=&start=0&max_results=3