Kavli Affiliate: Wei Gao
| First 5 Authors: Kai Huang, Haoming Wang, Wei Gao, ,
| Summary:
Text-to-image diffusion models can be fine-tuned in custom domains to adapt
to specific user preferences, but such adaptability has also been utilized for
illegal purposes, such as forging public figures’ portraits, duplicating
copyrighted artworks and generating explicit contents. Existing work focused on
detecting the illegally generated contents, but cannot prevent or mitigate
illegal adaptations of diffusion models. Other schemes of model unlearning and
reinitialization, similarly, cannot prevent users from relearning the knowledge
of illegal model adaptation with custom data. In this paper, we present
FreezeAsGuard, a new technique that addresses these limitations and enables
irreversible mitigation of illegal adaptations of diffusion models. Our
approach is that the model publisher selectively freezes tensors in pre-trained
diffusion models that are critical to illegal model adaptations, to mitigate
the fine-tuned model’s representation power in illegal adaptations, but
minimize the impact on other legal adaptations. Experiment results in multiple
text-to-image application domains show that FreezeAsGuard provides 37% stronger
power in mitigating illegal model adaptations compared to competitive
baselines, while incurring less than 5% impact on legal model adaptations. The
source code is available at: https://github.com/pittisl/FreezeAsGuard.
| Search Query: ArXiv Query: search_query=au:”Wei Gao”&id_list=&start=0&max_results=3