Kavli Affiliate: Zheng Zhu
| First 5 Authors: Linkang Du, Zheng Zhu, Min Chen, Zhou Su, Shouling Ji
| Summary:
Text-to-image models based on diffusion processes, such as DALL-E, Stable
Diffusion, and Midjourney, are capable of transforming texts into detailed
images and have widespread applications in art and design. As such, amateur
users can easily imitate professional-level paintings by collecting an artist’s
work and fine-tuning the model, leading to concerns about artworks’ copyright
infringement. To tackle these issues, previous studies either add visually
imperceptible perturbation to the artwork to change its underlying styles
(perturbation-based methods) or embed post-training detectable watermarks in
the artwork (watermark-based methods). However, when the artwork or the model
has been published online, i.e., modification to the original artwork or model
retraining is not feasible, these strategies might not be viable.
To this end, we propose a novel method for data-use auditing in the
text-to-image generation model. The general idea of ArtistAuditor is to
identify if a suspicious model has been finetuned using the artworks of
specific artists by analyzing the features related to the style. Concretely,
ArtistAuditor employs a style extractor to obtain the multi-granularity style
representations and treats artworks as samplings of an artist’s style. Then,
ArtistAuditor queries a trained discriminator to gain the auditing decisions.
The experimental results on six combinations of models and datasets show that
ArtistAuditor can achieve high AUC values (> 0.937). By studying
ArtistAuditor’s transferability and core modules, we provide valuable insights
into the practical implementation. Finally, we demonstrate the effectiveness of
ArtistAuditor in real-world cases by an online platform Scenario. ArtistAuditor
is open-sourced at https://github.com/Jozenn/ArtistAuditor.
| Search Query: ArXiv Query: search_query=au:”Zheng Zhu”&id_list=&start=0&max_results=3