Kavli Affiliate: Jing Wang
| First 5 Authors: Yitong Yang, Yinglin Wang, Jing Wang, Tian Zhang,
| Summary:
Text-driven diffusion models have achieved remarkable success in image
editing, but a crucial component in these models-text embeddings-has not been
fully explored. The entanglement and opacity of text embeddings present
significant challenges to achieving precise image editing. In this paper, we
provide a comprehensive and in-depth analysis of text embeddings in Stable
Diffusion XL, offering three key insights. First, while the ‘aug_embedding’
captures the full semantic content of the text, its contribution to the final
image generation is relatively minor. Second, ‘BOS’ and ‘Padding_embedding’ do
not contain any semantic information. Lastly, the ‘EOS’ holds the semantic
information of all words and contains the most style features. Each word
embedding plays a unique role without interfering with one another. Based on
these insights, we propose a novel approach for controllable image editing
using a free-text embedding control method called PSP (Prompt-Softbox-Prompt).
PSP enables precise image editing by inserting or adding text embeddings within
the cross-attention layers and using Softbox to define and control the specific
area for semantic injection. This technique allows for obejct additions and
replacements while preserving other areas of the image. Additionally, PSP can
achieve style transfer by simply replacing text embeddings. Extensive
experimental results show that PSP achieves significant results in tasks such
as object replacement, object addition, and style transfer.
| Search Query: ArXiv Query: search_query=au:”Jing Wang”&id_list=&start=0&max_results=3