PLTON: Product-Level Try-on with Realistic Clothes Shading and Wrinkles
METADATA ONLY
Loading...
Author / Producer
Date
2024
Publication Type
Conference Paper
ETH Bibliography
yes
Citations
Altmetric
METADATA ONLY
Data
Rights / License
Abstract
Image-based virtual try-on systems, which fit new garments onto human portraits, are gaining research attention. An ideal pipeline should preserve the static features of clothes, such as textures and logos, while also generating the adaptive dynamic clothes elements, e.g., shadows and folds. Previous works fail specifically in generating dynamic clothes features, as they maintain the warped in-shop clothes trivially with predicted an alpha mask by composition. To break the dilemma of overpreserving and textures losses, we propose a novel diffusion-based Product-level virtual try-on pipeline, PLTON, which can preserve the fine details of logos and embroideries while producing realistic clothes shading and wrinkles. The main insights are in three folds: 1) Adaptive Dynamic Rendering. Harnessing the potential of a pre-trained diffusion model as a generative prior, we condition it based on image-based visual features and train a novel dynamic feature extractor. This extractor generates dynamic tokens retaining high-fidelity semantic content, enabling the creation of compellingly authentic shadows and wrinkles in virtual garments. 2) Static Characteristics Transformation. PLTON commences by mapping in-shop clothes to the target pose using a conventional warping network. Thereafter, a high-pass filter extracts an HighFrequency Maps (HF-Map) to scrupulously retain static fabric features. These HF-Maps inform the generation of modulation maps via our static extractor, which are subsequently integrated into a fixed U-net for the synthesis of the final, detailed garment images. 3) Retention Enhancement Strategy. We propose a Twostage hybrid denoising technique to steer the diffusion process towards accurate spatial layouts and color fidelity, thus enhancing retention of visual information. PLTON is fine-tuned solely with a modestly-sized try-on dataset. Extensive quantitative and qualitative experiments conducted on high-resolution datasets substantiate the superiority of our framework in emulating the complex dynamics found in real clothing scenarios.
Permanent link
Publication status
published
Editor
Book title
2024 International Joint Conference on Neural Networks (IJCNN)
Journal / series
Volume
Pages / Article No.
10650070
Publisher
IEEE
Event
International Joint Conference on Neural Networks (IJCNN 2024)
Edition / version
Methods
Software
Geographic location
Date collected
Date created
Subject
virtual try-on; diffusion model; shading and wrinkles