A System for Virtual Try-on of Clothes on a 2D Image
Year: 2018
Introduction
This is my research thesis from my graduate school of computer science at National Taiwan University.
To address the inconvenience of buying clothes online, we've designed a virtual try-on system: When a user provides a photo of themselves and a photo of a garment, our AI model creates an image showing the user wearing that specific garment.
My Contribution
Proposed a method to enhance existing approaches by addressing their limitations using Python
Collected and processed a dataset contains 18573 image pairs with people and target clothes using MATLAB
Published in International Computer Symposium 2018 as a first author: Paper Link
Gained 300+ stars on GitHub: Code Link
Development
Python
MATLAB
What’s the Problem?
More people are buying clothes online, but a big problem is that they can't try them on first. This makes deciding whether to buy difficult. As a result, people order and return items, wasting time and resources. So, we worked on a solution. It lets people try on clothes virtually using pictures. This will make online shopping easier and help save time and resources.
User Persona
My Solution
To address this issue, we aimed to allow users to see the results of trying on clothes while shopping on website. We approached this problem from a technical standpoint, understanding that if it's feasible to change clothes on an image, then such a system could be realized. Therefore, this paper primarily focuses on the clothing replacement algorithm.
We developed an AI model to build a clothing replacement system on images. When a user provides a photo of themselves along with a photo of a garment, our AI model generates a photo of the user wearing that particular garment.
My System Architecture
This is the overall architecture of the system I proposed. I provide a detailed introduction to the individual steps below.
Individual Steps
Step1: Generate an initial rough result
I employed the method from *CAGAN to first generate a rough initial result.
*CAGAN: The Conditional Analogy Gan: Swapping Fashion Articles on People Images (CAGAN)
Step2: Obtain the mask for the target clothing transformation
Since the mask generated by CAGAN from Step 1 is quite fragmented, it's challenging to transform a piece of clothing into such a mask. Therefore, we need to generate a more complete mask.
Step3: Conduct Clothing Transformation
Use the mask generated in Step 2 to transform the input clothing into this mask.
Step4: Combine elements to produce the final output
Combine the head part of the input image and the transformed clothes from Step 3 into the final output.
Partial Results Display
What Makes My Approach Different from Others?
Based on two related papers, I proposed a method to enhance existing approaches by addressing their limitations.
Evaluate the Quality of the Results
Our results outperformed the baseline in approximately 90% of cases.
The quality of results was evaluated in a user study with 29 participants. The study used 499 clothing items with patterns, plaid, or stripes, paired with corresponding people from the dataset as testing data sets. The baseline was CAGAN, and the pairs were randomly selected.
Two versions of a questionnaire were created, each containing 60 pairs of a person image and a piece of clothing to be worn, randomly sampled from the testing dataset. Respondents were shown two results, one from CAGAN and one from the authors' model, and were asked to indicate their preference.
In two questionnaires, A and B, our results received 88.3% and 92.5% of the votes respectively, with an average of 90.3%. Respondents preferred our results in about 90% of cases.
