Deep Image Analogy
The major contributors of this repository include Jing Liao, Yuan Yao, Lu Yuan, Gang Hua and Sing Bing Kang at Microsoft Research.
Introduction
Deep Image Analogy is a technique to find semantically-meaningful dense correspondences between two input images. It adapts the notion of image analogy with features extracted from a Deep Convolutional Neural Network.
Deep Image Analogy is initially described in a SIGGRAPH 2017 paper
Disclaimer
This is an official C++ combined with CUDA implementation of Deep Image Analogy. It is worth noticing that:
- Our codes are based on Caffe.
- Our codes only have been tested on Windows 10 and Windows Server 2012 R2 with CUDA 8 or 7.5.
- Our codes only have been tested on several Nvidia GPU: Titan X, Titan Z, K40, GTX770.
- The size of input image is limited, mostly should not be large than 700x500 if you use 1.0 for parameter ratio.
License
© Microsoft, 2017. Licensed under a MIT license.
Citation
If you find Deep Image Analogy (include deep patchmatch) helpful for your research, please consider citing:
@article{Liao:2017:VAT:3072959.3073683,
author = {Liao, Jing and Yao, Yuan and Yuan, Lu and Hua, Gang and Kang, Sing Bing},
title = {Visual Attribute Transfer Through Deep Image Analogy},
journal = {ACM Trans. Graph.},
issue_date = {July 2017},
volume = {36},
number = {4},
month = jul,
year = {2017},
issn = {0730-0301},
pages = {120:1--120:15},
articleno = {120},
numpages = {15},
url = {http://doi.acm.org/10.1145/3072959.3073683},
doi = {10.1145/3072959.3073683},
acmid = {3073683},
publisher = {ACM},
address = {New York, NY, USA},
keywords = {deep matching, image analogy, transfer},
}
Application
Photo to Style
One major application of our code is to transfer the style from a painting to a photo.
Style to Style
It can also swap the styles between two artworks.
Style to Photo
The most challenging application is converting a sketch or a painting to a photo.
Photo to Photo
It can do color transfer between two photos, such as generating time lapse.
Getting Started
Prerequisite
- Windows 7/8/10 (for linux or mac os x user, please check branch linux.)
- CUDA 8 or 7.5
- Visual Studio 2013
Build
- Build Caffe at first. Just follow the tutorial here.
- Edit
deep_image_analogy.vcxproj
underwindows/deep_image_analogy
to make the CUDA version in it match yours . - Open solution
Caffe
and adddeep_image_analogy
project. - Build project
deep_image_analogy
.
Download models
You need to download models VGG-19 model before start to run a demo. Go to windows/deep_image_analogy/models/vgg19/
folder and download:
Demo
Open main.cpp
in windows/deep_image_analogy/source/
to see how to run a demo. You need to set several parameters which have been mentioned in the paper. To be more specific, you need to set
- path_model, where the VGG-19 model is.
- path_A, the input image A.
- path_BP, the input image BP.
- path_output, the output path.
- GPU Number, GPU ID you want to run this experiment.
- Ratio, the ratio to resize the inputs before sending them into the network.
- Blend Weight, the level of weights in blending process.
- Flag of WLS Filter, if you are trying to do photo style transfer, we recommend to switch this on to keep the structure of original photo.
Direct Run
We also provide a pre-built executable file in folder windows/deep_image_analogy/exe/
, don't hesitate to try it.
To run this deep_image_analogy.exe
, you need to write a command line as:
deep_image_analogy.exe ../models/ ../demo/content.png ../demo/style.png ../demo/output/ 0 0.5 2 0
which means
- path_model=
../models/
- path_A=
../demo/content.png
- path_BP=
../demo/style.png
- path_output=
../demo/output/
- GPU Number=
0
- Ratio=
0.5
- Blend Weight=
2
- Flag of WLS Filter=
0
(0
: WLS filter disabled,1
: WLS filter enabled, only required for the case of photo to photo)
Tips
- We often test images of size 600x400 and 448x448.
- We set ratio to 1.0 by default. Specifically, for face (portrait) cases, we find ratio = 0.5 often make the results better.
- Blend weight controls the result appearance. If you want the result to be more like original content photo, please increase it; if you want the result more faithful to the style, please reduce it.
- For the four applications, our settings are mostly (but not definitely):
- Photo to Style: blend weight=3, ratio=0.5 for face and ratio=1 for other cases.
- Style to Style: blend weight=3, ratio=1.
- Style to Photo: blend weight=2, ratio=0.5.
- Photo to Photo: blend weight=3, ratio=1.
Acknowledgments
Our codes acknowledge Eigen, PatchMatch, CudaLBFGS and Caffe. We also acknowledge to the authors of our image and style examples but we do not own the copyrights of them.