Avatar-GAN (RoboCoDraw)

This is the open-source repository of the project Avatar-GAN from the paper RoboCoDraw: Robotic Avatar Drawing with GAN-based Style Transfer andTime-efficient Path Optimization published in AAAI 2020.

Please cite our paper if this repository is useful for your project:

@misc{wang2019robocodraw,
    title={RoboCoDraw: Robotic Avatar Drawing with GAN-based Style Transfer and Time-efficient Path Optimization},
    author={Tianying Wang and Wei Qi Toh and Hao Zhang and Xiuchao Sui and Shaohua Li and Yong Liu and Wei Jing},
    year={2019},
    eprint={1912.05099},
    archivePrefix={arXiv},
    primaryClass={cs.RO}
}

Note: The datasets and checkpoints in Avatar-GAN system and RoboCoDraw system are different versions. The datasets and checkpoints in Avatar-GAN system (code in /GAN folder) are the same as that in the paper for reproducibility. The datasets in RoboCoDraw system were further processed (e.g. automatic background removal) for automatic drawing.

Feel free to contact Tianying ([email protected]) or Wei Qi ([email protected]).

Avatar-GAN system

Avatar-GAN was implemented based on Cycle-GAN proposed in paper [Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networkssee] [github].

Prerequisites

• tensorflow-gpu 1.13.1
• numpy 1.17.1
• scipy 1.3.1
• Pillow 6.1.0
• scikit-image 0.16.2

Getting Started

Installation

• Clone this repo:

git clone https://github.com/Psyhce-mia/Avatar-GAN
cd Avatar-GAN/GAN

Train

• Download the training dataset [Avatar-GAN dataset](not available yet) to Avatar-GAN/datasets/ folder

• Train a model:

python main.py

• Use tensorboard to visualize the training details:

tensorboard --logdir=./logs

Test

• Finally, test the model:

python main.py --phase=test 

Datasets

The datasets used in Avatar-GAN were generated from:

• Chicago Face Dataset (CFD): CFD dataset.
• Avataaars Library: Avataaars.
• CUHK Face Sketch Database: CUFS.

Note that we have applied preprocesses to the raw datasets.

RoboCoDraw system

Prerequisites

• tensorflow 1.13.1
• numpy 1.17.1
• opencv 4.1.1
• deap 1.3.0

Getting Started

Installation

Clone this repo:

git clone https://github.com/Psyhce-mia/Avatar-GAN

Running the RoboCoDraw system

To run the RoboCoDraw system with a UR robot, install python-urx. Ensure the robot end effector with the marker is at the bottom left corner of the drawing surface before executing the code with the --robot_drawing flag:

python run_RoboCoDraw.py --filename=CFD/gray1.jpg --opt_algo=rkgaLK --drawing_size=0.25 --table_surface_z=0.0 --robot_drawing 

Note:

• --filename sets the input image filename
• --opt_algo sets the optimization algorithm used for robot drawing (greedy, greedy2opt, greedyLK, rkga2opt, rkgaLK)
• --drawing_size sets the size of the output robot drawing (in meters)
• --table_surface_z sets the position of drawing table surface along the robot's z-axis

To run the RoboCoDraw system (style transfer + optimization) without executing the final robot drawing, run the following:

python run_RoboCoDraw.py --filename=CFD/gray1.jpg --opt_algo=rkgaLK