Generative Adversarial Networks for Real-robot Missions

Information

Author: Pin-Wei "David" Chen
Email: ccpwearth@gmail.com
Code: github.com/championway/gan_rv_thesis
Thesis: PDF Download

Abstract

Leveraging highly developed deep learning and artificial intelligence, computer vision technology and applications reached new levels. Computers can now not only perform image processing, classification, and object detection, but also can ”create” images similarly to humans, due to generative model developments. In particular, the generative adversarial network (GAN) provides many architectures and applications, such as image style transfer, human face generation, image generation from text, etc. However, there has been little study regarding applying GAN to real-robot missions to replace and improve other approaches. Therefore, this work proposed two GANs: FCN-Pix2Pix and SSIM-CycleGAN, based on Pix2Pix and CycleGAN respectively, and implemented them for two real-robot missions which still face some challenges with modern solutions: semantic segmentation and virtual dataset from sim to real. The proposed approaches were also compared with current state-of-the-art approaches, verifying significant advantages for the proposed methods.

Network Architectures

Experiments

Tote Sim2Real Dataset

• Class: 2 (Real totes, Unity totes)
• Image size: 640*480
• Description:
  • Real tote: Captured by SR300 from real world with different view angles.
  • Unity tote: Generated from virtual environment (Unity).
  • Image: Unity tote images with objects inside.
  • Mask: Labeled masks from Unity RGB images.
• CSV Description:
  • menu.csv: Mapping between Unity image, mask, and Unity tote image.
  • real_totes_train/test.csv: Lists for real totes.
  • unity_totes_train/test.csv: Lists for Unity totes.

Dataset	Size	Image	Real totes	Unity totes	Download
Tote Sim2Real Sample	25.9MB	100	100	99	sim2real_sample.zip
Tote Sim2Real	19.8GB	68608	5690	9472	sim2real.zip

Full Thesis PDF