Initial commit

.gitignore

@@ -0,0 +1,105 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+.idea/
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+env/
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+.hypothesis/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# pyenv
+.python-version
+
+# celery beat schedule file
+celerybeat-schedule
+
+# SageMath parsed files
+*.sage.py
+
+# dotenv
+.env
+
+# virtualenv
+.venv
+venv/
+ENV/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+
+RUNS/
+

README.md

@@ -0,0 +1,158 @@
+[![Udacity - Robotics NanoDegree Program](https://s3-us-west-1.amazonaws.com/udacity-robotics/Extra+Images/RoboND_flag.png)](https://www.udacity.com/robotics)
+
+## Deep Learning Project ##
+
+In this project, you will train a deep neural network to identify and track a target in simulation and then issue commands to a drone to follow that target. So-called “follow me” applications like this are key to many fields of robotics and the very same techniques you apply here could be extended to scenarios like advanced cruise control in autonomous vehicles or human-robot collaboration in industry.
+
+[image_0]: ./docs/misc/sim_screenshot.png
+![alt text][image_0] 
+
+## Setup Instructions
+**Clone the repository**
+```
+$ git clone https://github.com/udacity/RoboND-DeepLearning-Private.git
+```
+
+**Download the QuadSim binary**
+
+To interface your neural net with the QuadSim simulator, you must use a version QuadSim that has been custom tailored for this project. The previous version that you might have used for the Controls lab will not work.
+
+The simulator binary can be downloaded [here](https://github.com/udacity/RoboND-DeepLearning-Private/releases)
+
+**Install Dependencies**
+
+You'll need Python 3 and Jupyter Notebooks installed to do this project.  The best way to get setup with these if you are not already is to use Anaconda following along with the [RoboND-Python-Starterkit](https://github.com/ryan-keenan/RoboND-Python-Starterkit).
+
+If for some reason you choose not to use Anaconda, you must install the following framworks and packages on your system:
+* Python 2.7
+* Tensorflow 1.2.1
+* NumPy 1.11
+* OpenCV 2
+* SciPy 0.17.0
+* eventlet 
+* Flask
+* h5py
+* PIL
+* python-socketio
+* scikit-image
+* socketIO-client
+
+## Implement the Segmentation Network
+1. Download the training dataset from [here](https://github.com/udacity/RoboND-DeepLearning-Private/releases), and extract to the project `data` directory.
+2. Complete `make_model.py`by following the TODOs in `make_model_template.py`
+3. Complete `data_iterator.py` by following the TODOs in `data_iterator_template.py`
+4. Complete `train.py` by following the TODOs in `train_template.py`
+5. Train the network locally, or on [AWS](docs/aws_setup.md).
+6. Continue to experiement with the training data and network until you attain the score you desire.
+7. Once you are comfortable with performance on the training dataset, see how it performs in live simulation!
+
+## Collecting Training Data ##
+A simple training dataset has been provided in the [releases](https://github.com/udacity/RoboND-DeepLearning-Private/releases) section of this repository. This dataset will allow you to verify that you're segmentation network is semi-functional. However, if you're interested in improving your score, you may be interested in collecting additional training data. To do, please see the following steps.
+
+The data directory is organized as  follows:
+```
+data/runs - contains the results of prediction runs
+data/train/images - contains images for the training set
+data/train/masks - contains masked (labeled) images for the training set
+data/validation/images - contains images for the validation set
+data/validation/masks - contains masked (labeled) images for the validation set
+data/weights - contains trained TensorFlow models
+```
+
+### Training Set: with Hero Present ###
+1. Run QuadSim
+2. Select `Use Hero Target`
+3. Select `With Other Poeple`
+4. Click the `DL Training` button
+5. With the simulator running, press "r" to begin recording.
+6. In the file slection menu navigate to the `data/train/target/hero_train1` directory
+7. **optional** to speed up data collection, press "9" (1-9 will slow down collection speed)
+8. When you have finished collecting data, hit "r" to stop recording.
+9. To exit the simulator, hit "`<esc>`"
+
+### Training Set: without Hero Present ###
+1. Run QuadSim
+2. Make sure `Use Hero Target` is **NOT** selected
+3. Select `With Other Poeple`
+4. Click the `DL Training` button
+5. With the simulator running, press "r" to begin recording.
+6. In the file slection menu navigate to the `data/train/non_target/run_train1` directory.
+7. **optional** to speed up data collection, press "9"  (1-9 will slow down collection speed)
+8. When you have finished collecting data, hit "r" to stop recording.
+9. To exit the simulator, hit "`<esc>`"
+
+### Validation Set ###
+To collect the validation set, repeat both sets of steps above, except using the directory `data/validation` instead rather than `data/train`.
+
+### Image Preprocessing ###
+Before the network is trained, the images first need to be undergo a preprocessing step.
+**TODO**: Explain what preprocessing does, approximately.
+To run preprocesing:
+```
+$ python preprocess_ims.py
+```
+**Note**: If your data is stored as suggested in the steps above, this script should run without error.
+
+## Training, Predicting and Scoring ##
+With your training and validation data having been generated (or downloaded from the [releases](https://github.com/udacity/RoboND-DeepLearning-Private/releases) section of this repository, you are free to begin working with the neural net.
+
+**Note**: Training CNNs is a very compute-intensive process. If your system does not have a recent Nvidia graphics card, with [cuDNN](https://developer.nvidia.com/cudnn) installed , you may need to perform the training step in the cloud. Instructions for using AWS to train your network in the cloud may be found [here](docs/aws_setup.md)
+
+### Training your Model ###
+**Prerequisites**
+- Net has been implemented as per these instructions
+- Training data is in `data` directory
+
+To train, simply run the training script, `train.py`, giving it the name of the model weights file as a parameter:
+```
+$ python train.py my_amazing_model.h5
+```
+After the trianing run has completed, your model will be stored in the `data/weights` directory as an [HDF5](https://en.wikipedia.org/wiki/Hierarchical_Data_Format) file.
+
+### Predicting on the Validation Set and Evaluating the Results ###
+**Prerequisites**
+-Model has been trained
+-Validation set has been collected
+
+Once the network has been trained, you can run inference on the validation set using `predict.py`. This script requires two parameters, the name of the model file you wish to perform prediction with, and the output directory where you would like to store your prediction results.
+
+```
+$ python predict.py my_amazing_model.h5 my_prediction_run
+```
+
+For the prediction run above, the results will be stored in `data/runs/my_prediction_run`.
+
+To get a sense of the overall performance of the your net on the prediction set, you can use `evaluation.py` as follows:
+
+```
+$ python evaluate.py validation my_prediction_run
+average intersection over union 0.34498680536
+number of validation samples evaluated on 1000
+number of images with target detected: 541
+number of images false positives is: 4
+average squared pixel distance error 11.0021170157
+average squared log pixel distance error 1.4663195103
+```
+
+## Scoring ##
+**TODO**
+
+**How the Final score is Calculated**
+
+**TODO**
+
+**Ideas for Improving your Score**
+
+**TODO**
+
+**Obtaining a leaderboard score**
+
+**TODO**
+
+## Experimentation: Testing in Simulation
+1. Copy your saved model to the weights directory `data/weights`.
+2. Launch the simulator, select "Spawn People", and then click the "Follow Me" button. 
+3. Run `server.py` to launch the socketio server.
+4. Run the relatime follower script `$ realtime_follower.py my_awesome_model`
+
+**Note:** If you'd like to see an overlay of the detected region on each camera frame from the drone, simply pass the `--overlay_viz` parameter to `realtime_follower.py`

code/build_model.py

@@ -0,0 +1,5 @@
+import model_builder
+
+# run this file with 'python model_builder.py' to generate plots of network defined in network config
+if __name__ == '__main__':
+    net = model_builder.make_example_network()

code/evaluate.py

@@ -0,0 +1,62 @@
+import glob
+import numpy as np
+import os
+import sys
+from scipy import misc
+
+from utils import scoring_utils
+
+
+def score_run(gt_dir, pred_dir):
+    gt_files = sorted(glob.glob(os.path.join(gt_dir, 'masks', '*.png')))
+    pred_files = sorted(glob.glob(os.path.join(pred_dir, '*.png')))
+
+    iou = 0
+    error1 = 0
+    error2 = 0
+    n_valid_detections = 0
+    n_invalid_detections = 0
+
+    for e, gt_file in enumerate(gt_files):
+        gt_mask = misc.imread(gt_file)[:, :, 1].clip(0, 1)
+        pred_mask = (misc.imread(pred_files[e])[:, :, 1] > 127).astype(np.int)
+
+        iou += scoring_utils.intersection_over_union(pred_mask, gt_mask)
+        # only compute centroid distance if it found some object
+        if pred_mask.sum() > 3:
+            if gt_mask.sum() > 3:
+                gt_centroid = scoring_utils.get_centroid_largest_blob(gt_mask)
+                pred_centroid = scoring_utils.get_centroid_largest_blob(pred_mask)
+                error1 += scoring_utils.average_squared_distance(pred_centroid, gt_centroid)
+                error2 += scoring_utils.average_squared_log_distance(pred_centroid, gt_centroid)
+                n_valid_detections += 1
+            else:
+                error1 += 1
+                error2 += 1
+                n_invalid_detections += 1
+
+    return iou, error1, error2, len(gt_files), n_valid_detections, n_invalid_detections
+
+
+if __name__ == '__main__':
+
+    if len(sys.argv) != 3:
+        raise ValueError('evaluate.py the ground truth folder name and prediction folder name as commandline input')
+
+    gt_folder = sys.argv[1]
+    pred_folder = sys.argv[2]
+
+    gt_path = os.path.join('..', 'data', gt_folder)
+    pred_path = os.path.join('..', 'data', 'runs', pred_folder)
+
+    (iou, err1, err2, n_preds, n_valid_detections, n_invalid_detections
+     ) = score_run(gt_path, pred_path)
+
+    print('average intersection over union {}'.format(iou / n_preds))
+    print('number of validation samples evaluated on {}'.format(n_preds))
+    print('number of images with target detected: {}'.format(n_valid_detections))
+    print('number of images false positives is: {}'.format(n_invalid_detections))
+    if n_valid_detections > 0 or n_invalid_detections > 0:
+        n_detections = n_valid_detections + n_invalid_detections
+        print('average squared pixel distance error {}'.format(err1 / n_detections))
+        print('average squared log pixel distance error {}'.format(err2 / n_detections))

code/make_model_template.py

@@ -0,0 +1,53 @@
+from tensorflow.contrib.keras.python import keras
+from tensorflow.contrib.keras.python.keras import layers as l
+
+from network_definitions.separable_conv2d import SeparableConv2DKeras
+
+
+# TODO implement a encode block, it should reduce spatial edge width by half
+def encode_block(x, filts, n_blocks=2):
+    pass
+
+# TODO implement a decode block, it should increase spatial edge_width by half
+def decode_block(x_small, x_large, filts, n_blocks=2):
+    pass
+
+def make_encode_decode_net(inputs, num_classes, n_blocks, base=32, filter_list=None):
+    """Flexible semantic segmentation architecture using seperable convolutions"""
+
+    if filter_list is None:
+        filter_list = list()
+        for i in range(n_blocks):
+            filter_list.append(base * (2 ** i))
+
+    x0 = l.Conv2D(base, 3, activation='relu', padding='same')(inputs)
+    x0 = l.BatchNormalization()(x0)
+
+    encode_layers = list()
+    for i in range(n_blocks):
+        if i == 0:
+            encode_layers.append(encode_block(x0, filter_list[i]))
+        else:
+            encode_layers.append(encode_block(encode_layers[i - 1], filter_list[i]))
+
+    encode_layers = [x0] + encode_layers
+    encode_layers = list(reversed(encode_layers))
+    filter_list = list(reversed(filter_list))
+
+    for e, layer in enumerate(encode_layers[1:]):
+        if e == 0:
+            x = decode_block(encode_layers[0], layer, filter_list[e])
+        else:
+            x = decode_block(x, layer, filter_list[e])
+
+    return l.Conv2D(num_classes, 3, activation='sigmoid', padding='same')(x)
+
+
+def make_example_model():
+    im_shape = (256, 256, 3)
+    inputs = keras.layers.Input(im_shape)
+    out_layer = make_encode_decode_net(inputs, num_classes=3, n_blocks=3, base=16)
+
+    model = keras.models.Model(inputs=inputs, outputs=out_layer)
+    model.compile(optimizer=keras.optimizers.Adam(0.001), loss='binary_crossentropy')
+    return model