osm_ml_training_pt2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# AWS Sagemaker Training and Deploying\n",
    "## Cyclone Kenneth 2019-04-25\n",
    "### Part II\n",
    "\n",
    "In this part II notebook, we will upload the data to AWS S3 that we generated for training in the previous notebook. We will kick off an AWS Sagemaker object detection job and monitor the results. At the end of this notebook, you will have trained your own OSM-based CNN object detector!\n",
    "\n",
    "![](assets/happycloud.png)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Neural Network (Super simplified)\n",
    "\n",
    "We have a bunch of stacked 'neurons' that are mathematical function with weights.\n",
    "\n",
    "The number of neurons and how they are connected to each other defines an 'architecture'.\n",
    "\n",
    "We have a loss function that is iteratively checked to assess whether the neurons (and the weights) are trending to 'good': do the predictions align with the truth (this is validation data)?\n",
    "\n",
    "Weights are defined randomly (typically) to start. The net is pretty dumb. It is through the iterative process of training with many examples that learning is achived through imrpoving the weights.\n",
    "\n",
    "### Goal: Minimize the loss function!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A couple of things worth noting:\n",
    "\n",
    "🤔 ML models are not super useful unless they are scaled across a large amount of data\n",
    "\n",
    "🤔 To effectively scale across data, you need to be efficient\n",
    "\n",
    "🤔 Because we will be passing sensitive data to this notebook in order to scale our cloud compute through Sagemaker, we will use papermill to run this notebook from within python. It creates a simple wrapper around the notebook so that we can specify variables.\n",
    "\n",
    "e.g.\n",
    "\n",
    "``` python\n",
    "import papermill as pm\n",
    "pm.execute_notebook('osm_ml_training_pt2.ipynb','osm_ml_training_pt2_out.ipynb', parameters = dict(sage_bucket='',my_bucket='', role=''))\n",
    "\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sagemaker\n",
    "from sagemaker import get_execution_role\n",
    "from sagemaker.amazon.amazon_estimator import get_image_uri"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will use 'papermill' (https://github.com/nteract/papermill) to pass sensitive variables to this jupyter notebook. Things like passwords, cloud locations, etc, should be paramterized as a best practice -- Never stored in a repo (especially public facing).\n",
    "\n",
    "You will need to run `aws configure --profile uw` and enter in the crednetials I give you."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "tags": [
     "parameters"
    ]
   },
   "outputs": [],
   "source": [
    "ACCESS_KEY=''\n",
    "SECRET_KEY=''\n",
    "sage_bucket=''         #this is the 'top-level' s3 bucket, in which you will have a team data-folder\n",
    "my_bucket=''           #this is the 'folder' where your sagemaker data lives\n",
    "prefix = my_bucket     #this is your model prefix\n",
    "sessname =''           #your model name prefix\n",
    "nclass = 1\n",
    "epochs =2              #number of iterations\n",
    "mini_batch_size =2       #amount of data to use per iteration\n",
    "lr = 0.001\n",
    "lr_scheduler_factor =0.1\n",
    "momentum =0.9\n",
    "weight_decay =0.0005\n",
    "overlap = 0.5\n",
    "momentum = 0.45\n",
    "weight_decay =0.0005\n",
    "nms_thresh = 0.45\n",
    "image_shape =256\n",
    "label_width =150\n",
    "n_train_samples = 16551\n",
    "network ='resnet-50'\n",
    "optim = 'sgd'           #Stochastic gradient descent is an iterative method for optimizing an objective function      \n",
    "role = ''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import boto3\n",
    "\n",
    "my_east_sesison = boto3.Session(region_name = 'us-east-2',profile_name='uw')\n",
    "s3_client = my_east_sesison.client(\n",
    "    's3',\n",
    "    aws_access_key_id=ACCESS_KEY,\n",
    "    aws_secret_access_key=SECRET_KEY\n",
    ")\n",
    "s3 = my_east_sesison.resource('s3')\n",
    "\n",
    "s3_client.upload_file('rec/val.rec', sage_bucket, my_bucket+'/validation/val.rec')\n",
    "s3_client.upload_file('rec/train.rec', sage_bucket, my_bucket+'/train/train.rec')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "sess = sagemaker.Session(boto_session=my_east_sesison)\n",
    "training_image = get_image_uri(sess.boto_region_name, 'object-detection', repo_version=\"latest\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "s3_train_data = 's3://{}/{}'.format(sage_bucket, my_bucket+'/train/')\n",
    "s3_validation_data = 's3://{}/{}'.format(sage_bucket, my_bucket+'/validation/')\n",
    "\n",
    "s3_output_location = 's3://{}/{}/output'.format(sage_bucket, my_bucket)\n",
    "\n",
    "od_model = sagemaker.estimator.Estimator(training_image,\n",
    "                                         role, \n",
    "                                         train_instance_count=1, \n",
    "                                         train_instance_type='ml.p2.xlarge',\n",
    "                                         train_volume_size = 50,\n",
    "                                         train_max_run = 360000,\n",
    "                                         input_mode= 'File',\n",
    "                                         output_path=s3_output_location,\n",
    "                                         sagemaker_session=sess)\n",
    "                                         \n",
    "od_model.set_hyperparameters(base_network=network,\n",
    "                             use_pretrained_model=1,\n",
    "                             num_classes=nclass,\n",
    "                             mini_batch_size=mini_batch_size,\n",
    "                             epochs=epochs,\n",
    "                             learning_rate=lr,\n",
    "                             lr_scheduler_step='3,6',\n",
    "                             lr_scheduler_factor=lr_scheduler_factor,\n",
    "                             optimizer=optim,\n",
    "                             momentum=momentum,\n",
    "                             weight_decay=weight_decay,\n",
    "                             overlap_threshold=overlap,\n",
    "                             nms_threshold=nms_thresh,\n",
    "                             image_shape=image_shape,   \n",
    "                             label_width=label_width,\t\t\n",
    "                             num_training_samples=n_train_samples)\n",
    "\n",
    "train_data = sagemaker.session.s3_input(s3_train_data, distribution='FullyReplicated', \n",
    "                        content_type='application/x-recordio', s3_data_type='S3Prefix')\n",
    "validation_data = sagemaker.session.s3_input(s3_validation_data, distribution='FullyReplicated', \n",
    "                             content_type='application/x-recordio', s3_data_type='S3Prefix')\n",
    "data_channels = {'train': train_data, 'validation': validation_data}\n",
    "od_model.fit(inputs=data_channels, logs=True)    \n",
    "\n",
    "  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So now you are training!!! This will take a little while. We are only training for a very small number of epochs (2!), so we don't expect to have a really robust model. Potentially many 100s of epochs may be required depeneding on the quality and amount of training data we have. \n",
    "\n",
    "To level set, this model will be CRAPPY. But that is ok. You now have the basic tools required to set up and improve upon your own problem.\n",
    "\n",
    "🤔 What are the big considerations as a data scientist?\n",
    "\n",
    "🤔 What could we do to improve our model?\n",
    "\n",
    "🤔 How could we evaluate the quality of our data?\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "object_detector = od_model.deploy(initial_instance_count = 1,instance_type = 'ml.m4.xlarge')   \n",
    "\n",
    "#response = object_detector.predict(data)\n",
    "\n",
    "# Tears down the SageMaker endpoint and endpoint configuration\n",
    "#object_detector.delete_endpoint()\n",
    "\n",
    "# Deletes the SageMaker model\n",
    "#object_detector.delete_model()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Outside of this notebook, you can investigate your endpoint in the sagemaker console and run the test.sh script with the appropriate aws keys/role. This will generate some stats.\n",
    "\n",
    "You can also edit the output in the 'endpoint_infer_slippygeo.py' to write out a geojson that you can then explore in QGIS, etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "celltoolbar": "Tags",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}