01_Lab_1_Introduction.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introductory applied machine learning (INFR10069)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lab 1: Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the first part of this lab we will set-up a working environment for the IAML course. In the second part we will introduce some packages we will be using throughout the course."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Setting up for DICE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Within this course we will be using Python along with a few open-source libraries (packages). These packages cannot be installed directly, so we will have to create a virtual environment. We are using virtual enviroments to make the installation of packages and retention of correct versions as simple as possible. You can read [here](https://virtualenv.pypa.io/en/stable/) if you want to learn about virtual environments, but this is **not** neccessary for this tutorial."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now open a terminal and follow these instructions. We are expecting you to enter these commands in **one-by-one**. Waiting for each command to complete will help catch any unexpected warnings and errors. Please read heed any warnings and errors you may encounter. We are on standby in the labs to help if required.\n",
    "\n",
    "1. Change directory to home and create a virtual enviroment\n",
    "```bash\n",
    "cd\n",
    "virtualenv --distribute virtualenvs/iaml_env  # Creates a virtual environment called iaml_env\n",
    "```\n",
    "   \n",
    "2. Navigate to and activate the virtual enviroment (you will need to activate the virtual environment every time you open a new terminal - this adds the correct python version with all installed packages to your system's `$PATH` environment variable)\n",
    "```bash\n",
    "cd virtualenvs/iaml_env\n",
    "source ./bin/activate  # Activates the environment, your shell prompt should now \n",
    "                          # change to reflect you are in the `iaml_env` enviornment\n",
    "```\n",
    "\n",
    "3. Install all the python packages we need (once the correct virtual environment is activated, pip install will install packages to the virtual environent - if you're ever unsure which python you are using, type `which python` in the terminal) **WATCH FOR WARNINGS AND ERRORS HERE**. We have split these commands up to encourage you to enter them one-by-one.\n",
    "   \n",
    "    * `pip install -U setuptools  # The -U flag upgrades the current version`\n",
    "    * `pip install -U pip`\n",
    "    * `pip install yolk`\n",
    "    * `pip install jupyter`\n",
    "    * `pip install numpy`\n",
    "    * `pip install scipy`\n",
    "    * `pip install matplotlib`\n",
    "    * `pip install pandas`\n",
    "    * `pip install statsmodels`\n",
    "    * `pip install scikit-learn`\n",
    "    * `pip install seaborn`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should now have all the required modules installed. Our next step is to make a new directory where we will keep all the lab notebooks, datasets and assignments. Within your terminal:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Navigate back to your home directory\n",
    "    * `cd`\n",
    "1. Make a new directory and navigate to it\n",
    "    * `mkdir iaml_2016`\n",
    "    * `cd iaml_2016`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now you have two options:\n",
    "\n",
    "1. We recommend that you directly download a .zip file from https://github.com/agamemnonc/iaml which will contain everything you need and save it in the folder you have just created. You can do this from the terminal by typing:\n",
    "    * `wget https://github.com/agamemnonc/iaml/archive/master.zip`\n",
    "    * `unzip master.zip`\n",
    "2. If **and only if** you are familiar and confident with using Git/GitHub, you can initiliaze a git directory, add the above repo as remote and pull everything into your local directory. Please use this option only if you really know what you are doing. Unfortunately, we won't be able to provide you with Git/Github support if you run into issues with syncing and using version control in general. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once you have downloaded the material, you are now ready to start working in the Jupyter environment. First you need to activate the `iaml_env` environment and start a Jupyter Notebook session from within the folder where the material is stored. **Note that you will have to follow this procedure for all labs and assignments.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Navigate home and ensure the `iaml_env` virtualenv is activated\n",
    "    * `cd`\n",
    "    * `source virtualenvs/iaml_env/bin/activate` (Activate environment)\n",
    "1. Enter the directory you downloaded the course material\n",
    "    * `cd iaml_2016/iaml-master`\n",
    "1. Start a jupyter notebook\n",
    "    * `jupyter notebook`\n",
    "1. This should automatically open your browser\n",
    "    * Click on `01_Lab_1_Introduction.ipynb` to open it"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Setting up for personal machine (Windows / OS X / Ubuntu)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you are using a personal machine, you can choose whether to do as above or use the Anaconda distribution (Python version 2.7, choose the appropriate installer according to your operating system). Anaconda is a standard set of packages used in scientific computing which the Anaconda team curate to keep them consistent.\n",
    "\n",
    "Once the installation is complete, we need to install the Seaborn package which is the only one not included in the distribution. It's also recommended that you set up a virtual environment for this project. This way, if you update anything in your anaconda base install, this virtual environment will remain unchanged. To create a virtual environment called `iaml`, open a Terminal (or Command Prompt window if you are running Windows) and type:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```bash\n",
    "conda create -n iaml python=2.7 anaconda seaborn=0.7.0\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Don't forget to activate the virtual environment every time you begin work from a new terminal:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```bash\n",
    "source activate iaml\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once you have finished installed everything, open a terminal (or Command Prompt in Windows), navigate to the folder where you have downloaded the course material and type:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```bash\n",
    "jupyter notebook\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then click on `01_Lab_1_Introduction.ipynb` to open it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introducing main packages"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the following section we introduce the main **packages** we will be using in this course. If you think that you are already familiar with the packages outlined below you are free to skip this section. We do advise you, however, to go quickly through this tutorial to make sure that you are familiar with all the tools you will need for the rest of the course.\n",
    "\n",
    "The packages that we will use are the following:\n",
    "* [numpy:](http://www.numpy.org/) scientific computing by using array objects\n",
    "\n",
    "\n",
    "* [pandas](http://pandas.pydata.org/): data structures and data analysis tools\n",
    "\n",
    "\n",
    "* [matplotlib](http://matplotlib.org/): plotting library (similar to MATLAB's plot interface)\n",
    "\n",
    "\n",
    "* [seaborn](https://seaborn.github.io/): data visualisation library which works on top of matplotlib\n",
    "\n",
    "Throughout the course, we will also make heavy use of [scikit-learn](http://scikit-learn.org/stable/) which is a machine learning library implementing many learning algorithms and useful tools. This is introduced in Lab 2."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## IPython / Jupyter environment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Basic knowledge of `python` and working with `notebooks` in the Jupyter environment is assumed for this course. If you haven't used python before, you are strongly advised to familiarise yourselves with basic python syntax and working in the Jupyter environment. There are many excellent tutorials available on the web and you can choose the ones you like the most. If you are not sure which ones to choose, these are good starting points:\n",
    "\n",
    "[Introduction to Python for scientific computing](http://bebi103.caltech.edu/2015/tutorials/t1a_intro_to_python.html)\n",
    "\n",
    "[Introduction to Jupyter notebooks](http://bebi103.caltech.edu/2015/tutorials/t0b_intro_to_jupyter_notebooks.html)\n",
    "\n",
    "[Python/Numpy tutorial](http://cs231n.github.io/python-numpy-tutorial/#python)\n",
    "(A large part of Lab 1 is based on this tutorial)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Line magics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sometimes we might want to connect to the same notebook kernel from multiple frontends. This is useful for excecuting quick calculations, or checking objects currently stored in the kernel, without having to create a new cell in the notebook.\n",
    "\n",
    "The `%connect_info` line magic displays the information necessary to connect another frontend to the Notebook kernel."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%connect_info"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tab completion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Tab completion is a powerful method of finding out objects attributes and available methods.\n",
    "\n",
    "Let's see an example of this by using a Python [list](http://www.tutorialspoint.com/python/python_lists.htm). We will create a list and then you can see what methods are available by typing the list name followed by `.` and then hitting the < Tab > key. Then you can access any method's help documentation by hitting the method's name followed by `?`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "l = [1, 4.2, 'hello']\n",
    "print(l)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "l.append?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import packages"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we start, we need to import the packages that we will be using later. If you are having trouble importing any of these packages make sure that these are properly installed. If you still encounter issues, refer to Installing instructions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from __future__ import division, print_function # Makes division and printing work like python 3 (we're using 2)\n",
    "import os\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from matplotlib import pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Numpy introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Numpy is a powerful scientific computing library. The fundmental object is the (n-dimensional) numpy array and the library provides a collection of tools for working with such arrays. If you are already familiar with MATLAB you might find this [tutorial](https://docs.scipy.org/doc/numpy-dev/user/numpy-for-matlab-users.html) useful. \n",
    "\n",
    "The following Numpy introduction is largely based on this [tutorial](http://cs231n.github.io/python-numpy-tutorial/#numpy)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Arrays"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A numpy array is a grid of values, all of the same type, and is indexed by a [tuple](http://www.tutorialspoint.com/python/python_tuples.htm) of nonnegative integers. The number of dimensions is the rank of the array; the shape of an array is a tuple of integers giving the size of the array along each dimension. *N.B. this use of the word 'rank' is not the same as the meaning in linear algebra.*\n",
    "\n",
    "We can initialize numpy arrays from nested Python [lists](http://www.tutorialspoint.com/python/python_lists.htm), and access elements using square brackets:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "a = np.array([1, 2, 3])  # Create a rank 1 array (i.e. vector)\n",
    "print('Array a:', a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print(type(a))                    # Prints the type of object a (array)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print(a.shape)                    # Prints the number of elements for each dimension"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print(a[0], a[1], a[2])           # Index array"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "a[0] = 5                          # Change an element of the array"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print(a)                          # Prints the whole array"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "b = np.array([[1,2,3],[4,5,6]])   # Create a rank 2 array"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print(b.shape)                   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print(b[0, 0], b[0, 1], b[1, 0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At this point it worth noting -in case you didn't know already- that Python follows [0-based indexing](https://en.wikipedia.org/wiki/Zero-based_numbering) (as opposed to MATLAB which follows 1-based indexing)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In an ipython notebook you can just run a cell with an object name and ipython will try to print it. For example, execute the below cell:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "a"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create arrays"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have many options here:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "a = np.zeros((2,2))  # Create an array of all zeros\n",
    "print(a)                    # Prints \"[[ 0.  0.]\n",
    "                     #          [ 0.  0.]]\"\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "b = np.ones((1,2))   # Create an array of all ones\n",
    "print(b)                    # Prints \"[[ 1.  1.]]\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "c = np.full((2,2), 7) # Create a constant array\n",
    "print(c)                     # Prints \"[[ 7.  7.]\n",
    "                      #          [ 7.  7.]]\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "d = np.eye(2)        # Create a 2x2 identity matrix\n",
    "print(d)                    # Prints \"[[ 1.  0.]\n",
    "                     #          [ 0.  1.]]\"\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "e = np.random.random((2,2)) # Create an array filled with random values\n",
    "print(e)                           # Might print \"[[ 0.91940167  0.08143941]\n",
    "                            #               [ 0.68744134  0.87236687]]\"\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "some_list = [1, 4, 6, 8]\n",
    "f = np.array(some_list, dtype=float) # Create an array from a list. Also sets the array type to float\n",
    "print(f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Array indexing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Slicing is the most common way to index arrays. This works in a similar way to indexing python lists. \n",
    "\n",
    "There are also other options, such as integer and boolean array indexing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "a = np.array([[1,2,3,4], [5,6,7,8], [9,10,11,12]])\n",
    "print(a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Use slicing to pull out the subarray consisting of the first 2 rows\n",
    "# and columns 1 and 2; b is the following array of shape (2, 2):\n",
    "# [[2 3]\n",
    "#  [6 7]]\n",
    "b = a[:2, 1:3]\n",
    "print(b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# A slice of an array is a view into the same data, so modifying it\n",
    "# will modify the original array. This is called \n",
    "print(a[0,1])  # Prints \"2\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "b[0, 0] = 77    # b[0, 0] is the same piece of data as a[0, 1]\n",
    "print(a[0,1])          # Prints \"77\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Array datatypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Every numpy array is a grid of elements of the same type. Numpy provides a large set of numeric datatypes that you can use to construct arrays. Numpy tries to guess a datatype when you create an array, but functions that construct arrays usually also include an optional argument to explicitly specify the datatype. Here is an example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "x = np.array([1, 2])  # Let numpy choose the datatype\n",
    "print(x.dtype)       "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "x = np.array([1.0, 2.0])  # Let numpy choose the datatype\n",
    "print(x.dtype)             "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "x = np.array([1, 2], dtype=np.int64)  # Force a particular datatype\n",
    "print(x.dtype)                 "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Array math"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Basic mathematical functions operate elementwise on arrays, and are available both as operator overloads and as functions in the numpy module:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "x = np.array([[1,2],[3,4]], dtype=np.float64)\n",
    "y = np.array([[5,6],[7,8]], dtype=np.float64)\n",
    "print('x = ',x)\n",
    "print('y = ',y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Elementwise sum, equivalent expressions:\n",
    "print('x + y =')\n",
    "print(x + y)\n",
    "print(np.add(x, y))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Elementwise difference, equivalent expressions:\n",
    "print('x-y:')\n",
    "print(x - y)\n",
    "print(np.subtract(x, y))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Elementwise product, equivalent expressions:\n",
    "print('x*y:')\n",
    "print(x * y)\n",
    "print(np.multiply(x, y))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Elementwise division, equivalent expressions:\n",
    "print('x/y:')\n",
    "print(x / y)\n",
    "print(np.divide(x, y))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Elementwise square root\n",
    "print('Square root of x:')\n",
    "print(np.sqrt(x))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that unlike MATLAB, * is elementwise multiplication, not matrix multiplication. We instead use the `dot` function to compute inner products of vectors, to multiply a vector by a matrix, and to multiply matrices. `dot` is available both as a function in the numpy module and as an instance method of array objects:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "x = np.array([[1,2],[3,4]])\n",
    "y = np.array([[5,6],[7,8]])\n",
    "\n",
    "v = np.array([9,10])\n",
    "w = np.array([11, 12])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Inner product of vectors, equivalent expressions\n",
    "print(np.dot(v, w))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Matrix / vector product, equivalent expressions\n",
    "print(x.dot(v))\n",
    "print(np.dot(x, v))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Matrix / matrix product, equivalent expressions\n",
    "print(x.dot(y))\n",
    "print(np.dot(x, y))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Mathematical functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Numpy provides many useful functions for performing computations on arrays; one of the most useful is `sum`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "x = np.array([[1,2],[3,4]])\n",
    "print(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "np.sum(x)  # Compute sum of all elements"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "np.sum(x, axis=0)  # Compute sum of each column"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "np.sum(x, axis=1)  # Compute sum of each row"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can find the full list of mathematical functions provided by numpy in the [documentation](http://docs.scipy.org/doc/numpy/reference/routines.math.html).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Apart from computing mathematical functions using arrays, we frequently need to reshape or otherwise manipulate data in arrays. The simplest example of this type of operation is transposing a matrix; to transpose a matrix, simply use the `T` attribute of an array object:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "x = np.array([[1,2], [3,4]])\n",
    "print(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "x.T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "np.transpose(x) # Equivalent expression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Note that taking the transpose of a rank 1 array does nothing:\n",
    "v = np.array([1,2,3])\n",
    "print(v)  # Prints \"[1 2 3]\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print(v.T)  # Prints \"[1 2 3]\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Broadcasting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Broadcasting is a powerful mechanism that allows numpy to work with arrays of different shapes when performing arithmetic operations. Frequently we have a smaller array and a larger array, and we want to use the smaller array multiple times to perform some operation on the larger array.\n",
    "\n",
    "For example, suppose that we want to add a constant vector to each row of a matrix. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "x = np.array([[1,2,3], [4,5,6], [7,8,9], [10, 11, 12]])\n",
    "v = np.array([1, 0, 1])\n",
    "y = x + v  # Add v to each row of x using broadcasting\n",
    "print(y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print('x shape:', x.shape)\n",
    "print('v shape:', v.shape)\n",
    "print('y shape:', y.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The line `y = x + v` works even though `x` has shape `(4, 3)` and `v` has shape `(3,)` due to broadcasting; this line works as if v actually had shape `(4, 3)`, where each row was a copy of `v`, and the sum was performed elementwise.\n",
    "\n",
    "Broadcasting two arrays together follows these rules:\n",
    "\n",
    "If the arrays do not have the same rank, prepend the shape of the lower rank array with 1s until both shapes have the same length.\n",
    "The two arrays are said to be compatible in a dimension if they have the same size in the dimension, or if one of the arrays has size 1 in that dimension.\n",
    "The arrays can be broadcast together if they are compatible in all dimensions.\n",
    "After broadcasting, each array behaves as if it had shape equal to the elementwise maximum of shapes of the two input arrays.\n",
    "In any dimension where one array had size 1 and the other array had size greater than 1, the first array behaves as if it were copied along that dimension."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Numpy documentation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This brief overview has touched on many of the important things that you need to know about numpy, but is far from complete. Check out the numpy reference to find out much more about numpy."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Numpy exercises"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following short exercises test your understanding of simple -yet important- numpy properties and functions. Make sure you can complete them and feel free to advise the official [documentation](http://docs.scipy.org/doc/) should you need it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ========== Question 1 ==========\n",
    "Print your numpy version."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ========== Question 2 ==========\n",
    "Create a null vector of size 5."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ========== Question 3 ==========\n",
    "Create a null vector of size 5 and integer type. Set the third element is 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ========== Question 4 ==========\n",
    "Create a vector ranging from 0 to 9."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ========== Question 5 ==========\n",
    "Create a vector ranging from 10 to 29."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ========== Question 6 ==========\n",
    "Create a vector ranging from 0 to 9 and reverse it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ========== Question 7 ==========\n",
    "Create a null matrix of size (5,3)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ========== Question 8 ==========\n",
    "Crate a 3 X 3 matrix ranging from 0 to 8 and print its transpose."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ========== Question 9 ==========\n",
    "Create a 3 X 3 identity matrix."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ========== Question 10 ==========\n",
    "Create a 2 X 2 X 2 array with random values (drawn from a normal distribution)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ========== Question 11 ==========\n",
    "Create a 5 x 4 array with random values and find the minimum and maximum values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ========== Question 12 ==========\n",
    "Find the mean value of the array in 11."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ========== Question 13 ==========\n",
    "Find the row means of the array in 11."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ========== Question 14 ==========\n",
    "Find the column means of the array in 11."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ========== Question 15 ==========\n",
    "Create a list with elements 2.2, 3.5, 0, 4, 0. and convert into numpy array. Find the indices of non-zero elements."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ========== Question 16 ==========\n",
    "Crate two random matrices of shape (5,4) and (4,2). Print their matrix product."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ========== Question 17 ==========\n",
    "Crate a random matrix of shape (5,3) and a random vector of size 3. Use broadcasting to add the two arrays."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pandas introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Pandas is a library for data manipulation and analysis. There are two fundamental data structures in pandas: the **Series** and **DataFrame** structures which are built on top of NumPy arrays.\n",
    "\n",
    "The following introduction is largely based on this [tutorial](http://www.gregreda.com/2013/10/26/intro-to-pandas-data-structures/). Another useful referece is the [Pandas introduction to data structures](http://pandas.pydata.org/pandas-docs/stable/dsintro.html). Pandas is well documented and you will find good information about all methods and structures in the [API reference](http://pandas.pydata.org/pandas-docs/stable/api.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Series"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A **Series** a one-dimensional object (similar to a vector). Each element has a corresponding *index*. By default the indices range from 0 to N, where N is the length of the Series."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# First we need to import the pandas package\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Let's create a Series by passing in a list without specifying the indices.\n",
    "s = pd.Series([1, 4.2, 'Hello'])\n",
    "print(s)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Now, let's specify the indices explicitly\n",
    "s = pd.Series([1, 4.2, 'Hello'], index=['A', 'B', 'C'])\n",
    "print(s)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Indexing the Series\n",
    "s['B']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# We can also index by using boolean logic\n",
    "s[s>2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### DataFrame"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A DataFrame is a tabular data structure comprised of rows and columns. You can also think of the DataFrame as a collection of Series objects that share an index. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Creating DataFrame structures"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can create an empty DataFrame by specifying the column names. Then we can insert data row by row."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "df = pd.DataFrame(columns=['Gender', 'Age', 'Height', 'Weight'])\n",
    "print(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Now let's add an observation\n",
    "df.loc[0] = ['Male', 23, 180, 73] # Note how we used .loc to specify the index\n",
    "df.loc['A'] = ['Female', 27, 167, 59]\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Creating DataFrame from other structures"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also create a dataframe from:\n",
    "* Dict of 1D ndarrays, lists, dicts, or Series\n",
    "* 2-D numpy.ndarray\n",
    "* Structured or record ndarray\n",
    "* A Series\n",
    "* Another DataFrame\n",
    "\n",
    "For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Create a DataFrame from a list\n",
    "some_list = [['Male', 23, 180, 73], ['Female', 27, 167, 59]]\n",
    "df = pd.DataFrame(some_list, index=[0, 'A'], columns=['Gender', 'Age', 'Height', 'Weight'])\n",
    "print(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Create a DataFrame from a dictionary\n",
    "some_dict = {0:['Male', 23, 180, 73], 'A':['Female', 27, 167, 59]}\n",
    "df2 = pd.DataFrame.from_dict(some_dict, orient='index')\n",
    "df2.columns = ['Gender', 'Age', 'Height', 'Weight']\n",
    "print(df2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# By default, dictionary keys are assumed to be the column headers, above we\n",
    "# explicitly made keys the row index using the `orient` argument. Below\n",
    "# We transpose using the `.T` method\n",
    "df3 = pd.DataFrame({0:['Male', 23, 180, 73], 'A':['Female', 27, 167, 59]}).T\n",
    "df3.columns = ['Gender', 'Age', 'Height', 'Weight']\n",
    "print(df3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Loading a CSV into a DataFrame"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Most commonly we create DataFrame structures by reading csv files. To run the following piece of code you need to download the datasets associated with the course and place them in a subdirectory called \"datasets\" under the same directory that your notebooks are located. Alternatively, you can specify the full path of the .csv file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "cpu_loc = os.path.join(os.getcwd(), 'datasets', 'cpu.csv')\n",
    "cpu = pd.read_csv(cpu_loc)\n",
    "cpu.head() # Head shows the first 10 elements (unless specified otherwise) of the DataFrame"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should see that each observation in our dataset comprises 8 measurements (attributes)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Basic methods\n",
    "* `head(N)`: displays the first N elements of the DataFrame\n",
    "* `tail(N)`: displays the last N elements of the DataFrame\n",
    "* `info()`:  displays basic information about the variables\n",
    "* `describe()`: displays summary statistics of the data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Execute the following cells and observe the outputs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "cpu.tail(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "cpu.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "cpu.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Selection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can think of a DataFrame as a group of Series that share an index (in this case the column headers). This makes it easy to select specific **columns**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "cpu['MMAX'].head(5) # Display the first 5 elements"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "type(cpu['MMAX'])   # This is a Series "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To select multiple columns we simple need to pass a list of column names. The resulting object is another DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "cpu[['MMIN', 'MMAX']].head(7)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "type(cpu[['MMIN', 'MMAX']].head(7)) # This is a DataFrame"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To select specific **observations (i.e. rows)** we need to pass in the corresponding indices. This operation is called *slicing*. The resulting structure is again a DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "cpu[0:3]  # Do not forget that Python follows 0-based indexing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# This is equivalent to using .iloc\n",
    "cpu.iloc[0:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Filtering"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now suppose that you want to select all the observations which have an MMAX value which is higher than 35000. It is easy to do that:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "cpu[cpu['MMAX'] > 35000]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or equivalently:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "cpu[cpu.MMAX > 35000]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also filter the data by using multiple attributes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "cpu[(cpu.MMAX > 35000) & (cpu.MMIN > 16000)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We saw before how we can select rows by passing the index numbers. This most of the time works but very often our indices are not in linear ascending order. \n",
    "\n",
    "There are two basic methods of indexing DataFrame structures:\n",
    "* `loc`: works on labels in the index\n",
    "* `iloc`: works on the position in the index (so it only takes integers)\n",
    "\n",
    "The following example should clarify the difference between label-based indexing (`loc`) and positional indexing (`iloc`)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# First let's create a new dataframe\n",
    "cpu_new = cpu[cpu['MMAX'] > 35000]\n",
    "cpu_new"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "cpu_new.loc[8:10] # Looks for the rows which are labelled 8 and 9"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "cpu_new.iloc[0:2] # Look for the first and second rows (this yields the same result as before)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# If we try the following we will get an empty DataFrame because there are no rows with labels 0 and 1.\n",
    "cpu_new.loc[0:2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# The result is another DataFrame\n",
    "type(cpu[0:2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A very common scenario will be the following. We want to select specific observations and columns of a DataFrame and convert to a NumPy array so that we can use it for feature extraction, classification etc. This can be achieved by using the `values` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Select the first 10 observations and the \"MMIN\" and \"MMAX\" columns only and convert to numpy array.\n",
    "cpu[:10][['MMIN', 'MMAX']].values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can confirm that by using the `values` method the resulting object is a NumPy array."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pandas exercises"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ========== Question 18 ==========\n",
    "Load the `credit` dataset and display the dataset basic information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ========== Question 19 ==========\n",
    "Display the summary statistics of the attributes of the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ========== Question 20 ==========\n",
    "Display the last 6 instances of the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ========== Question 21 ==========\n",
    "Print the 5th observation (do not forget that Python follows 0-base indexing)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ========== Question 22 ==========\n",
    "Print the standard deviation of the attribute `CreditAmount` by using the numpy method `std`. You can verify you get the correct result by inspecting the statistics table from Question 19."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Matplotlib introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Matplotlib is a 2D python plotting library with a similar interface to MATLAB's plot engine. The library is fully compatible with NumPy which means that we can -and most of the times will- pass numpy arrays as input arguments to the various plotting functions.\n",
    "\n",
    "There are many [resources](http://matplotlib.org/resources/index.html) for learning how to use Matplotlib. The following examples demonstrate only some basic plotting functions. When you are looking for a particular feature which you don't know yet how to implement a web search can prove very useful."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# First we need to import the package and force it to render plots inside the notebook\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# The first example creates a sine and a cosine and plots them.\n",
    "X = np.linspace(-np.pi, np.pi, 256,endpoint=True) # Evenly spaced numbers over the specified interval\n",
    "C, S = np.cos(X), np.sin(X) # Create a sine (S) and a cosine (C)\n",
    "\n",
    "plt.figure(figsize=(8,5)) # Instantiates figure with a specified size\n",
    "plt.plot(X,C, label='Cosine', color='blue')\n",
    "plt.plot(X,S, label='Sine', color='red')\n",
    "plt.legend() # Displays legend\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Create 4 subplots and plot cosines with different colours, linestyels and linewidths.\n",
    "X = np.linspace(-np.pi, np.pi, 256,endpoint=True) # Evenly spaced numbers over the specified interval\n",
    "C1, C2, C3, C4 = np.cos(X), np.cos(X+0.5), np.cos(X+1.), np.cos(X+1.5)\n",
    "\n",
    "f, ax = plt.subplots(figsize=(7,7))\n",
    "plt.subplot(2,2,1)\n",
    "plt.plot(X, C1)\n",
    "plt.title('C1')\n",
    "plt.xticks([]) # Removes xticks\n",
    "plt.subplot(2,2,2)\n",
    "plt.plot(X, C2, color='green',  linestyle='-.')\n",
    "plt.title('C2')\n",
    "plt.yticks([]) # Removes yticks\n",
    "plt.subplot(2,2,3)\n",
    "plt.plot(X, C3, color='k', linestyle='--')\n",
    "plt.title('C3')\n",
    "plt.subplot(2,2,4)\n",
    "plt.plot(X, C4, color='m', linewidth = 5.)\n",
    "plt.title('C4')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Plot a histogram and add some text\n",
    "mu, sigma = 100, 15\n",
    "x = mu + sigma * np.random.randn(10000) # Generate data from a normal distribution with mean mu and variance sigma\n",
    "\n",
    "# the histogram of the data\n",
    "n, bins, patches = plt.hist(x, 50, normed=1, facecolor='g', alpha=0.75)\n",
    "\n",
    "plt.xlabel('Smarts')\n",
    "plt.ylabel('Probability')\n",
    "plt.title('Histogram of IQ')\n",
    "plt.text(60, .025, r'$\\mu=100,\\ \\sigma=15$') # Adds some text by using TeX\n",
    "plt.axis([40, 160, 0, 0.03]) # Sets the axes limits\n",
    "plt.grid(True) # Enables grid\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Matplotlib exercises"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ========== Question 23 ==========\n",
    "Initialise an empty figure of size (8,6). Create a scatter plot of the observations in the `credit` dataset for the attributes `Duration` and `Age` (age should be shown on the xaxis). Use black coloured markers. Label the axes accordingly by using red colour for the xaxis and green for the yaxis. Finally, add a title of your choice by using a font size of 20. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ========== Question 24 ==========\n",
    "You should note above that there is an outlier (i.e. an observation that is very distant to the other observations). Now you think that there must be something wrong with that observation (why?) and as a next step you should remove it from both the age and duration variables and a make a scatter plot of the new data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Seaborn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Seaborn](https://seaborn.github.io/) is a visualisation library built on top of matplotlib which offers some aesthetic enhancement and, more importantly, provides some high-level functions for  \"exploring and understanding data\". Seaborn is also tightly integrated with pandas and provides support for both numpy and pandas data structures.\n",
    "\n",
    "In the following example we will load seaborn and use its default settings to plot again the sine and cosine waves we saw earlier.\n",
    "\n",
    "Note that this example does not actually use any seaborn functions, but as soon as the library is loaded, the Seaborn plotting style is adopted by the matplotlib.pyplot library."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import seaborn as sns # Load seaborn with default settings\n",
    "X = np.linspace(-np.pi, np.pi, 256,endpoint=True) # Evenly spaced numbers over the specified interval\n",
    "C, S = np.cos(X), np.sin(X) # Create a sine (S) and a cosine (C)\n",
    "plt.figure(figsize=(8,5))\n",
    "plt.plot(X,C, label='Cosine')\n",
    "plt.plot(X,S, label='Sine')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Seaborn implemented functions [Optional]\n",
    "The Seaborn website has many great examples. Below we show a few taken directly from the website. These are just to give you a flavour of the kinds of things that Seaborn can do. You can use it as reference later; by no means should you learn this code by heart! You should definitely not worry if some of the following commands do not make sense just yet."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "sns.set(style=\"white\", palette=\"muted\", color_codes=True)\n",
    "rs = np.random.RandomState(10)\n",
    "\n",
    "# Set up the matplotlib figure\n",
    "f, axes = plt.subplots(2, 2, figsize=(7, 7), sharex=True)\n",
    "sns.despine(left=True)\n",
    "\n",
    "# Generate a random univariate dataset\n",
    "d = rs.normal(size=100)\n",
    "\n",
    "# Plot a simple histogram with binsize determined automatically\n",
    "sns.distplot(d, kde=False, color=\"b\", ax=axes[0, 0])\n",
    "\n",
    "# Plot a kernel density estimate and rug plot\n",
    "sns.distplot(d, hist=False, rug=True, color=\"r\", ax=axes[0, 1])\n",
    "\n",
    "# Plot a filled kernel density estimate\n",
    "sns.distplot(d, hist=False, color=\"g\", kde_kws={\"shade\": True}, ax=axes[1, 0])\n",
    "\n",
    "# Plot a historgram and kernel density estimate\n",
    "sns.distplot(d, color=\"m\", ax=axes[1, 1])\n",
    "\n",
    "plt.setp(axes, yticks=[])\n",
    "plt.tight_layout()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "sns.set(style=\"white\")\n",
    "\n",
    "# Generate a random correlated bivariate dataset\n",
    "rs = np.random.RandomState(5)\n",
    "mean = [0, 0]\n",
    "cov = [(1, .5), (.5, 1)]\n",
    "x1, x2 = rs.multivariate_normal(mean, cov, 500).T\n",
    "x1 = pd.Series(x1, name=\"$X_1$\")\n",
    "x2 = pd.Series(x2, name=\"$X_2$\")\n",
    "\n",
    "# Show the joint distribution using kernel density estimation\n",
    "g = sns.jointplot(x1, x2, kind=\"kde\", size=7, space=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "sns.set(style=\"whitegrid\", palette=\"pastel\", color_codes=True)\n",
    "\n",
    "# Load the example tips dataset\n",
    "tips = sns.load_dataset(\"tips\")\n",
    "\n",
    "# Draw a nested violinplot and split the violins for easier comparison\n",
    "sns.violinplot(x=\"day\", y=\"total_bill\", hue=\"sex\", data=tips, split=True,\n",
    "               inner=\"quart\", palette={\"Male\": \"b\", \"Female\": \"y\"})\n",
    "sns.despine(left=True)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "sns.set(style=\"darkgrid\")\n",
    "\n",
    "# Load the example titanic dataset\n",
    "df = sns.load_dataset(\"titanic\")\n",
    "\n",
    "# Make a custom palette with gendered colors\n",
    "pal = dict(male=\"#6495ED\", female=\"#F08080\")\n",
    "\n",
    "# Show the survival proability as a function of age and sex\n",
    "g = sns.lmplot(x=\"age\", y=\"survived\", col=\"sex\", hue=\"sex\", data=df,\n",
    "               palette=pal, y_jitter=.02, logistic=True)\n",
    "g.set(xlim=(0, 80), ylim=(-.05, 1.05))\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "sns.set()\n",
    "\n",
    "# Load the example flights dataset and convert to long-form\n",
    "flights_long = sns.load_dataset(\"flights\")\n",
    "flights = flights_long.pivot(\"month\", \"year\", \"passengers\")\n",
    "\n",
    "# Draw a heatmap with the numeric values in each cell\n",
    "sns.heatmap(flights, annot=True, fmt=\"d\", linewidths=.5)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "sns.set(style=\"darkgrid\")\n",
    "\n",
    "tips = sns.load_dataset(\"tips\")\n",
    "g = sns.FacetGrid(tips, row=\"sex\", col=\"time\", margin_titles=True)\n",
    "bins = np.linspace(0, 60, 13)\n",
    "g.map(plt.hist, \"total_bill\", color=\"steelblue\", bins=bins, lw=0)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "sns.set(style=\"ticks\")\n",
    "\n",
    "# Create a dataset with many short random walks\n",
    "rs = np.random.RandomState(4)\n",
    "pos = rs.randint(-1, 2, (20, 5)).cumsum(axis=1)\n",
    "pos -= pos[:, 0, np.newaxis]\n",
    "step = np.tile(range(5), 20)\n",
    "walk = np.repeat(range(20), 5)\n",
    "df = pd.DataFrame(np.c_[pos.flat, step, walk],\n",
    "                  columns=[\"position\", \"step\", \"walk\"])\n",
    "\n",
    "# Initialize a grid of plots with an Axes for each walk\n",
    "grid = sns.FacetGrid(df, col=\"walk\", hue=\"walk\", col_wrap=5, size=1.5)\n",
    "\n",
    "# Draw a horizontal line to show the starting point\n",
    "grid.map(plt.axhline, y=0, ls=\":\", c=\".5\")\n",
    "\n",
    "# Draw a line plot to show the trajectory of each random walk\n",
    "grid.map(plt.plot, \"step\", \"position\", marker=\"o\", ms=4)\n",
    "\n",
    "# Adjust the tick positions and labels\n",
    "grid.set(xticks=np.arange(5), yticks=[-3, 3],\n",
    "         xlim=(-.5, 4.5), ylim=(-3.5, 3.5))\n",
    "\n",
    "# Adjust the arrangement of the plots\n",
    "grid.fig.tight_layout(w_pad=1)"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}