Worksheet 3, Pandas.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "this worksheets is part of the [mlvu machine learning course](https://mlvu.github.io)<br>\n",
    "setting up your environment: https://bit.ly/3bzpn5C\n",
    "\n",
    "For this worksheet, we'll need to install the pandas package. Run the cell below, or run ```pip install pandas``` in the terminal/command-line/command prompt."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install pandas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Worksheet 3: Pandas\n",
    "\n",
    "Pandas is a python package for data analysis. Where numpy is a package built around a the data structure of a <em>matrix</em>, pandas is a package built around the data structure of a _dataframe_. A dataframe is a lot like a matrix, with some key differences:\n",
    "\n",
    "* In a data frame, the columns have header names. \n",
    "* Different datatypes (int, string, boolean) are allowed within the same dataframe. Each column has its own datatype\n",
    "\n",
    "In short, dataframes represent datasets of the kind we've seen in the lectures: an instance per row, and a feature per column. \n",
    "\n",
    "Pandas is designed to work together with numpy and matplotlib. Let's import all of them"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To explore Pandas we'll import the ANSUR II dataset (we also used this to create the examples in the first lecture). ANSUR II is an _anthropometric_ dataset: it contains body measurements. ANSUR II contains 108 measurements for about 4000 men and about 2000 women (all US soldiers).\n",
    "\n",
    "We'll start by reading the data. Like numpy, pandas has a function for reading CSV files. Pandas' function is much more robust, and much less likely to give you trouble. It does come with [a lot of options](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_csv.html), so you may have to try a few things before you get it to read your data accurately.\n",
    "\n",
    "ANSUR II comes in two separate tables: one for male soldiers, and one for female soldiers:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "female = pd.read_csv('./ansur/ANSUR II FEMALE Public.csv')\n",
    "male = pd.read_csv('./ansur/ANSUR II MALE Public.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All the warnings from the first two social impact videos, and from the fourth lecture apply here. This is not a sample that is representative of the population, and the gender class is a sensitive attribute that is poorly captured by two classes.\n",
    "\n",
    "```male``` and ```female``` are pandas DataFrame objects. Jupyter notebooks will print these as tables:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "male"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "These are big dataframes. Pandas can give us some quick summary statistics per column very easily."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "male.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "In pandas these columns are called <em>Series</em>, and a dataframe is basically a list of Series objects with the same length (indexed in various ways for efficient access).\n",
    "\n",
    "Let's have a look at all the available measurements."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i, col in enumerate(female.columns):\n",
    "    print(i, col)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each column name with a lowercase letter represents a physical measurement. Some of these are quite technical (like <em>bizygomaticbreadth</em>). The dataset comes with a very helpful document that shows what each measurement means (and how it should be performed). It's [included with the worksheets](./ansur/Hotzman_2011_ANSURIII_Measurements_a548497.pdf). Scroll down to section 6.4 for the description of the measurements."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once the dataframe is loaded, you can refer to the columns by name as python objects. For instance:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "male.stature"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Jupyter notebooks even gives you dynamic autocomplete. Try putting your cursor at the end of the next line and pressing the ```TAB``` button on your keyboard (it may take a second)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "female."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now easily do scatterplots of different measurements. Let's plot the ```stature``` (height) against the ```span``` (distance between outstretched arms)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.scatter(female.stature, female.span, color='red')\n",
    "plt.scatter(male.stature, male.span, color='blue');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Even if we make the points transparent (```alpha=0.1```) and small (```s=1```), it's quite a dense cloud. We can easily select a small subset by [slicing](https://www.pythoncentral.io/how-to-slice-listsarrays-and-tuples-in-python/):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "male[:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that slicing like this only works over the rows. We can't do ```male[:3, 5:12]```, like we could in numpy.\n",
    "\n",
    "For now, we know enough to plot a small subset of the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "female_sub = female[:50]\n",
    "male_sub = male[:50]\n",
    "\n",
    "plt.scatter(female_sub.stature, female_sub.span, color='red')\n",
    "plt.scatter(male_sub.stature, male_sub.span, color='blue');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To select a set of columns, the best method is to pass a list of strings containing column names. The result is another dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "male[['bicepscircumferenceflexed', 'Age']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Simple arithmetic\n",
    "\n",
    "Like numpy, pandas objects overload basic arithmetic operations. For instance, the units in this dataset are in millimeters, which is a little hard to read. To convert them to meters, we can simply multiply by 0.001.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "stat = female.stature * 0.001\n",
    "span = female.span * 0.001\n",
    "\n",
    "plt.scatter(stat, span);\n",
    "\n",
    "plt.xlabel('height (m)')\n",
    "plt.ylabel('span (m)');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Descriptive statistics\n",
    "\n",
    "For most descriptive statistics, pandas provides member functions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('mean           ', female.stature.mean())\n",
    "print('std dev.       ', female.stature.std())\n",
    "print('median         ', female.stature.median())\n",
    "print('standard error ', female.stature.sem())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Concatenating and sampling\n",
    "\n",
    "To perform the classification task from the first lecture (predicting gender from physical measurements), we want the male and female data in a single dataframe. To accomplish this, we can concatenate the two dataframes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "people = pd.concat([male, female])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This gives us a dataset of all the male measurements first, and then all the female measurements. For many reasons, it's helpful to shuffle these, so that the order is random. The simplest way to do this is pandas is to _sample_ a new dataframe (without replacement) of the same size:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "people = people.sample(frac=1)\n",
    "people[:5].Gender"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the row indices from the original dataframe are retained and shuffled as well. For our purposes this doesn't matter."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Performing Classification\n",
    "\n",
    "### Categories and codes\n",
    "\n",
    "To perform classification on this dataset, we need to convert the target value from strings to categorical data.\n",
    "\n",
    "Since gender is a sensitive attribute and we're just looking for any example, let's instead try to predict handedness. This is indicated by the attribute ```WritingPreference```.\n",
    "\n",
    "Right now, pandas thinks the column can have any string value: when we convert it, it checks the existing values (```Left Hand```, ```Right hand``` and ```Either hand (No preference)```) and limits the column values to those two options (changing the datatype)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wp_cat = people.WritingPreference.astype('category')\n",
    "print(wp_cat.dtype)\n",
    "print(wp_cat.cat.categories)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that we haven't changed the original data. To insert the categorized column back into the original dataframe, we just re-assign it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "people.WritingPreference = people.WritingPreference.astype('category')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can quickly check the class balance using the ```value_counts()``` function. (For a more fancy display, try the hist() function)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "people.WritingPreference.value_counts(normalize=True) \n",
    "\n",
    "# normalize=False will give you absolute counts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What does this tell us about the performance of the majority class baseline?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For many tasks (including classification with sklearn), we need integers instead of categorical values. Pandas actually uses integer codes behind the scenes for its categories and it's a simple matter to get a column of integers from a column of categorical data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "people.WritingPreference.cat.codes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This allows us, for instance, to scatterplot the data using the categories for color."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can add a cmap argument like ```cmap='copper'``` to change the colors. A [colormap](https://matplotlib.org/examples/color/colormaps_reference.html) maps a range of numeric values to a range of colors. I our case, we only have the values 0 and 1, so those get mapped to the extremes of the chosen colormap."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sub = people[:500]\n",
    "plt.scatter(sub.stature, sub.span, c=sub.WritingPreference.cat.codes, alpha=0.3);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Clearly, stature and span are not very predictive. Perhaps the measurements of the right arm will tell us a little more??"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sub = people[:500]\n",
    "plt.scatter(sub.bicepscircumferenceflexed, sub.bideltoidbreadth, c=sub.WritingPreference.cat.codes, alpha=0.3);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Clearly, this is a difficult problem. It may even be impossible from the features we have. Let's try anyway."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Classification\n",
    "\n",
    "Running a classifier looks much the same as it did with a numpy array."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.svm import SVC\n",
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "train = people[:4000]\n",
    "test  = people[4000:]\n",
    "# NB: We can split like this because we know the data is shuffled\n",
    "\n",
    "cls = SVC()\n",
    "cls.fit(train[['stature','span']], train.WritingPreference)\n",
    "\n",
    "accuracy_score(cls.predict(test[['stature', 'span']]), test.WritingPreference)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Very slightly higher than the majority class (though your results may differ). Can we conclude that handedness can be predicted from height and span?\n",
    "\n",
    "Let's see what we get for a kNN classifier on all measurements."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.neighbors import KNeighborsClassifier as KNN\n",
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "train = people[:4000]\n",
    "test  = people[4000:]\n",
    "# NB: We can split like this because we know the data is shuffled\n",
    "\n",
    "traina, testa = train.iloc[:, 1:94], test.iloc[:, 1:94] # select all measurement columns\n",
    "\n",
    "cls = KNN(n_neighbors=1)\n",
    "cls.fit(traina, train.WritingPreference)\n",
    "\n",
    "accuracy_score(cls.predict(testa), test.WritingPreference)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lower than the majority class. Note that with the number of neigbhors at 1, kNN is likely to overfit. See what happens if you increase the number of neighbors.\n",
    "\n",
    "At this point, we're in danger of multiple testing, so we should make a proper train/validation/test split. Can you see how you would do that?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, sklearn integrates beautifully with pandas, making our training and testing code even simpler. Not all libraries integrate this well with pandas; mlxtend, for instance, only inderstands numpy data.\n",
    "\n",
    "Happily, pandas data contains numpy arrays in the background, and we can simply ask for those by retrieving the ```.values``` attribute."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from mlxtend.plotting import plot_decision_regions\n",
    "\n",
    "# Plot the decision boundary with the first 50 points in the test set\n",
    "numpy_x = train[['stature','span']].values\n",
    "numpy_y = train.WritingPreference.cat.codes.values\n",
    "\n",
    "# This is necessary if pandas read the CSV files as integers\n",
    "# (seems to depend on version/OS)\n",
    "numpy_x = numpy_x.astype(float)\n",
    "\n",
    "# Rebuild the classifier \n",
    "# (a classifier trained on pandas data doesn't interoperate well with pure numpy data)\n",
    "tree = KNN(n_neighbors=2)\n",
    "tree.fit(numpy_x, numpy_y)\n",
    "\n",
    "plot_decision_regions(numpy_x[:25, :], numpy_y[:25], clf=tree);\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Final comments\n",
    "\n",
    "As usual, there is a lot more to learn. A good place to start is the 10-minute quicktstart guide to pandas:\n",
    "https://pandas.pydata.org/pandas-docs/stable/10min.html\n",
    "\n",
    "One very useful feature we didn't mention is _grouping_ (which will be familiar if you've done a little SQL):\n",
    "https://pandas.pydata.org/pandas-docs/stable/groupby.html\n",
    "\n",
    "Here's a list of 12 random tips, which gives you a good idea of how far pandas can go:\n",
    "https://www.analyticsvidhya.com/blog/2016/01/12-pandas-techniques-python-data-manipulation/\n",
    "\n",
    "Next week, deep learning with _Keras_."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}