5_arithmetic_ops_and_data_alignment.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Qcm-BNiPgSsM",
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# ICS 434: DATA SCIENCE FUNDAMENTALS\n",
    "\n",
    "## Arithmetic Operations and Data Alignment\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Ft4T8eRUiJW9",
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Vectorization\n",
    "\n",
    "A vectorized operation applies an operation to a list of elements at once\n",
    "  * Implemented on `numpy` library\n",
    "  * Easily extends to Pandas, which uses NumPy to store data\n",
    "* Taking advantage of the parallel capabilities of modern CPUs, it is much more efficient than Python *for loops*\n",
    "  * Automatically handles collection of data in parallel\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Vectorization -- Cont'd\n",
    "\n",
    "* A `Series` contains only homogeneous data types\n",
    "* `Series` operations are delegated to optimized and compiled C code. \n",
    "  * This process is known as vectorization. \n",
    "  * The result is a tremendous speedup compared to an analogous computation in Python, which:\n",
    "    * Iterates through the arrays\n",
    "    * Each item is painstakingly checked for its data type \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "x = pd.Series(range(1_000_000))\n",
    "type(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "a = range(1_000_000)\n",
    "b = []\n",
    "for elem in a:\n",
    "    b.append(elem**2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "y = x**2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### The Numerical Python (NumPy) Library\n",
    "\n",
    "* `numpy` is a Python library for array-based computing implemented in the `C` language\n",
    "  * The *de facto* standard for scientific computing\n",
    "    * Foundation for the Python scientific stack\n",
    "  \n",
    "* Pandas uses `numpy` to store the data behind the scenes    \n",
    "\n",
    "```python\n",
    ">>> x = pd.Series([1, 2, 3])\n",
    ">>> print(type(x.values))\n",
    "\n",
    "<class 'numpy.ndarray'>\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "x = pd.Series([1, 2, 3])\n",
    "print(type(x.values))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### The Numerical Python (NumPy) Library -- Cont'd\n",
    "\n",
    "* Python lists and `numpy` arrays are similar in structure. Numpy arrays differ in that they are densely packed with uniform values, as opposed to references to objects. This is especially useful:\n",
    "  * When working with large amounts of data\n",
    "  * Relatively small data fits in cache\n",
    "  * Data can be traversed relatively quickly\n",
    "* `numpy` objects have various `methods` to work with the data they contain"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Broadcasting\n",
    "\n",
    "* Broadcasting is simply a set of rules for applying binary functions (e.g., addition, subtraction, multiplication, etc.) on arrays of different sizes\n",
    "* Broadcasting follows a strict set of rules to determine the interaction between the two arrays\n",
    "  * When working with arrays in one dimension, the rules are fairly simple:\n",
    "    1. If the shape of the two arrays does not match, the array with a shape equal to 1 is stretched to match the second array\n",
    "    2. If sizes disagree and neither is equal to 1, an error is raised"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "x = np.array([1, 2, 3])\n",
    "y = np.array([3, 2, 1])\n",
    "x+y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "x = np.array([1, 2, 3])\n",
    "y = np.array([1])\n",
    "x+y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# Second condition throws and error\n",
    "x = np.array([1, 2, 3])\n",
    "y = np.array([1, 2])\n",
    "x+y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Broadcasting\n",
    "\n",
    "<img src=\"https://www.dropbox.com/scl/fi/8fig4p3t4oz0el5vdbqkp/broadcasting.png?rlkey=qr4v5205nyuas81ke7rscy1ao&dl=1\" alt=\"drawing\" style=\"width:500px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Vectorization: Arithmetic Operations\n",
    "\n",
    "* Arithmetic operations in `pandas`  are vectorized\n",
    "  * You can sum or multiply two columns and the operation would be vectorized, i.e., parallelized\n",
    "  \n",
    "* We explore using a trivial dataset\n",
    "\n",
    "```python\n",
    "df_1 = pd.DataFrame({'AA':{'A':79, 'C':2, 'T':12, 'X':21},\n",
    "                     'BB':{'A':11, 'C':2, 'T':2, 'X':9}})\n",
    "df_1[\"AA\"] + df_2[\"BB\"]                  \n",
    "```\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "nPMiIVzJy7cc",
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "df_1 = pd.DataFrame({'AA':{'A':79, 'C':2, 'T':13, 'X':21},\n",
    "                     'BB':{'A':11, 'C':2, 'T':2, 'X':9}})\n",
    "\n",
    "df_1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "df_1[\"AA\"] + df_1[\"BB\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Arithmetic Operations -- Cont'd\n",
    "\n",
    "* Are not limited to arithmetic operations on the data with the same index \n",
    "  * i.e., `Series` of the same size\n",
    "\n",
    "```python\n",
    "df_2 = pd.DataFrame({'AA':{'A':21, 'D':14, 'T':5},\n",
    "                     'CC':{'A':12, 'D':28, 'T':121}})\n",
    "df_2\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "df_2 = pd.DataFrame({'AA':{'A':21, 'D':14, 'T':5},\n",
    "                     'CC':{'A':12, 'D':28, 'T':121}})\n",
    "df_2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "df_1[\"AA\"]  + df_2[\"AA\"] "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "YfChLH62O48W",
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Vectorized Arithmetic Between `Series` (`DataFrame` Column)\n",
    "\n",
    "* Column-wise vectorized operations require aligning the data on the index\n",
    "  * A new index is created from the union of the indices of both `Series`\n",
    "  * Missing values in either one of the Series are filled with missing values (`NaN`)\n",
    "\n",
    "<img src=\"https://www.dropbox.com/scl/fi/r0aeoaqfwblq3in3ymlkw/alignment_arithmetic_col.png?rlkey=iosbgr4rknf7mj8vsndwgcypt&dl=1\" alt=\"drawing\" style=\"width:500px\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ErZIn1TSP5KI",
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Vectorized Arithmetic Between `Series` (`DataFrame` Row)\n",
    "\n",
    "* Operations on row `Series` work the same way as columns \n",
    "\n",
    "```python\n",
    "df_1.loc[\"A\"] + df_2.loc[\"D\"]\n",
    "```\n",
    "<img src=\"https://www.dropbox.com/scl/fi/bdmi1636ihr5t54kffj9i/E5_5_alignment_arithmetic_row.png?rlkey=w1xfklk9jjag0fo4uf1ba3lqm&dl=1\" alt=\"drawing\" style=\"width:700px\">\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "41SxTmdV5wvU",
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "df_1.loc[\"A\"] + df_2.loc[\"D\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "PIG2G5gkPADE",
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Vectorized Arithmetic on DataFrames\n",
    "\n",
    "* We can also perform arithmetic operations on the whole `DataFrame`\n",
    "  * This is simply an extension of the alignments on columns or rows\n",
    " \n",
    "\n",
    "```python\n",
    "df_1 + df_2\n",
    "```\n",
    "\n",
    "<img src=\"https://www.dropbox.com/scl/fi/d5nq66w06u4c7v1y20plt/alignment_arithmetic_df.png?rlkey=avbdrlqpq77iof3cqtpoowogc&dl=1\" alt=\"drawing\" style=\"width:750px\">\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "df_1 = pd.DataFrame({'AA':{'A':79, 'C':2, 'T':12, 'X':21},\n",
    "                     'BB':{'A':11, 'C':2, 'T':2, 'X':9}})\n",
    "df_2 = pd.DataFrame({'AA':{'A':21, 'D':14, 'T':5},\n",
    "                     'BB':{'A':12, 'D':28, 'T':121}})\n",
    "df_1 + df_2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Question? \n",
    "\n",
    "* What would the outcome be if a column is present in one `DataFrame` and not in the other?\n",
    "\n",
    "* Ex.:  df_1 + df_2?\n",
    "<img src=\"https://www.dropbox.com/scl/fi/ppquie519614d9n6aiwnq/question.png?rlkey=3ox0vktrl3clfxji5k2h8v9h3&dl=1\" alt=\"drawing\" style=\"width:500px\">\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "df_2 = pd.DataFrame({'AA':{'A':21, 'D':14, 'T':5},\n",
    "                     'CC':{'A':12, 'D':28, 'T':121}})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 202
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 408,
     "status": "ok",
     "timestamp": 1534192063202,
     "user": {
      "displayName": "Charles Dickens",
      "photoUrl": "//lh6.googleusercontent.com/-LyFY-SmW2Wk/AAAAAAAAAAI/AAAAAAAAAJc/aBlebAB4Y_M/s50-c-k-no/photo.jpg",
      "userId": "116926057727252856181"
     },
     "user_tz": 600
    },
    "id": "oUlho1G3ne1P",
    "outputId": "643ae759-cf1d-4491-9e28-b60bd60f3cad",
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "df_1 + df_2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "CKrdPyVogezm",
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Vectorization Example with the Medical Spending Data Set\n",
    "\n",
    "* An arithmetic operation between two `Series` yields a `Series`\n",
    "  * We can use the resulting `Series` in new computations\n",
    "  \n",
    "\n",
    "* Example: we can compute the average spending per beneficiary of the `spending_df` `DataFrame` using the following step:\n",
    "  1. For each `unique_id` (row), we can divide `spending` by  `nb_beneficiaries`\n",
    "    * This gives us the spending for each beneficiary for each `unique_id`\n",
    "  2. Compute the mean of the resulting values\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "spending_df = pd.read_csv('data/spending_10k.csv', index_col='unique_id')\n",
    "spending_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "spending_benef =  spending_df['spending'] / spending_df['nb_beneficiaries']\n",
    "\n",
    "type(spending_benef)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "spending_benef"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "spending_benef.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "O9yZ2OYGftw6",
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# operation chaining\n",
    "(spending_df['spending'] / spending_df['nb_beneficiaries']).mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "r5pyeKdsiMyT",
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Broadcasting\n",
    "\n",
    "* What happens with an arithmetic operation that involves a `Series` and a single value (`scalar`)?\n",
    "  * The logic is the same for `DataFrame` and a single value\n",
    "\n",
    "* The `scalar` is expanded (broadcasted) to fit the dimension of the `Series` (or `DataFrame`)\n",
    "  \n",
    "```python\n",
    "df_1['AA'] + 1.2\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "UYL_NlZmnmFu",
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Broadcasting -- Cont'd\n",
    "\n",
    "<img src=\"https://www.dropbox.com/scl/fi/g7k95046sa33mbiqsao7y/alignment.png?rlkey=qwzuulgxwnwknx4u1yr1nrivf&dl=1\" alt=\"drawing\" style=\"width:600px\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "scMKa-1MgEWn",
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Subsetting\n",
    "\n",
    "\n",
    "* We often need to subset a dataset and we know how to subset a dataset to extract a single row\n",
    "\n",
    "```python \n",
    "spending_df.iloc[1]\n",
    "```\n",
    "* We know how to subset a dataset to extract a range of rows\n",
    "\n",
    "```python \n",
    "spending_df.iloc[1:32]\n",
    "```\n",
    "\n",
    "* But it's also very useful to subset the data based on some condition:\n",
    "  * Extract all the entries in cardiology\n",
    "  * How many flights out of HNL were going to Nebraska?\n",
    "  * How many Biki bicycles were checked out in a specific geographical region (e.g., Manoa)?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "dCjOGvOmh7OG",
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Comparison Operations\n",
    "\n",
    "* Comparison operators (\"`<`\" , \"`>`\" , \"`==`\" , \"`>=`\" , \"`<=`\" , \"`!=`\") are applied in the same fashion as arithmetic operations \n",
    "\n",
    "  * Result in Booleans (True or False)\n",
    "\n",
    "```python\n",
    ">>> df_3 = pd.DataFrame({'AA':{'A':64, 'C':2, 'T':6, 'X':22}, \n",
    "                         'BB':{'A':3,  'C':2, 'T':7, 'X':12}})\n",
    "```\n",
    "\n",
    "```python\n",
    ">>> df_1.loc[:, \"AA\"] > df_3.loc[:, \"AA\"]\n",
    "A     True\n",
    "C    False\n",
    "T     True\n",
    "X    False\n",
    "Name: AA, dtype: bool\n",
    "```\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Comparison Result\n",
    "\n",
    "* Result of the comparison is a `Series` with the same indices in `df_1` and `df_3` \n",
    "  * IMPORTANT: comparisons between two `Series` or `DataFrames` can only be carried out if both objects are identically labeled\n",
    "  * Have the same row **and** column names\n",
    "\n",
    "* All entries are of type `bool`\n",
    "  * `True` is the position passed the comparison test and `False` otherwise\n",
    "\n",
    "```python\n",
    ">>> df_1.loc[:, \"AA\"] > df_2.loc[:, \"AA\"]\n",
    "...\n",
    "ValueError: Can only compare identically-labeled Series objects\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "EDN5mfqQn19O",
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "df_1 = pd.DataFrame({'AA':{'A':79, 'C':2, 'T':13, 'X':21},\n",
    "                     'BB':{'A':11, 'C':2, 'T':2, 'X':9}})\n",
    "df_3 = pd.DataFrame({'AA':{'A':64, 'C':2, 'T':6, 'X':22},\n",
    "                     'BB':{'A':3,  'C':2, 'T':7, 'X':12}})\n",
    "\n",
    "df_1[\"AA\"] > df_3[\"AA\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "## Not the same number of rows now the same index\n",
    "\n",
    "df_4 = pd.DataFrame({'AA':{'I':64, 'J':2, 'K':6}})\n",
    "\n",
    "df_1[\"AA\"] > df_4[\"AA\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "E-MJyvA0V_kc",
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Comparison Operations  and Indexing\n",
    "\n",
    "* Comparison operators are ideal for querying and subsetting\n",
    "\n",
    "  * SQL-like functionality although at much lower efficiency (both RAM and CPU intensive compared to dedicated DBMS)\n",
    "  \n",
    "\n",
    "* We can subset a `Series` using a *list* (`[]`) of `Boolean`s that has the same size\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "x = pd.Series([95, 23, 67])\n",
    "\n",
    "my_filter = pd.Series([True, False, True])\n",
    "\n",
    "x[my_filter]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "X71kcE8ZpMWx",
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "boolean_list = [True, True, False, False]\n",
    "df_1.loc[:, 'AA'][boolean_list]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "df_1['AA'] > 3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Comparison Operations  and Indexing -- Cont'd\n",
    "\n",
    "* Subsetting can also leverage broadcasting\n",
    "<img src=\"https://www.dropbox.com/scl/fi/ly9rt948jrdznobeycatb/E4_2_filter_dataframe.png?rlkey=rbx13q2m3rwi8nr5ddzrhyhx3&dl=1\" alt=\"drawing\" style=\"width:600px\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "3YRBdeK5h--o",
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Boolean Operations\n",
    "\n",
    "* Conditional expressions allow us to create more sophisticated queries for our dataset\n",
    "  * Queries commonly rely on boolean logic (boolean operators)\n",
    "\n",
    "* Pandas' `Boolean` operations are slightly different from Python's\n",
    "\n",
    " * `and` is replaced with `&` \n",
    " * `or` is replaced with `|`\n",
    " * `not` is replaced with `~`\n",
    " \n",
    "* The operators above return a boolean object of the same shape as the input"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Boolean Operations Example\n",
    "\n",
    "* Find instances in df_1 where values in columns \"AA\" and \"BB\" are greater than 3.\n",
    "```python\n",
    ">>> true_false_series = (df_1['AA'] > 3) & (df_1['BB'] > 3) \n",
    ">>> true_false_series\n",
    "A     True\n",
    "C    False\n",
    "T    False\n",
    "X     True\n",
    "dtype: bool\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "true_false_series = (df_1['AA'] > 3) & (df_1['BB'] > 3) \n",
    "true_false_series"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "df_1[true_false_series]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "df_1[(df_1['AA'] > 3) & (df_1['BB'] > 3)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "3HzjmNVfuWXi",
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Boolean Operations Examples with the Medical Spending Data Set\n",
    "\n",
    "\n",
    "* Let's subset the `spending_df` to retain rows that:\n",
    "\n",
    "  * Have 'DENTIST' as `specialty` \n",
    "  * Have spending below \\$100 \n",
    "  \n",
    "* Save results to a new `DataFrame`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Examples - Cont'd\n",
    "\n",
    "* Have 'DENTIST' as `specialty`\n",
    "\n",
    "```python\n",
    "spending_df['specialty'] == 'DENTIST'\n",
    "``` "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "(spending_df['specialty'] == 'DENTIST').head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Examples -- Cont'd\n",
    "\n",
    "* Have spending below \\$100 \n",
    "    \n",
    "```python \n",
    "spending_df['spending'] < 100\n",
    "``` "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "(spending_df['spending'] < 100).head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "### Examples -- Cont'd\n",
    "\n",
    "\n",
    "* Combine both conditions\n",
    "\n",
    "```python\n",
    "(spending_df.loc[:, 'specialty'] == 'DENTIST') & (spending_df.loc[:, 'spending'] < 100)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Examples -- Cont'd\n",
    "\n",
    "\n",
    "* Combine both conditions\n",
    "\n",
    "```python\n",
    "(spending_df['specialty'] == 'DENTIST') & (spending_df['spending'] < 100)\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "(spending_df['specialty'] == 'DENTIST') & (spending_df['spending'] < 100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "### Examples -- Cont'd\n",
    "\n",
    "* Subset the original dataset\n",
    "\n",
    "```python\n",
    "dentist_and_small_spending = (spending_df.loc[:, 'specialty'] == 'DENTIST') & (spending_df.loc[:, 'spending'] < 100)\n",
    "spending_df.loc[dentist_and_small_spending]\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Examples -- Cont'd\n",
    "\n",
    "* Subset the original dataset\n",
    "\n",
    "```python\n",
    "dentist_and_small_spending = (spending_df['specialty'] == 'DENTIST') & (spending_df['spending'] < 100)\n",
    "spending_df.loc[dentist_and_small_spending]\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "dentist_and_small_spending = (spending_df['specialty'] == 'DENTIST') & (spending_df['spending'] < 100)\n",
    "spending_df.loc[dentist_and_small_spending].head()"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "colab": {
   "collapsed_sections": [
    "0LJYevev1Ppg"
   ],
   "name": "3_Exploring_Data.ipynb",
   "provenance": [],
   "toc_visible": true,
   "version": "0.3.2"
  },
  "hide_input": false,
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}