introduction_to_sframes.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction to SFrames"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What is an SFrame?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note: This notebook uses GraphLab Create 1.7.**\n",
    "\n",
    "An SFrame is a tabular data structure.  If you are familiar with R or the pandas python package, SFrames behave similarly to the dataframes available in those frameworks.  SFrames act like a table by consisting of 0 or more columns.  Each column has its own datatype and every column of a particular SFrame must have the same number of entries as the other columns that already exist.  There are two things that make SFrames very different from other dataframes:\n",
    "\n",
    "- Each column is an SArray, which is a series of elements stored on disk.  This makes SFrames disk-based and therefore able to hold datasets that are too large to fit in your system's memory.  You'll see this come in to play throughout this demo.\n",
    "- An SFrame's data is located on the server that is running the GraphLab toolkits, which is not necessarily on your client machine.  While this example does not demonstrate working with a GraphLab server on a different machine, you can see that in action <a href=\"https://dato.com/learn/notebooks/running_in_the_cloud.html\">here</a>.\n",
    "\n",
    "This tutorial shows you how to import data into an SFrame, do some basic data cleaning/exploration, and save your work for later.  If you are someone that likes to learn these things through reading comprehensive documentation instead of tutorials, then you can visit our <a href=\"https://dato.com/products/create/docs/generated/graphlab.SFrame.html\">API Reference</a> first.  If not, read on!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Getting Started: Creating SFrames"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we will get set up with import statements for this tutorial."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import graphlab as gl"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Reading a csv file from an S3 bucket is just one way to import your data into an SFrame.  The read_csv function gives you lots of control over where to read your data from and how to parse it, which you can read about <a href=\"https://dato.com/products/create/docs/generated/graphlab.SFrame.read_csv.html#graphlab.SFrame.read_csv\">here</a>.  The column_type_hints option is important to highlight though, as without hints SFrame will simply decide that every column it finds is a string.  Here, only the year column is of type string.\n",
    "\n",
    "(The csv file of song metadata comes from the [Million Song Dataset](http://labrosa.ee.columbia.edu/millionsong/). This data set was used for a [Kaggle challenge](https://www.kaggle.com/c/msdchallenge) and includes data from [The Echo Nest](http://the.echonest.com/), [SecondHandSongs](http://www.secondhandsongs.com/), [musiXmatch](http://musixmatch.com/), and [Last.fm](http://www.last.fm/).)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[INFO] This non-commercial license of GraphLab Create is assigned to chandan2495@gmail.com and will expire on January 23, 2017. For commercial licensing options, visit https://dato.com/buy/.\n",
      "\n",
      "[INFO] Start server at: ipc:///tmp/graphlab_server-7016 - Server binary: C:\\Anaconda\\lib\\site-packages\\graphlab\\unity_server.exe - Server log: C:\\Users\\CHANDAN\\AppData\\Local\\Temp\\graphlab_server_1453644183.log.0\n",
      "[INFO] GraphLab Server Version: 1.8\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PROGRESS: Finished parsing file s3://dato-datasets/millionsong/song_data.csv\n",
      "PROGRESS: Parsing completed. Parsed 100 lines in 3.4382 secs.\n",
      "PROGRESS: Read 637410 lines. Lines per second: 8035.07\n",
      "PROGRESS: Finished parsing file s3://dato-datasets/millionsong/song_data.csv\n",
      "PROGRESS: Parsing completed. Parsed 1000000 lines in 80.1546 secs.\n"
     ]
    }
   ],
   "source": [
    "# In order to interact with S3 we need to set our AWS credentials.\n",
    "# You can use your own credentials or use the ones below.\n",
    "gl.aws.set_credentials('AKIAJMHKEZGY6YP24BXA', 'vf/miz2Zx7V7VkCai9ZeJR45ZSimqu6/W7qdRLmN')\n",
    "\n",
    "# The below will download a 78 MB file.\n",
    "song_sf = gl.SFrame.read_csv('s3://dato-datasets/millionsong/song_data.csv',\n",
    "                             column_type_hints = {'year' : int})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1000000"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "song_sf.num_rows()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If the csv file we want to read does not have a header, we can still provide column_type_hints, but with GraphLab's default column names.  Below is the code that would accomplish this, but I have commented it out because I don't want to affect the dataset we work with in the rest of this tutorial."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#song_sf = gl.SFrame.read_csv('s3://dato-datasets/millionsong/song_data.csv', header=False,\n",
    "#                             column_type_hints = {'X5' : int})\n",
    "#song_sf.head(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we start playing with this data, I want to highlight that you can save and load an SFrame for later use.  This is great if you don't want to re-download a file from S3 a bunch of times, or re-parse a large csv file.  Here's how to save to your current directory:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "song_sf.save('orig_song_data')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That save operation takes some time because it copies the files SFrame uses to the given location (in this case, an auto-created directory called 'orig_song_data').  The load operation, however, is instantaneous.  This is one of the perks of using a disk-backed dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "song_sf = gl.load_sframe('orig_song_data')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Viewing data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I can emit several commands to see that we are working with a fairly tame dataset.  After all, we only have five columns.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\"><table frame=\"box\" rules=\"cols\">\n",
       "    <tr>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">song_id</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">title</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">release</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">artist_name</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">year</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOQMMHC12AB0180CB8</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Silent Night</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Monster Ballads X-Mas</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Faster Pussy cat</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2003</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOVFVAK12A8C1350D9</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Tanssi vaan</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Karkuteillä</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Karkkiautomaatti</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1995</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOGTUKN12AB017F4F1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">No One Could Ever</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Butter</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Hudson Mohawke</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2006</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOBNYVR12A8C13558C</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Si Vos Querés</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">De Culo</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Yerba Brava</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2003</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOHSBXH12A8C13B0DF</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Tangle Of Aspens</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Rene Ablaze Presents<br>Winter Sessions ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Der Mystic</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOZVAPQ12A8C13B63C</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Symphony No. 1 G minor<br>\"Sinfonie ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Berwald: Symphonies Nos.<br>1/2/3/4 ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">David Montgomery</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOQVRHI12A6D4FB2D7</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">We Have Got Love</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Strictly The Best Vol. 34</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Sasha / Turbulence</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOEYRFT12AB018936C</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2 Da Beat Ch'yall</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Da Bomb</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Kris Kross</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1993</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOPMIYT12A6D4F851E</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Goodbye</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Danny Boy</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Joseph Locke</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOJCFMH12A8C13B0C2</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Mama_ mama can't you see<br>? ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">March to cadence with the<br>US marines ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">The Sun Harbor's Chorus-<br>Documentary Recordings ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "    </tr>\n",
       "</table>\n",
       "[10 rows x 5 columns]<br/>\n",
       "</div>"
      ],
      "text/plain": [
       "Columns:\n",
       "\tsong_id\tstr\n",
       "\ttitle\tstr\n",
       "\trelease\tstr\n",
       "\tartist_name\tstr\n",
       "\tyear\tint\n",
       "\n",
       "Rows: 10\n",
       "\n",
       "Data:\n",
       "+--------------------+-------------------------------+\n",
       "|      song_id       |             title             |\n",
       "+--------------------+-------------------------------+\n",
       "| SOQMMHC12AB0180CB8 |          Silent Night         |\n",
       "| SOVFVAK12A8C1350D9 |          Tanssi vaan          |\n",
       "| SOGTUKN12AB017F4F1 |       No One Could Ever       |\n",
       "| SOBNYVR12A8C13558C |         Si Vos Querés         |\n",
       "| SOHSBXH12A8C13B0DF |        Tangle Of Aspens       |\n",
       "| SOZVAPQ12A8C13B63C | Symphony No. 1 G minor \"Si... |\n",
       "| SOQVRHI12A6D4FB2D7 |        We Have Got Love       |\n",
       "| SOEYRFT12AB018936C |       2 Da Beat Ch'yall       |\n",
       "| SOPMIYT12A6D4F851E |            Goodbye            |\n",
       "| SOJCFMH12A8C13B0C2 |   Mama_ mama can't you see ?  |\n",
       "+--------------------+-------------------------------+\n",
       "+-------------------------------+-------------------------------+------+\n",
       "|            release            |          artist_name          | year |\n",
       "+-------------------------------+-------------------------------+------+\n",
       "|     Monster Ballads X-Mas     |        Faster Pussy cat       | 2003 |\n",
       "|          Karkuteillä          |        Karkkiautomaatti       | 1995 |\n",
       "|             Butter            |         Hudson Mohawke        | 2006 |\n",
       "|            De Culo            |          Yerba Brava          | 2003 |\n",
       "| Rene Ablaze Presents Winte... |           Der Mystic          |  0   |\n",
       "| Berwald: Symphonies Nos. 1... |        David Montgomery       |  0   |\n",
       "|   Strictly The Best Vol. 34   |       Sasha / Turbulence      |  0   |\n",
       "|            Da Bomb            |           Kris Kross          | 1993 |\n",
       "|           Danny Boy           |          Joseph Locke         |  0   |\n",
       "| March to cadence with the ... | The Sun Harbor's Chorus-Do... |  0   |\n",
       "+-------------------------------+-------------------------------+------+\n",
       "[10 rows x 5 columns]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "song_sf.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\"><table frame=\"box\" rules=\"cols\">\n",
       "    <tr>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">song_id</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">title</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">release</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">artist_name</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">year</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOTXAME12AB018F136</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">O Samba Da Vida</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Pacha V.I.P.</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Kiko Navarro</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOXQYIQ12A8C137FBB</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Jago Chhadeo</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Naale Baba Lassi Pee Gya</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Kuldeep Manak</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOHODZI12A8C137BB3</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Novemba</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Dub_Connected: electronic<br>music ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Gabriel Le Mar</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOLXGOR12A81C21EB7</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Faraday</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">The Trance Collection<br>Vol. 2 ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Elude</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOWXJXQ12AB0189F43</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Fernweh feat. Sektion<br>Kuchikäschtli ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">So Oder So</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Texta</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2004</td>\n",
       "    </tr>\n",
       "</table>\n",
       "[5 rows x 5 columns]<br/>\n",
       "</div>"
      ],
      "text/plain": [
       "Columns:\n",
       "\tsong_id\tstr\n",
       "\ttitle\tstr\n",
       "\trelease\tstr\n",
       "\tartist_name\tstr\n",
       "\tyear\tint\n",
       "\n",
       "Rows: 5\n",
       "\n",
       "Data:\n",
       "+--------------------+-------------------------------+\n",
       "|      song_id       |             title             |\n",
       "+--------------------+-------------------------------+\n",
       "| SOTXAME12AB018F136 |        O Samba Da Vida        |\n",
       "| SOXQYIQ12A8C137FBB |          Jago Chhadeo         |\n",
       "| SOHODZI12A8C137BB3 |            Novemba            |\n",
       "| SOLXGOR12A81C21EB7 |            Faraday            |\n",
       "| SOWXJXQ12AB0189F43 | Fernweh feat. Sektion Kuch... |\n",
       "+--------------------+-------------------------------+\n",
       "+-------------------------------+----------------+------+\n",
       "|            release            |  artist_name   | year |\n",
       "+-------------------------------+----------------+------+\n",
       "|          Pacha V.I.P.         |  Kiko Navarro  |  0   |\n",
       "|    Naale Baba Lassi Pee Gya   | Kuldeep Manak  |  0   |\n",
       "| Dub_Connected: electronic ... | Gabriel Le Mar |  0   |\n",
       "|  The Trance Collection Vol. 2 |     Elude      |  0   |\n",
       "|           So Oder So          |     Texta      | 2004 |\n",
       "+-------------------------------+----------------+------+\n",
       "[5 rows x 5 columns]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "song_sf.tail(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1000000, 1000000)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "song_sf.num_rows(), len(song_sf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "song_sf.num_cols()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['song_id', 'title', 'release', 'artist_name', 'year']"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "song_sf.column_names()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[str, str, str, str, int]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "song_sf.column_types()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Modifying an SFrame"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alright, I want a little more out of this SFrame.  I want to add a few columns.  Let's say I care about the length of the title of each song, what I've rated the song, and how many years old I was when the song was created."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\"><table frame=\"box\" rules=\"cols\">\n",
       "    <tr>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">song_id</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">title</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">release</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">artist_name</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">year</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">title_length</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">how_old_was_i</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOQMMHC12AB0180CB8</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Silent Night</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Monster Ballads X-Mas</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Faster Pussy cat</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2003</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOVFVAK12A8C1350D9</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Tanssi vaan</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Karkuteillä</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Karkkiautomaatti</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1995</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOGTUKN12AB017F4F1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">No One Could Ever</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Butter</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Hudson Mohawke</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2006</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">4</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOBNYVR12A8C13558C</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Si Vos Querés</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">De Culo</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Yerba Brava</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2003</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">3</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOHSBXH12A8C13B0DF</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Tangle Of Aspens</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Rene Ablaze Presents<br>Winter Sessions ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Der Mystic</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">3</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">-1988</td>\n",
       "    </tr>\n",
       "</table>\n",
       "<table frame=\"box\" rules=\"cols\">\n",
       "    <tr>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">my_rating</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "    </tr>\n",
       "</table>\n",
       "[5 rows x 8 columns]<br/>\n",
       "</div>"
      ],
      "text/plain": [
       "Columns:\n",
       "\tsong_id\tstr\n",
       "\ttitle\tstr\n",
       "\trelease\tstr\n",
       "\tartist_name\tstr\n",
       "\tyear\tint\n",
       "\ttitle_length\tint\n",
       "\thow_old_was_i\tint\n",
       "\tmy_rating\tint\n",
       "\n",
       "Rows: 5\n",
       "\n",
       "Data:\n",
       "+--------------------+-------------------+-------------------------------+\n",
       "|      song_id       |       title       |            release            |\n",
       "+--------------------+-------------------+-------------------------------+\n",
       "| SOQMMHC12AB0180CB8 |    Silent Night   |     Monster Ballads X-Mas     |\n",
       "| SOVFVAK12A8C1350D9 |    Tanssi vaan    |          Karkuteillä          |\n",
       "| SOGTUKN12AB017F4F1 | No One Could Ever |             Butter            |\n",
       "| SOBNYVR12A8C13558C |   Si Vos Querés   |            De Culo            |\n",
       "| SOHSBXH12A8C13B0DF |  Tangle Of Aspens | Rene Ablaze Presents Winte... |\n",
       "+--------------------+-------------------+-------------------------------+\n",
       "+------------------+------+--------------+---------------+-----------+\n",
       "|   artist_name    | year | title_length | how_old_was_i | my_rating |\n",
       "+------------------+------+--------------+---------------+-----------+\n",
       "| Faster Pussy cat | 2003 |      2       |       15      |     0     |\n",
       "| Karkkiautomaatti | 1995 |      2       |       7       |     0     |\n",
       "|  Hudson Mohawke  | 2006 |      4       |       18      |     0     |\n",
       "|   Yerba Brava    | 2003 |      3       |       15      |     0     |\n",
       "|    Der Mystic    |  0   |      3       |     -1988     |     0     |\n",
       "+------------------+------+--------------+---------------+-----------+\n",
       "[5 rows x 8 columns]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "year_i_was_born = 1988\n",
    "\n",
    "# Count the number of words in each song title and add the word count as a new feature\n",
    "song_sf['title_length'] = song_sf['title'].apply(lambda x: len(x.split()))\n",
    "\n",
    "# Count how old I was when this song came out\n",
    "song_sf.add_column(song_sf.select_column('year').apply(lambda x: x - year_i_was_born),\n",
    "                   'how_old_was_i')\n",
    "\n",
    "# Add a 0 rating for every song\n",
    "song_sf['my_rating'] = 0\n",
    "song_sf.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Clearly songs with a '0' year are a problem, but we'll cover that later.\n",
    "\n",
    "A few things to cover from the snippet above: \n",
    "\n",
    "- I can either use 'add_column'/'select_column', or just use python's index syntax to complete the same task.\n",
    "- It is easy to create new columns from an existing one by using 'apply' to apply a function to each element.\n",
    "- If you want a column to have the same value for every entry, just assign a single value to it.  We can do this with existing columns as well:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\"><table frame=\"box\" rules=\"cols\">\n",
       "    <tr>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">song_id</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">title</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">release</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">artist_name</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">year</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">title_length</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">how_old_was_i</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOQMMHC12AB0180CB8</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Silent Night</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Monster Ballads X-Mas</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Faster Pussy cat</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2003</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOVFVAK12A8C1350D9</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Tanssi vaan</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Karkuteillä</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Karkkiautomaatti</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1995</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOGTUKN12AB017F4F1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">No One Could Ever</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Butter</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Hudson Mohawke</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2006</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">4</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOBNYVR12A8C13558C</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Si Vos Querés</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">De Culo</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Yerba Brava</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2003</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">3</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOHSBXH12A8C13B0DF</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Tangle Of Aspens</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Rene Ablaze Presents<br>Winter Sessions ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Der Mystic</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">3</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">-1988</td>\n",
       "    </tr>\n",
       "</table>\n",
       "<table frame=\"box\" rules=\"cols\">\n",
       "    <tr>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">my_rating</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "    </tr>\n",
       "</table>\n",
       "[5 rows x 8 columns]<br/>\n",
       "</div>"
      ],
      "text/plain": [
       "Columns:\n",
       "\tsong_id\tstr\n",
       "\ttitle\tstr\n",
       "\trelease\tstr\n",
       "\tartist_name\tstr\n",
       "\tyear\tint\n",
       "\ttitle_length\tint\n",
       "\thow_old_was_i\tint\n",
       "\tmy_rating\tint\n",
       "\n",
       "Rows: 5\n",
       "\n",
       "Data:\n",
       "+--------------------+-------------------+-------------------------------+\n",
       "|      song_id       |       title       |            release            |\n",
       "+--------------------+-------------------+-------------------------------+\n",
       "| SOQMMHC12AB0180CB8 |    Silent Night   |     Monster Ballads X-Mas     |\n",
       "| SOVFVAK12A8C1350D9 |    Tanssi vaan    |          Karkuteillä          |\n",
       "| SOGTUKN12AB017F4F1 | No One Could Ever |             Butter            |\n",
       "| SOBNYVR12A8C13558C |   Si Vos Querés   |            De Culo            |\n",
       "| SOHSBXH12A8C13B0DF |  Tangle Of Aspens | Rene Ablaze Presents Winte... |\n",
       "+--------------------+-------------------+-------------------------------+\n",
       "+------------------+------+--------------+---------------+-----------+\n",
       "|   artist_name    | year | title_length | how_old_was_i | my_rating |\n",
       "+------------------+------+--------------+---------------+-----------+\n",
       "| Faster Pussy cat | 2003 |      2       |       15      |     1     |\n",
       "| Karkkiautomaatti | 1995 |      2       |       7       |     1     |\n",
       "|  Hudson Mohawke  | 2006 |      4       |       18      |     1     |\n",
       "|   Yerba Brava    | 2003 |      3       |       15      |     1     |\n",
       "|    Der Mystic    |  0   |      3       |     -1988     |     1     |\n",
       "+------------------+------+--------------+---------------+-----------+\n",
       "[5 rows x 8 columns]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "song_sf['my_rating'] = 1\n",
    "song_sf.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also add several columns at a time:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\"><table frame=\"box\" rules=\"cols\">\n",
       "    <tr>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">song_id</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">title</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">release</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">artist_name</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">year</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">title_length</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">how_old_was_i</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOQMMHC12AB0180CB8</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Silent Night</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Monster Ballads X-Mas</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Faster Pussy cat</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2003</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOVFVAK12A8C1350D9</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Tanssi vaan</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Karkuteillä</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Karkkiautomaatti</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1995</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOGTUKN12AB017F4F1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">No One Could Ever</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Butter</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Hudson Mohawke</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2006</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">4</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOBNYVR12A8C13558C</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Si Vos Querés</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">De Culo</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Yerba Brava</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2003</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">3</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOHSBXH12A8C13B0DF</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Tangle Of Aspens</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Rene Ablaze Presents<br>Winter Sessions ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Der Mystic</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">3</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">-1988</td>\n",
       "    </tr>\n",
       "</table>\n",
       "<table frame=\"box\" rules=\"cols\">\n",
       "    <tr>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">my_rating</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">dumb_col</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">dumb_col2</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">4</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">3</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">3</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "    </tr>\n",
       "</table>\n",
       "[5 rows x 10 columns]<br/>\n",
       "</div>"
      ],
      "text/plain": [
       "Columns:\n",
       "\tsong_id\tstr\n",
       "\ttitle\tstr\n",
       "\trelease\tstr\n",
       "\tartist_name\tstr\n",
       "\tyear\tint\n",
       "\ttitle_length\tint\n",
       "\thow_old_was_i\tint\n",
       "\tmy_rating\tint\n",
       "\tdumb_col\tint\n",
       "\tdumb_col2\tint\n",
       "\n",
       "Rows: 5\n",
       "\n",
       "Data:\n",
       "+--------------------+-------------------+-------------------------------+\n",
       "|      song_id       |       title       |            release            |\n",
       "+--------------------+-------------------+-------------------------------+\n",
       "| SOQMMHC12AB0180CB8 |    Silent Night   |     Monster Ballads X-Mas     |\n",
       "| SOVFVAK12A8C1350D9 |    Tanssi vaan    |          Karkuteillä          |\n",
       "| SOGTUKN12AB017F4F1 | No One Could Ever |             Butter            |\n",
       "| SOBNYVR12A8C13558C |   Si Vos Querés   |            De Culo            |\n",
       "| SOHSBXH12A8C13B0DF |  Tangle Of Aspens | Rene Ablaze Presents Winte... |\n",
       "+--------------------+-------------------+-------------------------------+\n",
       "+------------------+------+--------------+---------------+-----------+----------+\n",
       "|   artist_name    | year | title_length | how_old_was_i | my_rating | dumb_col |\n",
       "+------------------+------+--------------+---------------+-----------+----------+\n",
       "| Faster Pussy cat | 2003 |      2       |       15      |     1     |    2     |\n",
       "| Karkkiautomaatti | 1995 |      2       |       7       |     1     |    2     |\n",
       "|  Hudson Mohawke  | 2006 |      4       |       18      |     1     |    4     |\n",
       "|   Yerba Brava    | 2003 |      3       |       15      |     1     |    3     |\n",
       "|    Der Mystic    |  0   |      3       |     -1988     |     1     |    3     |\n",
       "+------------------+------+--------------+---------------+-----------+----------+\n",
       "+-----------+\n",
       "| dumb_col2 |\n",
       "+-----------+\n",
       "|     1     |\n",
       "|     1     |\n",
       "|     1     |\n",
       "|     1     |\n",
       "|     1     |\n",
       "+-----------+\n",
       "[5 rows x 10 columns]"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "song_sf[['dumb_col','dumb_col2']] = [song_sf['title_length'],song_sf['my_rating']]\n",
    "song_sf.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But maybe that was a dumb idea.  Let's get rid of those.  Before I do that, I'll show you how to rename and swap column ordering. Why not?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\"><table frame=\"box\" rules=\"cols\">\n",
       "    <tr>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">song_id</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">title</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">release</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">artist_name</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">year</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">title_length</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">how_old_was_i</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOQMMHC12AB0180CB8</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Silent Night</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Monster Ballads X-Mas</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Faster Pussy cat</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2003</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOVFVAK12A8C1350D9</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Tanssi vaan</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Karkuteillä</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Karkkiautomaatti</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1995</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOGTUKN12AB017F4F1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">No One Could Ever</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Butter</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Hudson Mohawke</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2006</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">4</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOBNYVR12A8C13558C</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Si Vos Querés</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">De Culo</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Yerba Brava</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2003</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">3</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOHSBXH12A8C13B0DF</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Tangle Of Aspens</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Rene Ablaze Presents<br>Winter Sessions ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Der Mystic</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">3</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">-1988</td>\n",
       "    </tr>\n",
       "</table>\n",
       "<table frame=\"box\" rules=\"cols\">\n",
       "    <tr>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">my_rating</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "    </tr>\n",
       "</table>\n",
       "[5 rows x 8 columns]<br/>\n",
       "</div>"
      ],
      "text/plain": [
       "Columns:\n",
       "\tsong_id\tstr\n",
       "\ttitle\tstr\n",
       "\trelease\tstr\n",
       "\tartist_name\tstr\n",
       "\tyear\tint\n",
       "\ttitle_length\tint\n",
       "\thow_old_was_i\tint\n",
       "\tmy_rating\tint\n",
       "\n",
       "Rows: 5\n",
       "\n",
       "Data:\n",
       "+--------------------+-------------------+-------------------------------+\n",
       "|      song_id       |       title       |            release            |\n",
       "+--------------------+-------------------+-------------------------------+\n",
       "| SOQMMHC12AB0180CB8 |    Silent Night   |     Monster Ballads X-Mas     |\n",
       "| SOVFVAK12A8C1350D9 |    Tanssi vaan    |          Karkuteillä          |\n",
       "| SOGTUKN12AB017F4F1 | No One Could Ever |             Butter            |\n",
       "| SOBNYVR12A8C13558C |   Si Vos Querés   |            De Culo            |\n",
       "| SOHSBXH12A8C13B0DF |  Tangle Of Aspens | Rene Ablaze Presents Winte... |\n",
       "+--------------------+-------------------+-------------------------------+\n",
       "+------------------+------+--------------+---------------+-----------+\n",
       "|   artist_name    | year | title_length | how_old_was_i | my_rating |\n",
       "+------------------+------+--------------+---------------+-----------+\n",
       "| Faster Pussy cat | 2003 |      2       |       15      |     1     |\n",
       "| Karkkiautomaatti | 1995 |      2       |       7       |     1     |\n",
       "|  Hudson Mohawke  | 2006 |      4       |       18      |     1     |\n",
       "|   Yerba Brava    | 2003 |      3       |       15      |     1     |\n",
       "|    Der Mystic    |  0   |      3       |     -1988     |     1     |\n",
       "+------------------+------+--------------+---------------+-----------+\n",
       "[5 rows x 8 columns]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "song_sf.rename({'dumb_col2' : 'another_dumb_col'})\n",
    "song_sf.swap_columns('dumb_col', 'another_dumb_col')\n",
    "del song_sf['dumb_col']\n",
    "del song_sf['another_dumb_col']\n",
    "song_sf.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Still with me?  Notice that the column types for the transformed columns are correct."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[str, str, str, str, int, int, int, int]"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "song_sf.column_types()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hold on though, I think I'd actually like the rating to be a float."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[str, str, str, str, int, int, int, float]"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "song_sf['my_rating'] = song_sf['my_rating'].astype(float)\n",
    "song_sf.column_types()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "To create even more interesting feature columns, you may want to apply a function using multiple (or all) columns.  When you apply a function to an SFrame (instead of just an SArray like I did earlier), the input to the function is a dictionary where the keys are your column names.  Here I'd like to know what combination of song title, album title, and artist name mentions the word 'love' the most:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\"><table frame=\"box\" rules=\"cols\">\n",
       "    <tr>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">song_id</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">title</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">release</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">artist_name</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">year</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">title_length</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOBYJSK12A8C132021</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Love To Love To Love You<br>Baby ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Love To Love To Love You<br>Baby &amp; Break 4 Love ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">David Vendetta</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOXAVWF12A8AE4922C</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">One Piece</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Low Low Low La La La Love<br>Love Love ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Low Low Low La La La Love<br>Love Love ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2007</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOKXWMK12A8AE45BDB</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Only Let Things Be</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Low Low Low La La La Love<br>Love Love ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Low Low Low La La La Love<br>Love Love ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2007</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOHCQVM12A8AE49226</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Venus In The Evening I<br>Have Disappeared Again ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Low Low Low La La La Love<br>Love Love ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Low Low Low La La La Love<br>Love Love ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2007</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOEXSIX12A58A77BA2</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Rest Your Arms</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Low Low Low La La La Love<br>Love Love ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Low Low Low La La La Love<br>Love Love ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2007</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">3</td>\n",
       "    </tr>\n",
       "</table>\n",
       "<table frame=\"box\" rules=\"cols\">\n",
       "    <tr>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">how_old_was_i</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">my_rating</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">love_count</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">-1988</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">19</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">19</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">19</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">19</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">6</td>\n",
       "    </tr>\n",
       "</table>\n",
       "[5 rows x 9 columns]<br/>\n",
       "</div>"
      ],
      "text/plain": [
       "Columns:\n",
       "\tsong_id\tstr\n",
       "\ttitle\tstr\n",
       "\trelease\tstr\n",
       "\tartist_name\tstr\n",
       "\tyear\tint\n",
       "\ttitle_length\tint\n",
       "\thow_old_was_i\tint\n",
       "\tmy_rating\tfloat\n",
       "\tlove_count\tint\n",
       "\n",
       "Rows: 5\n",
       "\n",
       "Data:\n",
       "+--------------------+-------------------------------+\n",
       "|      song_id       |             title             |\n",
       "+--------------------+-------------------------------+\n",
       "| SOBYJSK12A8C132021 | Love To Love To Love You Baby |\n",
       "| SOXAVWF12A8AE4922C |           One Piece           |\n",
       "| SOKXWMK12A8AE45BDB |       Only Let Things Be      |\n",
       "| SOHCQVM12A8AE49226 | Venus In The Evening I Hav... |\n",
       "| SOEXSIX12A58A77BA2 |         Rest Your Arms        |\n",
       "+--------------------+-------------------------------+\n",
       "+-------------------------------+-------------------------------+------+--------------+\n",
       "|            release            |          artist_name          | year | title_length |\n",
       "+-------------------------------+-------------------------------+------+--------------+\n",
       "| Love To Love To Love You B... |         David Vendetta        |  0   |      7       |\n",
       "| Low Low Low La La La Love ... | Low Low Low La La La Love ... | 2007 |      2       |\n",
       "| Low Low Low La La La Love ... | Low Low Low La La La Love ... | 2007 |      4       |\n",
       "| Low Low Low La La La Love ... | Low Low Low La La La Love ... | 2007 |      8       |\n",
       "| Low Low Low La La La Love ... | Low Low Low La La La Love ... | 2007 |      3       |\n",
       "+-------------------------------+-------------------------------+------+--------------+\n",
       "+---------------+-----------+------------+\n",
       "| how_old_was_i | my_rating | love_count |\n",
       "+---------------+-----------+------------+\n",
       "|     -1988     |    1.0    |     7      |\n",
       "|       19      |    1.0    |     6      |\n",
       "|       19      |    1.0    |     6      |\n",
       "|       19      |    1.0    |     6      |\n",
       "|       19      |    1.0    |     6      |\n",
       "+---------------+-----------+------------+\n",
       "[5 rows x 9 columns]"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "song_sf['love_count'] = song_sf[['release', 'title', 'artist_name']].apply(\n",
    "    lambda row: sum(x.lower().split().count('love') for x in row.values()))\n",
    "song_sf.topk('love_count').head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see from these examples that adding and deleting columns is a simple task for an SFrame.  This is because an SFrame is essentially the keeper of references to columns (SArrays), so adding and deleting columns is a very cheap operation.  However, the fact that SFrames store their data on disk produces some important limitations when thinking about editing an SFrame:\n",
    "\n",
    "- SFrames are immutable with respect to column size and data.\n",
    "- SFrames do not support random access of elements and are not indexed.  \n",
    "    \n",
    "Sequential access is king on disk, and this is very useful to remember when working with SFrames.  This means that inspecting a specific row would perform quite poorly and writing to a specific row is not possible. However, while working with SFrames you'll find that you can still accomplish nearly all of what you would have done with a more classic dataframe using transform and filter operations, yet you'll still reap the huge benefit of creating SFrames that are larger than the size of your machine's memory.  So let's learn about filtering!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Filtering and Missing Values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I think I want to take care of those invalid year entries now.  I don't really know how many there are, so I'll find out, as the answer to that may change what I do."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+------+--------+\n",
      "| year | Count  |\n",
      "+------+--------+\n",
      "| 1982 |  3597  |\n",
      "| 1930 |   40   |\n",
      "| 1975 |  2482  |\n",
      "| 1942 |   24   |\n",
      "| 1959 |  592   |\n",
      "| 1990 |  7258  |\n",
      "| 1974 |  2186  |\n",
      "|  0   | 484424 |\n",
      "| 1947 |   57   |\n",
      "| 1996 | 14135  |\n",
      "+------+--------+\n",
      "[10 rows x 2 columns]\n",
      "\n",
      "Number of unique years: 90\n",
      "Number of invalid years: \n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\"><table frame=\"box\" rules=\"cols\">\n",
       "    <tr>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">year</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">Count</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">484424</td>\n",
       "    </tr>\n",
       "</table>\n",
       "[1 rows x 2 columns]<br/>\n",
       "</div>"
      ],
      "text/plain": [
       "Columns:\n",
       "\tyear\tint\n",
       "\tCount\tint\n",
       "\n",
       "Rows: 1\n",
       "\n",
       "Data:\n",
       "+------+--------+\n",
       "| year | Count  |\n",
       "+------+--------+\n",
       "|  0   | 484424 |\n",
       "+------+--------+\n",
       "[1 rows x 2 columns]"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "year_count = song_sf.groupby('year', gl.aggregate.COUNT)\n",
    "print year_count.head()\n",
    "print \"Number of unique years: \" + str(len(year_count))\n",
    "print \"Number of invalid years: \"\n",
    "year_count.topk('year', reverse=True, k=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Yikes, that's almost half of my dataset.  Maybe I don't want to just get rid of that data. SFrames support missing values, and these are represented using 'None' .  We will transform the apporpriate values to missing here:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\"><table frame=\"box\" rules=\"cols\">\n",
       "    <tr>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">song_id</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">title</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">release</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">artist_name</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">year</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">title_length</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">how_old_was_i</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOQMMHC12AB0180CB8</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Silent Night</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Monster Ballads X-Mas</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Faster Pussy cat</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2003</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOVFVAK12A8C1350D9</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Tanssi vaan</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Karkuteillä</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Karkkiautomaatti</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1995</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOGTUKN12AB017F4F1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">No One Could Ever</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Butter</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Hudson Mohawke</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2006</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">4</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOBNYVR12A8C13558C</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Si Vos Querés</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">De Culo</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Yerba Brava</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2003</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">3</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOHSBXH12A8C13B0DF</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Tangle Of Aspens</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Rene Ablaze Presents<br>Winter Sessions ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Der Mystic</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">None</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">3</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">-1988</td>\n",
       "    </tr>\n",
       "</table>\n",
       "<table frame=\"box\" rules=\"cols\">\n",
       "    <tr>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">my_rating</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">love_count</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "    </tr>\n",
       "</table>\n",
       "[5 rows x 9 columns]<br/>\n",
       "</div>"
      ],
      "text/plain": [
       "Columns:\n",
       "\tsong_id\tstr\n",
       "\ttitle\tstr\n",
       "\trelease\tstr\n",
       "\tartist_name\tstr\n",
       "\tyear\tint\n",
       "\ttitle_length\tint\n",
       "\thow_old_was_i\tint\n",
       "\tmy_rating\tfloat\n",
       "\tlove_count\tint\n",
       "\n",
       "Rows: 5\n",
       "\n",
       "Data:\n",
       "+--------------------+-------------------+-------------------------------+\n",
       "|      song_id       |       title       |            release            |\n",
       "+--------------------+-------------------+-------------------------------+\n",
       "| SOQMMHC12AB0180CB8 |    Silent Night   |     Monster Ballads X-Mas     |\n",
       "| SOVFVAK12A8C1350D9 |    Tanssi vaan    |          Karkuteillä          |\n",
       "| SOGTUKN12AB017F4F1 | No One Could Ever |             Butter            |\n",
       "| SOBNYVR12A8C13558C |   Si Vos Querés   |            De Culo            |\n",
       "| SOHSBXH12A8C13B0DF |  Tangle Of Aspens | Rene Ablaze Presents Winte... |\n",
       "+--------------------+-------------------+-------------------------------+\n",
       "+------------------+------+--------------+---------------+-----------+------------+\n",
       "|   artist_name    | year | title_length | how_old_was_i | my_rating | love_count |\n",
       "+------------------+------+--------------+---------------+-----------+------------+\n",
       "| Faster Pussy cat | 2003 |      2       |       15      |    1.0    |     0      |\n",
       "| Karkkiautomaatti | 1995 |      2       |       7       |    1.0    |     0      |\n",
       "|  Hudson Mohawke  | 2006 |      4       |       18      |    1.0    |     0      |\n",
       "|   Yerba Brava    | 2003 |      3       |       15      |    1.0    |     0      |\n",
       "|    Der Mystic    | None |      3       |     -1988     |    1.0    |     0      |\n",
       "+------------------+------+--------------+---------------+-----------+------------+\n",
       "[5 rows x 9 columns]"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "song_sf['year'] = song_sf['year'].apply(lambda x :None if x == 0 else x)\n",
    "song_sf.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To show that normal operations work on columns with missing values, we will do the 'how_old_was_i' transformation again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\"><table frame=\"box\" rules=\"cols\">\n",
       "    <tr>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">song_id</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">title</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">release</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">artist_name</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">year</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">title_length</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">how_old_was_i</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOQMMHC12AB0180CB8</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Silent Night</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Monster Ballads X-Mas</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Faster Pussy cat</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2003</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOVFVAK12A8C1350D9</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Tanssi vaan</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Karkuteillä</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Karkkiautomaatti</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1995</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOGTUKN12AB017F4F1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">No One Could Ever</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Butter</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Hudson Mohawke</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2006</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">4</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOBNYVR12A8C13558C</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Si Vos Querés</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">De Culo</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Yerba Brava</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2003</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">3</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOHSBXH12A8C13B0DF</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Tangle Of Aspens</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Rene Ablaze Presents<br>Winter Sessions ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Der Mystic</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">None</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">3</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">None</td>\n",
       "    </tr>\n",
       "</table>\n",
       "<table frame=\"box\" rules=\"cols\">\n",
       "    <tr>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">my_rating</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">love_count</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "    </tr>\n",
       "</table>\n",
       "[5 rows x 9 columns]<br/>\n",
       "</div>"
      ],
      "text/plain": [
       "Columns:\n",
       "\tsong_id\tstr\n",
       "\ttitle\tstr\n",
       "\trelease\tstr\n",
       "\tartist_name\tstr\n",
       "\tyear\tint\n",
       "\ttitle_length\tint\n",
       "\thow_old_was_i\tint\n",
       "\tmy_rating\tfloat\n",
       "\tlove_count\tint\n",
       "\n",
       "Rows: 5\n",
       "\n",
       "Data:\n",
       "+--------------------+-------------------+-------------------------------+\n",
       "|      song_id       |       title       |            release            |\n",
       "+--------------------+-------------------+-------------------------------+\n",
       "| SOQMMHC12AB0180CB8 |    Silent Night   |     Monster Ballads X-Mas     |\n",
       "| SOVFVAK12A8C1350D9 |    Tanssi vaan    |          Karkuteillä          |\n",
       "| SOGTUKN12AB017F4F1 | No One Could Ever |             Butter            |\n",
       "| SOBNYVR12A8C13558C |   Si Vos Querés   |            De Culo            |\n",
       "| SOHSBXH12A8C13B0DF |  Tangle Of Aspens | Rene Ablaze Presents Winte... |\n",
       "+--------------------+-------------------+-------------------------------+\n",
       "+------------------+------+--------------+---------------+-----------+------------+\n",
       "|   artist_name    | year | title_length | how_old_was_i | my_rating | love_count |\n",
       "+------------------+------+--------------+---------------+-----------+------------+\n",
       "| Faster Pussy cat | 2003 |      2       |       15      |    1.0    |     0      |\n",
       "| Karkkiautomaatti | 1995 |      2       |       7       |    1.0    |     0      |\n",
       "|  Hudson Mohawke  | 2006 |      4       |       18      |    1.0    |     0      |\n",
       "|   Yerba Brava    | 2003 |      3       |       15      |    1.0    |     0      |\n",
       "|    Der Mystic    | None |      3       |      None     |    1.0    |     0      |\n",
       "+------------------+------+--------------+---------------+-----------+------------+\n",
       "[5 rows x 9 columns]"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "song_sf['how_old_was_i'] = song_sf['year'].apply(lambda x : x - year_i_was_born)\n",
    "song_sf.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, if I actually did want to filter out these missing values, that is easy too."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Length of trimmed data: 515576\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\"><table frame=\"box\" rules=\"cols\">\n",
       "    <tr>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">song_id</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">title</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">release</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">artist_name</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">year</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">title_length</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">how_old_was_i</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOQMMHC12AB0180CB8</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Silent Night</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Monster Ballads X-Mas</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Faster Pussy cat</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2003</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOVFVAK12A8C1350D9</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Tanssi vaan</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Karkuteillä</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Karkkiautomaatti</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1995</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOGTUKN12AB017F4F1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">No One Could Ever</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Butter</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Hudson Mohawke</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2006</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">4</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOBNYVR12A8C13558C</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Si Vos Querés</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">De Culo</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Yerba Brava</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2003</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">3</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOEYRFT12AB018936C</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2 Da Beat Ch'yall</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Da Bomb</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Kris Kross</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1993</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">4</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">5</td>\n",
       "    </tr>\n",
       "</table>\n",
       "<table frame=\"box\" rules=\"cols\">\n",
       "    <tr>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">my_rating</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">love_count</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "    </tr>\n",
       "</table>\n",
       "[5 rows x 9 columns]<br/>\n",
       "</div>"
      ],
      "text/plain": [
       "Columns:\n",
       "\tsong_id\tstr\n",
       "\ttitle\tstr\n",
       "\trelease\tstr\n",
       "\tartist_name\tstr\n",
       "\tyear\tint\n",
       "\ttitle_length\tint\n",
       "\thow_old_was_i\tint\n",
       "\tmy_rating\tfloat\n",
       "\tlove_count\tint\n",
       "\n",
       "Rows: 5\n",
       "\n",
       "Data:\n",
       "+--------------------+-------------------+-----------------------+\n",
       "|      song_id       |       title       |        release        |\n",
       "+--------------------+-------------------+-----------------------+\n",
       "| SOQMMHC12AB0180CB8 |    Silent Night   | Monster Ballads X-Mas |\n",
       "| SOVFVAK12A8C1350D9 |    Tanssi vaan    |      Karkuteillä      |\n",
       "| SOGTUKN12AB017F4F1 | No One Could Ever |         Butter        |\n",
       "| SOBNYVR12A8C13558C |   Si Vos Querés   |        De Culo        |\n",
       "| SOEYRFT12AB018936C | 2 Da Beat Ch'yall |        Da Bomb        |\n",
       "+--------------------+-------------------+-----------------------+\n",
       "+------------------+------+--------------+---------------+-----------+------------+\n",
       "|   artist_name    | year | title_length | how_old_was_i | my_rating | love_count |\n",
       "+------------------+------+--------------+---------------+-----------+------------+\n",
       "| Faster Pussy cat | 2003 |      2       |       15      |    1.0    |     0      |\n",
       "| Karkkiautomaatti | 1995 |      2       |       7       |    1.0    |     0      |\n",
       "|  Hudson Mohawke  | 2006 |      4       |       18      |    1.0    |     0      |\n",
       "|   Yerba Brava    | 2003 |      3       |       15      |    1.0    |     0      |\n",
       "|    Kris Kross    | 1993 |      4       |       5       |    1.0    |     0      |\n",
       "+------------------+------+--------------+---------------+-----------+------------+\n",
       "[5 rows x 9 columns]"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "song_sf_valid_years = song_sf[song_sf['year'] > 0]\n",
    "print \"Length of trimmed data: \" + str(len(song_sf_valid_years))\n",
    "song_sf_valid_years.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What I'm showing off here is that we can filter an SFrame by an SArray, where only the SFrame rows that correspond to the given SArray entries evaluating to True will make it through the filter.  This happens when an SArray is given as the index of an SFrame.  Furthermore, we can create a new SArray from an existing one by using any of the comparison operators.  To execute this filter we did both of these things, but you can do them in isolation as well.  Here I show the resulting SArray from running the '> 0' operation in isolation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dtype: int\n",
       "Rows: 1000000\n",
       "[1L, 1L, 1L, 1L, None, None, None, 1L, None, None, 1L, 1L, None, 1L, None, 1L, 1L, 1L, 1L, None, None, None, 1L, 1L, None, None, 1L, 1L, None, None, None, None, None, 1L, 1L, 1L, None, 1L, None, 1L, None, 1L, None, 1L, 1L, 1L, None, 1L, None, None, None, None, 1L, 1L, 1L, None, None, 1L, None, None, 1L, 1L, None, 1L, 1L, 1L, None, None, None, 1L, None, None, 1L, None, 1L, 1L, None, None, None, None, 1L, None, None, 1L, 1L, None, 1L, None, 1L, None, None, None, 1L, 1L, 1L, 1L, None, 1L, 1L, 1L, ... ]"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tmp = song_sf['year'] > 0\n",
    "tmp"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Keep in mind that the SArray must be the same length of the SFrame in order to filter.  This also works with more complicated, chained filters with logical operators.  Here's a list of songs that came out while I was in high school by a couple of my favorite bands in that period of my life:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\"><table frame=\"box\" rules=\"cols\">\n",
       "    <tr>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">song_id</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">title</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">release</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">artist_name</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">year</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">title_length</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOQNUHJ12A6D4F9E19</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Let It All Out (Album<br>Version) ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">MMHMM</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Relient K</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2004</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOIONAH12A58A76FD1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">We Are The Few (Album<br>Version) ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Everything Goes Numb</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Streetlight Manifesto</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2003</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOUUYHK12A6D4F9E16</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">I So Hate Consequences<br>(Album Version) ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">MMHMM</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Relient K</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2004</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOCBUJT12A6D4F9E1A</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Who I Am Hates Who I've<br>Been (mmhmm Album ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">MMHMM</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Relient K</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2004</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOBUHJR12A6D4FDC7C</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Here's To Life (Album<br>Version) ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Everything Goes Numb</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Streetlight Manifesto</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2003</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SONKBWG12A6D4FB91D</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Giving Up_ Giving In (LP<br>Version) ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Keasbey Nights</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Streetlight Manifesto</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2006</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOSFAVU12A6D4FDC6A</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Everything Went Numb<br>(Album Version) ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Everything Goes Numb</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Streetlight Manifesto</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2003</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SORYDMW12A6D4FB923</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">This One Goes Out To....<br>(LP Version) ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Keasbey Nights</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Streetlight Manifesto</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2006</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOFMBIT12A6D4F9E1C</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">This Week The Trend<br>(Album Version) ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">MMHMM</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Relient K</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2004</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOINPKF12A6D4FDC75</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">A Better Place_ A Better<br>Time (Album Version) ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Everything Goes Numb</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Streetlight Manifesto</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2003</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">8</td>\n",
       "    </tr>\n",
       "</table>\n",
       "<table frame=\"box\" rules=\"cols\">\n",
       "    <tr>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">how_old_was_i</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">my_rating</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">love_count</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">16</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">15</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">16</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">16</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">15</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">18</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">15</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">18</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">16</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">15</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "    </tr>\n",
       "</table>\n",
       "[? rows x 9 columns]<br/>Note: Only the head of the SFrame is printed. This SFrame is lazily evaluated.<br/>You can use len(sf) to force materialization.\n",
       "</div>"
      ],
      "text/plain": [
       "Columns:\n",
       "\tsong_id\tstr\n",
       "\ttitle\tstr\n",
       "\trelease\tstr\n",
       "\tartist_name\tstr\n",
       "\tyear\tint\n",
       "\ttitle_length\tint\n",
       "\thow_old_was_i\tint\n",
       "\tmy_rating\tfloat\n",
       "\tlove_count\tint\n",
       "\n",
       "Rows: Unknown\n",
       "\n",
       "Data:\n",
       "+--------------------+--------------------------------+----------------------+\n",
       "|      song_id       |             title              |       release        |\n",
       "+--------------------+--------------------------------+----------------------+\n",
       "| SOQNUHJ12A6D4F9E19 | Let It All Out (Album Version) |        MMHMM         |\n",
       "| SOIONAH12A58A76FD1 | We Are The Few (Album Version) | Everything Goes Numb |\n",
       "| SOUUYHK12A6D4F9E16 | I So Hate Consequences (Al...  |        MMHMM         |\n",
       "| SOCBUJT12A6D4F9E1A | Who I Am Hates Who I've Be...  |        MMHMM         |\n",
       "| SOBUHJR12A6D4FDC7C | Here's To Life (Album Version) | Everything Goes Numb |\n",
       "| SONKBWG12A6D4FB91D | Giving Up_ Giving In (LP V...  |    Keasbey Nights    |\n",
       "| SOSFAVU12A6D4FDC6A | Everything Went Numb (Albu...  | Everything Goes Numb |\n",
       "| SORYDMW12A6D4FB923 | This One Goes Out To.... (...  |    Keasbey Nights    |\n",
       "| SOFMBIT12A6D4F9E1C | This Week The Trend (Album...  |        MMHMM         |\n",
       "| SOINPKF12A6D4FDC75 | A Better Place_ A Better T...  | Everything Goes Numb |\n",
       "+--------------------+--------------------------------+----------------------+\n",
       "+-----------------------+------+--------------+---------------+-----------+------------+\n",
       "|      artist_name      | year | title_length | how_old_was_i | my_rating | love_count |\n",
       "+-----------------------+------+--------------+---------------+-----------+------------+\n",
       "|       Relient K       | 2004 |      6       |       16      |    1.0    |     0      |\n",
       "| Streetlight Manifesto | 2003 |      6       |       15      |    1.0    |     0      |\n",
       "|       Relient K       | 2004 |      6       |       16      |    1.0    |     0      |\n",
       "|       Relient K       | 2004 |      10      |       16      |    1.0    |     0      |\n",
       "| Streetlight Manifesto | 2003 |      5       |       15      |    1.0    |     0      |\n",
       "| Streetlight Manifesto | 2006 |      6       |       18      |    1.0    |     0      |\n",
       "| Streetlight Manifesto | 2003 |      5       |       15      |    1.0    |     0      |\n",
       "| Streetlight Manifesto | 2006 |      7       |       18      |    1.0    |     0      |\n",
       "|       Relient K       | 2004 |      6       |       16      |    1.0    |     0      |\n",
       "| Streetlight Manifesto | 2003 |      8       |       15      |    1.0    |     0      |\n",
       "+-----------------------+------+--------------+---------------+-----------+------------+\n",
       "[? rows x 9 columns]\n",
       "Note: Only the head of the SFrame is printed. This SFrame is lazily evaluated.\n",
       "You can use len(sf) to force materialization."
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my_fav_hs_songs = song_sf[((song_sf['artist_name'] == 'Relient K')\n",
    "                           | (song_sf['artist_name'] == 'Streetlight Manifesto'))\n",
    "                          & (song_sf['how_old_was_i'] >= 14) & (song_sf['how_old_was_i'] <= 18)]\n",
    "my_fav_hs_songs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's not all of them, but that's a pretty decent selection for a dataset of a million songs.  Notice that I had to use the bitwise operators instead of the 'and'/'or' keyword.  Python does not allow the overloading of logical operators, so remember to use the bitwise ones."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Descriptive Statistics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The descriptive statistics below are operations done on the SArray, and cannot be done on the SFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mean: 3.369894\n",
      "std: 2.17226527587\n",
      "var: 4.71873642876\n",
      "min: 0\n",
      "max: 47\n",
      "sum: 3369894\n",
      "number of non-zero entries: 999985\n"
     ]
    }
   ],
   "source": [
    "# Look at lots of descriptive statistics of title_length\n",
    "print \"mean: \" + str(song_sf['title_length'].mean())\n",
    "print \"std: \" + str(song_sf['title_length'].std())\n",
    "print \"var: \" + str(song_sf['title_length'].var())\n",
    "print \"min: \" + str(song_sf['title_length'].min())\n",
    "print \"max: \" + str(song_sf['title_length'].max())\n",
    "print \"sum: \" + str(song_sf['title_length'].sum())\n",
    "print \"number of non-zero entries: \" + str(song_sf['title_length'].nnz())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can accomplish essentially the same thing by getting a sketch_summary on this column.  This will give the exact values of the descriptive statistics I asked for above, and then give approximate values of some other useful stuff like quantiles and counts of unique values.  These values are approximate because performing the real operation on a dataset that is larger than your memory size could exhaust your memory or take too long to compute.  Each operation has well-defined bounds on how wrong the answer will be, which are listed in our <a href=\"https://dato.com/products/create/docs/generated/graphlab.Sketch.html\">API Reference</a>."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "+--------------------+---------------+----------+\n",
      "|        item        |     value     | is exact |\n",
      "+--------------------+---------------+----------+\n",
      "|       Length       |    1000000    |   Yes    |\n",
      "|        Min         |      0.0      |   Yes    |\n",
      "|        Max         |      47.0     |   Yes    |\n",
      "|        Mean        |    3.369894   |   Yes    |\n",
      "|        Sum         |   3369894.0   |   Yes    |\n",
      "|      Variance      | 4.71873642876 |   Yes    |\n",
      "| Standard Deviation | 2.17226527587 |   Yes    |\n",
      "|  # Missing Values  |       0       |   Yes    |\n",
      "|  # unique values   |       44      |    No    |\n",
      "+--------------------+---------------+----------+\n",
      "\n",
      "Most frequent items:\n",
      "+-------+--------+--------+--------+--------+-------+-------+-------+-------+\n",
      "| value |   2    |   3    |   1    |   4    |   5   |   6   |   7   |   8   |\n",
      "+-------+--------+--------+--------+--------+-------+-------+-------+-------+\n",
      "| count | 241587 | 217674 | 164124 | 152569 | 92148 | 54674 | 30777 | 17826 |\n",
      "+-------+--------+--------+--------+--------+-------+-------+-------+-------+\n",
      "+-------+------+\n",
      "|   9   |  10  |\n",
      "+-------+------+\n",
      "| 10465 | 6633 |\n",
      "+-------+------+\n",
      "\n",
      "Quantiles: \n",
      "+-----+-----+-----+-----+-----+-----+-----+------+------+\n",
      "|  0% |  1% |  5% | 25% | 50% | 75% | 95% | 99%  | 100% |\n",
      "+-----+-----+-----+-----+-----+-----+-----+------+------+\n",
      "| 0.0 | 1.0 | 1.0 | 2.0 | 3.0 | 4.0 | 7.0 | 11.0 | 47.0 |\n",
      "+-----+-----+-----+-----+-----+-----+-----+------+------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "approx_sketch = song_sf['title_length'].sketch_summary()\n",
    "print approx_sketch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Saving the return value from sketch_summary gives you a graphlab.Sketch object, which can be queried further (details <a href=\"https://dato.com/products/create/docs/generated/graphlab.Sketch.html\">here</a>).  Here, I can drill deeper into those quantiles:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.0\n",
      "4.0\n",
      "12.0\n",
      "13.0\n",
      "14.0\n"
     ]
    }
   ],
   "source": [
    "print approx_sketch.quantile(.25)\n",
    "print approx_sketch.quantile(.75)\n",
    "print approx_sketch.quantile(.993)\n",
    "print approx_sketch.quantile(.995)\n",
    "print approx_sketch.quantile(.997)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But wow...47 words?!? I gotta see what that song is."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\"><table frame=\"box\" rules=\"cols\">\n",
       "    <tr>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">song_id</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">title</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">release</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">artist_name</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">year</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOAALOO12AC468C4ED</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Resolution Island Suite<br>I) A Vessel Sublime II) ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Allegory Of Hearing</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Roy Montgomery</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOSEWOR12AB018BDF3</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Guayacan Mix: Amor<br>Traicionero / Te Amo_ Te ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Como en un baile</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Guayacan</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOEBPPZ12AB0183730</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Son Of Scheherazade:<br>Pt.1- Fanfare; Pt 2 - ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">British Tour '76</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Renaissance</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SODRVPW12A8C13DDCB</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">And by our own hand did<br>every last bird lie ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Every Red Heart Shines<br>Toward the Red Sun ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Red Sparowes</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2006</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SODEYRT12A8AE47972</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Any Place I Hang My Hat<br>is Home - On the ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Too Marvelous For Words -<br>The Songs Of Johnny ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Lee Lessack</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOEICJI12AC3DFAAD4</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Throw Away Comedy Medley:<br>Dance With A Dolly / You ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Live At The Sands</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Dean Martin</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SORHPYP12AB017C661</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Happenings' Medley: Oh!<br>The Grand Old Duke Of ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">The Cat &amp; The Fiddle - 66<br>Nursery Rhyme Favourites ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">The Mother Goose Singers</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOKNNWV12AB017B473</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">If I Had My Way/Irish<br>Rose/Daisy/Down Our ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1999 International<br>Barbershop Quartet ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Swing City</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOWKFQF12AAA8C85E4</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Manitoba Ne Répond Plus<br>(Comme Un Lego_ Dans Un ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Manitoba Ne Répond Plus</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Gérard Manset</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2008</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SONMHGO12A6D4F8E6A</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Marcha Do Cordão Da Bola<br>Preta / Me Dá Dinheiro ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Elizeth No Bola Preta</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Elizeth Cardoso</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">None</td>\n",
       "    </tr>\n",
       "</table>\n",
       "<table frame=\"box\" rules=\"cols\">\n",
       "    <tr>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">title_length</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">how_old_was_i</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">my_rating</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">love_count</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">47</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">None</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">46</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">None</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">46</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">None</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">45</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">18</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">44</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">None</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">44</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">None</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">40</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">None</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">40</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">None</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">39</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">20</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">39</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">None</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "    </tr>\n",
       "</table>\n",
       "[10 rows x 9 columns]<br/>\n",
       "</div>"
      ],
      "text/plain": [
       "Columns:\n",
       "\tsong_id\tstr\n",
       "\ttitle\tstr\n",
       "\trelease\tstr\n",
       "\tartist_name\tstr\n",
       "\tyear\tint\n",
       "\ttitle_length\tint\n",
       "\thow_old_was_i\tint\n",
       "\tmy_rating\tfloat\n",
       "\tlove_count\tint\n",
       "\n",
       "Rows: 10\n",
       "\n",
       "Data:\n",
       "+--------------------+-------------------------------+\n",
       "|      song_id       |             title             |\n",
       "+--------------------+-------------------------------+\n",
       "| SOAALOO12AC468C4ED | Resolution Island Suite I)... |\n",
       "| SOSEWOR12AB018BDF3 | Guayacan Mix: Amor Traicio... |\n",
       "| SOEBPPZ12AB0183730 | Son Of Scheherazade: Pt.1-... |\n",
       "| SODRVPW12A8C13DDCB | And by our own hand did ev... |\n",
       "| SODEYRT12A8AE47972 | Any Place I Hang My Hat is... |\n",
       "| SOEICJI12AC3DFAAD4 | Throw Away Comedy Medley: ... |\n",
       "| SORHPYP12AB017C661 | Happenings' Medley: Oh! Th... |\n",
       "| SOKNNWV12AB017B473 | If I Had My Way/Irish Rose... |\n",
       "| SOWKFQF12AAA8C85E4 | Manitoba Ne Répond Plus (C... |\n",
       "| SONMHGO12A6D4F8E6A | Marcha Do Cordão Da Bola P... |\n",
       "+--------------------+-------------------------------+\n",
       "+-------------------------------+--------------------------+------+--------------+\n",
       "|            release            |       artist_name        | year | title_length |\n",
       "+-------------------------------+--------------------------+------+--------------+\n",
       "|      Allegory Of Hearing      |      Roy Montgomery      | None |      47      |\n",
       "|        Como en un baile       |         Guayacan         | None |      46      |\n",
       "|        British Tour '76       |       Renaissance        | None |      46      |\n",
       "| Every Red Heart Shines Tow... |       Red Sparowes       | 2006 |      45      |\n",
       "| Too Marvelous For Words - ... |       Lee Lessack        | None |      44      |\n",
       "|       Live At The Sands       |       Dean Martin        | None |      44      |\n",
       "| The Cat & The Fiddle - 66 ... | The Mother Goose Singers | None |      40      |\n",
       "| 1999 International Barbers... |        Swing City        | None |      40      |\n",
       "|    Manitoba Ne Répond Plus    |      Gérard Manset       | 2008 |      39      |\n",
       "|     Elizeth No Bola Preta     |     Elizeth Cardoso      | None |      39      |\n",
       "+-------------------------------+--------------------------+------+--------------+\n",
       "+---------------+-----------+------------+\n",
       "| how_old_was_i | my_rating | love_count |\n",
       "+---------------+-----------+------------+\n",
       "|      None     |    1.0    |     0      |\n",
       "|      None     |    1.0    |     0      |\n",
       "|      None     |    1.0    |     1      |\n",
       "|       18      |    1.0    |     0      |\n",
       "|      None     |    1.0    |     0      |\n",
       "|      None     |    1.0    |     1      |\n",
       "|      None     |    1.0    |     0      |\n",
       "|      None     |    1.0    |     1      |\n",
       "|       20      |    1.0    |     0      |\n",
       "|      None     |    1.0    |     0      |\n",
       "+---------------+-----------+------------+\n",
       "[10 rows x 9 columns]"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "top10_titles = song_sf.topk('title_length')\n",
    "top10_titles"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Resolution Island Suite I) A Vessel Sublime II) And But A Gentle Swell III) Hubris Fills The Rash And Young Iv) Now The Reef-Dashed Mariner V) The Sirens_ They Feel Pity Vi) Wind Upon The Sails_ Light Upon The Sea Vii) Cast Away The Island_ Cruel S'"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "top10_titles['title'][0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Makes sense...looks like a song with several movements.  I'm somewhat curious about the titles with no words too."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\"><table frame=\"box\" rules=\"cols\">\n",
       "    <tr>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">song_id</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">title</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">release</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">artist_name</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">year</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">title_length</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">how_old_was_i</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">my_rating</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">love_count</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOAGRAA12AB018D567</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\"></td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Puce de luxe</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Sébastien Roch</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">None</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">None</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOOAFJX12AB018A028</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\"></td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Puce de luxe</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Sébastien Roch</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">None</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">None</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOLDTFD12AB018AFE6</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\"></td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Puce de luxe</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Sébastien Roch</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">None</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">None</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOASSAS12AB018AFCF</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\"></td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Puce de luxe</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Sébastien Roch</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">None</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">None</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOQUGMS12AB018B01D</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\"></td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Puce de luxe</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Sébastien Roch</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">None</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">None</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "    </tr>\n",
       "</table>\n",
       "[5 rows x 9 columns]<br/>\n",
       "</div>"
      ],
      "text/plain": [
       "Columns:\n",
       "\tsong_id\tstr\n",
       "\ttitle\tstr\n",
       "\trelease\tstr\n",
       "\tartist_name\tstr\n",
       "\tyear\tint\n",
       "\ttitle_length\tint\n",
       "\thow_old_was_i\tint\n",
       "\tmy_rating\tfloat\n",
       "\tlove_count\tint\n",
       "\n",
       "Rows: 5\n",
       "\n",
       "Data:\n",
       "+--------------------+-------+--------------+----------------+------+--------------+\n",
       "|      song_id       | title |   release    |  artist_name   | year | title_length |\n",
       "+--------------------+-------+--------------+----------------+------+--------------+\n",
       "| SOAGRAA12AB018D567 |       | Puce de luxe | Sébastien Roch | None |      0       |\n",
       "| SOOAFJX12AB018A028 |       | Puce de luxe | Sébastien Roch | None |      0       |\n",
       "| SOLDTFD12AB018AFE6 |       | Puce de luxe | Sébastien Roch | None |      0       |\n",
       "| SOASSAS12AB018AFCF |       | Puce de luxe | Sébastien Roch | None |      0       |\n",
       "| SOQUGMS12AB018B01D |       | Puce de luxe | Sébastien Roch | None |      0       |\n",
       "+--------------------+-------+--------------+----------------+------+--------------+\n",
       "+---------------+-----------+------------+\n",
       "| how_old_was_i | my_rating | love_count |\n",
       "+---------------+-----------+------------+\n",
       "|      None     |    1.0    |     0      |\n",
       "|      None     |    1.0    |     0      |\n",
       "|      None     |    1.0    |     0      |\n",
       "|      None     |    1.0    |     0      |\n",
       "|      None     |    1.0    |     0      |\n",
       "+---------------+-----------+------------+\n",
       "[5 rows x 9 columns]"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "song_sf.topk('title_length', k=5, reverse=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here are a couple boolean operations too, with which I can prove that there were, in fact, songs before I was born.  Just not all of them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(False, True)"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "before_i_was_born = song_sf['how_old_was_i'] < 0\n",
    "before_i_was_born.all(), before_i_was_born.any()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Perhaps let's try some deeper analysis, like what albums have the most songs?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\"><table frame=\"box\" rules=\"cols\">\n",
       "    <tr>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">artist_name</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">release</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">num_songs_in_album</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Fanny</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">First Time In A Long<br>Time: The Reprise ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">85</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Bernard Herrmann</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">The Twilight Zone</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">81</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Spanky &amp; Our Gang</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">The Complete Mercury<br>Recordings ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">75</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">The Smashing Pumpkins</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Rarities &amp; B-Sides</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">72</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Big Star</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Keep An Eye On The Sky</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">71</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Jacques Dutronc</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Intégrale Les Cactus</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">69</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">The Stooges</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1970: The Complete Fun<br>House Sessions ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">67</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Lull</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Moments</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">64</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Willie Clancy</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Willie Clancy The Gold<br>Ring ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">61</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Jack Dangers</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Forbidden Planet Explored<br>/ Sci-Fi Sound Effects ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">60</td>\n",
       "    </tr>\n",
       "</table>\n",
       "[10 rows x 3 columns]<br/>\n",
       "</div>"
      ],
      "text/plain": [
       "Columns:\n",
       "\tartist_name\tstr\n",
       "\trelease\tstr\n",
       "\tnum_songs_in_album\tint\n",
       "\n",
       "Rows: 10\n",
       "\n",
       "Data:\n",
       "+-----------------------+-------------------------------+--------------------+\n",
       "|      artist_name      |            release            | num_songs_in_album |\n",
       "+-----------------------+-------------------------------+--------------------+\n",
       "|         Fanny         | First Time In A Long Time:... |         85         |\n",
       "|    Bernard Herrmann   |       The Twilight Zone       |         81         |\n",
       "|   Spanky & Our Gang   | The Complete Mercury Recor... |         75         |\n",
       "| The Smashing Pumpkins |       Rarities & B-Sides      |         72         |\n",
       "|        Big Star       |     Keep An Eye On The Sky    |         71         |\n",
       "|    Jacques Dutronc    |      Intégrale Les Cactus     |         69         |\n",
       "|      The Stooges      | 1970: The Complete Fun Hou... |         67         |\n",
       "|          Lull         |            Moments            |         64         |\n",
       "|     Willie Clancy     |  Willie Clancy The Gold Ring  |         61         |\n",
       "|      Jack Dangers     | Forbidden Planet Explored ... |         60         |\n",
       "+-----------------------+-------------------------------+--------------------+\n",
       "[10 rows x 3 columns]"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "song_sf.groupby(['artist_name', 'release'], {'num_songs_in_album' : gl.aggregate.COUNT}).topk('num_songs_in_album')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our groupby function only supports aggregation after grouping.  The aggregation functions you can use are listed <a href=\"https://dato.com/products/create/docs/generated/graphlab.data_structures.html#module-graphlab.aggregate\">here</a>."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can only go so far in analyzing this data though.  We might want to match this data with user information, like how many times a certain person played one of these songs.  For that, we need to the join function, but first we need to read this data in as an SFrame:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre>PROGRESS: Downloading http://s3.amazonaws.com/dato-datasets/millionsong/10000.txt to C:/Users/Evan/AppData/Local/Temp/graphlab-Evan/6216/bb0b5555-6152-493b-b13c-a4a68e0d376a.txt</pre>"
      ],
      "text/plain": [
       "PROGRESS: Downloading http://s3.amazonaws.com/dato-datasets/millionsong/10000.txt to C:/Users/Evan/AppData/Local/Temp/graphlab-Evan/6216/bb0b5555-6152-493b-b13c-a4a68e0d376a.txt"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>PROGRESS: Finished parsing file http://s3.amazonaws.com/dato-datasets/millionsong/10000.txt</pre>"
      ],
      "text/plain": [
       "PROGRESS: Finished parsing file http://s3.amazonaws.com/dato-datasets/millionsong/10000.txt"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>PROGRESS: Parsing completed. Parsed 100 lines in 0.680234 secs.</pre>"
      ],
      "text/plain": [
       "PROGRESS: Parsing completed. Parsed 100 lines in 0.680234 secs."
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>PROGRESS: Read 844838 lines. Lines per second: 1.19595e+006</pre>"
      ],
      "text/plain": [
       "PROGRESS: Read 844838 lines. Lines per second: 1.19595e+006"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>PROGRESS: Finished parsing file http://s3.amazonaws.com/dato-datasets/millionsong/10000.txt</pre>"
      ],
      "text/plain": [
       "PROGRESS: Finished parsing file http://s3.amazonaws.com/dato-datasets/millionsong/10000.txt"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>PROGRESS: Parsing completed. Parsed 2000000 lines in 1.12524 secs.</pre>"
      ],
      "text/plain": [
       "PROGRESS: Parsing completed. Parsed 2000000 lines in 1.12524 secs."
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\"><table frame=\"box\" rules=\"cols\">\n",
       "    <tr>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">user_id</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">song_id</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">listen_count</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">b80344d063b5ccb3212f76538<br>f3d9e43d87dca9e ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOAKIMP12A8C130995</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">b80344d063b5ccb3212f76538<br>f3d9e43d87dca9e ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOBBMDR12A8C13253B</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">b80344d063b5ccb3212f76538<br>f3d9e43d87dca9e ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOBXHDL12A81C204C0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">b80344d063b5ccb3212f76538<br>f3d9e43d87dca9e ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOBYHAJ12A6701BF1D</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">b80344d063b5ccb3212f76538<br>f3d9e43d87dca9e ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SODACBL12A8C13C273</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">b80344d063b5ccb3212f76538<br>f3d9e43d87dca9e ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SODDNQT12A6D4F5F7E</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">b80344d063b5ccb3212f76538<br>f3d9e43d87dca9e ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SODXRTY12AB0180F3B</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">b80344d063b5ccb3212f76538<br>f3d9e43d87dca9e ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOFGUAY12AB017B0A8</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">b80344d063b5ccb3212f76538<br>f3d9e43d87dca9e ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOFRQTD12A81C233C0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">b80344d063b5ccb3212f76538<br>f3d9e43d87dca9e ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOHQWYZ12A6D4FA701</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "    </tr>\n",
       "</table>\n",
       "[2000000 rows x 3 columns]<br/>Note: Only the head of the SFrame is printed.<br/>You can use print_rows(num_rows=m, num_columns=n) to print more rows and columns.\n",
       "</div>"
      ],
      "text/plain": [
       "Columns:\n",
       "\tuser_id\tstr\n",
       "\tsong_id\tstr\n",
       "\tlisten_count\tint\n",
       "\n",
       "Rows: 2000000\n",
       "\n",
       "Data:\n",
       "+-------------------------------+--------------------+--------------+\n",
       "|            user_id            |      song_id       | listen_count |\n",
       "+-------------------------------+--------------------+--------------+\n",
       "| b80344d063b5ccb3212f76538f... | SOAKIMP12A8C130995 |      1       |\n",
       "| b80344d063b5ccb3212f76538f... | SOBBMDR12A8C13253B |      2       |\n",
       "| b80344d063b5ccb3212f76538f... | SOBXHDL12A81C204C0 |      1       |\n",
       "| b80344d063b5ccb3212f76538f... | SOBYHAJ12A6701BF1D |      1       |\n",
       "| b80344d063b5ccb3212f76538f... | SODACBL12A8C13C273 |      1       |\n",
       "| b80344d063b5ccb3212f76538f... | SODDNQT12A6D4F5F7E |      5       |\n",
       "| b80344d063b5ccb3212f76538f... | SODXRTY12AB0180F3B |      1       |\n",
       "| b80344d063b5ccb3212f76538f... | SOFGUAY12AB017B0A8 |      1       |\n",
       "| b80344d063b5ccb3212f76538f... | SOFRQTD12A81C233C0 |      1       |\n",
       "| b80344d063b5ccb3212f76538f... | SOHQWYZ12A6D4FA701 |      1       |\n",
       "+-------------------------------+--------------------+--------------+\n",
       "[2000000 rows x 3 columns]\n",
       "Note: Only the head of the SFrame is printed.\n",
       "You can use print_rows(num_rows=m, num_columns=n) to print more rows and columns."
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "usage_data = gl.SFrame.read_csv(\"http://s3.amazonaws.com/dato-datasets/millionsong/10000.txt\", header=False, delimiter='\\t', column_type_hints={'X3':int})\n",
    "usage_data.rename({'X1':'user_id', 'X2':'song_id', 'X3':'listen_count'})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I could just join the listen data with the song data, but maybe I'll do something a bit more interesting.  Let's find out how many users from this dataset have listened to any one of those songs from my high school times, compared to the total number of users.  First we need the total number of users:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "76353\n"
     ]
    }
   ],
   "source": [
    "num_users = len(usage_data['user_id'].unique())\n",
    "print num_users"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "287\n",
      "0.0037588568884\n"
     ]
    }
   ],
   "source": [
    "fav_hs_listen_data = my_fav_hs_songs.join(usage_data, 'song_id')\n",
    "num_fav_hs_users = len(fav_hs_listen_data['user_id'].unique())\n",
    "print num_fav_hs_users\n",
    "print float(num_fav_hs_users) / float(num_users)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's really small.  Those other people don't know what they're missing.  Maybe the small proportion is simply because I'm only using a list of 42 songs.  For kicks, what is the most popular song of that set of songs?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\"><table frame=\"box\" rules=\"cols\">\n",
       "    <tr>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">song_id</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">num_unique_users</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">total_listens</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">title</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">release</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOCVOVH12A6D4FB912</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">107</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">555</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Keasbey Nights (LP<br>Version) ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Keasbey Nights</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOPSQOS12A6D4F9E15</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">79</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">207</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">High Of 75 (Album<br>Version) ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">MMHMM</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOINPKF12A6D4FDC75</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">63</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">196</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">A Better Place_ A Better<br>Time (Album Version) ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Everything Goes Numb</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOCLCYG12A6D4FDC71</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">67</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">188</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Point/Counterpoint (Album<br>Version) ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Everything Goes Numb</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOUUYHK12A6D4F9E16</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">79</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">171</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">I So Hate Consequences<br>(Album Version) ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">MMHMM</td>\n",
       "    </tr>\n",
       "</table>\n",
       "<table frame=\"box\" rules=\"cols\">\n",
       "    <tr>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">artist_name</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">year</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">title_length</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">how_old_was_i</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">my_rating</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">love_count</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Streetlight Manifesto</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2006</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">4</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">18</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Relient K</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2004</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">5</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">16</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Streetlight Manifesto</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2003</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">8</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">15</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Streetlight Manifesto</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2003</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">3</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">15</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Relient K</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2004</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">6</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">16</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "    </tr>\n",
       "</table>\n",
       "[5 rows x 11 columns]<br/>\n",
       "</div>"
      ],
      "text/plain": [
       "Columns:\n",
       "\tsong_id\tstr\n",
       "\tnum_unique_users\tint\n",
       "\ttotal_listens\tint\n",
       "\ttitle\tstr\n",
       "\trelease\tstr\n",
       "\tartist_name\tstr\n",
       "\tyear\tint\n",
       "\ttitle_length\tint\n",
       "\thow_old_was_i\tint\n",
       "\tmy_rating\tfloat\n",
       "\tlove_count\tint\n",
       "\n",
       "Rows: 5\n",
       "\n",
       "Data:\n",
       "+--------------------+------------------+---------------+\n",
       "|      song_id       | num_unique_users | total_listens |\n",
       "+--------------------+------------------+---------------+\n",
       "| SOCVOVH12A6D4FB912 |       107        |      555      |\n",
       "| SOPSQOS12A6D4F9E15 |        79        |      207      |\n",
       "| SOINPKF12A6D4FDC75 |        63        |      196      |\n",
       "| SOCLCYG12A6D4FDC71 |        67        |      188      |\n",
       "| SOUUYHK12A6D4F9E16 |        79        |      171      |\n",
       "+--------------------+------------------+---------------+\n",
       "+-------------------------------+----------------------+-----------------------+\n",
       "|             title             |       release        |      artist_name      |\n",
       "+-------------------------------+----------------------+-----------------------+\n",
       "|  Keasbey Nights (LP Version)  |    Keasbey Nights    | Streetlight Manifesto |\n",
       "|   High Of 75 (Album Version)  |        MMHMM         |       Relient K       |\n",
       "| A Better Place_ A Better T... | Everything Goes Numb | Streetlight Manifesto |\n",
       "| Point/Counterpoint (Album ... | Everything Goes Numb | Streetlight Manifesto |\n",
       "| I So Hate Consequences (Al... |        MMHMM         |       Relient K       |\n",
       "+-------------------------------+----------------------+-----------------------+\n",
       "+------+--------------+---------------+-----------+------------+\n",
       "| year | title_length | how_old_was_i | my_rating | love_count |\n",
       "+------+--------------+---------------+-----------+------------+\n",
       "| 2006 |      4       |       18      |    1.0    |     0      |\n",
       "| 2004 |      5       |       16      |    1.0    |     0      |\n",
       "| 2003 |      8       |       15      |    1.0    |     0      |\n",
       "| 2003 |      3       |       15      |    1.0    |     0      |\n",
       "| 2004 |      6       |       16      |    1.0    |     0      |\n",
       "+------+--------------+---------------+-----------+------------+\n",
       "[5 rows x 11 columns]"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "most_popular = fav_hs_listen_data.groupby(['song_id'], {'total_listens':gl.aggregate.SUM('listen_count'),\n",
    "                                                     'num_unique_users':gl.aggregate.COUNT('user_id')})\n",
    "most_popular.join(song_sf, 'song_id').topk('total_listens',k=20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "...and only 5 even got listens, but \"Keasbey Nights\" wins from this small subset.  Now, suppose I was a cheater and wanted to make this look a little better?  I'll pretend I am so you can see 'append' in action."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\"><table frame=\"box\" rules=\"cols\">\n",
       "    <tr>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">song_id</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">num_unique_users</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">total_listens</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">title</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">release</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOSFAVU12A6D4FDC6A</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">4000</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Everything Went Numb<br>(Album Version) ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Everything Goes Numb</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOCVOVH12A6D4FB912</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">107</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">555</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Keasbey Nights (LP<br>Version) ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Keasbey Nights</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOPSQOS12A6D4F9E15</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">79</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">207</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">High Of 75 (Album<br>Version) ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">MMHMM</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOINPKF12A6D4FDC75</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">63</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">196</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">A Better Place_ A Better<br>Time (Album Version) ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Everything Goes Numb</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOCLCYG12A6D4FDC71</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">67</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">188</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Point/Counterpoint (Album<br>Version) ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Everything Goes Numb</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">SOUUYHK12A6D4F9E16</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">79</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">171</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">I So Hate Consequences<br>(Album Version) ...</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">MMHMM</td>\n",
       "    </tr>\n",
       "</table>\n",
       "<table frame=\"box\" rules=\"cols\">\n",
       "    <tr>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">artist_name</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">year</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">title_length</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">how_old_was_i</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">my_rating</th>\n",
       "        <th style=\"padding-left: 1em; padding-right: 1em; text-align: center\">love_count</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Streetlight Manifesto</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2003</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">5</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">15</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Streetlight Manifesto</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2006</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">4</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">18</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Relient K</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2004</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">5</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">16</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Streetlight Manifesto</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2003</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">8</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">15</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Streetlight Manifesto</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2003</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">3</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">15</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">Relient K</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">2004</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">6</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">16</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">1.0</td>\n",
       "        <td style=\"padding-left: 1em; padding-right: 1em; text-align: center; vertical-align: top\">0</td>\n",
       "    </tr>\n",
       "</table>\n",
       "[6 rows x 11 columns]<br/>\n",
       "</div>"
      ],
      "text/plain": [
       "Columns:\n",
       "\tsong_id\tstr\n",
       "\tnum_unique_users\tint\n",
       "\ttotal_listens\tint\n",
       "\ttitle\tstr\n",
       "\trelease\tstr\n",
       "\tartist_name\tstr\n",
       "\tyear\tint\n",
       "\ttitle_length\tint\n",
       "\thow_old_was_i\tint\n",
       "\tmy_rating\tfloat\n",
       "\tlove_count\tint\n",
       "\n",
       "Rows: 6\n",
       "\n",
       "Data:\n",
       "+--------------------+------------------+---------------+\n",
       "|      song_id       | num_unique_users | total_listens |\n",
       "+--------------------+------------------+---------------+\n",
       "| SOSFAVU12A6D4FDC6A |        1         |      4000     |\n",
       "| SOCVOVH12A6D4FB912 |       107        |      555      |\n",
       "| SOPSQOS12A6D4F9E15 |        79        |      207      |\n",
       "| SOINPKF12A6D4FDC75 |        63        |      196      |\n",
       "| SOCLCYG12A6D4FDC71 |        67        |      188      |\n",
       "| SOUUYHK12A6D4F9E16 |        79        |      171      |\n",
       "+--------------------+------------------+---------------+\n",
       "+-------------------------------+----------------------+-----------------------+\n",
       "|             title             |       release        |      artist_name      |\n",
       "+-------------------------------+----------------------+-----------------------+\n",
       "| Everything Went Numb (Albu... | Everything Goes Numb | Streetlight Manifesto |\n",
       "|  Keasbey Nights (LP Version)  |    Keasbey Nights    | Streetlight Manifesto |\n",
       "|   High Of 75 (Album Version)  |        MMHMM         |       Relient K       |\n",
       "| A Better Place_ A Better T... | Everything Goes Numb | Streetlight Manifesto |\n",
       "| Point/Counterpoint (Album ... | Everything Goes Numb | Streetlight Manifesto |\n",
       "| I So Hate Consequences (Al... |        MMHMM         |       Relient K       |\n",
       "+-------------------------------+----------------------+-----------------------+\n",
       "+------+--------------+---------------+-----------+------------+\n",
       "| year | title_length | how_old_was_i | my_rating | love_count |\n",
       "+------+--------------+---------------+-----------+------------+\n",
       "| 2003 |      5       |       15      |    1.0    |     0      |\n",
       "| 2006 |      4       |       18      |    1.0    |     0      |\n",
       "| 2004 |      5       |       16      |    1.0    |     0      |\n",
       "| 2003 |      8       |       15      |    1.0    |     0      |\n",
       "| 2003 |      3       |       15      |    1.0    |     0      |\n",
       "| 2004 |      6       |       16      |    1.0    |     0      |\n",
       "+------+--------------+---------------+-----------+------------+\n",
       "[6 rows x 11 columns]"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "me = gl.SFrame({'user_id':['evan'],'song_id':['SOSFAVU12A6D4FDC6A'],'listen_count':[4000]})\n",
    "usage_data = usage_data.append(me)\n",
    "fav_hs_listen_data = my_fav_hs_songs.join(usage_data, 'song_id')\n",
    "most_popular = fav_hs_listen_data.groupby(['song_id'], {'total_listens':gl.aggregate.SUM('listen_count'),\n",
    "                                                     'num_unique_users':gl.aggregate.COUNT('user_id')})\n",
    "most_popular.join(song_sf, 'song_id').topk('total_listens',k=20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Splitting and Sampling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We're almost done with the tour of features.  For easy splitting into training and test sets, we have the random_split function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(800135, 199865)"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Randomly split data rows into two subsets\n",
    "first_set, second_set = song_sf.random_split(0.8, seed = 1)\n",
    "first_set.num_rows(), second_set.num_rows()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to split on a predicate though, you'll have to do that manually."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(69968, 445608)"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "songs_before = song_sf[song_sf['how_old_was_i'] < 0]\n",
    "songs_after = song_sf[song_sf['how_old_was_i'] >= 0]\n",
    "songs_before.num_rows(), songs_after.num_rows()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also get a random sample of the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "370008"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pct37 = song_sf.sample(.37)\n",
    "pct37.num_rows()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Other Cool Features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "SArrays support lots of mathematical operations.  They can be performed with a scalar"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2L, 4L, 6L]\n"
     ]
    }
   ],
   "source": [
    "sa = gl.SArray([1,2,3])\n",
    "sa2 = sa * 2\n",
    "print sa2\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "...or they can be performed element-wise with another SArray."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[3L, 6L, 9L]\n",
      "[0.5, 0.5, 0.5]\n"
     ]
    }
   ],
   "source": [
    "add = sa + sa2\n",
    "div = sa / sa2\n",
    "print add\n",
    "print div"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also iterate over SArrays and SFrames.  When iterating over an SFrame, the returned element is a Python dictionary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Whoa that's long!\n",
      "Whoa that's long!\n",
      "Whoa that's long!\n",
      "Whoa that's long!\n"
     ]
    }
   ],
   "source": [
    "for i in song_sf:\n",
    "    if i['title_length'] >= 45:\n",
    "        print \"Whoa that's long!\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Saving Our Work"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I think I'm done exploring this dataset, but I'd like to save it for later.  There's a couple ways I can do this.  I can save it to a csv:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "song_sf.save('new_song_data.csv', format='csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or I can just save it as an SFrame as I showed earlier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "song_sf.save('new_song_data_sframe')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And of course, we can do all of this on S3.  Note that if you download this notebook and run it, you won't be able to save to our dato-datasets bucket.  Simply set your AWS credentials and uncomment the code below (replacing our S3 bucket with yours) to see this in action."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# In order to save to S3, you will need to use your own bucket and your own credentials.\n",
    "# You can set your AWS credentials using the below function:\n",
    "# graphlab.aws.set_credentials(<access_key_id>, <secret_access_key>)\n",
    "\n",
    "#song_sf.save('s3://dato-datasets/my_sframes/new_song_sframe')       # S3://<bucket-name>/<file-path>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now to load an SFrame back, we use the handy 'load_sframe' function like before.  This takes the name of the sframe's top level directory:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# The below will download about 78 MB.\n",
    "#hello_again = gl.load_sframe('s3://dato-datasets/my_sframes/new_song_sframe')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "SArrays can be saved in a similar fashion.  That's it!  Our <a href=\"https://dato.com/products/create/docs\">API reference</a> covers every function associated with SFrames and SArrays in detail."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}