NYC Taxi 4 - Custom Featuretools Primitives.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<center><a href=\"https://www.featuretools.com/\"><img src=\"img/featuretools-logo.png\" width=\"400\" height=\"200\" /></a></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2> An Advanced Featuretools Approach with Custom Primitives</h2>\n",
    "<p>The following tutorial illustrates an advanced featuretools model for the NYC Taxi Trip Duration competition on Kaggle using our custom primitive API. You will need to download the following five files into a `data` folder in this repository.</p>\n",
    "\n",
    "<h2>Step 1: Download raw data </h2>\n",
    "<a href=\"https://www.kaggle.com/c/nyc-taxi-trip-duration/data\">test.csv and train.csv</a>\n",
    "\n",
    "The functions used here don't appear until Featuretools version 0.1.9. If you're using a lower version you will get an error."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'0.6.0'"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import featuretools as ft\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import taxi_utils\n",
    "ft.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>Step 2: Prepare the Data </h2>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "TRAIN_DIR = \"data/train.csv\"\n",
    "TEST_DIR = \"data/test.csv\"\n",
    "\n",
    "data_train, data_test = taxi_utils.read_data(TRAIN_DIR, TEST_DIR)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `fastest_routes` dataset has information on the shortest path between two coordinates in NYC. We can merge it in with our dataset here and then merge together our test and train datasets after marking them as such."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>dropoff_latitude</th>\n",
       "      <th>dropoff_longitude</th>\n",
       "      <th>id</th>\n",
       "      <th>passenger_count</th>\n",
       "      <th>pickup_datetime</th>\n",
       "      <th>pickup_latitude</th>\n",
       "      <th>pickup_longitude</th>\n",
       "      <th>store_and_fwd_flag</th>\n",
       "      <th>test_data</th>\n",
       "      <th>trip_duration</th>\n",
       "      <th>vendor_id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>40.765602</td>\n",
       "      <td>-73.964630</td>\n",
       "      <td>id2875421</td>\n",
       "      <td>1</td>\n",
       "      <td>2016-03-14 17:24:55</td>\n",
       "      <td>40.767937</td>\n",
       "      <td>-73.982155</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>455.0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>40.731152</td>\n",
       "      <td>-73.999481</td>\n",
       "      <td>id2377394</td>\n",
       "      <td>1</td>\n",
       "      <td>2016-06-12 00:43:35</td>\n",
       "      <td>40.738564</td>\n",
       "      <td>-73.980415</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>663.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>40.710087</td>\n",
       "      <td>-74.005333</td>\n",
       "      <td>id3858529</td>\n",
       "      <td>1</td>\n",
       "      <td>2016-01-19 11:35:24</td>\n",
       "      <td>40.763939</td>\n",
       "      <td>-73.979027</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>2124.0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>40.706718</td>\n",
       "      <td>-74.012268</td>\n",
       "      <td>id3504673</td>\n",
       "      <td>1</td>\n",
       "      <td>2016-04-06 19:32:31</td>\n",
       "      <td>40.719971</td>\n",
       "      <td>-74.010040</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>429.0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>40.782520</td>\n",
       "      <td>-73.972923</td>\n",
       "      <td>id2181028</td>\n",
       "      <td>1</td>\n",
       "      <td>2016-03-26 13:30:55</td>\n",
       "      <td>40.793209</td>\n",
       "      <td>-73.973053</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>435.0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   dropoff_latitude  dropoff_longitude         id  passenger_count  \\\n",
       "0         40.765602         -73.964630  id2875421                1   \n",
       "1         40.731152         -73.999481  id2377394                1   \n",
       "2         40.710087         -74.005333  id3858529                1   \n",
       "3         40.706718         -74.012268  id3504673                1   \n",
       "4         40.782520         -73.972923  id2181028                1   \n",
       "\n",
       "      pickup_datetime  pickup_latitude  pickup_longitude  store_and_fwd_flag  \\\n",
       "0 2016-03-14 17:24:55        40.767937        -73.982155               False   \n",
       "1 2016-06-12 00:43:35        40.738564        -73.980415               False   \n",
       "2 2016-01-19 11:35:24        40.763939        -73.979027               False   \n",
       "3 2016-04-06 19:32:31        40.719971        -74.010040               False   \n",
       "4 2016-03-26 13:30:55        40.793209        -73.973053               False   \n",
       "\n",
       "   test_data  trip_duration  vendor_id  \n",
       "0      False          455.0          2  \n",
       "1      False          663.0          1  \n",
       "2      False         2124.0          2  \n",
       "3      False          429.0          2  \n",
       "4      False          435.0          2  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Make a train/test column\n",
    "data_train['test_data'] = False\n",
    "data_test['test_data'] = True\n",
    "\n",
    "# Combine the data and convert some strings\n",
    "data = pd.concat([data_train, data_test], sort=True)\n",
    "data.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3: Custom Primitives\n",
    "The custom primitive API is new to Featuretools version 0.1.9. Our workflow will be as follows:\n",
    "1. Make a new LatLong class which is a tuple of a latitude and longitude\n",
    "2. Define some functions for LatLong\n",
    "3. Make new primitives from those functions\n",
    "\n",
    "For the first step, we'll make new columns for the pickup and dropoff locations which are tuples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "data[\"pickup_latlong\"] = data[['pickup_latitude', 'pickup_longitude']].apply(tuple, axis=1)\n",
    "data[\"dropoff_latlong\"] = data[['dropoff_latitude', 'dropoff_longitude']].apply(tuple, axis=1)\n",
    "data = data.drop([\"pickup_latitude\", \"pickup_longitude\", \"dropoff_latitude\", \"dropoff_longitude\"], axis = 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can define the `pickup_latlong` to be a `LatLong` type. This is a way to tell DFS that certain primitives are only important in one direction, from the beginning of a trip to the end. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from featuretools import variable_types as vtypes\n",
    "from featuretools.variable_types import LatLong\n",
    "    \n",
    "trip_variable_types = {\n",
    "    'passenger_count': vtypes.Ordinal, \n",
    "    'vendor_id': vtypes.Categorical,\n",
    "    'pickup_latlong': LatLong,\n",
    "    'dropoff_latlong': LatLong,\n",
    "}\n",
    "\n",
    "es = ft.EntitySet(\"taxi\")\n",
    "\n",
    "es.entity_from_dataframe(entity_id=\"trips\",\n",
    "                         dataframe=data,\n",
    "                         index=\"id\",\n",
    "                         time_index='pickup_datetime',\n",
    "                         variable_types=trip_variable_types)\n",
    "\n",
    "es.normalize_entity(base_entity_id=\"trips\",\n",
    "                    new_entity_id=\"vendors\",\n",
    "                    index=\"vendor_id\")\n",
    "\n",
    "es.normalize_entity(base_entity_id=\"trips\",\n",
    "                    new_entity_id=\"passenger_cnt\",\n",
    "                    index=\"passenger_count\")\n",
    "\n",
    "cutoff_time = es['trips'].df[['id', 'pickup_datetime']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualize EntitySet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.40.1 (20161225.0304)\n",
       " -->\n",
       "<!-- Title: taxi Pages: 1 -->\n",
       "<svg width=\"488pt\" height=\"273pt\"\n",
       " viewBox=\"0.00 0.00 488.00 273.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 269)\">\n",
       "<title>taxi</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"transparent\" points=\"-4,4 -4,-269 484,-269 484,4 -4,4\"/>\n",
       "<!-- trips -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>trips</title>\n",
       "<polygon fill=\"none\" stroke=\"#000000\" points=\"120,-98.5 120,-264.5 359,-264.5 359,-98.5 120,-98.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"239.5\" y=\"-249.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">trips</text>\n",
       "<polyline fill=\"none\" stroke=\"#000000\" points=\"120,-241.5 359,-241.5 \"/>\n",
       "<text text-anchor=\"start\" x=\"128\" y=\"-226.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">id : index</text>\n",
       "<text text-anchor=\"start\" x=\"128\" y=\"-211.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">pickup_datetime : datetime_time_index</text>\n",
       "<text text-anchor=\"start\" x=\"128\" y=\"-196.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">store_and_fwd_flag : boolean</text>\n",
       "<text text-anchor=\"start\" x=\"128\" y=\"-181.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">test_data : boolean</text>\n",
       "<text text-anchor=\"start\" x=\"128\" y=\"-166.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">trip_duration : numeric</text>\n",
       "<text text-anchor=\"start\" x=\"128\" y=\"-151.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">passenger_count : id</text>\n",
       "<text text-anchor=\"start\" x=\"128\" y=\"-136.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">vendor_id : id</text>\n",
       "<text text-anchor=\"start\" x=\"128\" y=\"-121.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">pickup_latlong : latlong</text>\n",
       "<text text-anchor=\"start\" x=\"128\" y=\"-106.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">dropoff_latlong : latlong</text>\n",
       "</g>\n",
       "<!-- vendors -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>vendors</title>\n",
       "<polygon fill=\"none\" stroke=\"#000000\" points=\"0,-.5 0,-61.5 231,-61.5 231,-.5 0,-.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"115.5\" y=\"-46.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">vendors</text>\n",
       "<polyline fill=\"none\" stroke=\"#000000\" points=\"0,-38.5 231,-38.5 \"/>\n",
       "<text text-anchor=\"start\" x=\"8\" y=\"-23.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">vendor_id : index</text>\n",
       "<text text-anchor=\"start\" x=\"8\" y=\"-8.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">first_trips_time : datetime_time_index</text>\n",
       "</g>\n",
       "<!-- trips&#45;&gt;vendors -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>trips&#45;&gt;vendors</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M175.5,-98.3266C175.5,-98.3266 175.5,-71.7282 175.5,-71.7282\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"179.0001,-71.7281 175.5,-61.7282 172.0001,-71.7282 179.0001,-71.7281\"/>\n",
       "<text text-anchor=\"middle\" x=\"147\" y=\"-73.8274\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">vendor_id</text>\n",
       "</g>\n",
       "<!-- passenger_cnt -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>passenger_cnt</title>\n",
       "<polygon fill=\"none\" stroke=\"#000000\" points=\"249,-.5 249,-61.5 480,-61.5 480,-.5 249,-.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"364.5\" y=\"-46.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">passenger_cnt</text>\n",
       "<polyline fill=\"none\" stroke=\"#000000\" points=\"249,-38.5 480,-38.5 \"/>\n",
       "<text text-anchor=\"start\" x=\"257\" y=\"-23.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">passenger_count : index</text>\n",
       "<text text-anchor=\"start\" x=\"257\" y=\"-8.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">first_trips_time : datetime_time_index</text>\n",
       "</g>\n",
       "<!-- trips&#45;&gt;passenger_cnt -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>trips&#45;&gt;passenger_cnt</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M304,-98.3266C304,-98.3266 304,-71.7282 304,-71.7282\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"307.5001,-71.7281 304,-61.7282 300.5001,-71.7282 307.5001,-71.7281\"/>\n",
       "<text text-anchor=\"middle\" x=\"257\" y=\"-73.8274\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">passenger_count</text>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.dot.Digraph at 0x118d88828>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "es.plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we can create primitives for our new `pickup_latlong` and `dropoff_latlong` which are of type `LatLong`. \n",
    "\n",
    "The distance between two `LatLong` types is most accurately represented by the Haversine distance: the nearest length along a sphere. We can also define the Cityblock distance (also called the taxicab metric), which takes into account that we can't move diagonally through buildings in Manhattan."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "from featuretools.primitives import make_agg_primitive, make_trans_primitive\n",
    "\n",
    "def haversine(latlong1, latlong2):\n",
    "    lat_1s = np.array([x[0] for x in latlong1])\n",
    "    lon_1s = np.array([x[1] for x in latlong1])\n",
    "    lat_2s = np.array([x[0] for x in latlong2])\n",
    "    lon_2s = np.array([x[1] for x in latlong2])\n",
    "    lon1, lat1, lon2, lat2 = map(np.radians, [lon_1s, lat_1s, lon_2s, lat_2s])\n",
    "    dlon = lon2 - lon1\n",
    "    dlat = lat2 - lat1\n",
    "    a = np.sin(dlat/2.0)**2 + np.cos(lat1) * np.cos(lat2) * np.sin(dlon/2.0)**2\n",
    "    km = 6367 * 2 * np.arcsin(np.sqrt(a))\n",
    "    return km\n",
    "\n",
    "def cityblock(latlong1, latlong2):\n",
    "    lon_dis = haversine(latlong1, latlong2)\n",
    "    lat_dist = haversine(latlong1, latlong2)\n",
    "    return lon_dis + lat_dist\n",
    "\n",
    "def bearing(latlong1, latlong2):\n",
    "    lat1 = np.array([x[0] for x in latlong1])\n",
    "    lon1 = np.array([x[1] for x in latlong1])\n",
    "    lat2 = np.array([x[0] for x in latlong2])\n",
    "    lon2 = np.array([x[1] for x in latlong2])\n",
    "    delta_lon = np.radians(lon2 - lon1)\n",
    "    lon1, lat1, lon2, lat2 = map(np.radians, [lon1, lat1, lon2, lat2])\n",
    "    x = np.cos(lat2) * np.sin(delta_lon)\n",
    "    y = np.cos(lat1) * np.sin(lat2) - np.sin(lat1) * np.cos(lat2) * np.cos(delta_lon)\n",
    "    return np.degrees(np.arctan2(x, y))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also make some primitives directly from `LatLong` types. In particular, we'll almost certainly want to extract the Latitude, the Longitude and the central point between two such coordinates."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p>Now lets make primitives from those functions!</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "Bearing = make_trans_primitive(function=bearing,\n",
    "                               input_types=[LatLong, LatLong],\n",
    "                               commutative=True,\n",
    "                               return_type=vtypes.Numeric)\n",
    "\n",
    "Cityblock = make_trans_primitive(function=cityblock,\n",
    "                                 input_types=[LatLong, LatLong],\n",
    "                                 commutative=True,\n",
    "                                 return_type=vtypes.Numeric)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "def is_rush_hour(datetime):\n",
    "    hour = pd.DatetimeIndex(datetime).hour\n",
    "    return (hour >= 7) & (hour <= 11)\n",
    "\n",
    "def is_noon_hour(datetime):\n",
    "    hour = pd.DatetimeIndex(datetime).hour\n",
    "    return (hour >= 11) & (hour <= 13)\n",
    "\n",
    "def is_night_hour(datetime):\n",
    "    hour = pd.DatetimeIndex(datetime).hour\n",
    "    return (hour >= 18) & (hour <= 23)\n",
    "\n",
    "RushHour = make_trans_primitive(function=is_rush_hour,\n",
    "                                input_types=[vtypes.Datetime],\n",
    "                                return_type = vtypes.Boolean)\n",
    "    \n",
    "NoonHour = make_trans_primitive(function=is_noon_hour,\n",
    "                                input_types=[vtypes.Datetime],\n",
    "                                return_type = vtypes.Boolean)\n",
    "        \n",
    "NightHour = make_trans_primitive(function=is_night_hour,\n",
    "                                 input_types=[vtypes.Datetime],\n",
    "                                 return_type = vtypes.Boolean)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have grouped together some times of day with `RushHour`, `NoonHour` and `NightHour`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "from featuretools.primitives import TransformPrimitive\n",
    "from featuretools.variable_types import Boolean, LatLong\n",
    "\n",
    "class GeoBox(TransformPrimitive):\n",
    "    name = \"GeoBox\"\n",
    "    input_types=[LatLong]\n",
    "    return_type=Boolean\n",
    "    \n",
    "    def __init__(self, bottomleft, topright):\n",
    "        self.bottomleft = bottomleft\n",
    "        self.topright = topright\n",
    "    \n",
    "    def get_function(self):\n",
    "        def geobox(latlong, bottomleft=self.bottomleft, topright=self.topright):\n",
    "            lat = np.array([x[0] for x in latlong])\n",
    "            lon = np.array([x[1] for x in latlong])\n",
    "            boxlats = [bottomleft[0],topright[0]]\n",
    "            boxlongs = [bottomleft[1], topright[1]]\n",
    "            output = []\n",
    "            for i, name in enumerate(lat):\n",
    "                if (min(boxlats) <= lat[i] and lat[i] <= max(boxlats) and \n",
    "                        min(boxlongs) <= lon[i] and lon[i] <= max(boxlongs)):\n",
    "                    output.append(True)\n",
    "                else: \n",
    "                    output.append(False)\n",
    "            return output\n",
    "        \n",
    "        return geobox\n",
    "\n",
    "    def generate_name(self, base_feature_names):\n",
    "        return u\"GEOBOX({}, {}, {})\".format(base_feature_names[0],\n",
    "                                            str(self.bottomleft),\n",
    "                                            str(self.topright))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The feature `Geobox` finds if a latlong is in a rectangle defined by two coordinate pairs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 4: Using custom primitives in DFS"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see what features are created if we run DFS now:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Built 14 features\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[<Feature: store_and_fwd_flag>,\n",
       " <Feature: test_data>,\n",
       " <Feature: trip_duration>,\n",
       " <Feature: passenger_count>,\n",
       " <Feature: vendor_id>,\n",
       " <Feature: BEARING(dropoff_latlong, pickup_latlong)>,\n",
       " <Feature: CITYBLOCK(dropoff_latlong, pickup_latlong)>,\n",
       " <Feature: GEOBOX(pickup_latlong, (40.62, -73.85), (40.7, -73.75))>,\n",
       " <Feature: GEOBOX(dropoff_latlong, (40.62, -73.85), (40.7, -73.75))>,\n",
       " <Feature: GEOBOX(pickup_latlong, (40.7, -73.97), (40.77, -73.9))>,\n",
       " <Feature: GEOBOX(dropoff_latlong, (40.7, -73.97), (40.77, -73.9))>,\n",
       " <Feature: IS_RUSH_HOUR(pickup_datetime)>,\n",
       " <Feature: IS_NOON_HOUR(pickup_datetime)>,\n",
       " <Feature: IS_NIGHT_HOUR(pickup_datetime)>]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "agg_primitives = []\n",
    "trans_primitives = [Bearing, Cityblock,\n",
    "                    GeoBox(bottomleft=(40.62, -73.85), topright=(40.70, -73.75)),\n",
    "                    GeoBox(bottomleft=(40.70, -73.97), topright=(40.77, -73.9)),\n",
    "                    RushHour,\n",
    "                    NoonHour,\n",
    "                    NightHour]\n",
    "\n",
    "# calculate feature_matrix using deep feature synthesis\n",
    "features = ft.dfs(entityset=es,\n",
    "                  target_entity=\"trips\",\n",
    "                  trans_primitives=trans_primitives,\n",
    "                  agg_primitives=agg_primitives,\n",
    "                  drop_contains=['trips.test_data'],\n",
    "                  verbose=True,\n",
    "                  cutoff_time=cutoff_time,\n",
    "                  approximate='36d',\n",
    "                  max_depth=3,\n",
    "                  max_features=40,\n",
    "                  features_only=True)\n",
    "features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our new features were applied exactly where they should be! The distances were only calcuated between the pickup_latlong and the dropoff_latlong, while primitives like `LATITUDE` were calculated on both latlong columns.\n",
    "\n",
    "For this dataset, we have approximately 2 million distinct cutoff times, times at which DFS will have to recalculate aggregation primitives. Calculating those features at specific times every hour and using that number instead will save computation time and perhaps not lose too much information. The `approximate` parameter in DFS lets you specify a window size to use when approximating.\n",
    "\n",
    "With the features from before, we calculate the whole feature matrix:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Built 103 features\n",
      "Elapsed: 46:12 | Remaining: 00:00 | Progress: 100%|██████████| Calculated: 11/11 chunks\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>store_and_fwd_flag</th>\n",
       "      <th>test_data</th>\n",
       "      <th>trip_duration</th>\n",
       "      <th>passenger_count</th>\n",
       "      <th>vendor_id</th>\n",
       "      <th>BEARING(dropoff_latlong, pickup_latlong)</th>\n",
       "      <th>CITYBLOCK(dropoff_latlong, pickup_latlong)</th>\n",
       "      <th>GEOBOX(pickup_latlong, (40.62, -73.85), (40.7, -73.75))</th>\n",
       "      <th>GEOBOX(dropoff_latlong, (40.62, -73.85), (40.7, -73.75))</th>\n",
       "      <th>GEOBOX(pickup_latlong, (40.7, -73.97), (40.77, -73.9))</th>\n",
       "      <th>...</th>\n",
       "      <th>passenger_cnt.MAX(trips.CITYBLOCK(dropoff_latlong, pickup_latlong))</th>\n",
       "      <th>passenger_cnt.MAX(trips.MINUTE(pickup_datetime))</th>\n",
       "      <th>passenger_cnt.NUM_UNIQUE(trips.DAY(pickup_datetime))</th>\n",
       "      <th>passenger_cnt.NUM_UNIQUE(trips.HOUR(pickup_datetime))</th>\n",
       "      <th>passenger_cnt.NUM_UNIQUE(trips.MONTH(pickup_datetime))</th>\n",
       "      <th>passenger_cnt.NUM_UNIQUE(trips.WEEKDAY(pickup_datetime))</th>\n",
       "      <th>passenger_cnt.NUM_UNIQUE(trips.WEEK(pickup_datetime))</th>\n",
       "      <th>passenger_cnt.SKEW(trips.BEARING(dropoff_latlong, pickup_latlong))</th>\n",
       "      <th>passenger_cnt.SKEW(trips.CITYBLOCK(dropoff_latlong, pickup_latlong))</th>\n",
       "      <th>passenger_cnt.SKEW(trips.MINUTE(pickup_datetime))</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>id0190469</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>849.0</td>\n",
       "      <td>5</td>\n",
       "      <td>2</td>\n",
       "      <td>-163.529609</td>\n",
       "      <td>25.497222</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>id0621643</th>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>-169.753936</td>\n",
       "      <td>11.856590</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>id1384355</th>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>-149.644870</td>\n",
       "      <td>40.735905</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>id1665586</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>1294.0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>-34.622058</td>\n",
       "      <td>8.015227</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>id1210365</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>408.0</td>\n",
       "      <td>5</td>\n",
       "      <td>2</td>\n",
       "      <td>-136.358219</td>\n",
       "      <td>4.339017</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 103 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "           store_and_fwd_flag  test_data  trip_duration  passenger_count  \\\n",
       "id                                                                         \n",
       "id0190469               False      False          849.0                5   \n",
       "id0621643               False       True            NaN                2   \n",
       "id1384355               False       True            NaN                1   \n",
       "id1665586               False      False         1294.0                1   \n",
       "id1210365               False      False          408.0                5   \n",
       "\n",
       "           vendor_id  BEARING(dropoff_latlong, pickup_latlong)  \\\n",
       "id                                                               \n",
       "id0190469          2                               -163.529609   \n",
       "id0621643          2                               -169.753936   \n",
       "id1384355          1                               -149.644870   \n",
       "id1665586          1                                -34.622058   \n",
       "id1210365          2                               -136.358219   \n",
       "\n",
       "           CITYBLOCK(dropoff_latlong, pickup_latlong)  \\\n",
       "id                                                      \n",
       "id0190469                                   25.497222   \n",
       "id0621643                                   11.856590   \n",
       "id1384355                                   40.735905   \n",
       "id1665586                                    8.015227   \n",
       "id1210365                                    4.339017   \n",
       "\n",
       "           GEOBOX(pickup_latlong, (40.62, -73.85), (40.7, -73.75))  \\\n",
       "id                                                                   \n",
       "id0190469                                              False         \n",
       "id0621643                                              False         \n",
       "id1384355                                              False         \n",
       "id1665586                                              False         \n",
       "id1210365                                              False         \n",
       "\n",
       "           GEOBOX(dropoff_latlong, (40.62, -73.85), (40.7, -73.75))  \\\n",
       "id                                                                    \n",
       "id0190469                                              False          \n",
       "id0621643                                              False          \n",
       "id1384355                                              False          \n",
       "id1665586                                              False          \n",
       "id1210365                                              False          \n",
       "\n",
       "           GEOBOX(pickup_latlong, (40.7, -73.97), (40.77, -73.9))  \\\n",
       "id                                                                  \n",
       "id0190469                                              False        \n",
       "id0621643                                              False        \n",
       "id1384355                                              False        \n",
       "id1665586                                              False        \n",
       "id1210365                                              False        \n",
       "\n",
       "                                 ...                          \\\n",
       "id                               ...                           \n",
       "id0190469                        ...                           \n",
       "id0621643                        ...                           \n",
       "id1384355                        ...                           \n",
       "id1665586                        ...                           \n",
       "id1210365                        ...                           \n",
       "\n",
       "           passenger_cnt.MAX(trips.CITYBLOCK(dropoff_latlong, pickup_latlong))  \\\n",
       "id                                                                               \n",
       "id0190469                                                NaN                     \n",
       "id0621643                                                NaN                     \n",
       "id1384355                                                NaN                     \n",
       "id1665586                                                NaN                     \n",
       "id1210365                                                NaN                     \n",
       "\n",
       "           passenger_cnt.MAX(trips.MINUTE(pickup_datetime))  \\\n",
       "id                                                            \n",
       "id0190469                                               NaN   \n",
       "id0621643                                               NaN   \n",
       "id1384355                                               NaN   \n",
       "id1665586                                               NaN   \n",
       "id1210365                                               NaN   \n",
       "\n",
       "           passenger_cnt.NUM_UNIQUE(trips.DAY(pickup_datetime))  \\\n",
       "id                                                                \n",
       "id0190469                                                NaN      \n",
       "id0621643                                                NaN      \n",
       "id1384355                                                NaN      \n",
       "id1665586                                                NaN      \n",
       "id1210365                                                NaN      \n",
       "\n",
       "           passenger_cnt.NUM_UNIQUE(trips.HOUR(pickup_datetime))  \\\n",
       "id                                                                 \n",
       "id0190469                                                NaN       \n",
       "id0621643                                                NaN       \n",
       "id1384355                                                NaN       \n",
       "id1665586                                                NaN       \n",
       "id1210365                                                NaN       \n",
       "\n",
       "           passenger_cnt.NUM_UNIQUE(trips.MONTH(pickup_datetime))  \\\n",
       "id                                                                  \n",
       "id0190469                                                NaN        \n",
       "id0621643                                                NaN        \n",
       "id1384355                                                NaN        \n",
       "id1665586                                                NaN        \n",
       "id1210365                                                NaN        \n",
       "\n",
       "           passenger_cnt.NUM_UNIQUE(trips.WEEKDAY(pickup_datetime))  \\\n",
       "id                                                                    \n",
       "id0190469                                                NaN          \n",
       "id0621643                                                NaN          \n",
       "id1384355                                                NaN          \n",
       "id1665586                                                NaN          \n",
       "id1210365                                                NaN          \n",
       "\n",
       "           passenger_cnt.NUM_UNIQUE(trips.WEEK(pickup_datetime))  \\\n",
       "id                                                                 \n",
       "id0190469                                                NaN       \n",
       "id0621643                                                NaN       \n",
       "id1384355                                                NaN       \n",
       "id1665586                                                NaN       \n",
       "id1210365                                                NaN       \n",
       "\n",
       "           passenger_cnt.SKEW(trips.BEARING(dropoff_latlong, pickup_latlong))  \\\n",
       "id                                                                              \n",
       "id0190469                                                NaN                    \n",
       "id0621643                                                NaN                    \n",
       "id1384355                                                NaN                    \n",
       "id1665586                                                NaN                    \n",
       "id1210365                                                NaN                    \n",
       "\n",
       "           passenger_cnt.SKEW(trips.CITYBLOCK(dropoff_latlong, pickup_latlong))  \\\n",
       "id                                                                                \n",
       "id0190469                                                NaN                      \n",
       "id0621643                                                NaN                      \n",
       "id1384355                                                NaN                      \n",
       "id1665586                                                NaN                      \n",
       "id1210365                                                NaN                      \n",
       "\n",
       "           passenger_cnt.SKEW(trips.MINUTE(pickup_datetime))  \n",
       "id                                                            \n",
       "id0190469                                                NaN  \n",
       "id0621643                                                NaN  \n",
       "id1384355                                                NaN  \n",
       "id1665586                                                NaN  \n",
       "id1210365                                                NaN  \n",
       "\n",
       "[5 rows x 103 columns]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "agg_primitives = ['Sum', 'Mean', 'Median', 'Std', 'Count', 'Min', 'Max', 'Num_Unique', 'Skew']\n",
    "trans_primitives = [Bearing, Cityblock,\n",
    "                    GeoBox(bottomleft=(40.62, -73.85), topright=(40.70, -73.75)),\n",
    "                    GeoBox(bottomleft=(40.70, -73.97), topright=(40.77, -73.9)),\n",
    "                    RushHour,\n",
    "                    NoonHour,\n",
    "                    NightHour,\n",
    "                    'Day', 'Hour', 'Minute', 'Month', 'Weekday', 'Week', 'Is_weekend']\n",
    "\n",
    "# this allows us to create features that are conditioned on a second value before we calculate.\n",
    "es.add_interesting_values()\n",
    "\n",
    "# calculate feature_matrix using deep feature synthesis\n",
    "feature_matrix, features = ft.dfs(entityset=es,\n",
    "                                  target_entity=\"trips\",\n",
    "                                  trans_primitives=trans_primitives,\n",
    "                                  agg_primitives=agg_primitives,\n",
    "                                  drop_contains=['trips.test_data'],\n",
    "                                  verbose=True,\n",
    "                                  cutoff_time=cutoff_time,\n",
    "                                  approximate='36d',\n",
    "                                  max_depth=4)\n",
    "\n",
    "feature_matrix.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 5: Build the Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p>As before, we need to retrieve our labels for the train dataset, so we should merge our current feature matrix with the original dataset. </p>\n",
    "\n",
    "<p>We use the `log` of the trip duration since that measure is better at distinguishing distances within the city</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# separates the whole feature matrix into train data feature matrix, train data labels, and test data feature matrix \n",
    "X_train, labels, X_test = taxi_utils.get_train_test_fm(feature_matrix)\n",
    "labels = np.log(labels.values + 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0]\ttrain-rmse:5.00551\tvalid-rmse:5.00532\n",
      "Multiple eval metrics have been passed: 'valid-rmse' will be used for early stopping.\n",
      "\n",
      "Will train until valid-rmse hasn't improved in 60 rounds.\n",
      "[10]\ttrain-rmse:0.946418\tvalid-rmse:0.946367\n",
      "[20]\ttrain-rmse:0.432013\tvalid-rmse:0.433546\n",
      "[30]\ttrain-rmse:0.391621\tvalid-rmse:0.394602\n",
      "[40]\ttrain-rmse:0.381306\tvalid-rmse:0.385665\n",
      "[50]\ttrain-rmse:0.377907\tvalid-rmse:0.383019\n",
      "[60]\ttrain-rmse:0.375852\tvalid-rmse:0.381634\n",
      "[70]\ttrain-rmse:0.373583\tvalid-rmse:0.380123\n",
      "[80]\ttrain-rmse:0.372274\tvalid-rmse:0.379605\n",
      "[90]\ttrain-rmse:0.370662\tvalid-rmse:0.378809\n",
      "[100]\ttrain-rmse:0.370041\tvalid-rmse:0.378617\n",
      "[110]\ttrain-rmse:0.369409\tvalid-rmse:0.378434\n",
      "[120]\ttrain-rmse:0.368797\tvalid-rmse:0.378434\n",
      "[130]\ttrain-rmse:0.367817\tvalid-rmse:0.378096\n",
      "[140]\ttrain-rmse:0.367116\tvalid-rmse:0.377942\n",
      "[150]\ttrain-rmse:0.36683\tvalid-rmse:0.377944\n",
      "[160]\ttrain-rmse:0.366317\tvalid-rmse:0.377901\n",
      "[170]\ttrain-rmse:0.365764\tvalid-rmse:0.377967\n",
      "[180]\ttrain-rmse:0.365013\tvalid-rmse:0.377859\n",
      "[190]\ttrain-rmse:0.364817\tvalid-rmse:0.377857\n",
      "[200]\ttrain-rmse:0.364266\tvalid-rmse:0.377746\n",
      "[210]\ttrain-rmse:0.363872\tvalid-rmse:0.377716\n",
      "[220]\ttrain-rmse:0.363538\tvalid-rmse:0.377674\n",
      "[226]\ttrain-rmse:0.363468\tvalid-rmse:0.377659\n",
      "Modeling RMSE 0.37766\n"
     ]
    }
   ],
   "source": [
    "model = taxi_utils.train_xgb(X_train, labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>trip_duration</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>id</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>id0621643</th>\n",
       "      <td>1315.669678</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>id1384355</th>\n",
       "      <td>1609.481201</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>id2568735</th>\n",
       "      <td>1715.196045</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>id3700764</th>\n",
       "      <td>1327.091553</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>id3008929</th>\n",
       "      <td>328.528320</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           trip_duration\n",
       "id                      \n",
       "id0621643    1315.669678\n",
       "id1384355    1609.481201\n",
       "id2568735    1715.196045\n",
       "id3700764    1327.091553\n",
       "id3008929     328.528320"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "submission = taxi_utils.predict_xgb(model, X_test)\n",
    "submission.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "submission.to_csv('trip_duration_ft_custom_primitives.csv', index=True, index_label='id')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<dl>\n",
    "<dt>This solution:</dt>\n",
    "<dd>&nbsp; &nbsp; Received a score of 0.39256 on the Kaggle competition.</dd>\n",
    "<dd>&nbsp; &nbsp; Placed 416 out of 1257.</dd>\n",
    "<dd>&nbsp; &nbsp; Beat 67% of competitors on the Kaggle competition.</dd>\n",
    "<dd>&nbsp; &nbsp; Had a modeling RMSE of 0.33454</dd>\n",
    "</dl>\n",
    "\n",
    "December 27, 2017.\n",
    "\n",
    "<h2>Additional Analysis</h2>\n",
    "<p>Lets look at how important the features we created were for the model.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>feature_name</th>\n",
       "      <th>importance</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>BEARING(dropoff_latlong, pickup_latlong)</td>\n",
       "      <td>4924.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>CITYBLOCK(dropoff_latlong, pickup_latlong)</td>\n",
       "      <td>4768.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>HOUR(pickup_datetime)</td>\n",
       "      <td>3682.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>46</th>\n",
       "      <td>DAY(pickup_datetime)</td>\n",
       "      <td>2427.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>MINUTE(pickup_datetime)</td>\n",
       "      <td>2066.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40</th>\n",
       "      <td>WEEK(pickup_datetime)</td>\n",
       "      <td>1666.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>WEEKDAY(pickup_datetime)</td>\n",
       "      <td>1353.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>GEOBOX(pickup_latlong, (40.7, -73.97), (40.77,...</td>\n",
       "      <td>653.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>GEOBOX(dropoff_latlong, (40.7, -73.97), (40.77...</td>\n",
       "      <td>549.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>IS_WEEKEND(pickup_datetime)</td>\n",
       "      <td>441.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>MONTH(pickup_datetime)</td>\n",
       "      <td>371.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>passenger_cnt.STD(trips.BEARING(dropoff_latlon...</td>\n",
       "      <td>366.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>passenger_cnt.MEAN(trips.trip_duration)</td>\n",
       "      <td>352.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>IS_RUSH_HOUR(pickup_datetime)</td>\n",
       "      <td>338.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>passenger_cnt.STD(trips.MINUTE(pickup_datetime))</td>\n",
       "      <td>332.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>GEOBOX(dropoff_latlong, (40.62, -73.85), (40.7...</td>\n",
       "      <td>316.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>55</th>\n",
       "      <td>passenger_cnt.STD(trips.CITYBLOCK(dropoff_latl...</td>\n",
       "      <td>315.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>passenger_cnt.SKEW(trips.trip_duration)</td>\n",
       "      <td>307.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>passenger_cnt.MEAN(trips.BEARING(dropoff_latlo...</td>\n",
       "      <td>297.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>GEOBOX(pickup_latlong, (40.62, -73.85), (40.7,...</td>\n",
       "      <td>294.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>passenger_cnt.SKEW(trips.MINUTE(pickup_datetime))</td>\n",
       "      <td>283.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41</th>\n",
       "      <td>vendors.MEAN(trips.CITYBLOCK(dropoff_latlong, ...</td>\n",
       "      <td>266.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>passenger_cnt.MEAN(trips.MINUTE(pickup_datetime))</td>\n",
       "      <td>262.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>vendors.MEAN(trips.trip_duration)</td>\n",
       "      <td>259.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45</th>\n",
       "      <td>vendors.MEAN(trips.MINUTE(pickup_datetime))</td>\n",
       "      <td>255.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>passenger_cnt.MEDIAN(trips.BEARING(dropoff_lat...</td>\n",
       "      <td>253.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>IS_NOON_HOUR(pickup_datetime)</td>\n",
       "      <td>248.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32</th>\n",
       "      <td>IS_NIGHT_HOUR(pickup_datetime)</td>\n",
       "      <td>241.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51</th>\n",
       "      <td>vendors.SUM(trips.trip_duration)</td>\n",
       "      <td>239.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>58</th>\n",
       "      <td>vendors.MEAN(trips.BEARING(dropoff_latlong, pi...</td>\n",
       "      <td>233.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39</th>\n",
       "      <td>passenger_cnt.MEAN(trips.CITYBLOCK(dropoff_lat...</td>\n",
       "      <td>226.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42</th>\n",
       "      <td>passenger_cnt.MEDIAN(trips.CITYBLOCK(dropoff_l...</td>\n",
       "      <td>216.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>passenger_cnt.SUM(trips.trip_duration)</td>\n",
       "      <td>201.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>passenger_cnt.STD(trips.trip_duration)</td>\n",
       "      <td>199.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48</th>\n",
       "      <td>vendors.STD(trips.MINUTE(pickup_datetime))</td>\n",
       "      <td>193.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>passenger_count</td>\n",
       "      <td>186.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>passenger_cnt.SKEW(trips.BEARING(dropoff_latlo...</td>\n",
       "      <td>184.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>67</th>\n",
       "      <td>passenger_cnt.SKEW(trips.CITYBLOCK(dropoff_lat...</td>\n",
       "      <td>182.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>vendors.STD(trips.CITYBLOCK(dropoff_latlong, p...</td>\n",
       "      <td>181.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>passenger_cnt.MEDIAN(trips.trip_duration)</td>\n",
       "      <td>180.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>53</th>\n",
       "      <td>vendors.MEDIAN(trips.CITYBLOCK(dropoff_latlong...</td>\n",
       "      <td>178.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>passenger_cnt.MAX(trips.trip_duration)</td>\n",
       "      <td>164.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>65</th>\n",
       "      <td>vendors.SKEW(trips.trip_duration)</td>\n",
       "      <td>158.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>52</th>\n",
       "      <td>vendors.MEDIAN(trips.BEARING(dropoff_latlong, ...</td>\n",
       "      <td>151.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>60</th>\n",
       "      <td>vendors.SKEW(trips.MINUTE(pickup_datetime))</td>\n",
       "      <td>126.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>passenger_cnt.MIN(trips.BEARING(dropoff_latlon...</td>\n",
       "      <td>122.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>vendor_id</td>\n",
       "      <td>122.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>62</th>\n",
       "      <td>vendors.MEDIAN(trips.trip_duration)</td>\n",
       "      <td>115.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>63</th>\n",
       "      <td>passenger_cnt.MIN(trips.trip_duration)</td>\n",
       "      <td>108.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50</th>\n",
       "      <td>passenger_cnt.COUNT(trips)</td>\n",
       "      <td>107.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                         feature_name  importance\n",
       "13           BEARING(dropoff_latlong, pickup_latlong)      4924.0\n",
       "3          CITYBLOCK(dropoff_latlong, pickup_latlong)      4768.0\n",
       "8                               HOUR(pickup_datetime)      3682.0\n",
       "46                               DAY(pickup_datetime)      2427.0\n",
       "27                            MINUTE(pickup_datetime)      2066.0\n",
       "40                              WEEK(pickup_datetime)      1666.0\n",
       "29                           WEEKDAY(pickup_datetime)      1353.0\n",
       "10  GEOBOX(pickup_latlong, (40.7, -73.97), (40.77,...       653.0\n",
       "11  GEOBOX(dropoff_latlong, (40.7, -73.97), (40.77...       549.0\n",
       "9                         IS_WEEKEND(pickup_datetime)       441.0\n",
       "34                             MONTH(pickup_datetime)       371.0\n",
       "20  passenger_cnt.STD(trips.BEARING(dropoff_latlon...       366.0\n",
       "16            passenger_cnt.MEAN(trips.trip_duration)       352.0\n",
       "33                      IS_RUSH_HOUR(pickup_datetime)       338.0\n",
       "22   passenger_cnt.STD(trips.MINUTE(pickup_datetime))       332.0\n",
       "4   GEOBOX(dropoff_latlong, (40.62, -73.85), (40.7...       316.0\n",
       "55  passenger_cnt.STD(trips.CITYBLOCK(dropoff_latl...       315.0\n",
       "30            passenger_cnt.SKEW(trips.trip_duration)       307.0\n",
       "36  passenger_cnt.MEAN(trips.BEARING(dropoff_latlo...       297.0\n",
       "0   GEOBOX(pickup_latlong, (40.62, -73.85), (40.7,...       294.0\n",
       "35  passenger_cnt.SKEW(trips.MINUTE(pickup_datetime))       283.0\n",
       "41  vendors.MEAN(trips.CITYBLOCK(dropoff_latlong, ...       266.0\n",
       "12  passenger_cnt.MEAN(trips.MINUTE(pickup_datetime))       262.0\n",
       "23                  vendors.MEAN(trips.trip_duration)       259.0\n",
       "45        vendors.MEAN(trips.MINUTE(pickup_datetime))       255.0\n",
       "1   passenger_cnt.MEDIAN(trips.BEARING(dropoff_lat...       253.0\n",
       "14                      IS_NOON_HOUR(pickup_datetime)       248.0\n",
       "32                     IS_NIGHT_HOUR(pickup_datetime)       241.0\n",
       "51                   vendors.SUM(trips.trip_duration)       239.0\n",
       "58  vendors.MEAN(trips.BEARING(dropoff_latlong, pi...       233.0\n",
       "39  passenger_cnt.MEAN(trips.CITYBLOCK(dropoff_lat...       226.0\n",
       "42  passenger_cnt.MEDIAN(trips.CITYBLOCK(dropoff_l...       216.0\n",
       "21             passenger_cnt.SUM(trips.trip_duration)       201.0\n",
       "6              passenger_cnt.STD(trips.trip_duration)       199.0\n",
       "48         vendors.STD(trips.MINUTE(pickup_datetime))       193.0\n",
       "2                                     passenger_count       186.0\n",
       "31  passenger_cnt.SKEW(trips.BEARING(dropoff_latlo...       184.0\n",
       "67  passenger_cnt.SKEW(trips.CITYBLOCK(dropoff_lat...       182.0\n",
       "24  vendors.STD(trips.CITYBLOCK(dropoff_latlong, p...       181.0\n",
       "28          passenger_cnt.MEDIAN(trips.trip_duration)       180.0\n",
       "53  vendors.MEDIAN(trips.CITYBLOCK(dropoff_latlong...       178.0\n",
       "38             passenger_cnt.MAX(trips.trip_duration)       164.0\n",
       "65                  vendors.SKEW(trips.trip_duration)       158.0\n",
       "52  vendors.MEDIAN(trips.BEARING(dropoff_latlong, ...       151.0\n",
       "60        vendors.SKEW(trips.MINUTE(pickup_datetime))       126.0\n",
       "26  passenger_cnt.MIN(trips.BEARING(dropoff_latlon...       122.0\n",
       "25                                          vendor_id       122.0\n",
       "62                vendors.MEDIAN(trips.trip_duration)       115.0\n",
       "63             passenger_cnt.MIN(trips.trip_duration)       108.0\n",
       "50                         passenger_cnt.COUNT(trips)       107.0"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "feature_names = X_train.columns.values\n",
    "ft_importances = taxi_utils.feature_importances(model, feature_names)\n",
    "ft_importances[:50]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save output files\n",
    "\n",
    "import os\n",
    "\n",
    "try:\n",
    "    os.mkdir(\"output\")\n",
    "except:\n",
    "    pass\n",
    "\n",
    "feature_matrix.to_csv('output/feature_matrix.csv')\n",
    "cutoff_time.to_csv('output/cutoff_times.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p>\n",
    "    <img src=\"https://www.featurelabs.com/wp-content/uploads/2017/12/logo.png\" alt=\"Featuretools\" />\n",
    "</p>\n",
    "\n",
    "Featuretools was created by the developers at [Feature Labs](https://www.featurelabs.com/). If building impactful data science pipelines is important to you or your business, please [get in touch](https://www.featurelabs.com/contact/)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}