feature_engineering.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Feature engineering\n",
    "\n",
    "In this notebook, we will try to create features from checkin dataset.\n",
    "We based our work on the paper: `Fine-Scale Prediction of People’s Home Location Using Social Media Footprints` by H. Kavak et al. The authors start by clustring the checkins of each user using Density-based spatial clustering of applications with noise (DBSCAN). The goal of this step is to group checkins into dense and small clusters. One of these clusters contain the home location. To predict the latter, they generated the following mobility features per cluster:\n",
    "\n",
    "- Check-in Ratio (CR)\n",
    "- Check-in Ration during Midnight (MR)\n",
    "- Check-in Ratio of Last Destination of a Day (EDR)\n",
    "- Check-in Ratio of Last Destination of a Day with Inactive Midnight (EIDR)\n",
    "- PageRank (PR)\n",
    "- Reverse PageRank (RPR)\n",
    "\n",
    "We will explain each feature as we progress in this work.\n",
    "\n",
    "\n",
    "These feature will be used to classify each cluster and we average the latitude and longitude to obtain users' home location\n",
    "\n",
    "# 1. Importing libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "# functions to compute the distance between two geocoordinates\n",
    "from haversine import haversine_vector, Unit\n",
    "# DBSCAN\n",
    "from sklearn.cluster import DBSCAN\n",
    "# To do operations on datetime\n",
    "from datetime import timedelta\n",
    "# PageRank and ReversePageRank\n",
    "import networkx as nx"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Helper functions\n",
    "Here we will redefine functions we used during previous milestones to clean our data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.1 Latitude correction\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def correct_latitude(lat):\n",
    "    \"\"\"\n",
    "    This function corrects for out of range latitude.\n",
    "    \n",
    "    Input: \n",
    "    -- lat: latitude coordinates in °\n",
    "    Output: \n",
    "    -- lat: latitude coordinates put between -90 and 90°\n",
    "    \"\"\"\n",
    "    while lat>90 or lat<-90:\n",
    "        if lat>90:\n",
    "            lat = -(lat-180)\n",
    "        elif lat<-90:\n",
    "            lat = -(lat+180)\n",
    "    return lat"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "def correct_longitude(long):\n",
    "    \"\"\"\n",
    "    This function corrects for out of range longitude.\n",
    "    \n",
    "    Input: \n",
    "    -- long: longitude coordiantes in °\n",
    "    Output: \n",
    "    -- long: longitude coordinates put between -180 and 180°\n",
    "    \"\"\"\n",
    "    while long>180 or long<-180:\n",
    "        if long>180:\n",
    "            long = long - 360\n",
    "        elif long<-180:\n",
    "            long = long +360\n",
    "    return long"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.2. Compute distance between two geocoordinates points"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_distance(df,columns):\n",
    "    '''\n",
    "    This function computes the distance between two geographic coordinates for a given dataframe.\n",
    "    \n",
    "    Input: \n",
    "        - df: Dataframe containing 4 columns latitude1, longitude1, latitude2 and longitude2\n",
    "        - columns: list of columns [latitude1, longitude1, latitude2 and longitude2]\n",
    "        \n",
    "    Output: \n",
    "        - numpy array containing the distance between geographic coordinates of each row\n",
    "    '''\n",
    "    points1 = list(zip(df[columns[0]],df[columns[1]]))\n",
    "    points2 = list(zip(df[columns[2]],df[columns[3]]))\n",
    "    # Use harvesine_vector to compute the distance between points\n",
    "    return np.round(haversine_vector(points1,points2,Unit.KILOMETERS),decimals=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.3. Selecting relevant homes\n",
    "\n",
    "As we want to use the training dataset to predict home locations, it's important to keep only users that doesn't change home. To measure that, we compute the variance their home's location Using the function defined below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def select_relevant_homes(df_homes):\n",
    "    '''\n",
    "    This function selects relevant homes. We consider a home location as relevant if it's latitude and\n",
    "    longitude doesn't \"vary much\". To measure this variation, we simply compute the mean and the std of\n",
    "    the latitude and longitude of homes for every user. Then we construct 4 points as follow:\n",
    "        - by adding and substracting the standard deviation of the latitude and longitude from their\n",
    "        respective mean\n",
    "        - Measure the diagonal in KM\n",
    "        - If the diagonal is less than 100m we can assume with confidence that the mean is indeed the\n",
    "        home location\n",
    "    \n",
    "    Input:\n",
    "        - df_homes: A dataframe containing all checkins labled as Home\n",
    "    Output:\n",
    "        - df_homes: Home location for each user\n",
    "    '''\n",
    "    \n",
    "    # Grouping df_homes according to the user id and compute std and mean for lat and lon\n",
    "    df_homes = df_homes.groupby('User_ID').agg({'lat':('std','mean'),'lon':('std','mean')})\n",
    "    \n",
    "    # Filling nan values with 0 (std return 0 if there is only one sample)\n",
    "    df_homes.fillna(0,inplace = True)\n",
    "    \n",
    "    # Construct the diagonal points\n",
    "    df_tmp = pd.DataFrame()\n",
    "    df_tmp['lat1'] = df_homes.lat['mean']-df_homes.lat['std']\n",
    "    df_tmp['lat2'] = df_homes.lat['mean']+df_homes.lat['std']\n",
    "    df_tmp['lon1'] = df_homes.lon['mean']-df_homes.lon['std']\n",
    "    df_tmp['lon2'] = df_homes.lon['mean']+df_homes.lon['std']\n",
    "    \n",
    "    # Compute diagonal length\n",
    "    df_tmp['home_radius'] = compute_distance(df_tmp,['lat1','lon1','lat2','lon2'])\n",
    "    \n",
    "    # Filter home and keep relevant home (estimated distance between homes checkins < 100m )\n",
    "    df_homes = df_homes[df_tmp['home_radius']<0.1][[('lat','mean'),('lon','mean')]].copy()\n",
    "    \n",
    "    # Flatten df_homes columns\n",
    "    df_homes.columns = df_homes.columns.get_level_values(0)\n",
    "    \n",
    "    return df_homes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Constructing Checkins dataframe\n",
    "## 3.1. Training dataset\n",
    "In this section, we define a function that construct a clean checkin dataframe from Foursquare data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def construct_df_checkins(path,sample_frac = 1):\n",
    "    '''\n",
    "    This function takes the path of the raw data, import it and construct a checkin dataframe where\n",
    "    all users have at least 5 checkins and 1 home location\n",
    "    \n",
    "    Input:\n",
    "        - Path: the Path of the file containing the data\n",
    "        - sample_frac: sample fraction from the raw dataframe\n",
    "    Output:\n",
    "        - df_checkins: Checkin dataframe where all users have at least 5 checkins and 1 home location\n",
    "    '''\n",
    "    \n",
    "    # Read data from the file and drop unnecessary columns\n",
    "    df_tmp = pd.read_csv(path).sample(frac=sample_frac).drop(columns=['Venue_ID','day'])\n",
    "    \n",
    "    # Latitude and Longitude correction\n",
    "    df_tmp.lat = df_tmp.lat.apply(correct_latitude)\n",
    "    df_tmp.lon = df_tmp.lon.apply(correct_longitude)\n",
    "    \n",
    "    # Construct df_homes and select only relevant homes\n",
    "    df_homes = df_tmp.loc[df_tmp.place.str.lower().str.contains('home' and 'private')].copy()\n",
    "    df_homes = select_relevant_homes(df_homes)\n",
    "    \n",
    "    # Select users with relevant homes from the raw data\n",
    "    df_tmp = df_tmp.loc[df_tmp['User_ID'].isin(df_homes.index)].copy()\n",
    "    \n",
    "    # Count the number of checkins for each user\n",
    "    df_tmp_grouped = df_tmp.groupby('User_ID').agg({'User_ID':'count'})\n",
    "    \n",
    "    # Define a set containing users with at least 5 checkins\n",
    "    users = set(df_tmp_grouped[df_tmp_grouped['User_ID']>5].index)\n",
    "    \n",
    "    # Construct df_checkins\n",
    "    df_checkins = df_tmp.loc[df_tmp['User_ID'].isin(users)].copy()\n",
    "    \n",
    "    # Convert 'local time' attribute to a pandas datetime\n",
    "    df_checkins['local_time'] = pd.to_datetime(df_checkins['local_time'])\n",
    "    \n",
    "    # Label Homes\n",
    "    df_checkins['Is_home'] = df_checkins.place.str.lower().str.contains('home' and 'private')\n",
    "    \n",
    "    # Drop unnecessary column\n",
    "    df_checkins.drop(columns = ['place'],inplace = True)\n",
    "    \n",
    "    return df_checkins.sort_values(by=['User_ID','local_time']).reset_index(drop=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.2. Prediction dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def construct_prediction_dataset(path, sample_frac = 1):\n",
    "    '''\n",
    "    This function takes the path of the raw data, import it and construct a checkin dataframe where\n",
    "    all users have at least 5 checkins. It's used to prepare the dataset we will use to predict home location\n",
    "    of the users.\n",
    "    \n",
    "    Input:\n",
    "        - Path: the Path of the file containing the data\n",
    "        - sample_frac: sample fraction from the raw dataframe\n",
    "    Output:\n",
    "        - df_checkins: Checkin dataframe where all users have at least 5 checkins\n",
    "    '''\n",
    "    \n",
    "    # Import dataset\n",
    "    df_checkins = pd.read_csv(path,sep='\\t',header=None,names=['User_ID','local_time','lat','lon','location_id'],\n",
    "                              parse_dates = ['local_time'])\n",
    "    # Drop NaN values\n",
    "    df_checkins.dropna(inplace = True)\n",
    "    \n",
    "    # Drop unnecessary column\n",
    "    df_checkins.drop(columns = ['location_id'],inplace=True)\n",
    "    \n",
    "    # Correct latitude and longitude\n",
    "    df_checkins.lat = df_checkins.lat.apply(correct_latitude)\n",
    "    df_checkins.lon = df_checkins.lon.apply(correct_longitude)\n",
    "    \n",
    "    # Drop checkins with both latitude and longitude set at zero\n",
    "    # Note: this is specific to gowalla and brightkite dataset\n",
    "    df_checkins = df_checkins[(df_checkins['lat'] !=0) & (df_checkins['lon'] != 0)]\n",
    "    \n",
    "    # Grouping by users\n",
    "    users = df_checkins.groupby(['User_ID']).agg({'User_ID':'count'})\n",
    "    \n",
    "    # Selecting users with more than 5 checkins\n",
    "    users = set(users.loc[users['User_ID']>5].index)\n",
    "    \n",
    "    df_checkins = df_checkins.loc[df_checkins['User_ID'].isin(users)]\n",
    "    \n",
    "    return df_checkins"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4. Building features\n",
    "Some descriptions are taken from the online supplemental provided by the authors.\n",
    "\n",
    "Source: https://github.com/hamdikavak/home-location-prediction/blob/master/supplemental_revised.pdf\n",
    "## 4.1. Compute cluster label\n",
    "\n",
    "Instead of discretizing the world, the authors use an unsupervised to create small cluster. The function below take the checkin of every user, cluster them using DBSCAN and return label of the cluster for each checkin."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def build_clusters_labels(df_user,clustering_method):\n",
    "    '''\n",
    "    This function clusters the checkins for a single user.\n",
    "    \n",
    "    Input:\n",
    "        - df_user: a dataframe containing the latitude and longitude for each checkin\n",
    "        - clustering_method: DBSCAN, we define this parameter to avoid unnecessary initialisations\n",
    "        when calling this funcrion\n",
    "    Output:\n",
    "        - clusters_labels: cluster label assigned to each checkin\n",
    "    '''\n",
    "    cluster_lables = clustering_method.fit(np.deg2rad(df_user[['lat','lon']])).labels_\n",
    "    \n",
    "    return cluster_lables"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.2. Cleaning users\n",
    "\n",
    "The paper suggest removing multiple successive checkin within 60 minutes and 100m to avoid biasing the dataset.\n",
    "We defined the function below for this purpose"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "def cleaning_user(df_user):\n",
    "    '''\n",
    "    To avoid biasing the dataset with multiple checkins in a small period of time or small distance traveled, \n",
    "    we drop checkins that are consecutively shared within 60 minutes and 100m.\n",
    "    \n",
    "    Input:\n",
    "        - df_user: datafame containing checkin time and location sorted by time\n",
    "    Output:\n",
    "        - df_user: cleaned df_user\n",
    "    \n",
    "    '''\n",
    "    \n",
    "    # Constructing a dataframe containing the actual checkin and the next checkin\n",
    "    df_tmp = df_user.reset_index().merge(df_user.iloc[1:].reset_index(drop=True),right_index=True,\n",
    "                                         left_index=True,how='inner')\n",
    "    \n",
    "    # Compute the time between two consecutive checkins\n",
    "    df_tmp['dt'] = df_tmp['local_time_y'] - df_tmp['local_time_x']\n",
    "    \n",
    "    # Compute the distance between two consecutive checkins\n",
    "    columns = ['lat_x','lon_x','lat_y','lon_y']\n",
    "    df_tmp['distance'] = compute_distance(df_tmp,columns)\n",
    "    \n",
    "    # Construct a mask to keep consecutive checkins if they are distant by 60 minutes or 100m\n",
    "    # We also ignore checkins with 0 dt\n",
    "    mask = (df_tmp['dt']!=timedelta(0))&((df_tmp['dt']>timedelta(hours=1))|(df_tmp['distance']>0.1))\n",
    "    \n",
    "    return df_user.reset_index().iloc[df_tmp[mask].index]\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4.3. Checkin During Midnight\n",
    "\n",
    "Midnight check-in ratio looks at all midnight check-ins (12:00 AM - 07:00 AM) of a user and calculate the ratio of midnight check-ins per visited location."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_checkin_during_midnight(df_user):\n",
    "    '''\n",
    "    This function lables the checkins after midnight.\n",
    "    \n",
    "    Input:\n",
    "        - df_user: dataframe containing and sorted by checkin time\n",
    "    Output:\n",
    "        - Labeles for each checkin. If it is happening after midnight and befor 7am it's set to True\n",
    "        and False otherwise\n",
    "    '''\n",
    "    df_tmp = (df_user['local_time'].dt.hour>=0) & (df_user['local_time'].dt.hour<7)\n",
    "    \n",
    "    return df_tmp"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.4. Last Checkin\n",
    "This feature captures the last destination of the day which is found to be important to predict home location.\n",
    "We identify all last check-ins of days and calculate the ratio per location using tweets shared between 05:00 PM\n",
    "in the evening until 03:00 AM in the morning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_last_checkin(df_user):\n",
    "    '''\n",
    "    This function labels the last checkin before 3 am.\n",
    "    \n",
    "    Input:\n",
    "        df_user: Dataframe containing and sorted by checkin time\n",
    "    Output:\n",
    "        - Labeles for each checkin. If it is the last checkin of the day, the label is set to True\n",
    "        and False otherwise\n",
    "    '''\n",
    "    # We subsctract 3 hours  so we can detect the last checkin whenever the date changes\n",
    "    tmp_date = (df_user['local_time']-timedelta(hours=3)).dt.date.values\n",
    "    tmp_hour = (df_user['local_time']-timedelta(hours=3)).dt.hour.values\n",
    "    last_checkin = []\n",
    "    \n",
    "    # Labeling last checkins\n",
    "    for i in range(len(tmp_date)-1):\n",
    "        if (tmp_hour[i]>=14) and (tmp_hour[i]<=23) and (tmp_date[i]<tmp_date[i+1]):\n",
    "            last_checkin.append(True)\n",
    "        else:\n",
    "            last_checkin.append(False)\n",
    "            \n",
    "    # The last checkin is always True by definition \n",
    "    if (tmp_hour[-1]>=14) and (tmp_hour[-1]<=23):\n",
    "        last_checkin.append(True)\n",
    "    else:\n",
    "        last_checkin.append(False)\n",
    "\n",
    "    return last_checkin"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.5. Last checkin with inactive midnight\n",
    "This feature is very similar to the EDR feature but ignores days when a user shares tweets during midnight."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_last_checkin_with_inactive_midnight(df_user):\n",
    "    '''\n",
    "    This function labels the last checkin with inactive midnight (no checkins between 0am and 7am).\n",
    "    \n",
    "    Input:\n",
    "        df_user: Dataframe containing and sorted by checkin time\n",
    "    Output:\n",
    "        - Labeles for each checkin. If it is the last checkin of the day and the user didn't checkin\n",
    "        between 0am and 7am the label is True and False otherwise\n",
    "    '''\n",
    "    \n",
    "    # Substract 7 hours to detect the change of the day whenever the date changes\n",
    "    tmp_date = (df_user['local_time']-timedelta(hours=7)).dt.date.values\n",
    "    tmp_hour = (df_user['local_time']-timedelta(hours=3)).dt.hour.values\n",
    "    \n",
    "    last_checkin_with_inactive_midnight = []\n",
    "    \n",
    "    # Compute the last checkin with inactive midnight\n",
    "    for i in range(len(tmp_date)-1):\n",
    "        # If the date changes and the hour is <= 23 the last checkin is happening before midnight\n",
    "        if (tmp_hour[i]>=14) and (tmp_hour[i]<=23) and (tmp_date[i]<tmp_date[i+1]):\n",
    "            last_checkin_with_inactive_midnight.append(True)\n",
    "        else:\n",
    "            last_checkin_with_inactive_midnight.append(False)\n",
    "    \n",
    "        # As the last checkin is by definition the last checkin of the day, we simply need to see if it is \n",
    "        # happening before midnight\n",
    "    \n",
    "    if (tmp_hour[-1]>=14) and (tmp_hour[-1]<=23):\n",
    "        last_checkin_with_inactive_midnight.append(True)\n",
    "    else:\n",
    "        last_checkin_with_inactive_midnight.append(False)\n",
    "        \n",
    "    return last_checkin_with_inactive_midnight"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.6. PageRank and ReversePageRank\n",
    "The authors used PageRank and ReversePageRank to measure the importance of nodes based on the time made between each transition. Nodes here are the clusters.\n",
    "\n",
    "We start by computing the time between two consecute checkins using the following function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_dt_to_next_checkin(df_user):\n",
    "    '''\n",
    "    This function compute delta time between two consecutive checkins in Hour.\n",
    "    \n",
    "    Input:\n",
    "        - df_user: dataframe containing checkin time\n",
    "    Output:\n",
    "        - Delta time between two consecutive checkins\n",
    "    '''\n",
    "    \n",
    "    # get checkin times\n",
    "    checkin_time = df_user['local_time'].values\n",
    "    \n",
    "    # Compute the difference (The result is in nanoseconds)\n",
    "    delta_time = checkin_time[1:]-checkin_time[:-1]\n",
    "    \n",
    "    # Convert delta_time to hours\n",
    "    delta_time = delta_time.astype(float)/(1e9*3600)\n",
    "    \n",
    "    # The last checkin doesn't have a next checkin so we append None\n",
    "    delta_time = np.append(delta_time,None)\n",
    "    return delta_time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we compute PageRank and ReversePageRank with the function below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_PR_RPR(df_user):\n",
    "    '''\n",
    "    This function computes the PageRank and ReversePageRank for each cluster. The ReversePageRank is computed\n",
    "    by inverting the the edges. The weight of an edge is obtained by computing the sum over clusters of the \n",
    "    inverse of the time made between two consecutive checkins\n",
    "    \n",
    "    Input:\n",
    "        - df_user: dataframe containing clusters labels for each checkin and the time until next checkin\n",
    "    Output:\n",
    "        - PageRank and ReversePageRank\n",
    "    '''\n",
    "    # Construct a dataframe containing the acutal checkin and next checkin\n",
    "    df_tmp = df_user.reset_index().iloc[:-1].merge(df_user.iloc[1:].reset_index(),\n",
    "                                                    right_index=True,left_index=True)\n",
    "    \n",
    "    # Compute the inverse time made between two consecutive checkins\n",
    "    df_tmp['inverse_time'] = 1/df_tmp['dt_to_next_checkin_x']\n",
    "    \n",
    "    # Construct the graph edges dataframe\n",
    "    df_graph = df_tmp.groupby(['cluster_label_x','cluster_label_y'],as_index = False).agg({'inverse_time':'sum'})\n",
    "    \n",
    "    # Initialise PageRank Graph\n",
    "    G = nx.DiGraph()\n",
    "    \n",
    "    # Initialise ReversePageRank Graph\n",
    "    RG = nx.DiGraph()\n",
    "    \n",
    "    # Building edges of the two graphs\n",
    "    for i, row in df_graph.iterrows():\n",
    "        G.add_edge(int(row['cluster_label_x']),int(row['cluster_label_y']),weight=row['inverse_time'])\n",
    "        RG.add_edge(int(row['cluster_label_y']),int(row['cluster_label_x']),weight=row['inverse_time'])\n",
    "    \n",
    "    PR = list(nx.pagerank(G, max_iter = 10000, weight='weight').values())\n",
    "    RPR = list(nx.pagerank(RG, max_iter = 10000, weight='weight').values())\n",
    "    return PR, RPR"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.7. Select most visited country\n",
    "Since we are using DBSCAN, a cluster may contain multiple countries. The function below selects the most visited country and assign it to the cluster"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_country(column):\n",
    "    '''\n",
    "    This function compute the most visited country in a cluster and assign it as label to the cluster.\n",
    "    \n",
    "    Input: \n",
    "        - column: column containing checkin countries\n",
    "    Output:\n",
    "        - most visited country\n",
    "    '''\n",
    "    \n",
    "    values,counts = np.unique(column.astype(str),return_counts=True)\n",
    "    \n",
    "    return values[np.argmax(counts)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 5. Building features\n",
    "The function below group all the previously defined functions. It constructs the features of checkin dataframes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "def build_features(path, training = True, sample_frac = 1):\n",
    "    '''\n",
    "    This is the main function to extract features from checkin data.\n",
    "    \n",
    "    Input:\n",
    "        - path: the path of the file containing raw checkins.\n",
    "        - training: if this parameter is True, construct training features and testing features if it is False.\n",
    "        - sample_frac: sample fraction from the raw dataframe\n",
    "    Output:\n",
    "        - Cleaned training data: dataframe containing features for each cluster of every user\n",
    "    '''\n",
    "    \n",
    "    if training:\n",
    "        # Initialize output dataframe for training dataset\n",
    "        df_tmp = pd.DataFrame(columns = ['user','CR','MR','EDR','EIDR','PR','RPR','Is_home',\n",
    "                                         'lat','lon','country'])\n",
    "        # Construct checkin dataframe\n",
    "        df_checkins  = construct_df_checkins(path, sample_frac=sample_frac)\n",
    "        \n",
    "    else:\n",
    "        # Initialize output dataframe for dataset to predict\n",
    "        df_tmp = pd.DataFrame(columns = ['user','CR','MR','EDR','EIDR','PR','RPR',\n",
    "                                         'lat','lon'])\n",
    "        # Copy dataset to predict\n",
    "        df_checkins = construct_prediction_dataset(path, sample_frac=sample_frac)\n",
    "    \n",
    "    # Exctract users from the checkin dataframe\n",
    "    users_id = np.unique(df_checkins['User_ID'])\n",
    "    \n",
    "    # Grouping the checkin dataframe by the 'User ID'\n",
    "    grouped_checkins = df_checkins.groupby('User_ID')\n",
    "    \n",
    "    # Initialize Clustering method with the right parameters\n",
    "    KMS_PER_RADIAN = 6371.0088\n",
    "    PRECISION = 0.1\n",
    "    clustering_method = DBSCAN(eps=PRECISION/KMS_PER_RADIAN,metric='haversine')\n",
    "    \n",
    "    # Compute features for each cluster of every user\n",
    "    for user in users_id:\n",
    "        \n",
    "        # Get the user Dataframe\n",
    "        df_user = grouped_checkins.get_group(user)\n",
    "        \n",
    "        # Clean df_user\n",
    "        df_user = cleaning_user(df_user).copy()\n",
    "        \n",
    "        # Consider only dataframes containing more than 1 cleaned entries\n",
    "        if len(df_user)>1:\n",
    "            \n",
    "            # Compute cluster_label\n",
    "            df_user['cluster_label'] = build_clusters_labels(df_user,clustering_method)\n",
    "            \n",
    "            # Compute Checkin during midnight\n",
    "            df_user['checkin_during_midnight'] = compute_checkin_during_midnight(df_user)\n",
    "            \n",
    "            # Compute last checkin\n",
    "            df_user['last_checkin'] = compute_last_checkin(df_user)\n",
    "            \n",
    "            # Compute last checkin with inactive midnight\n",
    "            df_user['last_checkin_with_inactive_midnight'] = compute_last_checkin_with_inactive_midnight(df_user)\n",
    "            \n",
    "            # Compute distance to next_checkin and classify edges\n",
    "            df_user['dt_to_next_checkin'] = compute_dt_to_next_checkin(df_user)\n",
    "            \n",
    "            #print(df_user)\n",
    "            # Construct aggregation dictionnary\n",
    "            if training:\n",
    "                agg_dic = {'cluster_label':'count','checkin_during_midnight':'sum',\n",
    "                            'last_checkin':'sum','last_checkin_with_inactive_midnight': 'sum',\n",
    "                            'Is_home': 'sum','lat':'mean','lon':'mean','country':compute_country}\n",
    "            else:\n",
    "                agg_dic = {'cluster_label':'count','checkin_during_midnight':'sum',\n",
    "                            'last_checkin':'sum','last_checkin_with_inactive_midnight': 'sum',\n",
    "                            'lat':'mean','lon':'mean'}\n",
    "                \n",
    "            # Construct rename dictiaonnary\n",
    "            rename_dic = {'cluster_label':'CR','checkin_during_midnight':'MR','last_checkin':'EDR',\n",
    "                          'last_checkin_with_inactive_midnight':'EIDR'}\n",
    "            \n",
    "            # Group by cluster_label\n",
    "            grouped_clusters = df_user.groupby('cluster_label')\n",
    "            \n",
    "            # Compute the first 4 features\n",
    "            features = grouped_clusters.agg(agg_dic).rename(columns = rename_dic)\n",
    "            \n",
    "            # Add user ID to the features\n",
    "            features['user'] = user\n",
    "            \n",
    "            # Compute Checkin Ration (CR)\n",
    "            features['CR'] = features['CR']/features['CR'].sum()\n",
    "            \n",
    "            # Compute Checkin Ration (MR)\n",
    "            features['MR'] = features['MR']/features['MR'].sum()\n",
    "            \n",
    "            # Compute Checkin Ration (EDR)\n",
    "            features['EDR'] = features['EDR']/features['EDR'].sum()\n",
    "            \n",
    "            # Compute Checkin Ration (EIDR)\n",
    "            features['EIDR'] = features['EIDR']/features['EIDR'].sum()\n",
    "            \n",
    "            if training:\n",
    "                # Label the clusters that contain the home location\n",
    "                features['Is_home'] = features['Is_home'] == features['Is_home'].max()\n",
    "            \n",
    "            # Compute PageRank and ReversePageRank\n",
    "            features['PR'],features['RPR'] = compute_PR_RPR(df_user)\n",
    "            \n",
    "            # Append results to the output dataframe\n",
    "            df_tmp = df_tmp.append(features)\n",
    "    \n",
    "    return df_tmp.reset_index(drop=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We apply the function on the foursquare dataset which is also our training dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user</th>\n",
       "      <th>CR</th>\n",
       "      <th>MR</th>\n",
       "      <th>EDR</th>\n",
       "      <th>EIDR</th>\n",
       "      <th>PR</th>\n",
       "      <th>RPR</th>\n",
       "      <th>Is_home</th>\n",
       "      <th>lat</th>\n",
       "      <th>lon</th>\n",
       "      <th>country</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>19</td>\n",
       "      <td>0.756158</td>\n",
       "      <td>0.846154</td>\n",
       "      <td>0.777027</td>\n",
       "      <td>0.772414</td>\n",
       "      <td>0.821211</td>\n",
       "      <td>0.801806</td>\n",
       "      <td>True</td>\n",
       "      <td>38.652989</td>\n",
       "      <td>-73.973113</td>\n",
       "      <td>US</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>19</td>\n",
       "      <td>0.056650</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.054054</td>\n",
       "      <td>0.055172</td>\n",
       "      <td>0.022950</td>\n",
       "      <td>0.037273</td>\n",
       "      <td>False</td>\n",
       "      <td>40.725046</td>\n",
       "      <td>-73.992639</td>\n",
       "      <td>US</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>19</td>\n",
       "      <td>0.059113</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.114865</td>\n",
       "      <td>0.117241</td>\n",
       "      <td>0.026504</td>\n",
       "      <td>0.026315</td>\n",
       "      <td>False</td>\n",
       "      <td>40.726305</td>\n",
       "      <td>-73.984104</td>\n",
       "      <td>US</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>19</td>\n",
       "      <td>0.036946</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.013514</td>\n",
       "      <td>0.013793</td>\n",
       "      <td>0.024816</td>\n",
       "      <td>0.024017</td>\n",
       "      <td>False</td>\n",
       "      <td>40.724191</td>\n",
       "      <td>-73.997563</td>\n",
       "      <td>US</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>19</td>\n",
       "      <td>0.032020</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.013514</td>\n",
       "      <td>0.013793</td>\n",
       "      <td>0.024730</td>\n",
       "      <td>0.026114</td>\n",
       "      <td>False</td>\n",
       "      <td>40.722940</td>\n",
       "      <td>-73.995724</td>\n",
       "      <td>US</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  user        CR        MR       EDR      EIDR        PR       RPR Is_home  \\\n",
       "0   19  0.756158  0.846154  0.777027  0.772414  0.821211  0.801806    True   \n",
       "1   19  0.056650  0.000000  0.054054  0.055172  0.022950  0.037273   False   \n",
       "2   19  0.059113  0.000000  0.114865  0.117241  0.026504  0.026315   False   \n",
       "3   19  0.036946  0.000000  0.013514  0.013793  0.024816  0.024017   False   \n",
       "4   19  0.032020  0.000000  0.013514  0.013793  0.024730  0.026114   False   \n",
       "\n",
       "         lat        lon country  \n",
       "0  38.652989 -73.973113      US  \n",
       "1  40.725046 -73.992639      US  \n",
       "2  40.726305 -73.984104      US  \n",
       "3  40.724191 -73.997563      US  \n",
       "4  40.722940 -73.995724      US  "
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_training = build_features(path = 'data/foursquare_checkin_data.csv.zip')\n",
    "df_training.dropna(inplace = True)\n",
    "df_training.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Exporting results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_training.to_csv('data/training_dataset.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Buil"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user</th>\n",
       "      <th>CR</th>\n",
       "      <th>MR</th>\n",
       "      <th>EDR</th>\n",
       "      <th>EIDR</th>\n",
       "      <th>PR</th>\n",
       "      <th>RPR</th>\n",
       "      <th>lat</th>\n",
       "      <th>lon</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>0.725962</td>\n",
       "      <td>0.894737</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.748403</td>\n",
       "      <td>0.800407</td>\n",
       "      <td>34.862264</td>\n",
       "      <td>-98.091842</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>0.110577</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.071980</td>\n",
       "      <td>0.051105</td>\n",
       "      <td>30.269103</td>\n",
       "      <td>-97.749395</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>0.062500</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.056549</td>\n",
       "      <td>0.048330</td>\n",
       "      <td>30.267910</td>\n",
       "      <td>-97.749312</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>0.033654</td>\n",
       "      <td>0.017544</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.035480</td>\n",
       "      <td>0.027163</td>\n",
       "      <td>30.244860</td>\n",
       "      <td>-97.757163</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>0.028846</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.038535</td>\n",
       "      <td>0.035271</td>\n",
       "      <td>30.264854</td>\n",
       "      <td>-97.743845</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  user        CR        MR  EDR  EIDR        PR       RPR        lat  \\\n",
       "0    0  0.725962  0.894737  1.0   1.0  0.748403  0.800407  34.862264   \n",
       "1    0  0.110577  0.000000  0.0   0.0  0.071980  0.051105  30.269103   \n",
       "2    0  0.062500  0.000000  0.0   0.0  0.056549  0.048330  30.267910   \n",
       "3    0  0.033654  0.017544  0.0   0.0  0.035480  0.027163  30.244860   \n",
       "4    0  0.028846  0.000000  0.0   0.0  0.038535  0.035271  30.264854   \n",
       "\n",
       "         lon  \n",
       "0 -98.091842  \n",
       "1 -97.749395  \n",
       "2 -97.749312  \n",
       "3 -97.757163  \n",
       "4 -97.743845  "
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_gowalla_features = build_features(path = 'data/loc-gowalla_totalCheckins.txt.gz',training=False)\n",
    "df_gowalla_features.dropna(inplace = True)\n",
    "df_gowalla_features.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_gowalla_features.to_csv('data/gowalla_checkin_features.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user</th>\n",
       "      <th>CR</th>\n",
       "      <th>MR</th>\n",
       "      <th>EDR</th>\n",
       "      <th>EIDR</th>\n",
       "      <th>PR</th>\n",
       "      <th>RPR</th>\n",
       "      <th>lat</th>\n",
       "      <th>lon</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>0.285204</td>\n",
       "      <td>0.277950</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.420218</td>\n",
       "      <td>0.241797</td>\n",
       "      <td>39.693950</td>\n",
       "      <td>-98.427854</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>0.011437</td>\n",
       "      <td>0.021739</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.005258</td>\n",
       "      <td>0.020780</td>\n",
       "      <td>39.891383</td>\n",
       "      <td>-105.070814</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>0.076483</td>\n",
       "      <td>0.141304</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.015565</td>\n",
       "      <td>0.054985</td>\n",
       "      <td>39.891120</td>\n",
       "      <td>-105.068526</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>0.042888</td>\n",
       "      <td>0.013975</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.017623</td>\n",
       "      <td>0.023467</td>\n",
       "      <td>39.750728</td>\n",
       "      <td>-104.999579</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>0.012152</td>\n",
       "      <td>0.006211</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.015682</td>\n",
       "      <td>0.007052</td>\n",
       "      <td>39.752790</td>\n",
       "      <td>-104.996794</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  user        CR        MR  EDR  EIDR        PR       RPR        lat  \\\n",
       "0    0  0.285204  0.277950  1.0   1.0  0.420218  0.241797  39.693950   \n",
       "1    0  0.011437  0.021739  0.0   0.0  0.005258  0.020780  39.891383   \n",
       "2    0  0.076483  0.141304  0.0   0.0  0.015565  0.054985  39.891120   \n",
       "3    0  0.042888  0.013975  0.0   0.0  0.017623  0.023467  39.750728   \n",
       "4    0  0.012152  0.006211  0.0   0.0  0.015682  0.007052  39.752790   \n",
       "\n",
       "          lon  \n",
       "0  -98.427854  \n",
       "1 -105.070814  \n",
       "2 -105.068526  \n",
       "3 -104.999579  \n",
       "4 -104.996794  "
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_brightkite_features = build_features(path = 'data/loc-brightkite_totalCheckins.txt.gz',training=False)\n",
    "df_brightkite_features.dropna(inplace = True)\n",
    "df_brightkite_features.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_brightkite_features.to_csv('data/brightkite_checkin_features.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}