01_main_cleaning.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "01_main_cleaning.ipynb",
      "provenance": [],
      "authorship_tag": "ABX9TyNfzBKOvVVkEmk+PLT3e2IC",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/evanh1393/dsi_capstone/blob/main/01_main_cleaning.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Y9WhU1Yyuzy-"
      },
      "source": [
        "# Imports\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ESJhBsiUtxx7"
      },
      "source": [
        "import pandas as pd\n",
        "import numpy as np\n",
        "\n",
        "from datetime import datetime"
      ],
      "execution_count": 1,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IR7cNUrqu_rv"
      },
      "source": [
        "# Load Dataframes\n",
        "\n",
        "The data we are dealing with come from  the [MovieLens 1M](https://grouplens.org/datasets/movielens/) dataset published by GroupLens. It contains a *movies*, *users*, and a *ratings* dataset. Their file format is `.dat` and they are seperated by **::** instead of standard commas. They will need a slightly more robust `read_csv()` to load without throwing warnings.  \n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "g8soQ2MivC5V"
      },
      "source": [
        "## Mounting Drive"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "kpp7WeDrvBvh",
        "outputId": "475a891a-bed8-4d56-88ad-962f86073e40"
      },
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Mounted at /content/drive\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Z5yOJEdVvH1o"
      },
      "source": [
        "# Creating the path of data\n",
        "DATA_PATH = '/content/drive/MyDrive/Colab Notebooks/one-m-capstone/data/main/'"
      ],
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cOOWZko4va4g"
      },
      "source": [
        "## Movies Dataframe"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "9Y6Kcsbgyoiz",
        "outputId": "0f45b593-13f3-41b6-beaa-49d336daaff6"
      },
      "source": [
        "movies = pd.read_csv(DATA_PATH + 'movies.dat', sep='::', engine='python', names=['movie_id', 'title', 'genres'])\n",
        "movies.head()"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>movie_id</th>\n",
              "      <th>title</th>\n",
              "      <th>genres</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1</td>\n",
              "      <td>Toy Story (1995)</td>\n",
              "      <td>Animation|Children's|Comedy</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2</td>\n",
              "      <td>Jumanji (1995)</td>\n",
              "      <td>Adventure|Children's|Fantasy</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>3</td>\n",
              "      <td>Grumpier Old Men (1995)</td>\n",
              "      <td>Comedy|Romance</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>4</td>\n",
              "      <td>Waiting to Exhale (1995)</td>\n",
              "      <td>Comedy|Drama</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>5</td>\n",
              "      <td>Father of the Bride Part II (1995)</td>\n",
              "      <td>Comedy</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   movie_id                               title                        genres\n",
              "0         1                    Toy Story (1995)   Animation|Children's|Comedy\n",
              "1         2                      Jumanji (1995)  Adventure|Children's|Fantasy\n",
              "2         3             Grumpier Old Men (1995)                Comedy|Romance\n",
              "3         4            Waiting to Exhale (1995)                  Comedy|Drama\n",
              "4         5  Father of the Bride Part II (1995)                        Comedy"
            ]
          },
          "metadata": {},
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "4W20ruDxvW9f",
        "outputId": "599bb4ed-cad0-4877-efb7-50ac4ef90350"
      },
      "source": [
        "movies.info()"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "<class 'pandas.core.frame.DataFrame'>\n",
            "RangeIndex: 3883 entries, 0 to 3882\n",
            "Data columns (total 3 columns):\n",
            " #   Column    Non-Null Count  Dtype \n",
            "---  ------    --------------  ----- \n",
            " 0   movie_id  3883 non-null   int64 \n",
            " 1   title     3883 non-null   object\n",
            " 2   genres    3883 non-null   object\n",
            "dtypes: int64(1), object(2)\n",
            "memory usage: 91.1+ KB\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Wiqm8S1Ny2nh"
      },
      "source": [
        "### Genres\n",
        "The genres data will be crucial to our modeling later on. In order to make it more wieldy we will change the seperation technique to  a simple space-seperated string.  "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dQEWiN6FyyVv"
      },
      "source": [
        "movies['genres'] = movies['genres'].str.split('|')\n",
        "movies['genres'] = movies['genres'].str.join(' ').astype(str)"
      ],
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 112
        },
        "id": "EBJ1oiBN0IVT",
        "outputId": "5bf05a83-40bc-4050-83d3-6196ea550263"
      },
      "source": [
        "movies.head(2)"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>movie_id</th>\n",
              "      <th>title</th>\n",
              "      <th>genres</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1</td>\n",
              "      <td>Toy Story (1995)</td>\n",
              "      <td>Animation Children's Comedy</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2</td>\n",
              "      <td>Jumanji (1995)</td>\n",
              "      <td>Adventure Children's Fantasy</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   movie_id             title                        genres\n",
              "0         1  Toy Story (1995)   Animation Children's Comedy\n",
              "1         2    Jumanji (1995)  Adventure Children's Fantasy"
            ]
          },
          "metadata": {},
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MI4H2i3K2ENu"
      },
      "source": [
        "# Users Dataframe"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "doOMhRIj2Pgw",
        "outputId": "d21152ab-85dc-48f6-fdac-10ba909ea5d4"
      },
      "source": [
        "users = pd.read_csv(DATA_PATH + 'users.dat', \n",
        "                    sep='::', \n",
        "                    engine='python', \n",
        "                    names=['user_id', 'gender', 'age', 'occupation', 'zip'])\n",
        "users.head()"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>user_id</th>\n",
              "      <th>gender</th>\n",
              "      <th>age</th>\n",
              "      <th>occupation</th>\n",
              "      <th>zip</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1</td>\n",
              "      <td>F</td>\n",
              "      <td>1</td>\n",
              "      <td>10</td>\n",
              "      <td>48067</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2</td>\n",
              "      <td>M</td>\n",
              "      <td>56</td>\n",
              "      <td>16</td>\n",
              "      <td>70072</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>3</td>\n",
              "      <td>M</td>\n",
              "      <td>25</td>\n",
              "      <td>15</td>\n",
              "      <td>55117</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>4</td>\n",
              "      <td>M</td>\n",
              "      <td>45</td>\n",
              "      <td>7</td>\n",
              "      <td>02460</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>5</td>\n",
              "      <td>M</td>\n",
              "      <td>25</td>\n",
              "      <td>20</td>\n",
              "      <td>55455</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   user_id gender  age  occupation    zip\n",
              "0        1      F    1          10  48067\n",
              "1        2      M   56          16  70072\n",
              "2        3      M   25          15  55117\n",
              "3        4      M   45           7  02460\n",
              "4        5      M   25          20  55455"
            ]
          },
          "metadata": {},
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "CcAM6vqO2TJz",
        "outputId": "02b3858b-baca-4d7a-fa01-e73544a3a63e"
      },
      "source": [
        "users.info()"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "<class 'pandas.core.frame.DataFrame'>\n",
            "RangeIndex: 6040 entries, 0 to 6039\n",
            "Data columns (total 5 columns):\n",
            " #   Column      Non-Null Count  Dtype \n",
            "---  ------      --------------  ----- \n",
            " 0   user_id     6040 non-null   int64 \n",
            " 1   gender      6040 non-null   object\n",
            " 2   age         6040 non-null   int64 \n",
            " 3   occupation  6040 non-null   int64 \n",
            " 4   zip         6040 non-null   object\n",
            "dtypes: int64(3), object(2)\n",
            "memory usage: 236.1+ KB\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dbR83QcZ2WP0"
      },
      "source": [
        "## Elaborating on the numeric values\n",
        "\n",
        "This will make it easier to interpret our EDA. The mappings below are taken from the dataset's `README`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gDWqAh_22ez8"
      },
      "source": [
        "age_map = {\n",
        "    1  : 'Under 18',\n",
        "    18 : '18-24',\n",
        "    25 : '25-34',\n",
        "    35 : '35-44',\n",
        "    45 : '45-49',\n",
        "    50 : '50-55',\n",
        "    56 : '56+'\n",
        "} \n",
        "occ_map = {\n",
        "    0:  \"other or not specified\",\n",
        "    1:  \"academic/educator\",\n",
        "    2:  \"artist\",\n",
        "    3:  \"clerical/admin\",\n",
        "    4:  \"college/grad student\",\n",
        "    5:  \"customer service\",\n",
        "    6:  \"doctor/health care\",\n",
        "    7:  \"executive/managerial\",\n",
        "    8:  \"farmer\",\n",
        "    9:  \"homemaker\",\n",
        "    10:  \"K-12 student\",\n",
        "    11:  \"lawyer\",\n",
        "    12:  \"programmer\",\n",
        "    13:  \"retired\",\n",
        "    14:  \"sales/marketing\",\n",
        "    15:  \"scientist\",\n",
        "    16:  \"self-employed\",\n",
        "    17:  \"technician/engineer\",\n",
        "    18:  \"tradesman/craftsman\",\n",
        "    19:  \"unemployed\",\n",
        "    20:  \"writer\"\n",
        "}\n",
        "users['age_elab'] = users['age'].map(age_map)\n",
        "users['occ_elab'] = users['occupation'].map(occ_map)"
      ],
      "execution_count": 10,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "7WycJcE020XK",
        "outputId": "e2f35d7b-a913-4bf5-a5af-37b08e8c4509"
      },
      "source": [
        "users.head()"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>user_id</th>\n",
              "      <th>gender</th>\n",
              "      <th>age</th>\n",
              "      <th>occupation</th>\n",
              "      <th>zip</th>\n",
              "      <th>age_elab</th>\n",
              "      <th>occ_elab</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1</td>\n",
              "      <td>F</td>\n",
              "      <td>1</td>\n",
              "      <td>10</td>\n",
              "      <td>48067</td>\n",
              "      <td>Under 18</td>\n",
              "      <td>K-12 student</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2</td>\n",
              "      <td>M</td>\n",
              "      <td>56</td>\n",
              "      <td>16</td>\n",
              "      <td>70072</td>\n",
              "      <td>56+</td>\n",
              "      <td>self-employed</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>3</td>\n",
              "      <td>M</td>\n",
              "      <td>25</td>\n",
              "      <td>15</td>\n",
              "      <td>55117</td>\n",
              "      <td>25-34</td>\n",
              "      <td>scientist</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>4</td>\n",
              "      <td>M</td>\n",
              "      <td>45</td>\n",
              "      <td>7</td>\n",
              "      <td>02460</td>\n",
              "      <td>45-49</td>\n",
              "      <td>executive/managerial</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>5</td>\n",
              "      <td>M</td>\n",
              "      <td>25</td>\n",
              "      <td>20</td>\n",
              "      <td>55455</td>\n",
              "      <td>25-34</td>\n",
              "      <td>writer</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   user_id gender  age  occupation    zip  age_elab              occ_elab\n",
              "0        1      F    1          10  48067  Under 18          K-12 student\n",
              "1        2      M   56          16  70072       56+         self-employed\n",
              "2        3      M   25          15  55117     25-34             scientist\n",
              "3        4      M   45           7  02460     45-49  executive/managerial\n",
              "4        5      M   25          20  55455     25-34                writer"
            ]
          },
          "metadata": {},
          "execution_count": 11
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UQkx5eg_vvbP"
      },
      "source": [
        "# Ratings Dataframe"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "56nv6CDDvfpm",
        "outputId": "2a11c4a2-71f4-4e42-9917-a4fd25634276"
      },
      "source": [
        "ratings = pd.read_csv(DATA_PATH + 'ratings.dat', sep='::', \n",
        "                     engine='python', \n",
        "                     names=['user_id', 'movie_id', 'rating', 'timestamp'])\n",
        "ratings.head()"
      ],
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>user_id</th>\n",
              "      <th>movie_id</th>\n",
              "      <th>rating</th>\n",
              "      <th>timestamp</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1</td>\n",
              "      <td>1193</td>\n",
              "      <td>5</td>\n",
              "      <td>978300760</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1</td>\n",
              "      <td>661</td>\n",
              "      <td>3</td>\n",
              "      <td>978302109</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>1</td>\n",
              "      <td>914</td>\n",
              "      <td>3</td>\n",
              "      <td>978301968</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>1</td>\n",
              "      <td>3408</td>\n",
              "      <td>4</td>\n",
              "      <td>978300275</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>1</td>\n",
              "      <td>2355</td>\n",
              "      <td>5</td>\n",
              "      <td>978824291</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   user_id  movie_id  rating  timestamp\n",
              "0        1      1193       5  978300760\n",
              "1        1       661       3  978302109\n",
              "2        1       914       3  978301968\n",
              "3        1      3408       4  978300275\n",
              "4        1      2355       5  978824291"
            ]
          },
          "metadata": {},
          "execution_count": 12
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9PyXA1900ZaX"
      },
      "source": [
        "## Unrated movies\n",
        "\n",
        "Not the kind of unrated as in *X-rated* but movies that appear in movies but not in ratings. To resolve this we will generate ratings from a fake user. In our final modeling and implementation it will be critical that we preserve every movie in our data. The fake user will generate ratings as to totall encompass every movie in our dataframe. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "81L05Zh_vnpm",
        "outputId": "7bfb9033-e47d-4d87-cfb9-c2a2255534d0"
      },
      "source": [
        "all_movie_ids = movies['movie_id'].values\n",
        "rated_movie_ids = ratings['movie_id'].values\n",
        "unrated_ids = [x for x in all_movie_ids if x not in rated_movie_ids]\n",
        "\n",
        "print(f'There are  {len(unrated_ids)}  movies out of that do not have ratings')"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "There are  177  movies out of that do not have ratings\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oy7QJPa73YEO"
      },
      "source": [
        "### Fake User"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 81
        },
        "id": "wkS5_nls0tOu",
        "outputId": "b06d8e0c-3b69-41f1-bfaa-1ba6506f7433"
      },
      "source": [
        "# creating a fake user that we can use to make our test predictions in ratings from\n",
        "fake_user = pd.Series([6041, 'M', 30, 0, '07974', '25-34', 'other or not specified'], index=users.columns)\n",
        "users = users.append(fake_user, ignore_index=True)\n",
        "users.tail(1)"
      ],
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>user_id</th>\n",
              "      <th>gender</th>\n",
              "      <th>age</th>\n",
              "      <th>occupation</th>\n",
              "      <th>zip</th>\n",
              "      <th>age_elab</th>\n",
              "      <th>occ_elab</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>6040</th>\n",
              "      <td>6041</td>\n",
              "      <td>M</td>\n",
              "      <td>30</td>\n",
              "      <td>0</td>\n",
              "      <td>07974</td>\n",
              "      <td>25-34</td>\n",
              "      <td>other or not specified</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "      user_id gender  age  occupation    zip age_elab                occ_elab\n",
              "6040     6041      M   30           0  07974    25-34  other or not specified"
            ]
          },
          "metadata": {},
          "execution_count": 14
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1mOVGi-V3zoW"
      },
      "source": [
        "### Creating Fake (Imputed) Ratings"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "D9tDtfSA43QV",
        "outputId": "57b460dd-b06a-4b7f-d118-af533895f009"
      },
      "source": [
        "round(np.mean(ratings['rating']))"
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "4"
            ]
          },
          "metadata": {},
          "execution_count": 15
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ltGIAQGX3wnQ"
      },
      "source": [
        "# First create a mapping of genre to average ratings\n",
        "mean_score = round(np.mean(ratings['rating']))\n",
        "\n",
        "# create a fake ratings dataframe\n",
        "fake_ratings = []\n",
        "\n",
        "for movie_id in unrated_ids:\n",
        "  # check the movies genre and get the genre's average rating\n",
        "  movie_series = movies.query(f'movie_id == {movie_id}')\n",
        "  genre = str(movie_series['genres'].values).strip('[]').replace(\"'\",'')\n",
        "  \n",
        "  # if genre not listed just use 3... which is the global average\n",
        "  try:\n",
        "    g_avg = ugenre_map[genre]\n",
        "  except:\n",
        "    g_avg = 3\n",
        "\n",
        "  # our fake user from above will be the user submitting the rating\n",
        "  user_id = 6041\n",
        "\n",
        "  # we need a timestamp for the rating data\n",
        "  time_stamp = int(datetime.timestamp(datetime.now()))\n",
        "\n",
        "  # create the rating to attach\n",
        "  fake_rating = {\n",
        "      'user_id'   : 6041,\n",
        "      'movie_id'  : movie_id,\n",
        "      'rating'    : mean_score,\n",
        "      'timestamp' : np.int(datetime.timestamp(datetime.now()))\n",
        "  }\n",
        "  fake_ratings.append(fake_rating)"
      ],
      "execution_count": 16,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nLuIsGzv4t-f"
      },
      "source": [
        "fakedf = pd.DataFrame(data=fake_ratings, columns=['user_id','movie_id','rating','timestamp'])"
      ],
      "execution_count": 17,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oZ7NkzKP5VUe"
      },
      "source": [
        "ratings = ratings.append(fakedf,ignore_index=True)"
      ],
      "execution_count": 18,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VsKfza5u5nyp"
      },
      "source": [
        "### Confirming Changes"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "tKa9WAY85hFC",
        "outputId": "c2aa57d8-3377-4df6-f56c-29f19fd46383"
      },
      "source": [
        "all_movie_ids = movies['movie_id'].values\n",
        "rated_movie_ids = ratings['movie_id'].values\n",
        "unrated_ids = [x for x in all_movie_ids if x not in rated_movie_ids]\n",
        "\n",
        "print(f'There are  {len(unrated_ids)}  movies out of that do not have ratings')"
      ],
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "There are  0  movies out of that do not have ratings\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sRPZ10li6kdc"
      },
      "source": [
        "# Combined Dataframes\n",
        "\n",
        "This dataframe will make modeling easier later on. By combining all of our data, we create a monolithic dataframe that is easily accessible and reduces the amount of importing we need to do later on. We will use a simple inner join, the `pd.merge` default, to merge our data."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "im44tIUZ5l5q"
      },
      "source": [
        "combined = pd.merge(movies, ratings, on='movie_id')\n",
        "combined = pd.merge(combined, users, on='user_id')"
      ],
      "execution_count": 20,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sFgEpsgy7PJN"
      },
      "source": [
        "# Saving our Data"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7zqW2vhr7IzH"
      },
      "source": [
        "movies.to_csv(DATA_PATH + 'processed_movies.csv', sep=',', header=True, columns=['movie_id', 'title', 'genres'])\n",
        "ratings.to_csv(DATA_PATH + 'processed_ratings.csv', sep=',', header=True, columns=['user_id','movie_id','rating','timestamp'])\n",
        "users.to_csv(DATA_PATH + 'processed_users.csv', sep=',', header=True, columns=['user_id', 'gender', 'age', 'occupation', 'zip', 'age_elab', 'occ_elab'])\n",
        "combined.to_csv(DATA_PATH + 'combined.csv', header=True)"
      ],
      "execution_count": 22,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Qv5mGFbBITD-"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}