Copy_of_DMA24_Lab_1__Data_Preprocessing.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.9"
    },
    "colab": {
      "provenance": []
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vKduyi9dItO4"
      },
      "source": [
        "DATA MINING & ANALYTICS (2024)\n",
        "\n",
        "Make sure you fill in any place that says `YOUR CODE HERE` or `YOUR ANSWER HERE`, as well as your name below:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FsEVbU41ItO7"
      },
      "source": [
        "NAME = \"Min Cheah\""
      ],
      "execution_count": 1,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "M7sDljEMItPA"
      },
      "source": [
        "---"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "checksum": "720281502b9b9c7f8930d61c682cb061",
          "grade": false,
          "grade_id": "cell-731531e531b99c39",
          "locked": true,
          "schema_version": 1,
          "solution": false
        },
        "id": "LJNmSc7sItPB"
      },
      "source": [
        "# Lab 1 - Data Preprocessing\n",
        "\n",
        "Data transformations are useful for preparing a dataset for answering a particular question. Part of this process involves generating features from the dataset you find relevant to the question at hand. For this lab, we will be using a Yelp reviews dataset. Each row in the dataset depicts one review along with the features of the review (the reviewer, the review text, etc.). The goal of this lab is to eventually convert this reviews dataset into a *reviewers* dataset by creating different features describing each reviewer.\n",
        "\n",
        "The submission for this assignment should be done *individually*, but you are allowed to work in groups of 2.\n",
        "\n",
        "# Google Colab\n",
        "\n",
        "Colab is a free online platform provided by Google that allows you to execute python code without any installations on your local machine. Without Colab (using Jupyter notebooks or the command line), you would have to install various packages and manage dependencies.\n",
        "\n",
        "In Colab, you can simply import them, or even install them (for that particular session). Colab can be accessed at the link: https://colab.research.google.com\n",
        "\n",
        "\n",
        "**IMPORTANT: This lab has been shared with only read permissions to you. Make sure to click File -> \"Save a Copy in Drive\" so that you can get your own copy that WILL SAVE YOUR PROGRESS in your own Colab environment.**\n",
        "\n",
        "**If you download the .ipynb and want to further edit the notebook, you will need to make sure you have [Jupyter](https://jupyter.org/install) installed locally so you can view the notebook properly (not as a JSON file).**\n",
        "\n",
        "# Environment Setup\n",
        "Run this cell to setup your environment."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "checksum": "c4f9a2a35aae087d70dccad8514e5e4f",
          "grade": false,
          "grade_id": "cell-88d067a4407f02bf",
          "locked": true,
          "schema_version": 1,
          "solution": false
        },
        "id": "LIH96tFzItPC",
        "outputId": "65ffab76-c8e3-4a56-f80d-1c4c8ba8c823",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "# Importing libraries\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "import math\n",
        "import os\n",
        "print('Libraries Imported')\n",
        "\n",
        "#DOWNLOADING DATASET IF NOT PRESENT\n",
        "!wget -nc http://quadro.ist.berkeley.edu:1331/yelp_reviews.csv\n",
        "\n",
        "#!unzip yelp_reviews.zip\n",
        "print('Dataset Downloaded: yelp_reviews.csv')\n",
        "df=pd.read_csv('yelp_reviews.csv')\n",
        "print(df.head())\n",
        "\n",
        "print('Setup Complete')"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Libraries Imported\n",
            "--2024-09-11 04:51:39--  http://quadro.ist.berkeley.edu:1331/yelp_reviews.csv\n",
            "Resolving quadro.ist.berkeley.edu (quadro.ist.berkeley.edu)... 169.229.194.98\n",
            "Connecting to quadro.ist.berkeley.edu (quadro.ist.berkeley.edu)|169.229.194.98|:1331... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 376638166 (359M) [text/csv]\n",
            "Saving to: ‘yelp_reviews.csv’\n",
            "\n",
            "yelp_reviews.csv    100%[===================>] 359.19M  82.9MB/s    in 4.4s    \n",
            "\n",
            "2024-09-11 04:51:43 (81.5 MB/s) - ‘yelp_reviews.csv’ saved [376638166/376638166]\n",
            "\n",
            "Dataset Downloaded: yelp_reviews.csv\n",
            "     type             business_id                 user_id  stars  \\\n",
            "0  review  mxrXVZWc6PWk81gvOVNOUw  mv7shusL4Xb6TylVYBv4CA      4   \n",
            "1  review  mxrXVZWc6PWk81gvOVNOUw  0aN5QPhs-VwK2vusKG0waQ      5   \n",
            "2  review  kK4AzZ0YWI-U2G-paAL7Fg  0aN5QPhs-VwK2vusKG0waQ      5   \n",
            "3  review  mxrXVZWc6PWk81gvOVNOUw  1JUwyYab-uJzEx_FRd81Zg      5   \n",
            "4  review  mxrXVZWc6PWk81gvOVNOUw  2Zd3Xy8hUVmZkNg7RyNjhg      4   \n",
            "\n",
            "                                                text        date  cool_votes  \\\n",
            "0  Definitely try the duck dish.   I rank it amon...  2011-06-13           0   \n",
            "1  Big Ass Burger was awesome! Great $5 mojitos. ...  2011-06-25           1   \n",
            "2             Unbelievable sandwiches! Good service.  2011-06-25           0   \n",
            "3  Awesome, awesome, awesome! My mom and sister a...  2011-07-18           1   \n",
            "4  I had the ribs they were great.  The beer sele...  2011-07-19           1   \n",
            "\n",
            "   useful_votes  funny_votes  \n",
            "0             0            0  \n",
            "1             0            0  \n",
            "2             0            0  \n",
            "3             1            0  \n",
            "4             0            1  \n",
            "Setup Complete\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "checksum": "51be56b978a555a682bd1a02d85a7434",
          "grade": false,
          "grade_id": "cell-63034c0f0a81dc85",
          "locked": true,
          "schema_version": 1,
          "solution": false
        },
        "id": "esN8qRHkItPG"
      },
      "source": [
        "**Q1: What was the highest number of reviews for any one `business_id`?**\n",
        "- For this task, we will need to group the reviews dataset by `business_id`. This will aggregate data for each business, which is what we need for this task. This can be done using the [groupby](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.groupby.html) method. Some pointers of how you could go about this question are listed below:\n",
        "    - ```yelp_businesses = yelp_dataset.groupby('business_id').size() ```\n",
        "    - The `.size()` function counts the number of instances for each `business_id`, which gives us the number of reviews as each instance in this dataset is a review.\n",
        "    - The following command will sort this list, after which you can take note of the highest value: ```sorted_yelp_businesses = yelp_businesses.sort_values(ascending=False, inplace=False)```\n",
        "    - This approach allows you to see the data structure being used in the sort. A quicker approach to getting the max would be to use the max function: ```max(yelp_businesses)```\n",
        "<br>\n",
        "<br>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "deletable": false,
        "nbgrader": {
          "checksum": "a4b3a58ab839a93b83e514e1db0c11e3",
          "grade": false,
          "grade_id": "cell-b9fd95d084732ea0",
          "locked": false,
          "schema_version": 1,
          "solution": true
        },
        "id": "AbKk0rohItPG"
      },
      "source": [
        "\n",
        "#Make sure you return the answer value in this function\n",
        "def q1(df):\n",
        "    yelp_businesses = df.groupby('business_id').size()\n",
        "    return yelp_businesses.max()"
      ],
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "checksum": "e989434a5e20d93eb1500308f7ecf163",
          "grade": true,
          "grade_id": "cell-1d1402069f0f336b",
          "locked": true,
          "points": 0.5,
          "schema_version": 1,
          "solution": false
        },
        "id": "rzD83og_ItPJ",
        "outputId": "bbdef3fd-6605-452e-c746-8992409885b8",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "#This is a graded cell, do not edit\n",
        "print(q1(df))"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "4128\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "checksum": "6cde0278ab840df1fb90b16ca8c14a8a",
          "grade": false,
          "grade_id": "cell-c998b50208612779",
          "locked": true,
          "schema_version": 1,
          "solution": false
        },
        "id": "fZ_McPZ-ItPL"
      },
      "source": [
        "**Q2: On average, how many reviews did each business get?**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "deletable": false,
        "nbgrader": {
          "checksum": "6ff0153f15cd01aa6d958646c22cd2eb",
          "grade": false,
          "grade_id": "cell-bda529e7277d1978",
          "locked": false,
          "schema_version": 1,
          "solution": true
        },
        "id": "0V-m-GMAItPM"
      },
      "source": [
        "#Make sure you return the answer value in this function\n",
        "def q2(df):\n",
        "    yelp_businesses_avg = df.groupby('business_id').size()\n",
        "    return yelp_businesses_avg.mean()\n"
      ],
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "checksum": "72d8375074f6e5eb294215044c460696",
          "grade": true,
          "grade_id": "cell-febce0a51b6da9da",
          "locked": true,
          "points": 0.5,
          "schema_version": 1,
          "solution": false
        },
        "id": "V_9IXDmvItPO",
        "outputId": "fc723d7c-36c7-410a-b8e6-509708e3b6a1",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "#This is a graded cell, do not edit\n",
        "print(q2(df))\n"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "12.63413902163123\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "checksum": "d6874779b6c6d2e361fc9891455b6960",
          "grade": false,
          "grade_id": "cell-a568a611d600caa2",
          "locked": true,
          "schema_version": 1,
          "solution": false
        },
        "id": "T9bRVaYZItPQ"
      },
      "source": [
        "**Q3: What is the average number of reviews per reviewer?**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "deletable": false,
        "nbgrader": {
          "checksum": "369c594937118aa98e0aea635c345943",
          "grade": false,
          "grade_id": "cell-f92e05c4b7a28174",
          "locked": false,
          "schema_version": 1,
          "solution": true
        },
        "id": "J58jfsKTItPQ"
      },
      "source": [
        "#Make sure you return the answer value in this function\n",
        "def q3(df):\n",
        "    reviews_per_reviewer = df.groupby('user_id').size()\n",
        "    return reviews_per_reviewer.mean()"
      ],
      "execution_count": 7,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "checksum": "7ed7ddd4ec46674bdd89445dc1034175",
          "grade": true,
          "grade_id": "cell-d76279e8e7d4a79f",
          "locked": true,
          "points": 0.5,
          "schema_version": 1,
          "solution": false
        },
        "id": "Wx5wcx6pItPT",
        "outputId": "7619c4be-03e5-46d1-c8e3-f691bc601f8b",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "#This is a graded cell, do not edit\n",
        "print(q3(df))\n"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "3.188511934933203\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "checksum": "e8d5fd42acde59c3bd19cc07db76ac1c",
          "grade": false,
          "grade_id": "cell-c76f8dc8692f2472",
          "locked": true,
          "schema_version": 1,
          "solution": false
        },
        "id": "oa0doTGiItPW"
      },
      "source": [
        "**Q4: Calculate the total number of cool votes per reviewer, then average these totals across reviewers.**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "deletable": false,
        "nbgrader": {
          "checksum": "2db144055be1d1f506af2bbdcf924e58",
          "grade": false,
          "grade_id": "cell-f06192fbabe75195",
          "locked": false,
          "schema_version": 1,
          "solution": true
        },
        "id": "sWuWYjrSItPX"
      },
      "source": [
        "#Make sure you return the answer value in this function\n",
        "def q4(df):\n",
        "    total_cool_votes = df.groupby('user_id')['cool_votes'].sum()\n",
        "    return total_cool_votes.mean()\n"
      ],
      "execution_count": 9,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "checksum": "70b555da34836694349c4923beb1950b",
          "grade": true,
          "grade_id": "cell-cabbe00e6fee304e",
          "locked": true,
          "points": 0.5,
          "schema_version": 1,
          "solution": false
        },
        "id": "z3o9lMo-ItPZ",
        "outputId": "b280c67a-0dea-4c11-ae1d-2a057bdac66e",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "#This is a graded cell, do not edit\n",
        "print(q4(df))\n"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "1.2417282785380945\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "checksum": "1e80dfdd5311c1dd540fafe446ad10c3",
          "grade": false,
          "grade_id": "cell-9e278b8cd31ace62",
          "locked": true,
          "schema_version": 1,
          "solution": false
        },
        "id": "fDD4Lh8fItPb"
      },
      "source": [
        "**Q5: Calculate the total number of funny votes per reviewer, then average these totals across reviewers.**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "deletable": false,
        "nbgrader": {
          "checksum": "cb6393f6479792516d064ab368490d90",
          "grade": false,
          "grade_id": "cell-93635f0a6f297df9",
          "locked": false,
          "schema_version": 1,
          "solution": true
        },
        "id": "KiVt8qB7ItPc"
      },
      "source": [
        "#Make sure you return the answer value in this function\n",
        "def q5(df):\n",
        "     total_funny_votes = df.groupby('user_id')['funny_votes'].sum()\n",
        "     return total_funny_votes.mean()\n"
      ],
      "execution_count": 11,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "checksum": "d3be5fcefa0f798be6504de7f8f1b344",
          "grade": true,
          "grade_id": "cell-4c41413214d2d4f1",
          "locked": true,
          "points": 0.5,
          "schema_version": 1,
          "solution": false
        },
        "id": "bWzUljWAItPe",
        "outputId": "b43d8af6-086a-42bd-fa01-a6a294959fff",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "#This is a graded cell, do not edit\n",
        "print(q5(df))\n"
      ],
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "1.10126486404605\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "checksum": "e62dd2a28323b8d943f63dda321c3aea",
          "grade": false,
          "grade_id": "cell-9554ed7bdee76fb1",
          "locked": true,
          "schema_version": 1,
          "solution": false
        },
        "id": "qx0rBKw9ItPg"
      },
      "source": [
        "**Q6: Calculate the total number of useful votes per reviewer, then average these totals across reviewers.**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "deletable": false,
        "nbgrader": {
          "checksum": "4375ea8e9efa96b5ea65a2f4f553faad",
          "grade": false,
          "grade_id": "cell-2cebd87b7f8b37d1",
          "locked": false,
          "schema_version": 1,
          "solution": true
        },
        "id": "KIYaRM3uItPh"
      },
      "source": [
        "#Make sure you return the answer in this function\n",
        "def q6(df):\n",
        "\n",
        "    total_useful_votes = df.groupby('user_id')['useful_votes'].sum()\n",
        "    return total_useful_votes.mean()"
      ],
      "execution_count": 13,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "checksum": "2435ef28744576ab65e21558b65e0a5e",
          "grade": true,
          "grade_id": "cell-1a546329acbcd0e4",
          "locked": true,
          "points": 0.5,
          "schema_version": 1,
          "solution": false
        },
        "id": "xrNe4FWdItPj",
        "outputId": "209fda58-7c44-4eb6-d2a8-0392b782054c",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "#This is a graded cell, do not edit\n",
        "print(q6(df))\n"
      ],
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "2.484476138872867\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "checksum": "560323764f6c0effd9615228a4dd9537",
          "grade": false,
          "grade_id": "cell-328aa1a4c9a958d8",
          "locked": true,
          "schema_version": 1,
          "solution": false
        },
        "id": "yUBt7rj9ItP-"
      },
      "source": [
        "**Q7: On average, what percentage of a reviewer's votes are cool votes?**\n",
        "\n",
        "(hint1: calculate the percentage of cool votes for each reviewer, then average this percentage across reviewers)\n",
        "\n",
        "(hint2: you should discard reviewers who have absolutely no votes - from cool, funny, or useful votes - from your calculation)\n",
        "\n",
        "(hint3: if you run into a data type issue, convert your values to numeric using the pd.to_numeric function)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "deletable": false,
        "nbgrader": {
          "checksum": "c4ca0edbf727b2adc41527ebb2129cef",
          "grade": false,
          "grade_id": "cell-54fe5dd9777347a1",
          "locked": false,
          "schema_version": 1,
          "solution": true
        },
        "id": "LuFzcyytItP-"
      },
      "source": [
        "#Make sure you return the answer in this function\n",
        "#Remember to multiply by 100 for percentages\n",
        "def q7(df):\n",
        "  df = df.copy()\n",
        "  df =  df[(df['funny_votes'] > 0) | (df['cool_votes'] > 0) | (df['useful_votes'] > 0)]\n",
        "  df['sum'] = df['cool_votes'] + df['useful_votes'] + df['funny_votes']\n",
        "  df['cool_percent'] =  ( df['cool_votes'] / df['sum'] ) * 100\n",
        "  return df['cool_percent'].mean()"
      ],
      "execution_count": 15,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "checksum": "6ba2d96b43cc0796b9c8ec31e17fefe9",
          "grade": true,
          "grade_id": "cell-594b1121d785d15e",
          "locked": true,
          "points": 0.5,
          "schema_version": 1,
          "solution": false
        },
        "id": "ftY6isjeItQA",
        "outputId": "3b0328a9-87e5-4ca4-fa94-a077bc3d0e7c",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "#This is a graded cell, do not edit\n",
        "#Remember to multiply by 100 for percentages\n",
        "print(round(q7(df),2))\n"
      ],
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "21.68\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "checksum": "f7ca49268591f6d357d5e4d4be17e3c9",
          "grade": false,
          "grade_id": "cell-8b434399df89705e",
          "locked": true,
          "schema_version": 1,
          "solution": false
        },
        "id": "zKe8LJJLItQC"
      },
      "source": [
        "**Q8: On average, what percentage of a reviewer's votes are funny votes?**\n",
        "\n",
        "(hint1: calculate the percentage of funny votes for each reviewer, then average this percentage across reviewers)\n",
        "\n",
        "(hint2: you should discard reviewers who have zero total votes from your calculation)\n",
        "\n",
        "(hint3: if you run into a data type issue, convert your values to numeric using the pd.to_numeric function)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "deletable": false,
        "nbgrader": {
          "checksum": "b869bd5447e8ea81d0af3cae06012b94",
          "grade": false,
          "grade_id": "cell-a3dc739b16a53957",
          "locked": false,
          "schema_version": 1,
          "solution": true
        },
        "id": "DkXVxKdXItQC"
      },
      "source": [
        "#Make sure you return the answer in this function\n",
        "#Remember to multiply by 100 for percentages\n",
        "def q8(df):\n",
        "  df = df.copy()\n",
        "  df =  df[(df['funny_votes'] > 0) | (df['cool_votes'] > 0) | (df['useful_votes'] > 0)]\n",
        "  df['total'] = df['cool_votes'] + df['useful_votes'] + df['funny_votes']\n",
        "  df['funny_per'] =  ( df['funny_votes'] / df['total'] ) * 100\n",
        "  return df['funny_per'].mean()"
      ],
      "execution_count": 17,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "checksum": "2ef3655caaaedc3e433262c7db94fb77",
          "grade": true,
          "grade_id": "cell-e7fdee0fd548baf1",
          "locked": true,
          "points": 0.5,
          "schema_version": 1,
          "solution": false
        },
        "id": "q6uEzM9HItQE",
        "outputId": "20e6c472-03d5-4730-ba53-669c7e5105cc",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "#This is a graded cell, do not edit\n",
        "print(round(q8(df),2))\n"
      ],
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "18.1\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "checksum": "3ce1c8eefc6053b8515dc4218c5a0afb",
          "grade": false,
          "grade_id": "cell-40cd96c922286117",
          "locked": true,
          "schema_version": 1,
          "solution": false
        },
        "id": "wYs2eqhSItQI"
      },
      "source": [
        "**Q9: On average, what percentage of a reviewer's votes are useful votes?**\n",
        "\n",
        "(hint1: calculate the percentage of useful votes for each reviewer, then average this percentage across reviewers)\n",
        "\n",
        "(hint2: you should discard reviewers who have zero total votes from your calculation)\n",
        "\n",
        "(hint3: if you run into a data type issue, convert your values to numeric using the pd.to_numeric function)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "deletable": false,
        "nbgrader": {
          "checksum": "c8e322eafbf209672785a2c9f7c97f52",
          "grade": false,
          "grade_id": "cell-38769f4cbbe9cb8f",
          "locked": false,
          "schema_version": 1,
          "solution": true
        },
        "id": "56CBDjgrItQI"
      },
      "source": [
        "#Make sure you return the answer in this function\n",
        "def q9(df):\n",
        "  df = df.copy()\n",
        "  df =  df[(df['funny_votes'] > 0) | (df['cool_votes'] > 0) | (df['useful_votes'] > 0)]\n",
        "  df['total'] = df['cool_votes'] + df['useful_votes'] + df['funny_votes']\n",
        "  df['useful_per'] =  ( df['useful_votes'] / df['total'] ) * 100\n",
        "  return df['useful_per'].mean()"
      ],
      "execution_count": 19,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "checksum": "d3aeca2366b4c12d564167261c6bc6a2",
          "grade": true,
          "grade_id": "cell-590fb1425068cc06",
          "locked": true,
          "points": 0.5,
          "schema_version": 1,
          "solution": false
        },
        "id": "LvE8ZWSOItQL",
        "outputId": "b9efe531-faa0-4521-ba5c-1487174a896a",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "#This is a graded cell, do not edit\n",
        "print(round(q9(df),2))\n"
      ],
      "execution_count": 20,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "60.22\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "checksum": "2efdbde3ea13f0e1829d7488da2c6d92",
          "grade": false,
          "grade_id": "cell-615a3d7b2c4e0e26",
          "locked": true,
          "schema_version": 1,
          "solution": false
        },
        "id": "2ixankjPItQO"
      },
      "source": [
        "**Q10: Find the average review text length (in non-space characters).**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "deletable": false,
        "nbgrader": {
          "checksum": "59cc81340d9a1967bd6d3a0ffcacdcf3",
          "grade": false,
          "grade_id": "cell-75ec15303c64c506",
          "locked": false,
          "schema_version": 1,
          "solution": true
        },
        "id": "qgwHiHaVItQP"
      },
      "source": [
        "#Make sure you return the answer in this function\n",
        "def q10(df):\n",
        "    df['text_length'] = df['text'].apply(lambda x: len(x.replace(\" \", \"\")))\n",
        "    return df['text_length'].mean()"
      ],
      "execution_count": 21,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "checksum": "083e5ff6b77629d8784b8f6a30352251",
          "grade": true,
          "grade_id": "cell-d9fb4fc39a0b21c4",
          "locked": true,
          "points": 1,
          "schema_version": 1,
          "solution": false
        },
        "id": "_XmxRXdFItQQ",
        "outputId": "65f67144-a254-4736-b3ff-b37ab20e42fa",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "#This is a graded cell, do not edit\n",
        "print(round(q10(df),0))\n"
      ],
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "499.0\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "checksum": "54c8c584ece619ea786165e50a4e1002",
          "grade": false,
          "grade_id": "cell-a21a423546eb8654",
          "locked": true,
          "schema_version": 1,
          "solution": false
        },
        "id": "03gWN2PAItQS"
      },
      "source": [
        "**Q11: Find the year in which each reviewer wrote the least reviews. Once you have this for each reviewer, subtract the minimum possible year (2004) from each year so that your final feature values are 0, 1, 2, etc.**\n",
        "\n",
        "**Note: we are looking for the `answer` to be in the format of a Pandas Series with `user_id` as the index and the year (in 0, 1, 2 format as listed above) as the value.**"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "df['year'] = pd.to_datetime(df['date']).dt.year\n",
        "reviews_per_year = df.groupby(['user_id', 'year']).size().reset_index(name='review_count')\n",
        "least_reviews_year = reviews_per_year.loc[reviews_per_year.groupby('user_id')['review_count'].idxmin()]\n",
        "least_reviews_year['normalized_year'] = least_reviews_year['year'] - 2004\n",
        "answer = least_reviews_year.set_index('user_id')['normalized_year']\n"
      ],
      "metadata": {
        "id": "NEhNIzzVDU2s"
      },
      "execution_count": 23,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#This is a graded cell, do not edit\n",
        "print(answer.sort_index().head())"
      ],
      "metadata": {
        "id": "sVSBWoYcJSLl",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "17c286ad-b677-414e-f2b3-9c5dd923ece7"
      },
      "execution_count": 24,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "user_id\n",
            "--1Y03CEKR3WDbBjYnsW7A     7\n",
            "--2QZsyXGz1OhiD4-0FQLQ    10\n",
            "--82_AVgRBsLw6Dhy8sEnA     4\n",
            "--8A9o_NeGyt_3kzlXtSdg    12\n",
            "--8WbseBk1NjfPiZWjQ-XQ    12\n",
            "Name: normalized_year, dtype: int32\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deletable": false,
        "editable": false,
        "nbgrader": {
          "checksum": "93f7e9ef2385bdac5184802eb7e1ce75",
          "grade": false,
          "grade_id": "cell-da33d981591f0ff3",
          "locked": true,
          "schema_version": 1,
          "solution": false
        },
        "id": "waqvrQeDItQV"
      },
      "source": [
        "**Q12: Come up with a new feature for each review. This may be derived from existing features.\n",
        "Give your feature the name *my_new_feature.* Display head() of this new feature.**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "deletable": false,
        "nbgrader": {
          "checksum": "787d48b01fdbfeab0810cceb64c1abb3",
          "grade": true,
          "grade_id": "cell-09bbdc6a2946b441",
          "locked": false,
          "points": 1,
          "schema_version": 1,
          "solution": true
        },
        "id": "j3TZ3TD2ItQW"
      },
      "source": [
        "my_new_feature = df['text'].apply(len) * (df['useful_votes'] + df['funny_votes'] + df['cool_votes'])"
      ],
      "execution_count": 25,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#This is a graded cell, do not edit\n",
        "print(my_new_feature.head())"
      ],
      "metadata": {
        "id": "9vuRIWeEKFVb",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "ac8daa6d-1c06-42ac-ca41-94e451b1ab1f"
      },
      "execution_count": 26,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "0      0\n",
            "1    100\n",
            "2      0\n",
            "3    628\n",
            "4    372\n",
            "dtype: int64\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "*ⓒ Prof. Zachary Pardos, 2024*"
      ],
      "metadata": {
        "id": "IQulSenZDOf9"
      }
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "VvtFgYV77Pvf"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}