dataframe_notebooks/Movies_Notebook.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "c9626fb6",
   "metadata": {},
   "source": [
    "# Movies"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5a9624f5",
   "metadata": {},
   "source": [
    "EXISTING"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d3c326c0",
   "metadata": {},
   "source": [
    "This data frame lists all movies, their year, IMBD rating, IMBD votes, and their genres. The `movie_ID` column is later referenced in other data frames."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f7dac6e7",
   "metadata": {},
   "source": [
    "## Table of Contents\n",
    "- [1st Section](#Loading-in-the-data-and-basic-summary) is where I import the files and get an idea of the data shape and contents\n",
    "- [2nd section](#Cleaning-the-data) is where I clean up the year column and add decade information\n",
    "- [3rd section](#Pickling-the-data-and-creating-a-csv) is where the data are saved and exported\n",
    "- [Conclusion](#Conclusion) summarizes the notebook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "12146bf6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# import packages\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e0f29e9e",
   "metadata": {},
   "source": [
    "## Loading in the data and basic summary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "ea141c53",
   "metadata": {},
   "outputs": [],
   "source": [
    "movies_df = pd.read_csv('../data/movie_titles_metadata.txt', sep='\\s+\\+\\+\\+\\$\\+\\+\\+\\s?',\n",
    "                        names=['movie_ID', 'movie_title' , 'movie_year', 'IMDB_rating', 'IMBD_votes', 'genres'], \n",
    "                        engine='python', encoding='ISO-8859-1')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9797725",
   "metadata": {},
   "source": [
    "The data were all separated with ' +++$+++ ' and did not have column names. The README from the original data described what each column was in the data so I used that to create column names."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "e9c965d8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(617, 6)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "movies_df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "4bede941",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 617 entries, 0 to 616\n",
      "Data columns (total 6 columns):\n",
      " #   Column       Non-Null Count  Dtype  \n",
      "---  ------       --------------  -----  \n",
      " 0   movie_ID     617 non-null    object \n",
      " 1   movie_title  617 non-null    object \n",
      " 2   movie_year   617 non-null    object \n",
      " 3   IMDB_rating  617 non-null    float64\n",
      " 4   IMBD_votes   617 non-null    int64  \n",
      " 5   genres       617 non-null    object \n",
      "dtypes: float64(1), int64(1), object(4)\n",
      "memory usage: 29.0+ KB\n"
     ]
    }
   ],
   "source": [
    "# no missing values here\n",
    "movies_df.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "996837f5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>movie_ID</th>\n",
       "      <th>movie_title</th>\n",
       "      <th>movie_year</th>\n",
       "      <th>IMDB_rating</th>\n",
       "      <th>IMBD_votes</th>\n",
       "      <th>genres</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>m0</td>\n",
       "      <td>10 things i hate about you</td>\n",
       "      <td>1999</td>\n",
       "      <td>6.9</td>\n",
       "      <td>62847</td>\n",
       "      <td>['comedy', 'romance']</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>m1</td>\n",
       "      <td>1492: conquest of paradise</td>\n",
       "      <td>1992</td>\n",
       "      <td>6.2</td>\n",
       "      <td>10421</td>\n",
       "      <td>['adventure', 'biography', 'drama', 'history']</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>m2</td>\n",
       "      <td>15 minutes</td>\n",
       "      <td>2001</td>\n",
       "      <td>6.1</td>\n",
       "      <td>25854</td>\n",
       "      <td>['action', 'crime', 'drama', 'thriller']</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>m3</td>\n",
       "      <td>2001: a space odyssey</td>\n",
       "      <td>1968</td>\n",
       "      <td>8.4</td>\n",
       "      <td>163227</td>\n",
       "      <td>['adventure', 'mystery', 'sci-fi']</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>m4</td>\n",
       "      <td>48 hrs.</td>\n",
       "      <td>1982</td>\n",
       "      <td>6.9</td>\n",
       "      <td>22289</td>\n",
       "      <td>['action', 'comedy', 'crime', 'drama', 'thrill...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  movie_ID                 movie_title movie_year  IMDB_rating  IMBD_votes  \\\n",
       "0       m0  10 things i hate about you       1999          6.9       62847   \n",
       "1       m1  1492: conquest of paradise       1992          6.2       10421   \n",
       "2       m2                  15 minutes       2001          6.1       25854   \n",
       "3       m3       2001: a space odyssey       1968          8.4      163227   \n",
       "4       m4                     48 hrs.       1982          6.9       22289   \n",
       "\n",
       "                                              genres  \n",
       "0                              ['comedy', 'romance']  \n",
       "1     ['adventure', 'biography', 'drama', 'history']  \n",
       "2           ['action', 'crime', 'drama', 'thriller']  \n",
       "3                 ['adventure', 'mystery', 'sci-fi']  \n",
       "4  ['action', 'comedy', 'crime', 'drama', 'thrill...  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "movies_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "bac54d88",
   "metadata": {},
   "outputs": [],
   "source": [
    "# I do not need the IMDB information\n",
    "movies_df = movies_df[['movie_ID', 'movie_title', 'movie_year', 'genres']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "35137d43",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'1927'"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# what is the earliest movie in the df?\n",
    "movies_df['movie_year'].min()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "8b2e1a2a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'2010'"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# what is the most recent movie in the df?\n",
    "movies_df['movie_year'].max()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "7ec2093e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>movie_ID</th>\n",
       "      <th>movie_title</th>\n",
       "      <th>genres</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>movie_year</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1927</th>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1931</th>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1932</th>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1933</th>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1934</th>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2007/I</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2008</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2009</th>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2009/I</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>88 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "            movie_ID  movie_title  genres\n",
       "movie_year                               \n",
       "1927               2            2       2\n",
       "1931               2            2       2\n",
       "1932               4            4       4\n",
       "1933               2            2       2\n",
       "1934               3            3       3\n",
       "...              ...          ...     ...\n",
       "2007/I             1            1       1\n",
       "2008               1            1       1\n",
       "2009               3            3       3\n",
       "2009/I             1            1       1\n",
       "2010               1            1       1\n",
       "\n",
       "[88 rows x 3 columns]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# how many movies per year?\n",
    "movies_df.groupby('movie_year').count()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "281d2c32",
   "metadata": {},
   "source": [
    "## Cleaning the data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1f89cfef",
   "metadata": {},
   "source": [
    "Too many years to display all at once, I will add decade information for easier analysis. Some of this data will need to be cleaned up as well, extra '/I' after some of the movie years for some reason."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "1a21d7d9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# removing extra characters after the year\n",
    "movies_df['movie_year'] = movies_df['movie_year'].str.rstrip(r'/I')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "2b683ef4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# convert the movie_year to integer to generate decade\n",
    "movies_df = movies_df.astype({'movie_year':'int'})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "f9a5e48b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>movie_ID</th>\n",
       "      <th>movie_title</th>\n",
       "      <th>genres</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>movie_year</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1927</th>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1931</th>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1932</th>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1933</th>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1934</th>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2006</th>\n",
       "      <td>6</td>\n",
       "      <td>6</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2007</th>\n",
       "      <td>5</td>\n",
       "      <td>5</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2008</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2009</th>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>74 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "            movie_ID  movie_title  genres\n",
       "movie_year                               \n",
       "1927               2            2       2\n",
       "1931               2            2       2\n",
       "1932               4            4       4\n",
       "1933               2            2       2\n",
       "1934               3            3       3\n",
       "...              ...          ...     ...\n",
       "2006               6            6       6\n",
       "2007               5            5       5\n",
       "2008               1            1       1\n",
       "2009               4            4       4\n",
       "2010               1            1       1\n",
       "\n",
       "[74 rows x 3 columns]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "movies_df.groupby('movie_year').count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "854108da",
   "metadata": {},
   "outputs": [],
   "source": [
    "# add decades for easier grouping\n",
    "movies_df['movie_decade'] = 10 * (movies_df['movie_year'] // 10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "d4502345",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>movie_ID</th>\n",
       "      <th>movie_title</th>\n",
       "      <th>movie_year</th>\n",
       "      <th>genres</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>movie_decade</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1920</th>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1930</th>\n",
       "      <td>16</td>\n",
       "      <td>16</td>\n",
       "      <td>16</td>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1940</th>\n",
       "      <td>15</td>\n",
       "      <td>15</td>\n",
       "      <td>15</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1950</th>\n",
       "      <td>17</td>\n",
       "      <td>17</td>\n",
       "      <td>17</td>\n",
       "      <td>17</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1960</th>\n",
       "      <td>19</td>\n",
       "      <td>19</td>\n",
       "      <td>19</td>\n",
       "      <td>19</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1970</th>\n",
       "      <td>51</td>\n",
       "      <td>51</td>\n",
       "      <td>51</td>\n",
       "      <td>51</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1980</th>\n",
       "      <td>108</td>\n",
       "      <td>108</td>\n",
       "      <td>108</td>\n",
       "      <td>108</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1990</th>\n",
       "      <td>244</td>\n",
       "      <td>244</td>\n",
       "      <td>244</td>\n",
       "      <td>244</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2000</th>\n",
       "      <td>144</td>\n",
       "      <td>144</td>\n",
       "      <td>144</td>\n",
       "      <td>144</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              movie_ID  movie_title  movie_year  genres\n",
       "movie_decade                                           \n",
       "1920                 2            2           2       2\n",
       "1930                16           16          16      16\n",
       "1940                15           15          15      15\n",
       "1950                17           17          17      17\n",
       "1960                19           19          19      19\n",
       "1970                51           51          51      51\n",
       "1980               108          108         108     108\n",
       "1990               244          244         244     244\n",
       "2000               144          144         144     144\n",
       "2010                 1            1           1       1"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# how many movies per decade?\n",
    "movies_df.groupby('movie_decade').count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "28ca319b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEHCAYAAABBW1qbAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAWAUlEQVR4nO3dfbRddZ3f8feHB5kqOPIQaAxgUid2hA4PQwbt4CiVKoxdTsABGkYkrXSha4HFVm2JzBpjbaaOo1grPsBUEB8BUca4xoqYcaGiAyTIU4iMUShEMhDELtC2dALf/rH33TmEm5tLcvc5N/e+X2vddff57acvO4fzufu39/ntVBWSJAHsNuoCJEnTh6EgSeoYCpKkjqEgSeoYCpKkzh6jLmBnHHDAATV//vxRlyFJu5Q1a9Y8UlVzxpu3S4fC/PnzWb169ajLkKRdSpL/ua15vXUfJTkkybeTrEuyNsn5bfvyJD9Lclv787qBdZYlWZ/kniQn9lWbJGl8fZ4pbAbeUVW3JtkHWJPk+nbeh6vqg4MLJzkMWAIcDrwQ+FaSl1TVkz3WKEka0NuZQlVtrKpb2+nHgXXAvAlWWQxcWVVPVNW9wHrg2L7qkyQ901DuPkoyHzgauKltOi/JHUkuS7Jv2zYPeGBgtQ1MHCKSpCnWeygk2Rv4MvD2qnoM+ATwYuAoYCPwobFFx1n9GQMzJTknyeokqzdt2tRP0ZI0S/UaCkn2pAmEz1fVVwCq6qGqerKqngL+gi1dRBuAQwZWPxh4cOttVtWlVbWoqhbNmTPuHVWSpB3U591HAT4FrKuqiwba5w4sdgpwVzu9EliSZK8kC4CFwM191SdJeqY+7z46DngTcGeS29q2dwNnJDmKpmvoPuAtAFW1NsnVwN00dy6d651HkjRcvYVCVX2P8a8TfH2CdVYAK/qqSZI0sV36G82Sdj3Lly+fkfuaKRwQT5LUMRQkSR1DQZLUMRQkSR1DQZLUMRQkSR1DQZLUMRQkSR1DQZLUMRQkSR1DQZLUMRQkSR1DQZLUMRQkSR1DQZLUMRQkSR1DQZLUMRQkSR1DQZLUMRQkSR1DQZLUMRQkSR1DQZLUMRQkSR1DQZLUMRQkSR1DQZLUMRQkSR1DQZLUMRQkSR1DQZLUMRQkSR1DQZLU6S0UkhyS5NtJ1iVZm+T8tn2/JNcn+XH7e9+BdZYlWZ/kniQn9lWbJGl8fZ4pbAbeUVUvBV4OnJvkMOACYFVVLQRWta9p5y0BDgdOAj6eZPce65MkbaW3UKiqjVV1azv9OLAOmAcsBq5oF7sCOLmdXgxcWVVPVNW9wHrg2L7qkyQ901CuKSSZDxwN3AQcVFUboQkO4MB2sXnAAwOrbWjbtt7WOUlWJ1m9adOmXuuWpNmm91BIsjfwZeDtVfXYRIuO01bPaKi6tKoWVdWiOXPmTFWZkiR6DoUke9IEwuer6itt80NJ5rbz5wIPt+0bgEMGVj8YeLDP+iRJT9fn3UcBPgWsq6qLBmatBJa200uBrw60L0myV5IFwELg5r7qkyQ90x49bvs44E3AnUlua9veDbwfuDrJ2cD9wGkAVbU2ydXA3TR3Lp1bVU/2WJ8kaSu9hUJVfY/xrxMAnLCNdVYAK/qqSZI0Mb/RLEnqGAqSpI6hIEnqGAqSpI6hIEnqGAqSpI6hIEnqGAqSpI6hIEnqGAqSpI6hIEnqGAqSpI6hIEnqGAqSpI6hIEnqGAqSpI6hIEnqGAqSpI6hIEnqGAqSpI6hIEnqGAqSpI6hIEnqGAqSpI6hIEnqGAqSpI6hIEnqGAqSpI6hIEnqGAqSpI6hIEnqGAqSpI6hIEnq9BYKSS5L8nCSuwbalif5WZLb2p/XDcxblmR9knuSnNhXXZKkbevzTOHTwEnjtH+4qo5qf74OkOQwYAlweLvOx5Ps3mNtkqRx9BYKVfUd4NFJLr4YuLKqnqiqe4H1wLF91SZJGt8orimcl+SOtntp37ZtHvDAwDIb2jZJ0hANOxQ+AbwYOArYCHyobc84y9Z4G0hyTpLVSVZv2rSplyIlabYaaihU1UNV9WRVPQX8BVu6iDYAhwwsejDw4Da2cWlVLaqqRXPmzOm3YEmaZYYaCknmDrw8BRi7M2klsCTJXkkWAAuBm4dZmyQJ9uhrw0m+CBwPHJBkA/Ae4PgkR9F0Dd0HvAWgqtYmuRq4G9gMnFtVT/ZVmyRpfL2FQlWdMU7zpyZYfgWwoq96JEnb11soSNJ0dvWXhnPX++mn7Vo94Q5zIUnqTCoUkqyaTJskadc2YfdRkl8DnktzsXhftnyf4PnAC3uuTZI0ZNu7pvAW4O00AbCGLaHwGPCx/sqSJI3ChKFQVR8BPpLkbVX10SHVJEkakUndfVRVH03yu8D8wXWq6jM91SVJGoFJhUKSz9KMWXQbMPalsgIMBUmaQSb7PYVFwGFVNe4gdZKkmWGy31O4C/iHfRYiSRq9yZ4pHADcneRm4Imxxqr6g16qkiSNxGRDYXmfRUiSpofJ3n10Q9+FSJJGb7J3Hz3OliehPQfYE/hVVT2/r8IkScM32TOFfQZfJzmZLU9NkyTNEDs0SmpV/SXw6qktRZI0apPtPnrDwMvdaL634HcWJGmGmezdR68fmN5M8yjNxVNejSRppCZ7TeFf912IJGn0JvuQnYOTXJvk4SQPJflykoP7Lk6SNFyTvdB8ObCS5rkK84CvtW2SpBlksqEwp6our6rN7c+ngTk91iVJGoHJhsIjSc5Msnv7cybw8z4LkyQN32RD4c3A6cDfARuBUwEvPkvSDDPZW1LfByytql8AJNkP+CBNWEiSZojJnikcMRYIAFX1KHB0PyVJkkZlsqGwW5J9x160ZwqTPcuQJO0iJvvB/iHg+0muoRne4nRgRW9VSZJGYrLfaP5MktU0g+AFeENV3d1rZZKkoZt0F1AbAgaBJM1gXheQZpF1K/56KPt56YWOrL+r2qHnKUiSZiZDQZLUMRQkSZ3eQiHJZe1Q23cNtO2X5PokP25/D373YVmS9UnuSXJiX3VJkratzzOFTwMnbdV2AbCqqhYCq9rXJDkMWAIc3q7z8SS791ibJGkcvYVCVX0HeHSr5sXAFe30FcDJA+1XVtUTVXUvsB44tq/aJEnjG/Y1hYOqaiNA+/vAtn0e8MDAchvatmdIck6S1UlWb9q0qddiJWm2mS4XmjNOW423YFVdWlWLqmrRnDk+50eSptKwQ+GhJHMB2t8Pt+0bgEMGljsYeHDItUnSrDfsUFgJLG2nlwJfHWhfkmSvJAuAhcDNQ65Nkma93oa5SPJF4HjggCQbgPcA7weuTnI2cD9wGkBVrU1yNc3YSpuBc6vqyb5qkySNr7dQqKoztjHrhG0svwKH45akkZouF5olSdOAoSBJ6hgKkqSOoSBJ6hgKkqSOoSBJ6hgKkqSOoSBJ6hgKkqSOoSBJ6hgKkqSOoSBJ6hgKkqSOoSBJ6hgKkqSOoSBJ6hgKkqSOoSBJ6hgKkqSOoSBJ6hgKkqSOoSBJ6hgKkqSOoSBJ6hgKkqSOoSBJ6hgKkqSOoSBJ6hgKkqSOoSBJ6hgKkqSOoSBJ6hgKkqTOHqPYaZL7gMeBJ4HNVbUoyX7AVcB84D7g9Kr6xSjqk6TZapRnCv+sqo6qqkXt6wuAVVW1EFjVvpYkDdF06j5aDFzRTl8BnDy6UiRpdhpJ9xFQwDeTFHBJVV0KHFRVGwGqamOSA8dbMck5wDkAhx566LDqlXbKijNPHdq+LvzcNUPbl2aeUYXCcVX1YPvBf32SH012xTZALgVYtGhR9VWgJM1GI+k+qqoH298PA9cCxwIPJZkL0P5+eBS1SdJsNvRQSPK8JPuMTQOvBe4CVgJL28WWAl8ddm2SNNuNovvoIODaJGP7/0JVfSPJLcDVSc4G7gdOG0FtkjSrDT0UquqnwJHjtP8cOGHY9UiStphOt6RKkkbMUJAkdQwFSVLHUJAkdQwFSVLHUJAkdQwFSVLHUJAkdQwFSVLHUJAkdQwFSVLHUJAkdQwFSVLHUJAkdQwFSVLHUJAkdQwFSVLHUJAkdQwFSVLHUJAkdQwFSVJnj1EXIPXt4nd8bSj7Oe9Drx/KfqQ+GQrqzQ2vfNXQ9vWq79wwtH1JM5mhMMXu/0+/NbR9Hfondw5tX5JmB0Nhhjruo8cNZT83vu3GoexH0nB4oVmS1DEUJEkdQ0GS1DEUJEkdQ0GS1DEUJEkdQ0GS1DEUJEkdQ0GS1Jl2oZDkpCT3JFmf5IJR1yNJs8m0GuYiye7Ax4DXABuAW5KsrKq7J7P+Me/6TJ/lddb8+VlD2Y8kDdu0CgXgWGB9Vf0UIMmVwGJgUqEgSbuSI6+5bmj7uv3UEye1XKqq51ImL8mpwElV9W/a128CXlZV5w0scw5wTvvyHwP37ORuDwAe2cltTIXpUMd0qAGmRx3WsMV0qGM61ADTo46pqOFFVTVnvBnT7Uwh47Q9LbWq6lLg0inbYbK6qhZN1fZ25TqmQw3TpQ5rmF51TIcapksdfdcw3S40bwAOGXh9MPDgiGqRpFlnuoXCLcDCJAuSPAdYAqwccU2SNGtMq+6jqtqc5DzgOmB34LKqWtvzbqesK2onTYc6pkMNMD3qsIYtpkMd06EGmB519FrDtLrQLEkarenWfSRJGiFDQZLUmXGhkOSyJA8nuWug7cgkP0hyZ5KvJXl+2/6aJGva9jVJXj2wzjFt+/ok/y3JeLfLTlUdxya5rf25PckpU1HHs6lhYP6hSX6Z5J0jOhbzk/yfgePxyVEciyRHtPPWtvN/bQTH4o0Dx+G2JE8lOWqYxyLJnkmuaNvXJVk2sM4wj8Vzklzett+e5PipqCPJIUm+3f63rU1yftu+X5Lrk/y4/b3vwDrL2n3dk+TEgfYdquPZ1pBk/3b5Xya5eKtt7dS/CQBVNaN+gFcCvw3cNdB2C/CqdvrNwPva6aOBF7bT/wT42cA6NwP/lOa7E/8D+P0e63gusEc7PRd4eOD1DtfxbGoYmP9l4EvAO0d0LOYPLrfVdoZyLGhuwLgDOLJ9vT+w+7CPxVbr/Rbw0xEciz8Crhx4n94HzB/B++Jc4PJ2+kBgDbDbFByLucBvt9P7AH8LHAZ8ALigbb8A+LN2+jDgdmAvYAHwk519b+xADc8DXgG8Fbh4qv4f6bbxbFfYFX7Y6oMFeIwtF9UPAe4eZ50AP2//secCPxqYdwZwyZDqWAA8RPPBtNN1PJsagJOBPweW04bCsI/F1ssNLD+0YwG8DvhcHzXsxPviT4EVIzgWZwBfa9+P+9N8YO03gvfFx4AzB5ZbRTMszpTUMbD+V2nGXrsHmDtwvO9pp5cBywaWv47mQ3jK6theDQPL/SsGQmGqaphx3UfbcBfwB+30aTz9C3Jj/hD4YVU9Acyj+SLdmA1tW291JHlZkrXAncBbq2pzT3WMW0OS5wH/EXjvVssP/VgAC5L8MMkNSX6vxzq2VcNLgEpyXZJbk/yHHmuYqI5B/xL4Yo91bKuGa4BfARuB+4EPVtWjPdUwUR23A4uT7JFkAXBMO2/K6kgyn6b34CbgoKraCND+PrBdbB7wwDj7m5I6JlnDtkxJDbMlFN4MnJtkDc3p2f8bnJnkcODPgLeMNY2zjam4d3ebdVTVTVV1OPA7wLK2D7uPOrZVw3uBD1fVL7daftjHYiNwaFUdDfx74Attv/Iwj8UeNKfnb2x/n5LkhJ5qmKgOoPmDAfjfVTXW9z7MY3Es8CTwQpqz2Hck+Uc91TBRHZfRfMitBv4r8H1g81TVkWRvmq7Tt1fVYxMtuo397XQdz6KGZ1vbszKtvrzWl6r6EfBagCQvAf7F2LwkBwPXAmdV1U/a5g00Q2yMmZLhNiaqY2CZdUl+RXONY8rrmKCGlwGnJvkA8ALgqST/l+ZNOrRj0Z6pPdFOr0nyE5q/3Id5LDYAN1TVI+28r9P0fX9uqmvYTh1jlrDlLGGsvmEdiz8CvlFVfw88nORGYBHw3amuYaI62jPnfze2XJLvAz8GfrGzdSTZk+Z9/vmq+krb/FCSuVW1McnYdT7Y9lA8O/Vv8ixr2JYpeV/MijOFJAe2v3cD/hj4ZPv6BcBf0fQR3ji2fHuq9niSl7dX78+i6efrq44FSfZop19EM/rrfX3Usa0aqur3qmp+Vc2n+UvsT6vq4hEcizlpnqtB+xfpQpoLrEM7FjT9xEckeW777/Iqmr7toR6LgbbTgCvH2oZ8LO4HXp3G84CX0/RbD/t98dx2/yR5DbC5qnb636Rd51PAuqq6aGDWSmBpO710YJsrgSVJ9mq7sRYCN+9MHTtQw7im7N9kRy/ITNcfmr+oNgJ/T5OcZwPn01wg+1vg/Wy5kPXHNP2ltw38HNjOW0TTv/kT4OKxdXqq403A2nb/twInD2xnh+t4NjVstd5ynn730TCPxR+2x+L29li8fhTHAjizreMu4AOjOBbt8scDfzPOdoZyLIC9ae5GW0vzXJN3jeh9MZ/mwus64Fs0Qz9PxbF4BU0Xyx1s+Qx4Hc1F9VU0ZyOrgP0G1rmw3dc9DNzds6N17GAN9wGPAr9sj91hU/FvUlUOcyFJ2mJWdB9JkibHUJAkdQwFSVLHUJAkdQwFSVLHUJAkdQwFaTuSvDXJWVOwna2HEJlSSe5LckCf+9DMNyuGuZB2RlV9cvtLSTODZwqaUdI8pOdHSf57kruSfD7JP09yY5qHlRyb5uElf5nkjiR/k+aBOru1f2m/YGBb65MclGR52ocOJXlxkm+keSjTd5P85gS1LEjzwJhbkrxvq3nvatvvSPLegfaz2rbbk3y2bXt9kpvSjBz7rSQHte37J/lm234JAwOiJTkzyc1pHs5zydjQIdL2GAqaiX4D+AhwBPCbNIO6vQJ4J/BumhFhf1hVR7SvP1NVT9GME3MKdCOT3ldVD2217UuBt1XVMe32Pj5BHR8BPlFVvwP83VhjktfSjJlzLHAUcEySV6YZrfdC4NVVdSTNkA8A3wNeXs3IsVcCY0N5vwf4Xtu+Eji03f5LaYbaPq6qjqIZ5fSN2z9skt1Hmpnurao7AdI8o2JVVVWSO2nG0HkRzRhLVNVft39x/zpwFfAnwOU0I5NeNbjRNEMb/y7wpWx5yuFeE9Rx3Nh+gM/SDM8OzSigrwV+2L7emyYkjgSuqXZ01mqeWwDNaJdXpRkp8znAvW37K4E3tMv+VZJftO0n0Dxv4Ja2zn/A9kfYlABDQTPTEwPTTw28formPb95nHUK+AHwG0nm0DyF7j9vtcxuwP9q//qerPEGFwvwX6rqkqc1Jv92G8t/FLioqlameTbx8kls/4qqWjbOPGlCdh9pNvoObXdK+yH7SFU9Vs3okNcCF9EMY/zzwZWqefDJvUlOa9dNkiMn2M+NNGcc8PTum+uAN7dnHiSZ1w4ZvQo4Pcn+bft+7fK/DvysnV46sJ3B/47fB8YeLr+K5tkYY8NQ75dmSHZpuwwFzUbLgUVJ7qAZnnnwg/YqmmGzrxpnPWg+hM9OcjvNcNKLJ9jP+TRPEbuF5oMdgKr6JvAF4Adtl9Y1wD5VtRZYAdzQbn9sbP3lNF1W3wUeGdj+e4FXJrmVpjvq/nb7d9MMC//N9r/xeprn90rb5dDZkqSOZwqSpI4XmqWdlORCmkdmDvpSVa0YRT3SzrD7SJLUsftIktQxFCRJHUNBktQxFCRJnf8PU53WxevJIB4AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.countplot(x = \"movie_decade\",\n",
    "             data = movies_df)\n",
    "plt.savefig('../data_visualization/movie_decade_plot.png')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c79baa52",
   "metadata": {},
   "source": [
    "Wide span of movies, from the 1920s to the 2010s, but most of them come from the 1980s to the 2000s."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8ea900fc",
   "metadata": {},
   "source": [
    "## Pickling the data and creating a csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "671439e6",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pickle"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "22ffaaad",
   "metadata": {},
   "outputs": [],
   "source": [
    "# pickle the data to use in other notebooks for further analysis\n",
    "f = open('movies_df.pkl', 'wb')\n",
    "pickle.dump(movies_df, f, -1)\n",
    "f.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "fc7a7df2",
   "metadata": {},
   "outputs": [],
   "source": [
    "movies_df.to_csv('../new_data/movies_df.csv', header=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d0c2b55",
   "metadata": {},
   "source": [
    "## Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4e4eaa3c",
   "metadata": {},
   "source": [
    "There are 617 movies in this corpus. The data were mostly clean, but some extra characters were present in the year data. Most of the movies come from the 1980s-2000s, but the corpus has movies from 1927 to 2010."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}