Importing Oscar Runners.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Brincando com dados: Ganhadores do Oscar Parte 1\n",
    "\n",
    "Olá a todos, esse post será o primeiro de muitos (como diria a minhoca do worms) sobre análise de dados. O objetivo desses posts intitulados “Brincando com os dados” é dar exemplos de como importar, limpar, analisar dados e muito mais.\n",
    "\n",
    "O assunto desse primeiro exemplo vai ser “Ganhadores do Oscar” e o objetivo dele é ser bem simples e didático, mostrar alguns conceitos e exemplos de ferramentas e bibliotecas. \n",
    "\n",
    "A primeira parte será sobre como importar a base de dados. Irei mostrar como utilizar a biblioteca [IMDbPY](http://imdbpy.sourceforge.net) para buscar filmes e o [PyMongo](https://api.mongodb.org/python/current/) para salvar os dados no [MongoDB](https://www.mongodb.org/).\n",
    "\n",
    "De forma que o post fique mais simples em termos de código, estou usando aqui o html gerado pelo próprio IPython Notebook, então os comentários e o código estão entrelaçado.\n",
    "\n",
    "## Preparativos\n",
    "Para ficar mais simples e didático decidimos utilizar o IPython Notebook, para fazer a instalação dele siga o tutorial descrito aqui: http://jupyter.readthedocs.org/en/latest/install.html \n",
    "\n",
    "Para rodar o código ou acompanhar o post pelo IPython Notebook baixe o arquivo: ARQUIVO.PY\n",
    "\n",
    "## Base\n",
    "MongoDB: O MongoDB é um banco de dados não relacional, ele é orientado a documentos e utiliza um formato chamado BSON, bem próximo ao JSON. Ele é bem simples de entender e utilizar.\n",
    "\n",
    "IMDbPY: IMDbPy é uma biblioteca em python para acessar os dados do IMDb.\n",
    "\n",
    "PyMongo: PyMongo é uma biblioteca em python para acessar o MongoDB.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Let’s do this! (Leeroy Jenkins)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from imdb import IMDb\n",
    "import pymongo\n",
    "from pymongo import MongoClient   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "movies = {}\n",
    "movies['2017'] = ['Arrival','Fences', 'Hacksaw Ridge', 'Hell or High Water', 'Hidden Figures', 'La la land', 'Lion', 'Manchester by the sea', 'Moonlight']\n",
    "movies['2016'] = ['The Big Short','Bridge of Spies','Brooklyn','Mad Max: Fury Road','The Martian','The Revenant','Room','Spotlight']\n",
    "movies['2015'] = ['Birdman','American Sniper','Boyhood','The Grand Budapest Hotel','Imitation Game','Selma','The Theory of Everything','Whiplash']\n",
    "movies['2014'] = ['American Hustle','Captain Phillips','Dallas Buyers Club','Gravity','Her (2013)','Nebraska','Philomena','The Wolf of Wall Street', '12 Years a Slave']\n",
    "movies['2013'] = ['Argo','Amour','tt2125435','Django Unchained','tt1707386','Life of Pi','Lincoln','Silver Linings Playbook','Zero Dark Thirty']\n",
    "movies['2012'] = ['The Artist','The Descendants','Extremely Loud & Incredibly Close','The Help','Hugo','Midnight in Paris','Moneyball','The Tree of Life','War Horse']\n",
    "movies['2011'] = ['The King\\'s Speech','127 Hours','Black Swan','The Fighter','Inception','The Kids Are All Right','The Social Network','Toy Story 3','True Grit','Winter\\'s Bone']\n",
    "movies['2010'] = ['The Hurt Locker','Avatar','tt0878804','District 9','An Education (2009)','Inglorious Bastards','Precious','A Serious Man','up','Up in the air (2009)']\n",
    "movies['2009'] = ['Slumdog Millionaire','Benjamin Button','tt0870111','Milk','The Reader']\n",
    "movies['2008'] = ['No Country for Old Men','Atonement','Juno','Michael Clayton','There Will Be Blood']\n",
    "movies['2007'] = ['The Departed','Babel','Letters from Iwo Jima','Little Miss Sunshine (2006)','The Queen (2006)']\n",
    "movies['2006'] = ['Crash','Brokeback Mountain','Capote','Good Night, and Good Luck.','tt0408306']\n",
    "movies['2005'] = ['Million Dollar Baby','The Aviator','Finding Neverland','Ray','Sideways']\n",
    "movies['2004'] = ['The Lord of the Rings: The Return of the King','Lost in Translation','Master and Commander: The Far Side of the World','Mystic River','Seabiscuit']\n",
    "movies['2003'] = ['Chicago','tt0217505','tt0274558','The Lord of the Rings: The Two Towers','The Pianist']\n",
    "movies['2002'] = ['A Beautiful Mind','Gosford Park','tt0247425','The Lord of the Rings: The Fellowship of the Ring','Moulin Rouge!']      \n",
    "movies['2001'] = ['Gladiator','Chocolat','Crouching Tiger, Hidden Dragon','Erin Brockovich','Traffic']          \n",
    "\n",
    "exceptions = {'tt0878804':'The Blind Side','tt1707386':'Les Misérables', 'tt0870111':'Frost/Nixo', 'tt2125435': 'Beasts of the Southern Wild', 'tt0247425': 'In the Bedroom', 'tt0408306':'Munich', 'tt0274558':'The Hours', 'tt0217505':'Gangs of New York'}\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Primeiramente, foi feito o import das bibliotecas que vamos precisar. Logo depois, criamos a lista dos filmes indicados ao Oscar de “Melhor Filme” de 2000 a 2014. \n",
    "\n",
    "É importante salientar, que a melhor forma de utilizar o IMDbPy não é essa. O processo normal dessa biblioteca é importar a base inteira do IMDb para seu MySQL [tutorial](http://imdbpy.sourceforge.net/docs/README.sqldb.txt) , mas como queria demonstrar o MongoDB, decidi fazer dessa maneira.\n",
    "\n",
    "### First, we are going to insert the Oscar Nominees in the MongoDB database\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# initiating the imdb api\n",
    "ia = IMDb()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# starting the client\n",
    "client = MongoClient()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# getting the db for the imdb database\n",
    "db = client.imdb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# getting the movies colletion\n",
    "movie_db = db.movies"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Agora, iniciamos a api do IMDb e o cliente do MongoDB, logo estamos definindo que iremos trabalhar na database chamada ‘imdb’ e a variável movie_db será utilizada para trabalha na coleção ‘movies’. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# creating a function that maps the movie json to a dictionary\n",
    "def map_movie_to_dic(title, movie, year, exceptions):\n",
    "              \n",
    "    movie_dic = {}\n",
    "    movie_dic['_id'] = movie.movieID\n",
    "    movie_dic['oscar_year'] = year\n",
    "    map_person_list(movie,movie_dic,'assistant director')\n",
    "    map_person_list(movie,movie_dic,'director')   \n",
    "    map_person_list(movie,movie_dic,'writer')  \n",
    "    map_person_list(movie,movie_dic,'editor')\n",
    "    map_person_list(movie,movie_dic,'cast') \n",
    "    map_person_list(movie,movie_dic,'editor') \n",
    "    map_person_list(movie,movie_dic,'original music')\n",
    "    map_company_list(movie,movie_dic,'distributors')\n",
    "    map_company_list(movie,movie_dic,'production companies')  \n",
    "    map_company_list(movie,movie_dic,'special effects companies')         \n",
    "    map_person_list(movie,movie_dic,'casting director') \n",
    "    \n",
    "    if title in exceptions.keys():\n",
    "        movie_dic['title'] = exceptions[title]\n",
    "    else:\n",
    "        movie_dic['title'] = title\n",
    "    \n",
    "    print 'Title searched: ' + title + ' - Title in the result: ' + movie['title']\n",
    "        \n",
    "    if 'plot' in movie.data:\n",
    "        movie_dic['plot'] = movie['plot']\n",
    "    if 'rating' in movie.data:    \n",
    "        movie_dic['user_rating'] = movie['rating']\n",
    "    else:\n",
    "        print 'Movie ' + title + ' has no rating\\n' + str(movie.data)        \n",
    "    if 'countries' in movie.data:    \n",
    "        movie_dic['countries'] = movie['countries']\n",
    "    if 'genres'  in movie.data:\n",
    "        movie_dic['genres'] = movie['genres']      \n",
    "    else:\n",
    "        print 'Movie ' + title + ' has no genres\\n' + str(movie.data)\n",
    "    if 'runtime'  in movie.data:\n",
    "        movie_dic['runtime'] = movie['runtime']\n",
    "    if 'languages' in movie.data:\n",
    "        movie_dic['languages'] = movie['languages']  \n",
    "    if 'opening weekend' in movie['business']:\n",
    "        movie_dic['opening_weekend'] = get_money(movie['business']['opening weekend'])\n",
    "    else:\n",
    "        movie_dic['opening_weekend'] = 0  \n",
    "    if 'budget' in movie['business']:\n",
    "        movie_dic['budget'] = get_money(movie['business']['budget'])\n",
    "    else:\n",
    "        movie_dic['budget'] = 0        \n",
    "\n",
    "    return movie_dic"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A função *map_movie_to_dic()* mapeia o resultado da biblioteca IMDbPy (em formato JSON) para um dicionário de python. Mas para isso são necessárias as funções listadas abaixo."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import re\n",
    "# Creating a function that maps the money figure to a integer\n",
    "def get_money(raw):\n",
    "    regex = '\\$([0-9],*)*'\n",
    "    for item in raw:\n",
    "        match = re.match(regex, item)\n",
    "        if match:\n",
    "            # return the first group matched without the $ and commas\n",
    "            return int(match.group(0).strip('$').replace(',',''))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A função *get_money()* utiliza de um regex simples para transformar uma string no formato $100,00 em um inteiro 100."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Creating function that maps a person list. Saving only the name and id\n",
    "def map_person_list(movie,movie_dic,field):\n",
    "    movie_dic[field] = []  \n",
    "    if field in movie.data:\n",
    "        for person in movie[field]:    \n",
    "            subdoc = {}\n",
    "            subdoc['name'] = person['name']\n",
    "            subdoc['id'] = person.personID    \n",
    "            movie_dic[field].append(subdoc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A função *map_person_list()* cria uma lista com dicionários, a idéia é que em um filme pode existir mais de um diretor, escritor e ator. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Creating function that maps a company list. Saving only the name and id\n",
    "def map_company_list(movie,movie_dic,field):\n",
    "    movie_dic[field] = []  \n",
    "    if field in movie.data:\n",
    "        for person in movie[field]:    \n",
    "            subdoc = {}\n",
    "            subdoc['name'] = person['name']\n",
    "            subdoc['id'] = person.companyID    \n",
    "            movie_dic[field].append(subdoc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A função *map_company_list* é muito parecida com a *map_person_list()* mas é utilizada para campos relativos a empresas."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Movie Slumdog Millionaire already in mongodb\n",
      "Movie Benjamin Button already in mongodb\n",
      "Movie tt0870111 already in mongodb\n",
      "Movie Milk already in mongodb\n",
      "Movie The Reader already in mongodb\n",
      "Movie Chicago already in mongodb\n",
      "Movie tt0217505 already in mongodb\n",
      "Movie tt0274558 already in mongodb\n",
      "Title searched: The Lord of the Rings: The Two Towers - Title in the result: The Lord of the Rings: The Two Towers\n",
      "Just saved the movie The Lord of the Rings: The Two Towers into the database\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python2.7/dist-packages/ipykernel/__main__.py:40: DeprecationWarning: save is deprecated. Use insert_one or replace_one instead\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Title searched: The Pianist - Title in the result: The Pianist\n",
      "Just saved the movie The Pianist into the database\n",
      "Title searched: A Beautiful Mind - Title in the result: A Beautiful Mind\n",
      "Just saved the movie A Beautiful Mind into the database\n",
      "Title searched: Gosford Park - Title in the result: Gosford Park\n",
      "Just saved the movie Gosford Park into the database\n",
      "Title searched: tt0247425 - Title in the result: In the Bedroom\n",
      "Just saved the movie tt0247425 into the database\n",
      "Title searched: The Lord of the Rings: The Fellowship of the Ring - Title in the result: The Lord of the Rings: The Fellowship of the Ring\n",
      "Just saved the movie The Lord of the Rings: The Fellowship of the Ring into the database\n",
      "Title searched: Moulin Rouge! - Title in the result: Moulin Rouge!\n",
      "Just saved the movie Moulin Rouge! into the database\n",
      "Title searched: No Country for Old Men - Title in the result: No Country for Old Men\n",
      "Just saved the movie No Country for Old Men into the database\n",
      "Title searched: Atonement - Title in the result: Atonement\n",
      "Just saved the movie Atonement into the database\n",
      "Title searched: Juno - Title in the result: Juno\n",
      "Just saved the movie Juno into the database\n",
      "Title searched: Michael Clayton - Title in the result: Michael Clayton\n",
      "Just saved the movie Michael Clayton into the database\n",
      "Title searched: There Will Be Blood - Title in the result: There Will Be Blood\n",
      "Just saved the movie There Will Be Blood into the database\n",
      "Title searched: Gladiator - Title in the result: Gladiator\n",
      "Just saved the movie Gladiator into the database\n",
      "Title searched: Chocolat - Title in the result: Chocolat\n",
      "Just saved the movie Chocolat into the database\n",
      "Title searched: Crouching Tiger, Hidden Dragon - Title in the result: Wo hu cang long\n",
      "Just saved the movie Crouching Tiger, Hidden Dragon into the database\n",
      "Title searched: Erin Brockovich - Title in the result: Erin Brockovich\n",
      "Just saved the movie Erin Brockovich into the database\n",
      "Title searched: Traffic - Title in the result: Traffic\n",
      "Just saved the movie Traffic into the database\n",
      "Title searched: Crash - Title in the result: Crash\n",
      "Just saved the movie Crash into the database\n",
      "Title searched: Brokeback Mountain - Title in the result: Brokeback Mountain\n",
      "Just saved the movie Brokeback Mountain into the database\n",
      "Title searched: Capote - Title in the result: Capote\n",
      "Just saved the movie Capote into the database\n",
      "Title searched: Good Night, and Good Luck. - Title in the result: Good Night, and Good Luck.\n",
      "Just saved the movie Good Night, and Good Luck. into the database\n",
      "Title searched: tt0408306 - Title in the result: Munich\n",
      "Just saved the movie tt0408306 into the database\n",
      "Title searched: The Departed - Title in the result: The Departed\n",
      "Just saved the movie The Departed into the database\n",
      "Title searched: Babel - Title in the result: Babel\n",
      "Just saved the movie Babel into the database\n",
      "Title searched: Letters from Iwo Jima - Title in the result: Letters from Iwo Jima\n",
      "Just saved the movie Letters from Iwo Jima into the database\n",
      "Title searched: Little Miss Sunshine (2006) - Title in the result: Little Miss Sunshine\n",
      "Just saved the movie Little Miss Sunshine (2006) into the database\n",
      "Title searched: The Queen (2006) - Title in the result: The Queen\n",
      "Just saved the movie The Queen (2006) into the database\n",
      "Title searched: The Lord of the Rings: The Return of the King - Title in the result: The Lord of the Rings: The Return of the King\n",
      "Just saved the movie The Lord of the Rings: The Return of the King into the database\n",
      "Title searched: Lost in Translation - Title in the result: Lost in Translation\n",
      "Just saved the movie Lost in Translation into the database\n",
      "Title searched: Master and Commander: The Far Side of the World - Title in the result: Master and Commander: The Far Side of the World\n",
      "Just saved the movie Master and Commander: The Far Side of the World into the database\n",
      "Title searched: Mystic River - Title in the result: Mystic River\n",
      "Just saved the movie Mystic River into the database\n",
      "Title searched: Seabiscuit - Title in the result: Seabiscuit\n",
      "Just saved the movie Seabiscuit into the database\n",
      "Title searched: Million Dollar Baby - Title in the result: Million Dollar Baby\n",
      "Just saved the movie Million Dollar Baby into the database\n",
      "Title searched: The Aviator - Title in the result: The Aviator\n",
      "Just saved the movie The Aviator into the database\n",
      "Title searched: Finding Neverland - Title in the result: Finding Neverland\n",
      "Just saved the movie Finding Neverland into the database\n",
      "Title searched: Ray - Title in the result: Ray\n",
      "Just saved the movie Ray into the database\n",
      "Title searched: Sideways - Title in the result: Sideways\n",
      "Just saved the movie Sideways into the database\n",
      "Title searched: Birdman - Title in the result: Birdman or (The Unexpected Virtue of Ignorance)\n",
      "Just saved the movie Birdman into the database\n",
      "Title searched: American Sniper - Title in the result: American Sniper\n",
      "Just saved the movie American Sniper into the database\n",
      "Title searched: Boyhood - Title in the result: Boyhood\n",
      "Just saved the movie Boyhood into the database\n",
      "Title searched: The Grand Budapest Hotel - Title in the result: The Grand Budapest Hotel\n",
      "Just saved the movie The Grand Budapest Hotel into the database\n",
      "Title searched: Imitation Game - Title in the result: The Imitation Game\n",
      "Just saved the movie Imitation Game into the database\n",
      "Title searched: Selma - Title in the result: Selma\n",
      "Just saved the movie Selma into the database\n",
      "Title searched: The Theory of Everything - Title in the result: The Theory of Everything\n",
      "Just saved the movie The Theory of Everything into the database\n",
      "Title searched: Whiplash - Title in the result: Whiplash\n",
      "Just saved the movie Whiplash into the database\n",
      "Title searched: American Hustle - Title in the result: American Hustle\n",
      "Just saved the movie American Hustle into the database\n",
      "Title searched: Captain Phillips - Title in the result: Captain Phillips\n",
      "Just saved the movie Captain Phillips into the database\n",
      "Title searched: Dallas Buyers Club - Title in the result: Dallas Buyers Club\n",
      "Just saved the movie Dallas Buyers Club into the database\n",
      "Title searched: Gravity - Title in the result: Gravity\n",
      "Just saved the movie Gravity into the database\n",
      "Title searched: Her (2013) - Title in the result: Her\n",
      "Just saved the movie Her (2013) into the database\n",
      "Title searched: Nebraska - Title in the result: Nebraska\n",
      "Just saved the movie Nebraska into the database\n",
      "Title searched: Philomena - Title in the result: Philomena\n",
      "Just saved the movie Philomena into the database\n",
      "Title searched: The Wolf of Wall Street - Title in the result: The Wolf of Wall Street\n",
      "Just saved the movie The Wolf of Wall Street into the database\n",
      "Title searched: 12 Years a Slave - Title in the result: 12 Years a Slave\n",
      "Just saved the movie 12 Years a Slave into the database\n",
      "Movie Arrival already in mongodb\n",
      "Movie Fences already in mongodb\n",
      "Movie Hacksaw Ridge already in mongodb\n",
      "Movie Hell or High Water already in mongodb\n",
      "Movie Hidden Figures already in mongodb\n",
      "Movie La la land already in mongodb\n",
      "Movie Lion already in mongodb\n",
      "Movie Manchester by the sea already in mongodb\n",
      "Movie Moonlight already in mongodb\n",
      "Title searched: The Big Short - Title in the result: The Big Short\n",
      "Just saved the movie The Big Short into the database\n",
      "Title searched: Bridge of Spies - Title in the result: Bridge of Spies\n",
      "Just saved the movie Bridge of Spies into the database\n",
      "Title searched: Brooklyn - Title in the result: Brooklyn\n",
      "Just saved the movie Brooklyn into the database\n",
      "Title searched: Mad Max: Fury Road - Title in the result: Mad Max: Fury Road\n",
      "Just saved the movie Mad Max: Fury Road into the database\n",
      "Title searched: The Martian - Title in the result: The Martian\n",
      "Just saved the movie The Martian into the database\n",
      "Title searched: The Revenant - Title in the result: The Revenant\n",
      "Just saved the movie The Revenant into the database\n",
      "Title searched: Room - Title in the result: Room\n",
      "Just saved the movie Room into the database\n",
      "Title searched: Spotlight - Title in the result: Spotlight\n",
      "Just saved the movie Spotlight into the database\n",
      "Title searched: The King's Speech - Title in the result: The King's Speech\n",
      "Just saved the movie The King's Speech into the database\n",
      "Title searched: 127 Hours - Title in the result: 127 Hours\n",
      "Just saved the movie 127 Hours into the database\n",
      "Title searched: Black Swan - Title in the result: Black Swan\n",
      "Just saved the movie Black Swan into the database\n",
      "Title searched: The Fighter - Title in the result: The Fighter\n",
      "Just saved the movie The Fighter into the database\n",
      "Title searched: Inception - Title in the result: Inception\n",
      "Just saved the movie Inception into the database\n",
      "Title searched: The Kids Are All Right - Title in the result: The Kids Are All Right\n",
      "Just saved the movie The Kids Are All Right into the database\n",
      "Title searched: The Social Network - Title in the result: The Social Network\n",
      "Just saved the movie The Social Network into the database\n",
      "Title searched: Toy Story 3 - Title in the result: Toy Story 3\n",
      "Just saved the movie Toy Story 3 into the database\n",
      "Title searched: True Grit - Title in the result: True Grit\n",
      "Just saved the movie True Grit into the database\n",
      "Title searched: Winter's Bone - Title in the result: Winter's Bone\n",
      "Just saved the movie Winter's Bone into the database\n",
      "Title searched: The Hurt Locker - Title in the result: The Hurt Locker\n",
      "Just saved the movie The Hurt Locker into the database\n",
      "Title searched: Avatar - Title in the result: Avatar\n",
      "Just saved the movie Avatar into the database\n",
      "Title searched: tt0878804 - Title in the result: The Blind Side\n",
      "Just saved the movie tt0878804 into the database\n",
      "Title searched: District 9 - Title in the result: District 9\n",
      "Just saved the movie District 9 into the database\n",
      "Title searched: An Education (2009) - Title in the result: An Education\n",
      "Just saved the movie An Education (2009) into the database\n",
      "Title searched: Inglorious Bastards - Title in the result: Inglourious Basterds\n",
      "Just saved the movie Inglorious Bastards into the database\n",
      "Title searched: Precious - Title in the result: Precious\n",
      "Just saved the movie Precious into the database\n",
      "Title searched: A Serious Man - Title in the result: A Serious Man\n",
      "Just saved the movie A Serious Man into the database\n",
      "Title searched: up - Title in the result: Up\n",
      "Just saved the movie up into the database\n",
      "Title searched: Up in the air (2009) - Title in the result: Up in the Air\n",
      "Just saved the movie Up in the air (2009) into the database\n",
      "Title searched: Argo - Title in the result: Argo\n",
      "Just saved the movie Argo into the database\n",
      "Title searched: Amour - Title in the result: Amour\n",
      "Just saved the movie Amour into the database\n",
      "Title searched: tt2125435 - Title in the result: Beasts of the Southern Wild\n",
      "Just saved the movie tt2125435 into the database\n",
      "Title searched: Django Unchained - Title in the result: Django Unchained\n",
      "Just saved the movie Django Unchained into the database\n",
      "Title searched: tt1707386 - Title in the result: Les Misérables\n",
      "Just saved the movie tt1707386 into the database\n",
      "Title searched: Life of Pi - Title in the result: Life of Pi\n",
      "Just saved the movie Life of Pi into the database\n",
      "Title searched: Lincoln - Title in the result: Lincoln\n",
      "Just saved the movie Lincoln into the database\n",
      "Title searched: Silver Linings Playbook - Title in the result: Silver Linings Playbook\n",
      "Just saved the movie Silver Linings Playbook into the database\n",
      "Title searched: Zero Dark Thirty - Title in the result: Zero Dark Thirty\n",
      "Just saved the movie Zero Dark Thirty into the database\n",
      "Title searched: The Artist - Title in the result: The Artist\n",
      "Just saved the movie The Artist into the database\n",
      "Title searched: The Descendants - Title in the result: The Descendants\n",
      "Just saved the movie The Descendants into the database\n",
      "Title searched: Extremely Loud & Incredibly Close - Title in the result: Extremely Loud & Incredibly Close\n",
      "Just saved the movie Extremely Loud & Incredibly Close into the database\n",
      "Title searched: The Help - Title in the result: The Help\n",
      "Just saved the movie The Help into the database\n",
      "Title searched: Hugo - Title in the result: Hugo\n",
      "Just saved the movie Hugo into the database\n",
      "Title searched: Midnight in Paris - Title in the result: Midnight in Paris\n",
      "Just saved the movie Midnight in Paris into the database\n",
      "Title searched: Moneyball - Title in the result: Moneyball\n",
      "Just saved the movie Moneyball into the database\n",
      "Title searched: The Tree of Life - Title in the result: The Tree of Life\n",
      "Just saved the movie The Tree of Life into the database\n",
      "Title searched: War Horse - Title in the result: War Horse\n",
      "Just saved the movie War Horse into the database\n"
     ]
    }
   ],
   "source": [
    "import time\n",
    "\n",
    "# iterating in the movies list by year\n",
    "for year in movies:\n",
    "    \n",
    "    #for each movie\n",
    "    for movie in movies[year]:        \n",
    "        # setting the number of retries\n",
    "        for attempt in range(15):        \n",
    "            # Check if this movie is already in the MongoDB database\n",
    "            title = ''\n",
    "            if movie in exceptions.keys():\n",
    "                title = exceptions[movie]\n",
    "            else:\n",
    "                title = movie\n",
    "            \n",
    "            if movie_db.find_one({'title':title}):\n",
    "                print 'Movie '+ movie +' already in mongodb'\n",
    "                break;\n",
    "\n",
    "            # search in the IMDB database by title\n",
    "            try:\n",
    "                imdb_result = ia.search_movie(movie)\n",
    "            except Exception as e:\n",
    "                print 'Oops!  Error getting movie \\n' + str(e) \n",
    "                time.sleep(10)\n",
    "\n",
    "            # if the result is not empty\n",
    "            if len(imdb_result) > 0:\n",
    "                best_match = imdb_result[0]\n",
    "\n",
    "                #update necessary fiels\n",
    "                ia.update(best_match)\n",
    "                ia.update(best_match, 'business')\n",
    "\n",
    "                # saving the movie in the database. \n",
    "                # pymongo is easy, just save the dictionary\n",
    "                # Observation: if the database is going to be big, it is better to use\n",
    "                # short field names in MongoDB, example 'title' turns into 't'\n",
    "                movie_db.save(map_movie_to_dic(movie, best_match, int(year), exceptions))\n",
    "                print 'Just saved the movie ' + movie  +' into the database'\n",
    "                \n",
    "                # sucess will break the attempts while\n",
    "                break\n",
    "\n",
    "            else:\n",
    "                print 'No result for the movie: ' + movie +'. Trying again after 30 seconds.'\n",
    "                ia = IMDb()\n",
    "                time.sleep(30)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Importando os filmes\n",
    "O código acima itera nos anos selecionados, para cada ano ele pega a lista de filmes. \n",
    "\n",
    "Cada filme é requisitado no banco de dados pelo seu título( *movie_db.find_one({'title':title})* ), se não for encontrado então iremos na api do IMDb (usando a função *search_movie* ).\n",
    "\n",
    "Como a api não é muito confiável/estável coloquei um *for* com 15 tentativas por filme. \n",
    "\n",
    "A função *ia.update()* é utilizada para buscar mais campos de cada filme.\n",
    "\n",
    "A api retorna um JSON representando o filme, utilizamos então a função *map_movie_to_dic()* para transformar ela em um dicionário de python. Esse dicionário pode ser salvo diretamente na coleção 'movies' utilizando a função *save* do pyMongo(sim, é simples assim).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Fazendo atualização dos ganhadores do Oscar"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# first we are going to create a list with all oscar winners.\n",
    "# I could not find this information in the IMDB api\n",
    "oscar_winners = ['Gladiator', 'A Beautiful Mind','Chicago', 'The Lord of the Rings: The Return of the King', \n",
    "                 'Million Dollar Baby', 'Crash', 'The Departed', 'No Country for Old Men', 'Slumdog Millionaire',\n",
    "                 'The Hurt Locker', 'The King\\'s Speech','The Artist', 'Argo', '12 Years a Slave', 'Birdman', 'Spotlight']\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Primeiramente, listaremos aqui todos ganhadores do Oscar de melhor filme de 2000 a 2016"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The list to be updated has 16 movies, the query returned 16\n",
      "Spotlight\n",
      "Birdman\n",
      "12 Years a Slave\n",
      "Argo\n",
      "The Artist\n",
      "The King's Speech\n",
      "The Hurt Locker\n",
      "Slumdog Millionaire\n",
      "No Country for Old Men\n",
      "The Departed\n",
      "Crash\n",
      "Million Dollar Baby\n",
      "The Lord of the Rings: The Return of the King\n",
      "Chicago\n",
      "A Beautiful Mind\n",
      "Gladiator\n"
     ]
    }
   ],
   "source": [
    "# Now we are going to iterate trough this list updating a new field\n",
    "# with mongodb you do not have a schema, so we can update a non-existent field. It will create the field.\n",
    "\n",
    "db.movies.update_many({},{\"$set\": { \"winner\": False}})\n",
    "    \n",
    "for winner in oscar_winners:\n",
    "    # the update with pymongo is simple, is json-like operation\n",
    "    # .update_one({QUERY}, {UPDATE})  https://docs.mongodb.org/getting-started/python/update/\n",
    "    db.movies.update_one({'title': winner},{\"$set\": { \"winner\": True}})\n",
    "\n",
    "# Now we can check if everything went smoothly, lets query for the oscar winners\n",
    "# we are sorting by the oscar year (descending)\n",
    "cursor = movie_db.find({\"winner\": True}).sort('oscar_year', pymongo.DESCENDING)\n",
    "print 'The list to be updated has ' + str(len(oscar_winners)) + ' movies, the query returned '+ str(cursor.count())\n",
    "for document in cursor:\n",
    "    print(document['title'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Agora iteramos pela lista de ganhadores e realizamos um update no documento adicionando um novo campo chamado \"winner\", esse campo será *True* se esse filme ganhou o oscar e não existirá se ele não ganhou.\n",
    "\n",
    "O update no pymongo é simples: o primeiro parâmetro é a consulta que será realizada, no nosso caso, queremos o filme que o campo 'title' seja igual ao título do filme que ganhou o oscar. O segundo parâmetro é a operação de atualização que será realizada nos documentos que foram buscados pela consulta, nesse caso, estamos usando o operador de *$set* para atualizar/criar um campo chamado 'winner' e colocando o valor True para ele.\n",
    "\n",
    "Depois disso, fazemos uma consulta por todos os filmes que tem o campo *winner* igual a True ordenando por data(invertido). Conferimos então o total da lista buscada pelo *find* com a lista inicial."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The nominees for best picture in 2016 were:\n",
      "\n",
      "The Big Short\n",
      "Bridge of Spies\n",
      "Brooklyn\n",
      "Mad Max: Fury Road\n",
      "The Martian\n",
      "The Revenant\n",
      "Room\n",
      "Spotlight\n",
      "\n",
      "The Winner from 2016 was: \n",
      "Spotlight\n"
     ]
    }
   ],
   "source": [
    "# Now lets list all movies from 2013 and then that oscar winner from that year\n",
    "cursor = movie_db.find({'oscar_year':2016})\n",
    "print('The nominees for best picture in 2016 were:\\n')\n",
    "for document in cursor:\n",
    "    print(document['title'])\n",
    "    \n",
    "result = movie_db.find_one({'oscar_year':2016,'winner':True})    \n",
    "print('\\nThe Winner from 2016 was: \\n'+ result['title'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Por último, vamos listar todos os filmes de 2016 seguido pelo ganhador do mesmo.\n",
    "\n",
    "\n",
    "Finalizando esse código temos um banco de dados no Mongodb chamado **\"imdb\"** com uma coleção **\"movies\"** com todos os filmes indicados ao Oscar de melhor de 2000 a 2016.\n",
    "\n",
    "No próximo post iremos utilizar esses filmes para analisar os dados e tentar descobrir peculiaridades e padrões entre os filmes ganhadores.\n",
    "\n",
    "Abraços e até o próximo"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Remover esse codigo antes de enviar"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Add the Rotten Tomatoes score for each movie.\n",
      "\n",
      "\n",
      "\n",
      "Amour\n",
      "Score for critics: 93\n",
      "Score for people: 82\n",
      "\n",
      "\n",
      "Beasts of the Southern Wild\n",
      "Score for critics: 86\n",
      "Score for people: 76\n",
      "\n",
      "\n",
      "Django Unchained\n",
      "Score for critics: 88\n",
      "Score for people: 91\n",
      "\n",
      "\n",
      "Les Misérables\n",
      "Score for critics: 69\n",
      "Score for people: 79\n",
      "\n",
      "\n",
      "Life of Pi\n",
      "Score for critics: 87\n",
      "Score for people: 94\n",
      "\n",
      "\n",
      "Lincoln\n",
      "Score for critics: 90\n",
      "Score for people: 80\n",
      "\n",
      "\n",
      "Silver Linings Playbook\n",
      "Score for critics: 92\n",
      "Score for people: 86\n",
      "\n",
      "\n",
      "Zero Dark Thirty\n",
      "Score for critics: 92\n",
      "Score for people: 80\n",
      "\n",
      "\n",
      "The Artist\n",
      "Score for critics: 96\n",
      "Score for people: 87\n",
      "\n",
      "\n",
      "The Descendants\n",
      "Score for critics: 89\n",
      "Score for people: 79\n",
      "\n",
      "\n",
      "Extremely Loud & Incredibly Close\n",
      "Score for critics: 46\n",
      "Score for people: 61\n",
      "\n",
      "\n",
      "The Help\n",
      "Score for critics: 75\n",
      "Score for people: 89\n",
      "\n",
      "\n",
      "Hugo\n",
      "Score for critics: 94\n",
      "Score for people: 78\n",
      "\n",
      "\n",
      "Midnight in Paris\n",
      "Score for critics: 93\n",
      "Score for people: 83\n",
      "\n",
      "\n",
      "Moneyball\n",
      "Score for critics: 94\n",
      "Score for people: 86\n",
      "\n",
      "\n",
      "The Tree of Life\n",
      "Score for critics: 84\n",
      "Score for people: 60\n",
      "\n",
      "\n",
      "War Horse\n",
      "Score for critics: 76\n",
      "Score for people: 74\n"
     ]
    }
   ],
   "source": [
    "\n",
    "cursor = movie_db.find({\"rotten_critic\":{\"$exists\":False}})\n",
    "print('Add the Rotten Tomatoes score for each movie.\\n')\n",
    "for document in cursor:\n",
    "    print('\\n')\n",
    "    print(document['title'])\n",
    "    critic = raw_input('Score for critics: ')\n",
    "    people = raw_input('Score for people: ')\n",
    "    \n",
    "    db.movies.update_one({'title': document['title']},{\"$set\": { \"rotten_critic\": critic, \"rotten_people\": people}})    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}