GetData.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Get Reddit Data\n",
    "This notebook is used to pull and clean the reddit comments that will be used for training the various models.\n",
    "\n",
    "\n",
    "We will need praw, the Python Reddit API Wrapper, nltk, and Keras (which needs tensorflow) for data collection and cleaning. As the libraries not included in conda, here are the four installs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install praw\n",
    "!pip install nltk\n",
    "!pip install tensorflow\n",
    "!pip install keras"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# constants\n",
    "THRESHOLD = 100000"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import praw\n",
    "import nltk\n",
    "nltk.download('vader_lexicon')\n",
    "import pickle\n",
    "import numpy as np\n",
    "\n",
    "from keras.preprocessing import sequence\n",
    "from keras.preprocessing.text import Tokenizer\n",
    "\n",
    "from nltk.sentiment import SentimentIntensityAnalyzer\n",
    "from get_data import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Establishing reddit instance and scraping comments\n",
    "Having installed and imported all the neceassary libraries, we request THRESHOLD comments and their parents from reddit. The data is stored in 6 pickle files. We have,\n",
    "\n",
    "    comment_texts               ====>        PLaintext of each comment requested from reddit\n",
    "    comment_sentiment_scores    ====>        list of the scores returned by vader sentiment analyzer for each comment\n",
    "    comment_upvotes             ====>        The number of upvotes for each comment\n",
    "    parent_texts                ====>        Plaintext of the parent from each parent child pair\n",
    "    parent_sentiment_scores     ====>        list of the scores returned by vader sentiment analyzer for each parent\n",
    "    children_texts              ====>        Plaintext of the parent from each parent child pair"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "reddit = praw.Reddit(client_id = \"\",\n",
    "                     client_secret = \"\",\n",
    "                     user_agent='Python: Comment Scraper: v0.1(by /u/josmfred)')\n",
    "\n",
    "comment_texts, comment_sentiment_scores, comment_upvotes = [], [], []\n",
    "parent_texts, parent_sentiment_scores, children_texts = [], [], []\n",
    "\n",
    "while len(comment_texts) < THRESHOLD:\n",
    "    score_predict_data, word_predict_data = (\n",
    "            get_comments_and_parents(get_random_submission(\"ProgrammerHumor\", reddit))\n",
    "    )\n",
    "    comment_texts.extend(score_predict_data[0])\n",
    "    comment_sentiment_scores.extend(score_predict_data[1])\n",
    "    comment_upvotes.extend(score_predict_data[2])\n",
    "    parent_texts.extend(word_predict_data[0])\n",
    "    parent_sentiment_scores.extend(word_predict_data[1])\n",
    "    children_texts.extend(word_predict_data[2])\n",
    "\n",
    "pickle.dump(comment_texts, open(\"preclean/comment_texts.pkl\", \"wb\"))\n",
    "pickle.dump(comment_sentiment_scores, open(\"preclean/comment_sentiment_scores.pkl\", \"wb\"))\n",
    "pickle.dump(comment_upvotes, open(\"preclean/comment_upvotes.pkl\", \"wb\"))\n",
    "pickle.dump(parent_texts, open(\"preclean/parent_texts.pkl\", \"wb\"))\n",
    "pickle.dump(parent_sentiment_scores, open(\"preclean/parents_sentiment_scores.pkl\", \"wb\"))\n",
    "pickle.dump(children_texts, open(\"preclean/children_texts.pkl\", \"wb\"))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Cleaning\n",
    "Having finished aquiring and saving the data, we will begin cleaning the data under the assumption that the data is saved in the preclean directory. The following portion of the notebook does not require any of the previous cells to have been run. All they require is that the correct pickles are in the preclean directory. So, first we load all the data from the correct location, and the tokenizer, if it exists. We then run the cleaning function in get_data.py to prepare the data for the learn notebook. We are then left with the data that is saved stored as follows:\n",
    "\n",
    "    parent_texts               ====>        Padded, tokenized on words vectors of each parent comment\n",
    "    parent_sentiment_scores    ====>        Numpy array of the vader sentiment scores for parent comments\n",
    "    child_first_word           ====>        The tokenized first word in each child comment\n",
    "    comment_texts              ====>        Padded, tokenized on words vectors of each comment\n",
    "    comment_sentiment_scores   ====>        Numpy array of the vader sentiment scores for every comment\n",
    "    comment_upvotes            ====>        The number of upvotes for every comment\n",
    "    words                      ====>        The words associated to each token in order of token\n",
    "    tokenizer                  ====>        The tokenizer to reuse."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "comment_texts = pickle.load(open(\"preclean/comment_texts.pkl\", \"rb\"))\n",
    "sentiment_scores = pickle.load(open(\"preclean/comment_sentiment_scores.pkl\", \"rb\"))\n",
    "comment_upvotes = pickle.load(open(\"preclean/comment_upvotes.pkl\", \"rb\"))\n",
    "parent_texts = pickle.load(open(\"preclean/parent_texts.pkl\", \"rb\"))\n",
    "parent_sentiment_scores = pickle.load(open(\"preclean/parents_sentiment_scores.pkl\", \"rb\"))\n",
    "children_texts = pickle.load(open(\"preclean/children_texts.pkl\", \"rb\"))\n",
    "# The tokenizer might not exist. If the tokenizer does not exist, then\n",
    "# we assume that the data we are processing should have be used\n",
    "# to fit a new tokenizer, and then this tokenizer is saved to use in later\n",
    "# data processing. If the tokenizer does exist, we use the existing\n",
    "# tokenizer on the new data.\n",
    "try:\n",
    "    tokenizer = pickle.load(open(\"tokenizer.pkl\", \"rb\"))\n",
    "except:\n",
    "    tokenizer = None\n",
    "\n",
    "pad_parents, parent_scores, first_word, tokenizer =  prepare_next_word_data(parent_texts,\n",
    "                                                                            parent_sentiment_scores,\n",
    "                                                                            children_texts,\n",
    "                                                                            comment_texts,\n",
    "                                                                            tokenizer=tokenizer)\n",
    "texts_pad, sentiment_scores, upvotes, tokenizer = prepare_score_data(comment_texts,\n",
    "                                                                     comment_sentiment_scores,\n",
    "                                                                     comment_upvotes,\n",
    "                                                                     150,\n",
    "                                                                     tokenizer=tokenizer)\n",
    "\n",
    "# Save all the processed data, the word -> index dictionary of th\n",
    "# tokenizer, and the tokenizer itself.\n",
    "np.save(\"cleaned/parent_texts.npy\", pad_parents)\n",
    "np.save(\"cleaned/parent_sentiment_scores.npy\", parent_scores)\n",
    "np.save(\"cleaned/child_first_word.npy\", first_word)\n",
    "np.save(\"cleaned/comment_texts.npy\", texts_pad)\n",
    "np.save(\"cleaned/comment_sentiment_scores.npy\", sentiment_scores)\n",
    "np.save(\"cleaned/comment_upvotes.npy\", upvotes)\n",
    "pickle.dump(tokenizer.word_index, open(\"cleaned/words.pkl\", \"wb\"))\n",
    "pickle.dump(tokenizer, open(\"tokenizer.pkl\", \"wb\"))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}