notebooks/data_extraction/process_reddit_data.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Reddit Dataset - Extraction\n",
    "\n",
    "https://nbviewer.org/github/Data-Science-for-Linguists-2023/Text-Based-Age-Recognition/blob/main/notebooks/data_extraction/process_reddit_data.ipynb \n",
    "\n",
    "This notebook contains the code that extracts data that is generated by the scripts in `/scripts/`. Since the data is raw, there is some EDA being done here to make sure we get a nice curated dataset which is fairly balanced."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Cleaning"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Function to remove links and escape characters.\n",
    "\n",
    "Since punctuation, whitespace, and capitalization remain relevant, I am leaving in those features for now."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "import string\n",
    "\n",
    "def preprocess_text(text):\n",
    "    # Remove URLs\n",
    "    text = str(text)\n",
    "    text = re.sub(r'http\\S+', '', text)\n",
    "    # Remove escape characters\n",
    "    text = text.replace('\\n', ' ')\n",
    "    text = text.replace('\\t', ' ')        \n",
    "    return text"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I'll explain this down below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "def remove_mod_messages(texts):\n",
    "    return [text for text in texts if 'mod' not in text.lower().split() and 'moderator' not in text.lower().split()]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Loading"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Teens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>comments</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>Fax.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>What happened to my comment....it was soo good...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>A shit ton of censorship. And I don't mean \"de...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>Wasn't aware of the drama between /r/askmen an...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>Nice username\\n\\nI too am from Finland</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14896</th>\n",
       "      <td>14896</td>\n",
       "      <td>Not gay if youre a girl</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14897</th>\n",
       "      <td>14897</td>\n",
       "      <td>Masturbate and play with my temporary boobs</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14898</th>\n",
       "      <td>14898</td>\n",
       "      <td>And rape him first</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14899</th>\n",
       "      <td>14899</td>\n",
       "      <td>Prostitution.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14900</th>\n",
       "      <td>14900</td>\n",
       "      <td>Please go fuck yourself</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>14901 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       Unnamed: 0                                           comments\n",
       "0               0                                               Fax.\n",
       "1               1  What happened to my comment....it was soo good...\n",
       "2               2  A shit ton of censorship. And I don't mean \"de...\n",
       "3               3  Wasn't aware of the drama between /r/askmen an...\n",
       "4               4             Nice username\\n\\nI too am from Finland\n",
       "...           ...                                                ...\n",
       "14896       14896                            Not gay if youre a girl\n",
       "14897       14897        Masturbate and play with my temporary boobs\n",
       "14898       14898                                 And rape him first\n",
       "14899       14899                                      Prostitution.\n",
       "14900       14900                            Please go fuck yourself\n",
       "\n",
       "[14901 rows x 2 columns]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "filepath = '../../reddit_data/AskTeenBoys_comments.csv'\n",
    "df = pd.read_csv(filepath)\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "teens_1 = df['comments'].apply(preprocess_text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"#MOD MESSAGE    If anyone can help us get in touch with reddit admins, that would be great. To help us counter this situation.           If you see any pedos around here or on r/AskTeenGirls, report them to us AND directly to Reddit.          Edit: everyone who's confused, I'm a mod from ATG.\""
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "teens_1[90]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It looks like we randomly stumbled onto a mod in a teen subreddit whose text will probably get assigned the 'teen' label. It looks like an older person wrote it to me, but I could be wrong.\n",
    "\n",
    "I can check for 'mod' and remove text, but it will remove all text that mentions a mod, not just the moderator's messages.\n",
    "\n",
    "Either way, this is a good example of noise in a dataset. Hence the function (also repeated below)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def remove_mod_messages(texts):\n",
    "    return [text for text in texts if 'mod' not in text.lower().split() and 'moderator' not in text.lower().split()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "teens_1 = remove_mod_messages(teens_1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Fax.',\n",
       " 'What happened to my comment....it was soo good....  so ima just repeat myself  U callin yer selfs ***D O G S*** eh?',\n",
       " 'A shit ton of censorship. And I don\\'t mean \"dem libtards takin muh free spheech\" kind of censorship. Basically if you disagree with a commenter, and reply stating a counter, your comments will be removed. Or even factual fucking errors are counted as \"invalidating\".  Extreme example:  \"2+3 is 7 and you just got to accept that\"  \"It\\'s actually 5 but i get what you\\'re saying\"  \"Your comment was removed for Invalidation.',\n",
       " \"Wasn't aware of the drama between /r/askmen and /r/AskWomen, and yet hearing about it now is completely unsurprising.\",\n",
       " 'Nice username  I too am from Finland']"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "teens_1[0:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'#Come again nigga?'"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "teens_1[90]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " looks like that data point we had before is now gone (this data is crazy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>comments</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>you're damn right i gave you a kiss</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>how I missed you my comment</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>So are we married now? jk jk... unless😳</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>what if 😳😳😳</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>***I missed you too baby***</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21296</th>\n",
       "      <td>21296</td>\n",
       "      <td>1) Legend\\n\\n2) In Italy we don’t have dress c...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21297</th>\n",
       "      <td>21297</td>\n",
       "      <td>i left during lunch w a few friends and caught...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21298</th>\n",
       "      <td>21298</td>\n",
       "      <td>I used to skip one of my classes. Then I’d lie...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21299</th>\n",
       "      <td>21299</td>\n",
       "      <td>Yea but I didn't want to write an essay</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21300</th>\n",
       "      <td>21300</td>\n",
       "      <td>[deleted]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>21301 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       Unnamed: 0                                           comments\n",
       "0               0                you're damn right i gave you a kiss\n",
       "1               1                        how I missed you my comment\n",
       "2               2            So are we married now? jk jk... unless😳\n",
       "3               3                                        what if 😳😳😳\n",
       "4               4                        ***I missed you too baby***\n",
       "...           ...                                                ...\n",
       "21296       21296  1) Legend\\n\\n2) In Italy we don’t have dress c...\n",
       "21297       21297  i left during lunch w a few friends and caught...\n",
       "21298       21298  I used to skip one of my classes. Then I’d lie...\n",
       "21299       21299            Yea but I didn't want to write an essay\n",
       "21300       21300                                          [deleted]\n",
       "\n",
       "[21301 rows x 2 columns]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "filepath = '../../reddit_data/AskTeenGirls_comments.csv'\n",
    "df = pd.read_csv(filepath)\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "teens_2 = [preprocess_text(item) for item in list(df['comments'])]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "teens_2 = remove_mod_messages(teens_2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[\"you're damn right i gave you a kiss\",\n",
       " 'how I missed you my comment',\n",
       " 'So are we married now? jk jk... unless😳',\n",
       " 'what if 😳😳😳',\n",
       " '***I missed you too baby***']"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "teens_2[0:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "teens = teens_1.extend(teens_2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "36119"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(teens_1)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is now our teen data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "teens = teens_1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "36119"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(teens)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Twenties"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>comments</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>I honestly teared up a bit reading this letter...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>Yup. There's almost nothing better than a prof...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>My mom died Nov 11, just a few weeks ago. I am...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>That’s so touching that your professor cares s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>I fucking can’t. My mom died two years ago fro...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14696</th>\n",
       "      <td>14696</td>\n",
       "      <td>yes! with the past year being work from home a...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14697</th>\n",
       "      <td>14697</td>\n",
       "      <td>RIP for whichever lab hires me after grad lol</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14698</th>\n",
       "      <td>14698</td>\n",
       "      <td>Easier, but so boring</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14699</th>\n",
       "      <td>14699</td>\n",
       "      <td>I love online classes too, it’s easier to lear...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14700</th>\n",
       "      <td>14700</td>\n",
       "      <td>Since my major is graphic design absolutely</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>14701 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       Unnamed: 0                                           comments\n",
       "0               0  I honestly teared up a bit reading this letter...\n",
       "1               1  Yup. There's almost nothing better than a prof...\n",
       "2               2  My mom died Nov 11, just a few weeks ago. I am...\n",
       "3               3  That’s so touching that your professor cares s...\n",
       "4               4  I fucking can’t. My mom died two years ago fro...\n",
       "...           ...                                                ...\n",
       "14696       14696  yes! with the past year being work from home a...\n",
       "14697       14697      RIP for whichever lab hires me after grad lol\n",
       "14698       14698                              Easier, but so boring\n",
       "14699       14699  I love online classes too, it’s easier to lear...\n",
       "14700       14700        Since my major is graphic design absolutely\n",
       "\n",
       "[14701 rows x 2 columns]"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "filepath = '../../reddit_data/college_comments.csv'\n",
    "df = pd.read_csv(filepath)\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "14700"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "college = [preprocess_text(item) for item in list(df['comments'])]\n",
    "college = remove_mod_messages(college)\n",
    "len(college)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>comments</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>the agricultural revolution was a mistake</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>Why do people get degrees like engineering and...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>Instead of staring at the one 5 steps from my ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>I came to this subreddit for advice and direct...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>Exactly, sitting 8 hours behind a desk 5 days ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10596</th>\n",
       "      <td>10596</td>\n",
       "      <td>thanks so much! it’s sad but your right about ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10597</th>\n",
       "      <td>10597</td>\n",
       "      <td>Abbreviations mean Child Protective Services a...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10598</th>\n",
       "      <td>10598</td>\n",
       "      <td>As the other commenter said, I'd consider grad...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10599</th>\n",
       "      <td>10599</td>\n",
       "      <td>thanks for the tips!</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10600</th>\n",
       "      <td>10600</td>\n",
       "      <td>There’s usually plenty of jobs for Bachs in ps...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>10601 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       Unnamed: 0                                           comments\n",
       "0               0          the agricultural revolution was a mistake\n",
       "1               1  Why do people get degrees like engineering and...\n",
       "2               2  Instead of staring at the one 5 steps from my ...\n",
       "3               3  I came to this subreddit for advice and direct...\n",
       "4               4  Exactly, sitting 8 hours behind a desk 5 days ...\n",
       "...           ...                                                ...\n",
       "10596       10596  thanks so much! it’s sad but your right about ...\n",
       "10597       10597  Abbreviations mean Child Protective Services a...\n",
       "10598       10598  As the other commenter said, I'd consider grad...\n",
       "10599       10599                               thanks for the tips!\n",
       "10600       10600  There’s usually plenty of jobs for Bachs in ps...\n",
       "\n",
       "[10601 rows x 2 columns]"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "filepath = '../../reddit_data/LifeAfterSchool_comments.csv'\n",
    "df = pd.read_csv(filepath)\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "10600"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "life_after_school = [preprocess_text(item) for item in list(df['comments'])]\n",
    "life_after_school = remove_mod_messages(life_after_school)\n",
    "len(life_after_school)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "college.extend(life_after_school)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "25300"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(college)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is now our twenties data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "twenties = college"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Length based Filtering for Teens and Twenties"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I hypothesize that these subreddits by nature have shorter or longer comments. Let's look further.\n",
    "\n",
    "The function below counts words, and we use this to get average comment word count per data group."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "def word_counts(text_list):\n",
    "    return [len(text.split()) for text in text_list]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "45.894268774703555"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sum(word_counts(twenties)) / len(word_counts(twenties))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "19.068578864309643"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sum(word_counts(teens)) / len(word_counts(teens))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These are drastically different, and I'm not too concerned about length of text denoting age -- it could be subreddit specific, and it's an easy differentiator which tells in an obvious way where the data came from. Let's remove the data shorter than the first quartile of comments when arranged by length of words in the text.\n",
    "\n",
    "This should hopefully make the dataset more difficult and interesting to work with."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "teens_words = [text.split() for text in teens]\n",
    "teens_word_counts = [len(words) for words in teens_words]\n",
    "q1 = np.percentile(teens_word_counts, 25)\n",
    "teens_long_comments = [text for i, text in enumerate(teens) if teens_word_counts[i] >= q1]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "23.6631641623761"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sum(word_counts(teens_long_comments)) / len(word_counts(teens_long_comments))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "28551"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(teens_long_comments)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we're up to 23 words per comment, from 19.\n",
    "\n",
    "Let's also remove text longer than the third quartile in the `twenties` data to make the word length closer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "twenties_words = [text.split() for text in twenties]\n",
    "twenties_word_counts = [len(words) for words in twenties_words]\n",
    "q3 = np.percentile(twenties_word_counts, 75)\n",
    "twenties_short_comments = [text for i, text in enumerate(twenties) if twenties_word_counts[i] < q3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "20.002953898090517"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sum(word_counts(twenties_short_comments)) / len(word_counts(twenties_short_comments))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looks like the upper data points are really long, pulling up the average. Lets check how much data we have here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "18958"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(twenties_short_comments)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looks like there is a pretty big difference in data size -- I'll leave it for now, and stratify when I make my test set. I might come back and change it.\n",
    "\n",
    "Moving onto the older age group."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Thirties"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>comments</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>Definitely the best side of reddit in that pos...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>Yessss this is the glow-up we want to see!</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>I remember reading your original post and bein...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>I find it so satisfying when they come groveli...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>Very happy for you! Funny that the greener pas...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14796</th>\n",
       "      <td>14796</td>\n",
       "      <td>Yes. Thank you for this response. I don’t view...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14797</th>\n",
       "      <td>14797</td>\n",
       "      <td>Better hope that you're contacted before someo...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14798</th>\n",
       "      <td>14798</td>\n",
       "      <td>Thank you for this question. I also find mysel...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14799</th>\n",
       "      <td>14799</td>\n",
       "      <td>Yes. Being able to consider working part-time ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14800</th>\n",
       "      <td>14800</td>\n",
       "      <td>Tell that to hundreds of gay couples in the 70...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>14801 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       Unnamed: 0                                           comments\n",
       "0               0  Definitely the best side of reddit in that pos...\n",
       "1               1         Yessss this is the glow-up we want to see!\n",
       "2               2  I remember reading your original post and bein...\n",
       "3               3  I find it so satisfying when they come groveli...\n",
       "4               4  Very happy for you! Funny that the greener pas...\n",
       "...           ...                                                ...\n",
       "14796       14796  Yes. Thank you for this response. I don’t view...\n",
       "14797       14797  Better hope that you're contacted before someo...\n",
       "14798       14798  Thank you for this question. I also find mysel...\n",
       "14799       14799  Yes. Being able to consider working part-time ...\n",
       "14800       14800  Tell that to hundreds of gay couples in the 70...\n",
       "\n",
       "[14801 rows x 2 columns]"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "filepath = '../../reddit_data/AskWomenOver30_comments.csv'\n",
    "df = pd.read_csv(filepath)\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "14794"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "women_over_thirty = [preprocess_text(item) for item in list(df['comments'])]\n",
    "women_over_thirty = remove_mod_messages(women_over_thirty)\n",
    "len(women_over_thirty)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>comments</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>It should never be \"You vs. Me\" but rather \"Us...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>To build off this...my Father mentioned once.....</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>If you’re arguing to win you’ve already lost.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>Just recently I heard “if he loves you, you wi...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>\"Hey, can you please shut the fuck up?\"\\n\\nI'm...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14196</th>\n",
       "      <td>14196</td>\n",
       "      <td>Washing your balls primarily</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14197</th>\n",
       "      <td>14197</td>\n",
       "      <td>Yeah. I was born in 84 and really saw the adve...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14198</th>\n",
       "      <td>14198</td>\n",
       "      <td>I was 31, I was massively overweight, smoked 2...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14199</th>\n",
       "      <td>14199</td>\n",
       "      <td>I think the decline started in my early 40s, b...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14200</th>\n",
       "      <td>14200</td>\n",
       "      <td>Exercise, sleep, and nutrition have always bee...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>14201 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       Unnamed: 0                                           comments\n",
       "0               0  It should never be \"You vs. Me\" but rather \"Us...\n",
       "1               1  To build off this...my Father mentioned once.....\n",
       "2               2     If you’re arguing to win you’ve already lost. \n",
       "3               3  Just recently I heard “if he loves you, you wi...\n",
       "4               4  \"Hey, can you please shut the fuck up?\"\\n\\nI'm...\n",
       "...           ...                                                ...\n",
       "14196       14196                       Washing your balls primarily\n",
       "14197       14197  Yeah. I was born in 84 and really saw the adve...\n",
       "14198       14198  I was 31, I was massively overweight, smoked 2...\n",
       "14199       14199  I think the decline started in my early 40s, b...\n",
       "14200       14200  Exercise, sleep, and nutrition have always bee...\n",
       "\n",
       "[14201 rows x 2 columns]"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "filepath = '../../reddit_data/AskMenOver30_comments.csv'\n",
    "df = pd.read_csv(filepath)\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "14196"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "men_over_thirty = [preprocess_text(item) for item in list(df['comments'])]\n",
    "men_over_thirty = remove_mod_messages(men_over_thirty)\n",
    "len(men_over_thirty)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "men_over_thirty.extend(women_over_thirty)\n",
    "thirties = men_over_thirty"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's check the average word length."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "63.30131079682649"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sum(word_counts(thirties)) / len(word_counts(thirties))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wow, super high. Definitely some wise people here.\n",
    "\n",
    "How much can we normalize?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "thirties_words = [text.split() for text in thirties]\n",
    "thirties_word_counts = [len(words) for words in thirties_words]\n",
    "q3 = np.percentile(thirties_word_counts, 75)\n",
    "thirties_short_comments = [text for i, text in enumerate(thirties) if thirties_word_counts[i] < q3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "29.529132040627886"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sum(word_counts(thirties_short_comments)) / len(word_counts(thirties_short_comments))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "21660"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(thirties_short_comments)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since we have some more data here anyway, let's take out some more of the longer data points to make word count normalized."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "thirties_words = [text.split() for text in thirties_short_comments]\n",
    "thirties_word_counts = [len(words) for words in thirties_words]\n",
    "p90 = np.percentile(thirties_word_counts, 90)\n",
    "thirties_short_comments = [text for i, text in enumerate(thirties_short_comments) if thirties_word_counts[i] < p90]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "24.883458260601703"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sum(word_counts(thirties_short_comments)) / len(word_counts(thirties_short_comments))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "19478"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(thirties_short_comments)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Okay, much better. Putting it all together."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Combining the data into a set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "teen_labels = ['te'] * len(teens_long_comments)\n",
    "twenties_labels = ['tw'] * len(twenties_short_comments)\n",
    "thirties_labels = ['th'] * len(thirties_short_comments)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = list(zip(teens_long_comments + twenties_short_comments + thirties_short_comments, teen_labels + twenties_labels + thirties_labels))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(\"Yes. Being able to consider working part-time because I'll be on my partner's health insurance is super exciting for me. If I were to go part-time right now my health insurance costs would be astronomical.   Also, I absolutely want to be able to have my partner make medical decisions for me if something happens. He knows me best and knows what I want.\",\n",
       " 'th')"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data[-1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>age</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>What happened to my comment....it was soo good...</td>\n",
       "      <td>te</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>A shit ton of censorship. And I don't mean \"de...</td>\n",
       "      <td>te</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Wasn't aware of the drama between /r/askmen an...</td>\n",
       "      <td>te</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Nice username  I too am from Finland</td>\n",
       "      <td>te</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Your comment was on the [other post]( lol</td>\n",
       "      <td>te</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>64861</th>\n",
       "      <td>And, after 10 years of marriage, you can get 5...</td>\n",
       "      <td>th</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>64862</th>\n",
       "      <td>Yes. Thank you for this response. I don’t view...</td>\n",
       "      <td>th</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>64863</th>\n",
       "      <td>Better hope that you're contacted before someo...</td>\n",
       "      <td>th</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>64864</th>\n",
       "      <td>Thank you for this question. I also find mysel...</td>\n",
       "      <td>th</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>64865</th>\n",
       "      <td>Yes. Being able to consider working part-time ...</td>\n",
       "      <td>th</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>64866 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                    text age\n",
       "0      What happened to my comment....it was soo good...  te\n",
       "1      A shit ton of censorship. And I don't mean \"de...  te\n",
       "2      Wasn't aware of the drama between /r/askmen an...  te\n",
       "3                   Nice username  I too am from Finland  te\n",
       "4              Your comment was on the [other post]( lol  te\n",
       "...                                                  ...  ..\n",
       "64861  And, after 10 years of marriage, you can get 5...  th\n",
       "64862  Yes. Thank you for this response. I don’t view...  th\n",
       "64863  Better hope that you're contacted before someo...  th\n",
       "64864  Thank you for this question. I also find mysel...  th\n",
       "64865  Yes. Being able to consider working part-time ...  th\n",
       "\n",
       "[64866 rows x 2 columns]"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create a pandas DataFrame from the list of tuples\n",
    "df = pd.DataFrame(data, columns=['text', 'age'])\n",
    "df = df.loc[df['text'] != '[deleted]'].reset_index(drop=True)\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>age</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>What happened to my comment....it was soo good...</td>\n",
       "      <td>te</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>A shit ton of censorship. And I don't mean \"de...</td>\n",
       "      <td>te</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Wasn't aware of the drama between /r/askmen an...</td>\n",
       "      <td>te</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Nice username  I too am from Finland</td>\n",
       "      <td>te</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Your comment was on the [other post]( lol</td>\n",
       "      <td>te</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>64861</th>\n",
       "      <td>And, after 10 years of marriage, you can get 5...</td>\n",
       "      <td>th</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>64862</th>\n",
       "      <td>Yes. Thank you for this response. I don’t view...</td>\n",
       "      <td>th</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>64863</th>\n",
       "      <td>Better hope that you're contacted before someo...</td>\n",
       "      <td>th</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>64864</th>\n",
       "      <td>Thank you for this question. I also find mysel...</td>\n",
       "      <td>th</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>64865</th>\n",
       "      <td>Yes. Being able to consider working part-time ...</td>\n",
       "      <td>th</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>64866 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                    text age\n",
       "0      What happened to my comment....it was soo good...  te\n",
       "1      A shit ton of censorship. And I don't mean \"de...  te\n",
       "2      Wasn't aware of the drama between /r/askmen an...  te\n",
       "3                   Nice username  I too am from Finland  te\n",
       "4              Your comment was on the [other post]( lol  te\n",
       "...                                                  ...  ..\n",
       "64861  And, after 10 years of marriage, you can get 5...  th\n",
       "64862  Yes. Thank you for this response. I don’t view...  th\n",
       "64863  Better hope that you're contacted before someo...  th\n",
       "64864  Thank you for this question. I also find mysel...  th\n",
       "64865  Yes. Being able to consider working part-time ...  th\n",
       "\n",
       "[64866 rows x 2 columns]"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# save df as a pickle file\n",
    "df.to_pickle('../../data_samples/reddit_samples/all.pkl')\n",
    "\n",
    "# load df from pickle file to make sure it works\n",
    "loaded_df = pd.read_pickle('../../data_samples/reddit_samples/all.pkl')\n",
    "loaded_df"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looks like that's the mostly final form of the data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.6 (default, Oct 18 2022, 12:41:40) \n[Clang 14.0.0 (clang-1400.0.29.202)]"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "31f2aee4e71d21fbe5cf8b01ff0e069b9275f58929596ceb00d14d90e3e16cd6"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}