Feature_Extraction_Final.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "\n",
        "---\n",
        "\n",
        "\n",
        "# 📊 Proyecto de Análisis de Sentimientos de Tweets"
      ],
      "metadata": {
        "id": "7sQMkBgVmapT"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 🎯 Problema de Negocio\n",
        "\n",
        "El objetivo de este proyecto es analizar los sentimientos de los tweets, identificando si los mensajes son positivos o negativos. Utilizando un conjunto de datos de 1,600,000 tweets preprocesados, se pretende generar diversas características que se utilizarán para entrenar un modelo de clasificación."
      ],
      "metadata": {
        "id": "ibw5iSF8mk85"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## ❓ Preguntas Clave\n",
        "\n",
        "- **¿Qué variables pueden ser clave para la detección del sentimiento?**\n",
        "- **¿Qué tipo de modelos pueden servir para detectar el sentimiento en los tweets?**\n",
        "- **¿Qué métricas se pueden utilizar para evaluar el desempeño del modelo?**"
      ],
      "metadata": {
        "id": "DykzxaU-mq6H"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import pandas as pd\n",
        "import re\n",
        "import string\n",
        "import emoji\n",
        "from nltk.corpus import stopwords\n",
        "from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer\n",
        "from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer\n",
        "import nltk\n",
        "import math\n",
        "from textblob import TextBlob\n",
        "nltk.download('stopwords')\n",
        "\n",
        "# Cargar el dataset de tweets\n",
        "df = pd.read_csv(\"training_1600000_processed_noemoticon.csv\", encoding='latin-1')\n",
        "df.columns = ['target', 'ids', 'date', 'flag', 'user', 'text']\n",
        "df"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 770
        },
        "id": "7Nz1raqR7brZ",
        "outputId": "61c89ad0-9ec5-497b-e703-ada6faa5788c"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "         target         ids                          date      flag  \\\n",
              "0             0  1467810672  Mon Apr 06 22:19:49 PDT 2009  NO_QUERY   \n",
              "1             0  1467810917  Mon Apr 06 22:19:53 PDT 2009  NO_QUERY   \n",
              "2             0  1467811184  Mon Apr 06 22:19:57 PDT 2009  NO_QUERY   \n",
              "3             0  1467811193  Mon Apr 06 22:19:57 PDT 2009  NO_QUERY   \n",
              "4             0  1467811372  Mon Apr 06 22:20:00 PDT 2009  NO_QUERY   \n",
              "...         ...         ...                           ...       ...   \n",
              "1599994       4  2193601966  Tue Jun 16 08:40:49 PDT 2009  NO_QUERY   \n",
              "1599995       4  2193601969  Tue Jun 16 08:40:49 PDT 2009  NO_QUERY   \n",
              "1599996       4  2193601991  Tue Jun 16 08:40:49 PDT 2009  NO_QUERY   \n",
              "1599997       4  2193602064  Tue Jun 16 08:40:49 PDT 2009  NO_QUERY   \n",
              "1599998       4  2193602129  Tue Jun 16 08:40:50 PDT 2009  NO_QUERY   \n",
              "\n",
              "                    user                                               text  \n",
              "0          scotthamilton  is upset that he can't update his Facebook by ...  \n",
              "1               mattycus  @Kenichan I dived many times for the ball. Man...  \n",
              "2                ElleCTF    my whole body feels itchy and like its on fire   \n",
              "3                 Karoli  @nationwideclass no, it's not behaving at all....  \n",
              "4               joy_wolf                      @Kwesidei not the whole crew   \n",
              "...                  ...                                                ...  \n",
              "1599994  AmandaMarie1028  Just woke up. Having no school is the best fee...  \n",
              "1599995      TheWDBoards  TheWDB.com - Very cool to hear old Walt interv...  \n",
              "1599996           bpbabe  Are you ready for your MoJo Makeover? Ask me f...  \n",
              "1599997     tinydiamondz  Happy 38th Birthday to my boo of alll time!!! ...  \n",
              "1599998   RyanTrevMorris  happy #charitytuesday @theNSPCC @SparksCharity...  \n",
              "\n",
              "[1599999 rows x 6 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-bf3a50ca-7433-48d8-a51b-043022bc6c44\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>target</th>\n",
              "      <th>ids</th>\n",
              "      <th>date</th>\n",
              "      <th>flag</th>\n",
              "      <th>user</th>\n",
              "      <th>text</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0</td>\n",
              "      <td>1467810672</td>\n",
              "      <td>Mon Apr 06 22:19:49 PDT 2009</td>\n",
              "      <td>NO_QUERY</td>\n",
              "      <td>scotthamilton</td>\n",
              "      <td>is upset that he can't update his Facebook by ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>0</td>\n",
              "      <td>1467810917</td>\n",
              "      <td>Mon Apr 06 22:19:53 PDT 2009</td>\n",
              "      <td>NO_QUERY</td>\n",
              "      <td>mattycus</td>\n",
              "      <td>@Kenichan I dived many times for the ball. Man...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>0</td>\n",
              "      <td>1467811184</td>\n",
              "      <td>Mon Apr 06 22:19:57 PDT 2009</td>\n",
              "      <td>NO_QUERY</td>\n",
              "      <td>ElleCTF</td>\n",
              "      <td>my whole body feels itchy and like its on fire</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>0</td>\n",
              "      <td>1467811193</td>\n",
              "      <td>Mon Apr 06 22:19:57 PDT 2009</td>\n",
              "      <td>NO_QUERY</td>\n",
              "      <td>Karoli</td>\n",
              "      <td>@nationwideclass no, it's not behaving at all....</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>0</td>\n",
              "      <td>1467811372</td>\n",
              "      <td>Mon Apr 06 22:20:00 PDT 2009</td>\n",
              "      <td>NO_QUERY</td>\n",
              "      <td>joy_wolf</td>\n",
              "      <td>@Kwesidei not the whole crew</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1599994</th>\n",
              "      <td>4</td>\n",
              "      <td>2193601966</td>\n",
              "      <td>Tue Jun 16 08:40:49 PDT 2009</td>\n",
              "      <td>NO_QUERY</td>\n",
              "      <td>AmandaMarie1028</td>\n",
              "      <td>Just woke up. Having no school is the best fee...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1599995</th>\n",
              "      <td>4</td>\n",
              "      <td>2193601969</td>\n",
              "      <td>Tue Jun 16 08:40:49 PDT 2009</td>\n",
              "      <td>NO_QUERY</td>\n",
              "      <td>TheWDBoards</td>\n",
              "      <td>TheWDB.com - Very cool to hear old Walt interv...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1599996</th>\n",
              "      <td>4</td>\n",
              "      <td>2193601991</td>\n",
              "      <td>Tue Jun 16 08:40:49 PDT 2009</td>\n",
              "      <td>NO_QUERY</td>\n",
              "      <td>bpbabe</td>\n",
              "      <td>Are you ready for your MoJo Makeover? Ask me f...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1599997</th>\n",
              "      <td>4</td>\n",
              "      <td>2193602064</td>\n",
              "      <td>Tue Jun 16 08:40:49 PDT 2009</td>\n",
              "      <td>NO_QUERY</td>\n",
              "      <td>tinydiamondz</td>\n",
              "      <td>Happy 38th Birthday to my boo of alll time!!! ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1599998</th>\n",
              "      <td>4</td>\n",
              "      <td>2193602129</td>\n",
              "      <td>Tue Jun 16 08:40:50 PDT 2009</td>\n",
              "      <td>NO_QUERY</td>\n",
              "      <td>RyanTrevMorris</td>\n",
              "      <td>happy #charitytuesday @theNSPCC @SparksCharity...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>1599999 rows × 6 columns</p>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-bf3a50ca-7433-48d8-a51b-043022bc6c44')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-bf3a50ca-7433-48d8-a51b-043022bc6c44 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-bf3a50ca-7433-48d8-a51b-043022bc6c44');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-38116193-b969-4d4b-b255-274bd8233b44\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-38116193-b969-4d4b-b255-274bd8233b44')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-38116193-b969-4d4b-b255-274bd8233b44 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "  <div id=\"id_63fd838c-af3a-47f1-bed6-842516945fa6\">\n",
              "    <style>\n",
              "      .colab-df-generate {\n",
              "        background-color: #E8F0FE;\n",
              "        border: none;\n",
              "        border-radius: 50%;\n",
              "        cursor: pointer;\n",
              "        display: none;\n",
              "        fill: #1967D2;\n",
              "        height: 32px;\n",
              "        padding: 0 0 0 0;\n",
              "        width: 32px;\n",
              "      }\n",
              "\n",
              "      .colab-df-generate:hover {\n",
              "        background-color: #E2EBFA;\n",
              "        box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "        fill: #174EA6;\n",
              "      }\n",
              "\n",
              "      [theme=dark] .colab-df-generate {\n",
              "        background-color: #3B4455;\n",
              "        fill: #D2E3FC;\n",
              "      }\n",
              "\n",
              "      [theme=dark] .colab-df-generate:hover {\n",
              "        background-color: #434B5C;\n",
              "        box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "        filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "        fill: #FFFFFF;\n",
              "      }\n",
              "    </style>\n",
              "    <button class=\"colab-df-generate\" onclick=\"generateWithVariable('df')\"\n",
              "            title=\"Generate code using this dataframe.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M7,19H8.4L18.45,9,17,7.55,7,17.6ZM5,21V16.75L18.45,3.32a2,2,0,0,1,2.83,0l1.4,1.43a1.91,1.91,0,0,1,.58,1.4,1.91,1.91,0,0,1-.58,1.4L9.25,21ZM18.45,9,17,7.55Zm-12,3A5.31,5.31,0,0,0,4.9,8.1,5.31,5.31,0,0,0,1,6.5,5.31,5.31,0,0,0,4.9,4.9,5.31,5.31,0,0,0,6.5,1,5.31,5.31,0,0,0,8.1,4.9,5.31,5.31,0,0,0,12,6.5,5.46,5.46,0,0,0,6.5,12Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "    <script>\n",
              "      (() => {\n",
              "      const buttonEl =\n",
              "        document.querySelector('#id_63fd838c-af3a-47f1-bed6-842516945fa6 button.colab-df-generate');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      buttonEl.onclick = () => {\n",
              "        google.colab.notebook.generateWithVariable('df');\n",
              "      }\n",
              "      })();\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "variable_name": "df"
            }
          },
          "metadata": {},
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "El dataset contiene texto que se puede considerar como \"sucio\", es por ello que partamos de crear una función que pueda limpiar este texto eliminando URLs, menciones, emojis y caracteres especiales.\n",
        "Recuerda que la limpieza de texto es crucial para que el análisis sea más preciso."
      ],
      "metadata": {
        "id": "jMIXw8hRj6Cv"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def clean_text(text):\n",
        "    # Eliminar URLs\n",
        "    text = re.sub(r'http\\S+|www\\S+|https\\S+', '', text, flags=re.MULTILINE)\n",
        "    # Eliminar menciones de usuarios (@)\n",
        "    text = re.sub(r'\\@\\w+', '', text)\n",
        "    # Eliminar emojis\n",
        "    text = emoji.replace_emoji(text, replace=\"\")\n",
        "    # Eliminar signos de puntuación y números\n",
        "    text = text.translate(str.maketrans('', '', string.punctuation))\n",
        "    # Convertir a minúsculas\n",
        "    text = text.lower()\n",
        "    return text\n",
        "df['cleaned_text'] = df['text'].apply(clean_text)"
      ],
      "metadata": {
        "id": "QoJSN_K6i-1m"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "stop_words = set(stopwords.words('english'))"
      ],
      "metadata": {
        "id": "VDgYG6EbqCuE"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def count_emojis(text):\n",
        "    return sum(1 for char in text if char in emoji.EMOJI_DATA)\n",
        "def capital_word_density(text):\n",
        "    words = text.split()\n",
        "    if len(words) == 0:\n",
        "        return 0\n",
        "    uppercase_words = [word for word in words if word.isupper()]\n",
        "    return len(uppercase_words) / len(words)\n",
        "def stopword_count(text):\n",
        "    words = text.split()\n",
        "    stopword_count = sum(1 for word in words if word in stop_words)\n",
        "    return stopword_count\n",
        "def repeated_word_proportion(text):\n",
        "    words = text.split()\n",
        "    if len(words) == 0:\n",
        "        return 0\n",
        "    unique_words = set(words)\n",
        "    return (len(words) - len(unique_words)) / len(words)\n",
        "def calculate_entropy(text):\n",
        "    prob = [text.count(c) / len(text) for c in set(text)]\n",
        "    entropy = -sum([p * math.log(p) for p in prob])\n",
        "    return entropy\n",
        "def detect_sarcasm(text):\n",
        "    if '!' in text and analyzer.polarity_scores(text)['neg'] > 0.5:\n",
        "        return 1\n",
        "    return 0"
      ],
      "metadata": {
        "id": "uzapr9VSp4su"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "df = df[['target', 'text','date', 'cleaned_text']]\n",
        "df['tweet_length'] = df['cleaned_text'].apply(len)\n",
        "df['emoji_count'] = df['text'].apply(count_emojis)\n",
        "df['exclamation_count'] = df['text'].apply(lambda x: x.count('!'))\n",
        "df['question_count'] = df['text'].apply(lambda x: x.count('?'))\n",
        "df['capital_word_density'] = df['text'].apply(capital_word_density)\n",
        "df['word_count'] = df['cleaned_text'].apply(lambda x: len(x.split()))\n",
        "df['stopword_count'] = df['cleaned_text'].apply(stopword_count)\n",
        "df['unique_word_count'] = df['cleaned_text'].apply(lambda x: len(set(x.split())))\n",
        "df['repeated_word_proportion'] = df['cleaned_text'].apply(repeated_word_proportion)\n",
        "df['textblob_polarity'] = df['cleaned_text'].apply(lambda x: TextBlob(x).sentiment.polarity)\n",
        "df['textblob_subjectivity'] = df['cleaned_text'].apply(lambda x: TextBlob(x).sentiment.subjectivity)\n",
        "df['punctuation_count'] = df['text'].apply(lambda x: len([char for char in x if char in string.punctuation]))\n",
        "df['mention_count'] = df['text'].apply(lambda x: x.count('@'))\n",
        "df['hashtag_count'] = df['text'].apply(lambda x: x.count('#'))\n",
        "df['text_entropy'] = df['cleaned_text'].apply(calculate_entropy)\n",
        "df['sarcasm'] = df['text'].apply(detect_sarcasm)"
      ],
      "metadata": {
        "id": "faWw-02rpxWh"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Añade también otras variables que has creado y que crees que puedan hacer que el modelo sea más robusto 😉"
      ],
      "metadata": {
        "id": "q20ejZbely1q"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Existen librerías que ya detectan los sentimientos de texto, crees que debas de usarlas?**"
      ],
      "metadata": {
        "id": "mhpp87IVm2ut"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "df.to_csv('datatotal.csv')"
      ],
      "metadata": {
        "id": "IsKQuapuCUmy"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}