tse2020-roberta-cnn-nlp20.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.017623,
     "end_time": "2022-10-22T07:09:20.255560",
     "exception": false,
     "start_time": "2022-10-22T07:09:20.237937",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "<a class=\"anchor\" id=\"0\"></a>\n",
    "# [Tweet Sentiment Extraction](https://www.kaggle.com/c/tweet-sentiment-extraction)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.015855,
     "end_time": "2022-10-22T07:09:20.287897",
     "exception": false,
     "start_time": "2022-10-22T07:09:20.272042",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Overview\n",
    "\n",
    "In this notebook, I analyze and visualize the outliers of the NLP solution from very good notebook \"[TSE2020] RoBERTa (CNN) & Random Seed Distribution\"(https://www.kaggle.com/khoongweihao/tse2020-roberta-cnn-random-seed-distribution) using the functions from my notebook [NLP - EDA, Bag of Words, TF IDF, GloVe, BERT](https://www.kaggle.com/vbmokin/nlp-eda-bag-of-words-tf-idf-glove-bert) including PCA processing, Kmeans clustering, WordCloud and others. More over I try to improve the original solution.\n",
    "\n",
    "Add chapters \"**Subtext analysis**\" and \"**Metric analysis**\" from the commit 10."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.016039,
     "end_time": "2022-10-22T07:09:20.319797",
     "exception": false,
     "start_time": "2022-10-22T07:09:20.303758",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Results of analysis:\n",
    "1. Outlier analysis of the best solutions on basic roBERTa - pls. see https://www.kaggle.com/c/tweet-sentiment-extraction/discussion/155419\n",
    "2. Analysis of the predictions with the worst score=0 from roBERTa - pls. see https://www.kaggle.com/c/tweet-sentiment-extraction/discussion/155616\n",
    "3. New (commit 22): **analysis of 3 or more repetitions of characters in words**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.015809,
     "end_time": "2022-10-22T07:09:20.351608",
     "exception": false,
     "start_time": "2022-10-22T07:09:20.335799",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Acknowledgements\n",
    "* [NLP - EDA, Bag of Words, TF IDF, GloVe, BERT](https://www.kaggle.com/vbmokin/nlp-eda-bag-of-words-tf-idf-glove-bert)\n",
    "* [COVID-19 (Week5) Global Forecasting - EDA&ExtraTR](https://www.kaggle.com/vbmokin/covid-19-week5-global-forecasting-eda-extratr)\n",
    "* [TSE2020] RoBERTa (CNN) & Random Seed Distribution (https://www.kaggle.com/khoongweihao/tse2020-roberta-cnn-random-seed-distribution)\n",
    "* Chris Deotte's post: https://www.kaggle.com/c/tweet-sentiment-extraction/discussion/142404#809872\n",
    "* [Faster (2x) TF roBERTa](https://www.kaggle.com/seesee/faster-2x-tf-roberta)\n",
    "* Many thanks to Chris Deotte for his TF roBERTa dataset at https://www.kaggle.com/cdeotte/tf-roberta\n",
    "* https://www.kaggle.com/abhishek/roberta-inference-5-folds"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.017346,
     "end_time": "2022-10-22T07:09:20.385367",
     "exception": false,
     "start_time": "2022-10-22T07:09:20.368021",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "<a class=\"anchor\" id=\"0.1\"></a>\n",
    "## Table of Contents\n",
    "\n",
    "1. [Import libraries](#1)\n",
    "1. [Data fetching](#2)\n",
    "1. [Model (TF: RoBerta, DNN: CNN)](#3)\n",
    "   - [My upgrade of parameters](#3.1)\n",
    "   - [Model training](#3.2)\n",
    "1. [Submission](#4)\n",
    "1. [Outlier analysis](#5)\n",
    "    - [Training prediction result visualization](#5.1)\n",
    "    - [WordCloud](#5.2)\n",
    "    - [Subtext analysis](#5.3)\n",
    "    - [Metric analysis](#5.4)\n",
    "    - [PCA visualization](#5.5)\n",
    "    - [Clustering](#5.6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.016446,
     "end_time": "2022-10-22T07:09:20.417979",
     "exception": false,
     "start_time": "2022-10-22T07:09:20.401533",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## 1. Import libraries <a class=\"anchor\" id=\"1\"></a>\n",
    "\n",
    "[Back to Table of Contents](#0.1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "_kg_hide-input": false,
    "execution": {
     "iopub.execute_input": "2022-10-22T07:09:20.462715Z",
     "iopub.status.busy": "2022-10-22T07:09:20.461963Z",
     "iopub.status.idle": "2022-10-22T07:09:28.489479Z",
     "shell.execute_reply": "2022-10-22T07:09:28.488663Z",
     "shell.execute_reply.started": "2022-10-22T07:05:12.793477Z"
    },
    "papermill": {
     "duration": 8.054956,
     "end_time": "2022-10-22T07:09:28.489604",
     "exception": false,
     "start_time": "2022-10-22T07:09:20.434648",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib\n",
    "import matplotlib.patches as mpatches\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "import plotly.express as px\n",
    "import seaborn as sns; sns.set(style='white')\n",
    "from mpl_toolkits.mplot3d import Axes3D\n",
    "\n",
    "from wordcloud import WordCloud\n",
    "from sklearn.decomposition import PCA, TruncatedSVD\n",
    "import math\n",
    "import pickle\n",
    "\n",
    "from scipy.cluster import hierarchy\n",
    "from scipy.spatial.distance import pdist\n",
    "from sklearn.manifold import TSNE\n",
    "from sklearn.cluster import KMeans\n",
    "\n",
    "import tensorflow as tf\n",
    "import tensorflow.keras.backend as K\n",
    "from transformers import *\n",
    "import tokenizers\n",
    "from sklearn.model_selection import StratifiedKFold\n",
    "\n",
    "pd.set_option('max_colwidth', 40)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.016135,
     "end_time": "2022-10-22T07:09:28.521776",
     "exception": false,
     "start_time": "2022-10-22T07:09:28.505641",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## 2. Data fetching <a class=\"anchor\" id=\"2\"></a>\n",
    "\n",
    "[Back to Table of Contents](#0.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.015746,
     "end_time": "2022-10-22T07:09:28.553199",
     "exception": false,
     "start_time": "2022-10-22T07:09:28.537453",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Code from notebook https://www.kaggle.com/khoongweihao/tse2020-roberta-cnn-random-seed-distribution?scriptVersionId=34448972"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-10-22T07:09:28.594890Z",
     "iopub.status.busy": "2022-10-22T07:09:28.594171Z",
     "iopub.status.idle": "2022-10-22T07:09:28.597245Z",
     "shell.execute_reply": "2022-10-22T07:09:28.596603Z",
     "shell.execute_reply.started": "2022-10-22T07:05:20.553455Z"
    },
    "papermill": {
     "duration": 0.027813,
     "end_time": "2022-10-22T07:09:28.597359",
     "exception": false,
     "start_time": "2022-10-22T07:09:28.569546",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# global variables\n",
    "\n",
    "MAX_LEN = 96\n",
    "EPOCHS = 3 # originally 3\n",
    "BATCH_SIZE = 64 # originally 32\n",
    "PAD_ID = 1\n",
    "SEED = 88888\n",
    "LABEL_SMOOTHING = 0.15"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-10-22T07:09:28.635608Z",
     "iopub.status.busy": "2022-10-22T07:09:28.635058Z",
     "iopub.status.idle": "2022-10-22T07:09:28.754805Z",
     "shell.execute_reply": "2022-10-22T07:09:28.754254Z",
     "shell.execute_reply.started": "2022-10-22T07:05:20.566796Z"
    },
    "papermill": {
     "duration": 0.141408,
     "end_time": "2022-10-22T07:09:28.754920",
     "exception": false,
     "start_time": "2022-10-22T07:09:28.613512",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# get train data \n",
    "\n",
    "train = pd.read_csv('../input/tweet-sentiment-extraction/train.csv').fillna('')\n",
    "\n",
    "# initialize tenserflow and numpy random seed\n",
    "tf.random.set_seed(SEED)\n",
    "np.random.seed(SEED)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-10-22T07:09:28.797687Z",
     "iopub.status.busy": "2022-10-22T07:09:28.796783Z",
     "iopub.status.idle": "2022-10-22T07:09:38.857348Z",
     "shell.execute_reply": "2022-10-22T07:09:38.856300Z",
     "shell.execute_reply.started": "2022-10-22T07:05:20.746489Z"
    },
    "papermill": {
     "duration": 10.085866,
     "end_time": "2022-10-22T07:09:38.857476",
     "exception": false,
     "start_time": "2022-10-22T07:09:28.771610",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "27481\n",
      "1699\n"
     ]
    }
   ],
   "source": [
    "# remove noises in train data\n",
    "\n",
    "def find_failed_start(text,selected_text):\n",
    "    \"\"\"find the case that lable is not truncated by space\"\"\"\n",
    "    begin = text.find(selected_text)\n",
    "    end = begin + len(selected_text)\n",
    "    if begin == 0:\n",
    "        return False\n",
    "    return text[begin -1].isalpha()\n",
    "\n",
    "def find_failed_end(text,selected_text):\n",
    "    \"\"\"find the case that lable is not truncated by space\"\"\"\n",
    "    begin = text.find(selected_text)\n",
    "    end = begin + len(selected_text)\n",
    "    if end == len(text):\n",
    "        return False\n",
    "    return text[end].isalpha()\n",
    "\n",
    "failed_start = train[train.apply(lambda row: find_failed_start(row.text, row.selected_text),axis=1)]\n",
    "failed_end = train[train.apply(lambda row: find_failed_end(row.text, row.selected_text),axis=1)]\n",
    "failed_ids = set(failed_start.textID.tolist() + failed_end.textID.tolist())\n",
    "\n",
    "print(len(train))\n",
    "print(len(failed_ids))\n",
    "\n",
    "for ids in failed_ids:\n",
    "    idx=train[train['textID']==ids].index\n",
    "    train=train.drop(idx)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-10-22T07:09:38.901711Z",
     "iopub.status.busy": "2022-10-22T07:09:38.900885Z",
     "iopub.status.idle": "2022-10-22T07:09:38.911469Z",
     "shell.execute_reply": "2022-10-22T07:09:38.912052Z",
     "shell.execute_reply.started": "2022-10-22T07:05:30.955546Z"
    },
    "papermill": {
     "duration": 0.037472,
     "end_time": "2022-10-22T07:09:38.912197",
     "exception": false,
     "start_time": "2022-10-22T07:09:38.874725",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "25782\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>textID</th>\n",
       "      <th>text</th>\n",
       "      <th>selected_text</th>\n",
       "      <th>sentiment</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>25777</th>\n",
       "      <td>a208770a32</td>\n",
       "      <td>in spoke to you yesterday and u did...</td>\n",
       "      <td>in spoke to you yesterday and u didn...</td>\n",
       "      <td>neutral</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25778</th>\n",
       "      <td>b78ec00df5</td>\n",
       "      <td>enjoy ur night</td>\n",
       "      <td>enjoy</td>\n",
       "      <td>positive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25779</th>\n",
       "      <td>f67aae2310</td>\n",
       "      <td>Yay good for both of you. Enjoy the...</td>\n",
       "      <td>Yay good for both of you.</td>\n",
       "      <td>positive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25780</th>\n",
       "      <td>ed167662a5</td>\n",
       "      <td>But it was worth it  ****.</td>\n",
       "      <td>But it was worth it  ****.</td>\n",
       "      <td>positive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25781</th>\n",
       "      <td>6f7127d9d7</td>\n",
       "      <td>All this flirting going on - The ...</td>\n",
       "      <td>All this flirting going on - The ATG...</td>\n",
       "      <td>neutral</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           textID                                     text  \\\n",
       "25777  a208770a32   in spoke to you yesterday and u did...   \n",
       "25778  b78ec00df5                           enjoy ur night   \n",
       "25779  f67aae2310   Yay good for both of you. Enjoy the...   \n",
       "25780  ed167662a5               But it was worth it  ****.   \n",
       "25781  6f7127d9d7     All this flirting going on - The ...   \n",
       "\n",
       "                                 selected_text sentiment  \n",
       "25777  in spoke to you yesterday and u didn...   neutral  \n",
       "25778                                    enjoy  positive  \n",
       "25779                Yay good for both of you.  positive  \n",
       "25780               But it was worth it  ****.  positive  \n",
       "25781  All this flirting going on - The ATG...   neutral  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# display train data after removing noises\n",
    "\n",
    "train.reset_index(drop=True, inplace=True)\n",
    "print(len(train))\n",
    "train.tail()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "_kg_hide-input": true,
    "execution": {
     "iopub.execute_input": "2022-10-22T07:09:38.951815Z",
     "iopub.status.busy": "2022-10-22T07:09:38.951285Z",
     "iopub.status.idle": "2022-10-22T07:09:38.970969Z",
     "shell.execute_reply": "2022-10-22T07:09:38.970451Z",
     "shell.execute_reply.started": "2022-10-22T07:05:30.976602Z"
    },
    "papermill": {
     "duration": 0.041806,
     "end_time": "2022-10-22T07:09:38.971067",
     "exception": false,
     "start_time": "2022-10-22T07:09:38.929261",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# get test data\n",
    "\n",
    "test = pd.read_csv('../input/tweet-sentiment-extraction/test.csv').fillna('')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.016701,
     "end_time": "2022-10-22T07:09:39.004680",
     "exception": false,
     "start_time": "2022-10-22T07:09:38.987979",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## 3. Model (TF: RoBerta, DNN: CNN) <a class=\"anchor\" id=\"3\"></a>\n",
    "\n",
    "[Back to Table of Contents](#0.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.016614,
     "end_time": "2022-10-22T07:09:39.038643",
     "exception": false,
     "start_time": "2022-10-22T07:09:39.022029",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## 3.1. Tunable parameter <a class=\"anchor\" id=\"3.1\"></a>\n",
    "\n",
    "[Back to Table of Contents](#0.1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-10-22T07:09:39.077489Z",
     "iopub.status.busy": "2022-10-22T07:09:39.076622Z",
     "iopub.status.idle": "2022-10-22T07:09:39.079240Z",
     "shell.execute_reply": "2022-10-22T07:09:39.079709Z",
     "shell.execute_reply.started": "2022-10-22T07:05:31.004451Z"
    },
    "papermill": {
     "duration": 0.023704,
     "end_time": "2022-10-22T07:09:39.079859",
     "exception": false,
     "start_time": "2022-10-22T07:09:39.056155",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "dropout_new = 0.15     # originally 0.1\n",
    "n_split = 5            # originally 5\n",
    "lr = 3e-5              # (lr: learning rate) originally 3e-5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.017235,
     "end_time": "2022-10-22T07:09:39.114072",
     "exception": false,
     "start_time": "2022-10-22T07:09:39.096837",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## 3.2. Jaccard score function <a class=\"anchor\" id=\"3.2\"></a>\n",
    "\n",
    "[Back to Table of Contents](#0.1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "_kg_hide-input": true,
    "execution": {
     "iopub.execute_input": "2022-10-22T07:09:39.154343Z",
     "iopub.status.busy": "2022-10-22T07:09:39.153625Z",
     "iopub.status.idle": "2022-10-22T07:09:39.156725Z",
     "shell.execute_reply": "2022-10-22T07:09:39.156300Z",
     "shell.execute_reply.started": "2022-10-22T07:05:31.011039Z"
    },
    "papermill": {
     "duration": 0.025201,
     "end_time": "2022-10-22T07:09:39.156848",
     "exception": false,
     "start_time": "2022-10-22T07:09:39.131647",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "def jaccard(str1, str2): \n",
    "    a = set(str1.lower().split()) \n",
    "    b = set(str2.lower().split())\n",
    "    if (len(a)==0) & (len(b)==0): return 0.5\n",
    "    c = a.intersection(b)\n",
    "    return float(len(c)) / (len(a) + len(b) - len(c))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.018131,
     "end_time": "2022-10-22T07:09:39.193078",
     "exception": false,
     "start_time": "2022-10-22T07:09:39.174947",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## 3.3. Build RoBerta <a class=\"anchor\" id=\"3.3\"></a>\n",
    "\n",
    "[Back to Table of Contents](#0.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.016568,
     "end_time": "2022-10-22T07:09:39.227109",
     "exception": false,
     "start_time": "2022-10-22T07:09:39.210541",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Code from notebook https://www.kaggle.com/khoongweihao/tse2020-roberta-cnn-random-seed-distribution?scriptVersionId=34448972\n",
    "\n",
    "**Upgrade:** add prediction for training data for Outlier analysis and parameters tuning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-10-22T07:09:39.395750Z",
     "iopub.status.busy": "2022-10-22T07:09:39.394835Z",
     "iopub.status.idle": "2022-10-22T07:09:39.398339Z",
     "shell.execute_reply": "2022-10-22T07:09:39.397416Z",
     "shell.execute_reply.started": "2022-10-22T07:05:31.020639Z"
    },
    "papermill": {
     "duration": 0.154334,
     "end_time": "2022-10-22T07:09:39.398457",
     "exception": false,
     "start_time": "2022-10-22T07:09:39.244123",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "PATH = '../input/tf-roberta/'\n",
    "\n",
    "# get RoBerta tokenizer\n",
    "\n",
    "tokenizer = tokenizers.ByteLevelBPETokenizer(\n",
    "    vocab_file = PATH +'vocab-roberta-base.json', \n",
    "    merges_file = PATH +'merges-roberta-base.txt', \n",
    "    lowercase = True,\n",
    "    add_prefix_space = True\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.018258,
     "end_time": "2022-10-22T07:09:39.435737",
     "exception": false,
     "start_time": "2022-10-22T07:09:39.417479",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "tokenization of the train data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-10-22T07:09:39.486245Z",
     "iopub.status.busy": "2022-10-22T07:09:39.480511Z",
     "iopub.status.idle": "2022-10-22T07:09:51.962601Z",
     "shell.execute_reply": "2022-10-22T07:09:51.963226Z",
     "shell.execute_reply.started": "2022-10-22T07:05:31.162069Z"
    },
    "papermill": {
     "duration": 12.509035,
     "end_time": "2022-10-22T07:09:51.963362",
     "exception": false,
     "start_time": "2022-10-22T07:09:39.454327",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# tokenization of the train data\n",
    "\n",
    "sentiment_id = {'positive': 1313, 'negative': 2430, 'neutral': 7974}\n",
    "\n",
    "ct = train.shape[0]\n",
    "input_ids_train = np.ones((ct,MAX_LEN),dtype='int32')\n",
    "attention_mask_train = np.zeros((ct,MAX_LEN),dtype='int32')\n",
    "token_type_ids_train = np.zeros((ct,MAX_LEN),dtype='int32')\n",
    "start_tokens_train = np.zeros((ct,MAX_LEN),dtype='int32')\n",
    "end_tokens_train = np.zeros((ct,MAX_LEN),dtype='int32')\n",
    "\n",
    "for k in range(train.shape[0]):\n",
    "    # find overlap\n",
    "    text1 = \" \"+\" \".join(train.loc[k,'text'].split())\n",
    "    text2 = \" \".join(train.loc[k,'selected_text'].split())\n",
    "    idx = text1.find(text2)\n",
    "    chars = np.zeros((len(text1)))\n",
    "    chars[idx:idx+len(text2)]=1\n",
    "    if text1[idx-1]==' ': chars[idx-1] = 1 \n",
    "    enc = tokenizer.encode(text1) \n",
    "        \n",
    "    # id offsets\n",
    "    offsets = []; idx=0\n",
    "    for t in enc.ids:\n",
    "        w = tokenizer.decode([t])\n",
    "        offsets.append((idx,idx+len(w)))\n",
    "        idx += len(w)\n",
    "    \n",
    "    # start and end tokens\n",
    "    toks = []\n",
    "    for i,(a,b) in enumerate(offsets):\n",
    "        sm = np.sum(chars[a:b])\n",
    "        if sm>0: toks.append(i) \n",
    "        \n",
    "    s_tok = sentiment_id[train.loc[k,'sentiment']]\n",
    "    input_ids_train[k,:len(enc.ids)+3] = [0, s_tok] + enc.ids + [2]\n",
    "    attention_mask_train[k,:len(enc.ids)+3] = 1\n",
    "    if len(toks)>0:\n",
    "        start_tokens_train[k,toks[0]+2] = 1\n",
    "        end_tokens_train[k,toks[-1]+2] = 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.016949,
     "end_time": "2022-10-22T07:09:51.998017",
     "exception": false,
     "start_time": "2022-10-22T07:09:51.981068",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "tokenization of the test data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-10-22T07:09:52.041448Z",
     "iopub.status.busy": "2022-10-22T07:09:52.040527Z",
     "iopub.status.idle": "2022-10-22T07:09:52.697185Z",
     "shell.execute_reply": "2022-10-22T07:09:52.697631Z",
     "shell.execute_reply.started": "2022-10-22T07:05:43.491638Z"
    },
    "papermill": {
     "duration": 0.682678,
     "end_time": "2022-10-22T07:09:52.697806",
     "exception": false,
     "start_time": "2022-10-22T07:09:52.015128",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# tokenization of the test data\n",
    "\n",
    "ct = test.shape[0]\n",
    "input_ids_test = np.ones((ct,MAX_LEN),dtype='int32')\n",
    "attention_mask_test = np.zeros((ct,MAX_LEN),dtype='int32')\n",
    "token_type_ids_test = np.zeros((ct,MAX_LEN),dtype='int32')\n",
    "\n",
    "# store input_ids and attention_mask\n",
    "\n",
    "for k in range(test.shape[0]):\n",
    "    text1 = \" \"+\" \".join(test.loc[k,'text'].split())\n",
    "    enc = tokenizer.encode(text1)                \n",
    "    s_tok = sentiment_id[test.loc[k,'sentiment']]\n",
    "    input_ids_test[k,:len(enc.ids)+3] = [0, s_tok] + enc.ids + [2]\n",
    "    attention_mask_test[k,:len(enc.ids)+3] = 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "_kg_hide-input": true,
    "execution": {
     "iopub.execute_input": "2022-10-22T07:09:52.754528Z",
     "iopub.status.busy": "2022-10-22T07:09:52.753770Z",
     "iopub.status.idle": "2022-10-22T07:09:52.756781Z",
     "shell.execute_reply": "2022-10-22T07:09:52.756350Z",
     "shell.execute_reply.started": "2022-10-22T07:05:44.590023Z"
    },
    "papermill": {
     "duration": 0.041281,
     "end_time": "2022-10-22T07:09:52.756904",
     "exception": false,
     "start_time": "2022-10-22T07:09:52.715623",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "def save_weights(model, dst_fn):\n",
    "    weights = model.get_weights()\n",
    "    with open(dst_fn, 'wb') as f:\n",
    "        pickle.dump(weights, f)\n",
    "\n",
    "\n",
    "def load_weights(model, weight_fn):\n",
    "    with open(weight_fn, 'rb') as f:\n",
    "        weights = pickle.load(f)\n",
    "    model.set_weights(weights)\n",
    "    return model\n",
    "\n",
    "def loss_fn(y_true, y_pred):\n",
    "    # adjust the targets for sequence bucketing\n",
    "    ll = tf.shape(y_pred)[1]\n",
    "    y_true = y_true[:, :ll]\n",
    "    loss = tf.keras.losses.categorical_crossentropy(y_true, y_pred,\n",
    "        from_logits=False, label_smoothing=LABEL_SMOOTHING)\n",
    "    loss = tf.reduce_mean(loss)\n",
    "    return loss\n",
    "\n",
    "\n",
    "def build_model():\n",
    "    ids = tf.keras.layers.Input((MAX_LEN,), dtype=tf.int32)\n",
    "    att = tf.keras.layers.Input((MAX_LEN,), dtype=tf.int32)\n",
    "    tok = tf.keras.layers.Input((MAX_LEN,), dtype=tf.int32)\n",
    "    padding = tf.cast(tf.equal(ids, PAD_ID), tf.int32)\n",
    "\n",
    "    lens = MAX_LEN - tf.reduce_sum(padding, -1)\n",
    "    max_len = tf.reduce_max(lens)\n",
    "    ids_ = ids[:, :max_len]\n",
    "    att_ = att[:, :max_len]\n",
    "    tok_ = tok[:, :max_len]\n",
    "\n",
    "    config = RobertaConfig.from_pretrained(PATH+'config-roberta-base.json')\n",
    "    bert_model = TFRobertaModel.from_pretrained(PATH+'pretrained-roberta-base.h5',config=config)\n",
    "    x = bert_model(ids_,attention_mask=att_,token_type_ids=tok_)\n",
    "    \n",
    "    x1 = tf.keras.layers.Dropout(dropout_new)(x[0])\n",
    "    x1 = tf.keras.layers.Conv1D(768, 2,padding='same')(x1)\n",
    "    x1 = tf.keras.layers.LeakyReLU()(x1)\n",
    "    x1 = tf.keras.layers.Conv1D(64, 2,padding='same')(x1)\n",
    "    x1 = tf.keras.layers.LeakyReLU()(x1)\n",
    "    x1 = tf.keras.layers.Conv1D(64, 2,padding='same')(x1)\n",
    "    x1 = tf.keras.layers.Dense(1)(x1)\n",
    "    x1 = tf.keras.layers.Flatten()(x1)\n",
    "    x1 = tf.keras.layers.Activation('softmax')(x1)\n",
    "    \n",
    "    x2 = tf.keras.layers.Dropout(dropout_new)(x[0]) \n",
    "    x2 = tf.keras.layers.Conv1D(768, 2,padding='same')(x2)\n",
    "    x2 = tf.keras.layers.LeakyReLU()(x2)\n",
    "    x2 = tf.keras.layers.Conv1D(64, 2, padding='same')(x2)\n",
    "    x2 = tf.keras.layers.LeakyReLU()(x2)\n",
    "    x2 = tf.keras.layers.Conv1D(64, 2, padding='same')(x2)\n",
    "    x2 = tf.keras.layers.Dense(1)(x2)\n",
    "    x2 = tf.keras.layers.Flatten()(x2)\n",
    "    x2 = tf.keras.layers.Activation('softmax')(x2)\n",
    "\n",
    "    model = tf.keras.models.Model(inputs=[ids, att, tok], outputs=[x1,x2])\n",
    "    optimizer = tf.keras.optimizers.Adam(learning_rate=lr) \n",
    "    model.compile(loss=loss_fn, optimizer=optimizer)\n",
    "    \n",
    "    # this is required as `model.predict` needs a fixed size!\n",
    "    x1_padded = tf.pad(x1, [[0, 0], [0, MAX_LEN - max_len]], constant_values=0.)\n",
    "    x2_padded = tf.pad(x2, [[0, 0], [0, MAX_LEN - max_len]], constant_values=0.)\n",
    "    \n",
    "    padded_model = tf.keras.models.Model(inputs=[ids, att, tok], outputs=[x1_padded,x2_padded])\n",
    "    return model, padded_model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.016684,
     "end_time": "2022-10-22T07:09:52.791047",
     "exception": false,
     "start_time": "2022-10-22T07:09:52.774363",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Train RoBerta"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-10-22T07:09:52.838252Z",
     "iopub.status.busy": "2022-10-22T07:09:52.837400Z",
     "iopub.status.idle": "2022-10-22T07:40:45.634456Z",
     "shell.execute_reply": "2022-10-22T07:40:45.635042Z"
    },
    "papermill": {
     "duration": 1852.827089,
     "end_time": "2022-10-22T07:40:45.635188",
     "exception": false,
     "start_time": "2022-10-22T07:09:52.808099",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "#########################\n",
      "### FOLD 1\n",
      "#########################\n",
      "Train on 20625 samples, validate on 5157 samples\n",
      "20625/20625 [==============================] - 110s 5ms/sample - loss: 3.3530 - activation_loss: 1.6777 - activation_1_loss: 1.6748 - val_loss: 2.8666 - val_activation_loss: 1.4297 - val_activation_1_loss: 1.4435\n",
      "Train on 20625 samples, validate on 5157 samples\n",
      "Epoch 2/2\n",
      "20625/20625 [==============================] - 78s 4ms/sample - loss: 2.8477 - activation_loss: 1.4237 - activation_1_loss: 1.4224 - val_loss: 2.7008 - val_activation_loss: 1.3598 - val_activation_1_loss: 1.3471\n",
      "Train on 20625 samples, validate on 5157 samples\n",
      "Epoch 3/3\n",
      "20625/20625 [==============================] - 87s 4ms/sample - loss: 2.8049 - activation_loss: 1.4103 - activation_1_loss: 1.3966 - val_loss: 2.7375 - val_activation_loss: 1.3830 - val_activation_1_loss: 1.3596\n",
      "Loading model...\n",
      "Predicting OOF...\n",
      "5157/5157 [==============================] - 16s 3ms/sample\n",
      "Predicting all TRAIN for outlier analysis...\n",
      "25782/25782 [==============================] - 60s 2ms/sample\n",
      "Predicting TEST...\n",
      "3534/3534 [==============================] - 9s 2ms/sample\n",
      ">>>> FOLD 1 Jaccard = 0.7423914403296816\n",
      "\n",
      "#########################\n",
      "### FOLD 2\n",
      "#########################\n",
      "Train on 20625 samples, validate on 5157 samples\n",
      "20625/20625 [==============================] - 102s 5ms/sample - loss: 3.2770 - activation_loss: 1.6327 - activation_1_loss: 1.6443 - val_loss: 2.8413 - val_activation_loss: 1.4328 - val_activation_1_loss: 1.4163\n",
      "Train on 20625 samples, validate on 5157 samples\n",
      "Epoch 2/2\n",
      "20625/20625 [==============================] - 82s 4ms/sample - loss: 2.8365 - activation_loss: 1.4181 - activation_1_loss: 1.4198 - val_loss: 2.7738 - val_activation_loss: 1.4050 - val_activation_1_loss: 1.3770\n",
      "Train on 20625 samples, validate on 5157 samples\n",
      "Epoch 3/3\n",
      "20625/20625 [==============================] - 80s 4ms/sample - loss: 2.7263 - activation_loss: 1.3680 - activation_1_loss: 1.3587 - val_loss: 2.7356 - val_activation_loss: 1.3815 - val_activation_1_loss: 1.3615\n",
      "Loading model...\n",
      "Predicting OOF...\n",
      "5157/5157 [==============================] - 15s 3ms/sample\n",
      "Predicting all TRAIN for outlier analysis...\n",
      "25782/25782 [==============================] - 60s 2ms/sample\n",
      "Predicting TEST...\n",
      "3534/3534 [==============================] - 8s 2ms/sample\n",
      ">>>> FOLD 2 Jaccard = 0.736493141561854\n",
      "\n",
      "#########################\n",
      "### FOLD 3\n",
      "#########################\n",
      "Train on 20626 samples, validate on 5156 samples\n",
      "20626/20626 [==============================] - 98s 5ms/sample - loss: 3.2414 - activation_loss: 1.6137 - activation_1_loss: 1.6263 - val_loss: 2.7313 - val_activation_loss: 1.3772 - val_activation_1_loss: 1.3621\n",
      "Train on 20626 samples, validate on 5156 samples\n",
      "Epoch 2/2\n",
      "20626/20626 [==============================] - 81s 4ms/sample - loss: 2.8205 - activation_loss: 1.4145 - activation_1_loss: 1.4070 - val_loss: 2.7190 - val_activation_loss: 1.3629 - val_activation_1_loss: 1.3631\n",
      "Train on 20626 samples, validate on 5156 samples\n",
      "Epoch 3/3\n",
      "20626/20626 [==============================] - 88s 4ms/sample - loss: 2.7884 - activation_loss: 1.3964 - activation_1_loss: 1.3931 - val_loss: 2.7094 - val_activation_loss: 1.3607 - val_activation_1_loss: 1.3549\n",
      "Loading model...\n",
      "Predicting OOF...\n",
      "5156/5156 [==============================] - 14s 3ms/sample\n",
      "Predicting all TRAIN for outlier analysis...\n",
      "25782/25782 [==============================] - 60s 2ms/sample\n",
      "Predicting TEST...\n",
      "3534/3534 [==============================] - 8s 2ms/sample\n",
      ">>>> FOLD 3 Jaccard = 0.7332598735422187\n",
      "\n",
      "#########################\n",
      "### FOLD 4\n",
      "#########################\n",
      "Train on 20626 samples, validate on 5156 samples\n",
      "20626/20626 [==============================] - 103s 5ms/sample - loss: 3.3117 - activation_loss: 1.6599 - activation_1_loss: 1.6496 - val_loss: 2.8098 - val_activation_loss: 1.4135 - val_activation_1_loss: 1.4023\n",
      "Train on 20626 samples, validate on 5156 samples\n",
      "Epoch 2/2\n",
      "20626/20626 [==============================] - 82s 4ms/sample - loss: 2.8477 - activation_loss: 1.4225 - activation_1_loss: 1.4241 - val_loss: 2.7273 - val_activation_loss: 1.3804 - val_activation_1_loss: 1.3531\n",
      "Train on 20626 samples, validate on 5156 samples\n",
      "Epoch 3/3\n",
      "20626/20626 [==============================] - 84s 4ms/sample - loss: 2.7414 - activation_loss: 1.3753 - activation_1_loss: 1.3690 - val_loss: 2.7424 - val_activation_loss: 1.3848 - val_activation_1_loss: 1.3631\n",
      "Loading model...\n",
      "Predicting OOF...\n",
      "5156/5156 [==============================] - 16s 3ms/sample\n",
      "Predicting all TRAIN for outlier analysis...\n",
      "25782/25782 [==============================] - 60s 2ms/sample\n",
      "Predicting TEST...\n",
      "3534/3534 [==============================] - 8s 2ms/sample\n",
      ">>>> FOLD 4 Jaccard = 0.729638400196351\n",
      "\n",
      "#########################\n",
      "### FOLD 5\n",
      "#########################\n",
      "Train on 20626 samples, validate on 5156 samples\n",
      "20626/20626 [==============================] - 101s 5ms/sample - loss: 3.2782 - activation_loss: 1.6383 - activation_1_loss: 1.6400 - val_loss: 2.8904 - val_activation_loss: 1.4442 - val_activation_1_loss: 1.4485\n",
      "Train on 20626 samples, validate on 5156 samples\n",
      "Epoch 2/2\n",
      "20626/20626 [==============================] - 80s 4ms/sample - loss: 2.8404 - activation_loss: 1.4251 - activation_1_loss: 1.4149 - val_loss: 2.8132 - val_activation_loss: 1.4138 - val_activation_1_loss: 1.4026\n",
      "Train on 20626 samples, validate on 5156 samples\n",
      "Epoch 3/3\n",
      "20626/20626 [==============================] - 86s 4ms/sample - loss: 2.7780 - activation_loss: 1.3979 - activation_1_loss: 1.3824 - val_loss: 2.7657 - val_activation_loss: 1.3968 - val_activation_1_loss: 1.3723\n",
      "Loading model...\n",
      "Predicting OOF...\n",
      "5156/5156 [==============================] - 14s 3ms/sample\n",
      "Predicting all TRAIN for outlier analysis...\n",
      "25782/25782 [==============================] - 61s 2ms/sample\n",
      "Predicting TEST...\n",
      "3534/3534 [==============================] - 8s 2ms/sample\n",
      ">>>> FOLD 5 Jaccard = 0.7183574174020708\n",
      "\n",
      "CPU times: user 17min 36s, sys: 2min 40s, total: 20min 17s\n",
      "Wall time: 30min 52s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "jac = []\n",
    "VER='v0'\n",
    "DISPLAY=1 # use display=1 for interactive\n",
    "\n",
    "oof_start = np.zeros((input_ids_train.shape[0],MAX_LEN))\n",
    "oof_end = np.zeros((input_ids_train.shape[0],MAX_LEN))\n",
    "\n",
    "preds_start_train = np.zeros((input_ids_train.shape[0],MAX_LEN))\n",
    "preds_end_train = np.zeros((input_ids_train.shape[0],MAX_LEN))\n",
    "\n",
    "preds_start = np.zeros((input_ids_test.shape[0],MAX_LEN))\n",
    "preds_end = np.zeros((input_ids_test.shape[0],MAX_LEN))\n",
    "\n",
    "skf = StratifiedKFold(n_splits=n_split,shuffle=True,random_state=SEED)\n",
    "\n",
    "for fold,(idxT,idxV) in enumerate(skf.split(input_ids_train,train.sentiment.values)):\n",
    "    print('#'*25)\n",
    "    print('### FOLD %i'%(fold+1))\n",
    "    print('#'*25)\n",
    "    \n",
    "    K.clear_session()\n",
    "    model, padded_model = build_model()\n",
    "        \n",
    "    #sv = tf.keras.callbacks.ModelCheckpoint(\n",
    "    #    '%s-roberta-%i.h5'%(VER,fold), monitor='val_loss', verbose=1, save_best_only=True,\n",
    "    #    save_weights_only=True, mode='auto', save_freq='epoch')\n",
    "    inpT = [input_ids_train[idxT,], attention_mask_train[idxT,], token_type_ids_train[idxT,]]\n",
    "    targetT = [start_tokens_train[idxT,], end_tokens_train[idxT,]]\n",
    "    inpV = [input_ids_train[idxV,],attention_mask_train[idxV,],token_type_ids_train[idxV,]]\n",
    "    targetV = [start_tokens_train[idxV,], end_tokens_train[idxV,]]\n",
    "    \n",
    "    # sort the validation data\n",
    "    shuffleV = np.int32(sorted(range(len(inpV[0])), key=lambda k: (inpV[0][k] == PAD_ID).sum(), reverse=True))\n",
    "    inpV = [arr[shuffleV] for arr in inpV]\n",
    "    targetV = [arr[shuffleV] for arr in targetV]\n",
    "    weight_fn = '%s-roberta-%i.h5'%(VER,fold)\n",
    "    for epoch in range(1, EPOCHS + 1):\n",
    "        # sort and shuffle: We add random numbers to not have the same order in each epoch\n",
    "        shuffleT = np.int32(sorted(range(len(inpT[0])), key=lambda k: (inpT[0][k] == PAD_ID).sum() + np.random.randint(-3, 3), reverse=True))\n",
    "        # shuffle in batches, otherwise short batches will always come in the beginning of each epoch\n",
    "        num_batches = math.ceil(len(shuffleT) / BATCH_SIZE)\n",
    "        batch_inds = np.random.permutation(num_batches)\n",
    "        shuffleT_ = []\n",
    "        for batch_ind in batch_inds:\n",
    "            shuffleT_.append(shuffleT[batch_ind * BATCH_SIZE: (batch_ind + 1) * BATCH_SIZE])\n",
    "        shuffleT = np.concatenate(shuffleT_)\n",
    "        # reorder the input data\n",
    "        inpT = [arr[shuffleT] for arr in inpT]\n",
    "        targetT = [arr[shuffleT] for arr in targetT]\n",
    "        model.fit(inpT, targetT, \n",
    "            epochs=epoch, initial_epoch=epoch - 1, batch_size=BATCH_SIZE, verbose=DISPLAY, callbacks=[],\n",
    "            validation_data=(inpV, targetV), shuffle=False)  # don't shuffle in `fit`\n",
    "        save_weights(model, weight_fn)\n",
    "\n",
    "    print('Loading model...')\n",
    "    # model.load_weights('%s-roberta-%i.h5'%(VER,fold))\n",
    "    load_weights(model, weight_fn)\n",
    "\n",
    "    print('Predicting OOF...')\n",
    "    oof_start[idxV,],oof_end[idxV,] = padded_model.predict([input_ids_train[idxV,],attention_mask_train[idxV,],token_type_ids_train[idxV,]],verbose=DISPLAY)\n",
    "    \n",
    "    print('Predicting all TRAIN for outlier analysis...')\n",
    "    preds_train = padded_model.predict([input_ids_train,attention_mask_train,token_type_ids_train],verbose=DISPLAY)\n",
    "    preds_start_train += preds_train[0]/skf.n_splits\n",
    "    preds_end_train += preds_train[1]/skf.n_splits\n",
    "\n",
    "    print('Predicting TEST...')\n",
    "    preds = padded_model.predict([input_ids_test,attention_mask_test,token_type_ids_test],verbose=DISPLAY)\n",
    "    preds_start += preds[0]/skf.n_splits\n",
    "    preds_end += preds[1]/skf.n_splits\n",
    "    \n",
    "    # display fold jaccard\n",
    "    all = []\n",
    "    for k in idxV:\n",
    "        a = np.argmax(oof_start[k,])\n",
    "        b = np.argmax(oof_end[k,])\n",
    "        if a>b: \n",
    "            st = train.loc[k,'text'] # improve CV/LB with better choice here\n",
    "        else:\n",
    "            text1 = \" \"+\" \".join(train.loc[k,'text'].split())\n",
    "            enc = tokenizer.encode(text1)\n",
    "            st = tokenizer.decode(enc.ids[a-2:b-1])\n",
    "        all.append(jaccard(st,train.loc[k,'selected_text']))\n",
    "    jac.append(np.mean(all))\n",
    "    print('>>>> FOLD %i Jaccard ='%(fold+1),np.mean(all))\n",
    "    print()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-10-22T07:40:50.995154Z",
     "iopub.status.busy": "2022-10-22T07:40:50.992457Z",
     "iopub.status.idle": "2022-10-22T07:40:50.997340Z",
     "shell.execute_reply": "2022-10-22T07:40:50.996891Z"
    },
    "papermill": {
     "duration": 2.546097,
     "end_time": "2022-10-22T07:40:50.997438",
     "exception": false,
     "start_time": "2022-10-22T07:40:48.451341",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "OVERALL 5Fold CV Jaccard = 0.7320280546064353\n",
      "[0.7423914403296816, 0.736493141561854, 0.7332598735422187, 0.729638400196351, 0.7183574174020708]\n"
     ]
    }
   ],
   "source": [
    "print('OVERALL 5Fold CV Jaccard =',np.mean(jac))\n",
    "print(jac) # Jaccard CVs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 2.570158,
     "end_time": "2022-10-22T07:40:56.168329",
     "exception": false,
     "start_time": "2022-10-22T07:40:53.598171",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## 4. Submission <a class=\"anchor\" id=\"4\"></a>\n",
    "\n",
    "[Back to Table of Contents](#0.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 2.829935,
     "end_time": "2022-10-22T07:41:01.643297",
     "exception": false,
     "start_time": "2022-10-22T07:40:58.813362",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Code from notebook https://www.kaggle.com/khoongweihao/tse2020-roberta-cnn-random-seed-distribution?scriptVersionId=34448972"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "_kg_hide-input": true,
    "execution": {
     "iopub.execute_input": "2022-10-22T07:41:06.921866Z",
     "iopub.status.busy": "2022-10-22T07:41:06.920561Z",
     "iopub.status.idle": "2022-10-22T07:41:07.408730Z",
     "shell.execute_reply": "2022-10-22T07:41:07.408267Z"
    },
    "papermill": {
     "duration": 3.227697,
     "end_time": "2022-10-22T07:41:07.408881",
     "exception": false,
     "start_time": "2022-10-22T07:41:04.181184",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "all = []\n",
    "for k in range(input_ids_test.shape[0]):\n",
    "    a = np.argmax(preds_start[k,])\n",
    "    b = np.argmax(preds_end[k,])\n",
    "    if a>b: \n",
    "        st = test.loc[k,'text']\n",
    "    else:\n",
    "        text1 = \" \"+\" \".join(test.loc[k,'text'].split())\n",
    "        enc = tokenizer.encode(text1)\n",
    "        st = tokenizer.decode(enc.ids[a-2:b-1])\n",
    "    all.append(st)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-10-22T07:41:13.202422Z",
     "iopub.status.busy": "2022-10-22T07:41:13.201507Z",
     "iopub.status.idle": "2022-10-22T07:41:13.310230Z",
     "shell.execute_reply": "2022-10-22T07:41:13.309328Z"
    },
    "papermill": {
     "duration": 3.287617,
     "end_time": "2022-10-22T07:41:13.310354",
     "exception": false,
     "start_time": "2022-10-22T07:41:10.022737",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>textID</th>\n",
       "      <th>text</th>\n",
       "      <th>sentiment</th>\n",
       "      <th>selected_text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2319</th>\n",
       "      <td>ced0bb0d25</td>\n",
       "      <td>I meant looking like a tiger - stupi...</td>\n",
       "      <td>negative</td>\n",
       "      <td>stupid</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>940</th>\n",
       "      <td>261a5bbd14</td>\n",
       "      <td>tell me what you think of Pride Pre...</td>\n",
       "      <td>neutral</td>\n",
       "      <td>tell me what you think of pride pre...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1271</th>\n",
       "      <td>66762b14cd</td>\n",
       "      <td>I need one of these! Was just thin...</td>\n",
       "      <td>neutral</td>\n",
       "      <td>i need one of these! was just think...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1284</th>\n",
       "      <td>a22d319e5f</td>\n",
       "      <td>- That Jasper clip is my first 'fav...</td>\n",
       "      <td>neutral</td>\n",
       "      <td>- that jasper clip is my first 'fav...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1493</th>\n",
       "      <td>b14b4d5209</td>\n",
       "      <td>haha i agree ! i am her test dummy....</td>\n",
       "      <td>positive</td>\n",
       "      <td>love it. she is magic!</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>836</th>\n",
       "      <td>105b761a2c</td>\n",
       "      <td>i dont even know now lenaaa  when y...</td>\n",
       "      <td>neutral</td>\n",
       "      <td>i dont even know now lenaaa when yo...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1505</th>\n",
       "      <td>762a44afb6</td>\n",
       "      <td>Mrs.Bates left</td>\n",
       "      <td>neutral</td>\n",
       "      <td>mrs.bates left</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1974</th>\n",
       "      <td>b4031d0753</td>\n",
       "      <td>Yay me!  But, what in particular?</td>\n",
       "      <td>neutral</td>\n",
       "      <td>yay me! but, what in particular?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1219</th>\n",
       "      <td>cea5b3f191</td>\n",
       "      <td>Best friend is leaving to go back to...</td>\n",
       "      <td>negative</td>\n",
       "      <td>sad</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1423</th>\n",
       "      <td>a9e176a76e</td>\n",
       "      <td>hey I didn`t get the email yet</td>\n",
       "      <td>neutral</td>\n",
       "      <td>hey i didn`t get the email yet</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          textID                                     text sentiment  \\\n",
       "2319  ced0bb0d25  I meant looking like a tiger - stupi...  negative   \n",
       "940   261a5bbd14   tell me what you think of Pride Pre...   neutral   \n",
       "1271  66762b14cd    I need one of these! Was just thin...   neutral   \n",
       "1284  a22d319e5f   - That Jasper clip is my first 'fav...   neutral   \n",
       "1493  b14b4d5209   haha i agree ! i am her test dummy....  positive   \n",
       "836   105b761a2c   i dont even know now lenaaa  when y...   neutral   \n",
       "1505  762a44afb6                           Mrs.Bates left   neutral   \n",
       "1974  b4031d0753        Yay me!  But, what in particular?   neutral   \n",
       "1219  cea5b3f191  Best friend is leaving to go back to...  negative   \n",
       "1423  a9e176a76e           hey I didn`t get the email yet   neutral   \n",
       "\n",
       "                                selected_text  \n",
       "2319                                   stupid  \n",
       "940    tell me what you think of pride pre...  \n",
       "1271   i need one of these! was just think...  \n",
       "1284   - that jasper clip is my first 'fav...  \n",
       "1493                   love it. she is magic!  \n",
       "836    i dont even know now lenaaa when yo...  \n",
       "1505                           mrs.bates left  \n",
       "1974         yay me! but, what in particular?  \n",
       "1219                                      sad  \n",
       "1423           hey i didn`t get the email yet  "
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test['selected_text'] = all\n",
    "test[['textID','selected_text']].to_csv('submission.csv',index=False)\n",
    "test.sample(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 2.569916,
     "end_time": "2022-10-22T07:41:18.461512",
     "exception": false,
     "start_time": "2022-10-22T07:41:15.891596",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## 5. Outlier analysis <a class=\"anchor\" id=\"5\"></a>\n",
    "\n",
    "[Back to Table of Contents](#0.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 2.646841,
     "end_time": "2022-10-22T07:41:23.815128",
     "exception": false,
     "start_time": "2022-10-22T07:41:21.168287",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Code from notebook https://www.kaggle.com/khoongweihao/tse2020-roberta-cnn-random-seed-distribution?scriptVersionId=34448972"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 2.513167,
     "end_time": "2022-10-22T07:41:28.869933",
     "exception": false,
     "start_time": "2022-10-22T07:41:26.356766",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "[Go to Top](#0)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  },
  "papermill": {
   "duration": 1935.00606,
   "end_time": "2022-10-22T07:41:31.726644",
   "environment_variables": {},
   "exception": null,
   "input_path": "__notebook__.ipynb",
   "output_path": "__notebook__.ipynb",
   "parameters": {},
   "start_time": "2022-10-22T07:09:16.720584",
   "version": "2.1.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}