6Final Project (Predict Future Sales).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "FSRh1nm6Phaw"
   },
   "source": [
    "# Final Project\n",
    "\n",
    "# *1C Company* - Predict Future Sales\n",
    "\n",
    "**Motivation**\n",
    "\n",
    "This challenge serves as final project for the Coursera course \"How to win a data science competition\". In this competition I will work with a challenging time-series dataset consisting of daily sales data. The sales data is kindly provided by one of the largest Russian software firms - 1C Company. \n",
    "\n",
    "The task is to **predict the total amount of products sold in every shop for the test set in the next month**. \n",
    "\n",
    "Submissions are evaluated by **root mean squared error** (RMSE). True target values are clipped into [0,20] range.\n",
    "\n",
    "Source: https://www.kaggle.com/c/competitive-data-science-predict-future-sales/overview\n",
    "\n",
    "## Table of contents\n",
    "\n",
    "###  [1. Importing data](#Getting1)\n",
    "###  [2. Data preprocessing and feature extraction](#Preprocessing2)\n",
    "&nbsp;&nbsp;&nbsp;[2.1 Data cleaning](#Clear21)  \n",
    "&nbsp;&nbsp;&nbsp;[2.2 Extracting categorical features from item names](#Extracting22)  \n",
    "&nbsp;&nbsp;&nbsp;[2.3 ID values of Train and test data](#Making23)   \n",
    "&nbsp;&nbsp;&nbsp;[2.4 Text features with TF-IDF](#Text24)      \n",
    "&nbsp;&nbsp;&nbsp;[2.5 Feature matrix](#Feature27)  \n",
    "###  [3. Exploratory data analysis](#Exploratory3) \n",
    "###  [4. Advanced feature engineering](#Advanced4)   \n",
    "&nbsp;&nbsp;&nbsp;[4.1 Matrix factorization on text processed features](#Matrix42)  \n",
    "&nbsp;&nbsp;&nbsp;[4.2 Mean encoding on categorical features](#Mean41)  \n",
    "&nbsp;&nbsp;&nbsp;[4.3 Last sales month](#Last)  \n",
    "&nbsp;&nbsp;&nbsp;[4.4 Creating lag features](#lag)  \n",
    "&nbsp;&nbsp;&nbsp;[4.5 Sale stats in recent](#stats)  \n",
    "&nbsp;&nbsp;&nbsp;[4.6 Consecutive_month_number](#consecutive)  \n",
    "&nbsp;&nbsp;&nbsp;[4.7 StandardScaler](#Simple43)   \n",
    "&nbsp;&nbsp;&nbsp;[4.8 Simple feature selection](#Simple48)  \n",
    "###  [5. Model training](#Model5)   \n",
    "&nbsp;&nbsp;&nbsp;[5.1 Validation scheme](#Validation51)   \n",
    "&nbsp;&nbsp;&nbsp;[5.2 Baseline](#XGBoost53)  \n",
    "&nbsp;&nbsp;&nbsp;[5.3 Feature importance](#Linear54)  \n",
    "###  [6. Ensembling and test predictions](#Ensembling6)   \n",
    "&nbsp;&nbsp;&nbsp;[6.1 Ensembling: XGBoost + Linear Regression](#Ensembling61)  \n",
    "&nbsp;&nbsp;&nbsp;[6.2 Leaderboard](#Leaderboard62)  \n",
    "###  [7. Conclusion](#Conclusion)       \n",
    "###  [8. Reference](#Reference)     \n",
    "&nbsp;&nbsp;&nbsp;  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "_cell_guid": "b1076dfc-b9ad-4769-8c92-a6c4dae69d19",
    "_uuid": "8f2839f25d086af736a60e9eeb907d3b93b6e0e5",
    "colab": {},
    "colab_type": "code",
    "id": "7OXETkyCPha0"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/brian/miniconda3/envs/data_science_env/lib/python3.7/site-packages/lightgbm/__init__.py:46: UserWarning: Starting from version 2.2.1, the library file in distribution wheels for macOS is built by the Apple Clang (Xcode_9.4.1) compiler.\n",
      "This means that in case of installing LightGBM from PyPI via the ``pip install lightgbm`` command, you don't need to install the gcc compiler anymore.\n",
      "Instead of that, you need to install the OpenMP library, which is required for running LightGBM on the system with the Apple Clang compiler.\n",
      "You can install the OpenMP library by the following command: ``brew install libomp``.\n",
      "  \"You can install the OpenMP library by the following command: ``brew install libomp``.\", UserWarning)\n"
     ]
    }
   ],
   "source": [
    "# This Python 3 environment comes with many helpful analytics libraries installed\n",
    "# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n",
    "\n",
    "import numpy as np # linear algebra\n",
    "import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n",
    "import matplotlib.pyplot as plt\n",
    "plt.style.use('fivethirtyeight')\n",
    "%matplotlib inline\n",
    "\n",
    "import seaborn as sns\n",
    "\n",
    "import pickle\n",
    "import gc\n",
    "import os\n",
    "import time\n",
    "from tqdm import tqdm\n",
    "\n",
    "from sklearn.preprocessing import LabelEncoder\n",
    "from sklearn.feature_extraction import text\n",
    "from sklearn.model_selection import KFold #StratifiedKFold\n",
    "from sklearn.decomposition import NMF #PCA, TruncatedSVD, \n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.metrics import r2_score, mean_squared_error\n",
    "from sklearn.linear_model import LinearRegression, Ridge, Lasso, SGDRegressor\n",
    "from sklearn.ensemble import RandomForestRegressor\n",
    "from catboost import CatBoostRegressor\n",
    "import xgboost as xgb\n",
    "import lightgbm as lgb\n",
    "\n",
    "pd.set_option('display.max_columns', None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "KKQ2GDcXPhbI"
   },
   "source": [
    "# Part 1/8: Importing data<a name=\"Getting1\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 74
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 3435,
     "status": "ok",
     "timestamp": 1561965839300,
     "user": {
      "displayName": "Shengbing YU",
      "photoUrl": "",
      "userId": "03994843105687918628"
     },
     "user_tz": -600
    },
    "id": "eVP6GZGAPhbM",
    "outputId": "534376fb-48a7-4c7d-adc1-6ba9ed15e3c1"
   },
   "source": [
    "# mount my google drive, only used when i use colab to do computation\n",
    "from google.colab import drive\n",
    "drive.mount('/content/drive')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "_cell_guid": "79c7e3d0-c299-4dcb-8224-4455121ee9b0",
    "_uuid": "d629ff2d2480ee46fbb7e2d37f6b5fab8052498a",
    "colab": {},
    "colab_type": "code",
    "id": "xcHDYiH3PhbX"
   },
   "outputs": [],
   "source": [
    "#DATA_FOLDER = '../input/'  # kaggle data folder\n",
    "DATA_FOLDER = '../readonly/final_project_data/' # data folder on my Mac Pro\n",
    "# data folder on google colab\n",
    "#DATA_FOLDER = './drive/My Drive/Colab Notebooks/final_project_data/'  \n",
    "\n",
    "sales_train    = pd.read_csv(os.path.join(DATA_FOLDER, 'sales_train.csv'))\n",
    "items          = pd.read_csv(os.path.join(DATA_FOLDER, 'items.csv'))\n",
    "test           = pd.read_csv(os.path.join(DATA_FOLDER, 'test.csv'))\n",
    "item_categories = pd.read_csv(os.path.join(DATA_FOLDER, 'item_categories.csv'))\n",
    "shops          = pd.read_csv(os.path.join(DATA_FOLDER, 'shops.csv'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 204
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 7327,
     "status": "ok",
     "timestamp": 1561965843322,
     "user": {
      "displayName": "Shengbing YU",
      "photoUrl": "",
      "userId": "03994843105687918628"
     },
     "user_tz": -600
    },
    "id": "QbTJ8H6xPhbd",
    "outputId": "779c5629-117f-445f-f9b2-8f9170ce9624"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>date</th>\n",
       "      <th>date_block_num</th>\n",
       "      <th>shop_id</th>\n",
       "      <th>item_id</th>\n",
       "      <th>item_price</th>\n",
       "      <th>item_cnt_day</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>02.01.2013</td>\n",
       "      <td>0</td>\n",
       "      <td>59</td>\n",
       "      <td>22154</td>\n",
       "      <td>999.00</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>03.01.2013</td>\n",
       "      <td>0</td>\n",
       "      <td>25</td>\n",
       "      <td>2552</td>\n",
       "      <td>899.00</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>05.01.2013</td>\n",
       "      <td>0</td>\n",
       "      <td>25</td>\n",
       "      <td>2552</td>\n",
       "      <td>899.00</td>\n",
       "      <td>-1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>06.01.2013</td>\n",
       "      <td>0</td>\n",
       "      <td>25</td>\n",
       "      <td>2554</td>\n",
       "      <td>1709.05</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>15.01.2013</td>\n",
       "      <td>0</td>\n",
       "      <td>25</td>\n",
       "      <td>2555</td>\n",
       "      <td>1099.00</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         date  date_block_num  shop_id  item_id  item_price  item_cnt_day\n",
       "0  02.01.2013               0       59    22154      999.00           1.0\n",
       "1  03.01.2013               0       25     2552      899.00           1.0\n",
       "2  05.01.2013               0       25     2552      899.00          -1.0\n",
       "3  06.01.2013               0       25     2554     1709.05           1.0\n",
       "4  15.01.2013               0       25     2555     1099.00           1.0"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sales_train.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 391
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 8885,
     "status": "ok",
     "timestamp": 1561965844953,
     "user": {
      "displayName": "Shengbing YU",
      "photoUrl": "",
      "userId": "03994843105687918628"
     },
     "user_tz": -600
    },
    "id": "SP7MRm2XPhb0",
    "outputId": "05ad1494-a042-4d63-d91a-09ee6423b6f6"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shape: \n",
      " (2935849, 6)\n",
      "\n",
      "\n",
      "dtypes: \n",
      " date               object\n",
      "date_block_num      int64\n",
      "shop_id             int64\n",
      "item_id             int64\n",
      "item_price        float64\n",
      "item_cnt_day      float64\n",
      "dtype: object\n",
      "\n",
      "\n",
      "number of unique value: \n",
      " date               1034\n",
      "date_block_num       34\n",
      "shop_id              60\n",
      "item_id           21807\n",
      "item_price        19993\n",
      "item_cnt_day        198\n",
      "dtype: int64\n"
     ]
    }
   ],
   "source": [
    "#Check shape\n",
    "print('shape: \\n', sales_train.shape)\n",
    "print('\\n')\n",
    "print('dtypes: \\n', sales_train.dtypes)\n",
    "print('\\n')\n",
    "print('number of unique value: \\n', sales_train.nunique())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 297
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 9602,
     "status": "ok",
     "timestamp": 1561965845726,
     "user": {
      "displayName": "Shengbing YU",
      "photoUrl": "",
      "userId": "03994843105687918628"
     },
     "user_tz": -600
    },
    "id": "7qtb_s8KPhcX",
    "outputId": "85991ea4-67d4-4fc6-8678-619fee39c447"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>date_block_num</th>\n",
       "      <th>shop_id</th>\n",
       "      <th>item_id</th>\n",
       "      <th>item_price</th>\n",
       "      <th>item_cnt_day</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>2.935849e+06</td>\n",
       "      <td>2.935849e+06</td>\n",
       "      <td>2.935849e+06</td>\n",
       "      <td>2.935849e+06</td>\n",
       "      <td>2.935849e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>1.456991e+01</td>\n",
       "      <td>3.300173e+01</td>\n",
       "      <td>1.019723e+04</td>\n",
       "      <td>8.908532e+02</td>\n",
       "      <td>1.242641e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>9.422988e+00</td>\n",
       "      <td>1.622697e+01</td>\n",
       "      <td>6.324297e+03</td>\n",
       "      <td>1.729800e+03</td>\n",
       "      <td>2.618834e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>-1.000000e+00</td>\n",
       "      <td>-2.200000e+01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>7.000000e+00</td>\n",
       "      <td>2.200000e+01</td>\n",
       "      <td>4.476000e+03</td>\n",
       "      <td>2.490000e+02</td>\n",
       "      <td>1.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>1.400000e+01</td>\n",
       "      <td>3.100000e+01</td>\n",
       "      <td>9.343000e+03</td>\n",
       "      <td>3.990000e+02</td>\n",
       "      <td>1.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>2.300000e+01</td>\n",
       "      <td>4.700000e+01</td>\n",
       "      <td>1.568400e+04</td>\n",
       "      <td>9.990000e+02</td>\n",
       "      <td>1.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>3.300000e+01</td>\n",
       "      <td>5.900000e+01</td>\n",
       "      <td>2.216900e+04</td>\n",
       "      <td>3.079800e+05</td>\n",
       "      <td>2.169000e+03</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       date_block_num       shop_id       item_id    item_price  item_cnt_day\n",
       "count    2.935849e+06  2.935849e+06  2.935849e+06  2.935849e+06  2.935849e+06\n",
       "mean     1.456991e+01  3.300173e+01  1.019723e+04  8.908532e+02  1.242641e+00\n",
       "std      9.422988e+00  1.622697e+01  6.324297e+03  1.729800e+03  2.618834e+00\n",
       "min      0.000000e+00  0.000000e+00  0.000000e+00 -1.000000e+00 -2.200000e+01\n",
       "25%      7.000000e+00  2.200000e+01  4.476000e+03  2.490000e+02  1.000000e+00\n",
       "50%      1.400000e+01  3.100000e+01  9.343000e+03  3.990000e+02  1.000000e+00\n",
       "75%      2.300000e+01  4.700000e+01  1.568400e+04  9.990000e+02  1.000000e+00\n",
       "max      3.300000e+01  5.900000e+01  2.216900e+04  3.079800e+05  2.169000e+03"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sales_train.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ojhJFVc8Phcx"
   },
   "source": [
    "#### Note\n",
    "Here we can see min item_price is negative. It is strange and need to deal with."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 204
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 9545,
     "status": "ok",
     "timestamp": 1561965845727,
     "user": {
      "displayName": "Shengbing YU",
      "photoUrl": "",
      "userId": "03994843105687918628"
     },
     "user_tz": -600
    },
    "id": "s99K3JdIPhcy",
    "outputId": "021d259e-00b7-4fde-b598-6ad95640d938"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>shop_id</th>\n",
       "      <th>item_id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>5037</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>5320</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>5</td>\n",
       "      <td>5233</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>5</td>\n",
       "      <td>5232</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>5268</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   ID  shop_id  item_id\n",
       "0   0        5     5037\n",
       "1   1        5     5320\n",
       "2   2        5     5233\n",
       "3   3        5     5232\n",
       "4   4        5     5268"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 289
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 9487,
     "status": "ok",
     "timestamp": 1561965845729,
     "user": {
      "displayName": "Shengbing YU",
      "photoUrl": "",
      "userId": "03994843105687918628"
     },
     "user_tz": -600
    },
    "id": "qKcS4sw2Phc9",
    "outputId": "a7dd1d84-ee3c-41d8-ab43-538f1bbfa586"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shape: \n",
      " (214200, 3)\n",
      "\n",
      "\n",
      "dtypes: \n",
      " ID         int64\n",
      "shop_id    int64\n",
      "item_id    int64\n",
      "dtype: object\n",
      "\n",
      "\n",
      "number of unique value: \n",
      " ID         214200\n",
      "shop_id        42\n",
      "item_id      5100\n",
      "dtype: int64\n"
     ]
    }
   ],
   "source": [
    "print('shape: \\n', test.shape)\n",
    "print('\\n')\n",
    "print('dtypes: \\n', test.dtypes)\n",
    "print('\\n')\n",
    "print('number of unique value: \\n', test.nunique())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 204
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 9427,
     "status": "ok",
     "timestamp": 1561965845732,
     "user": {
      "displayName": "Shengbing YU",
      "photoUrl": "",
      "userId": "03994843105687918628"
     },
     "user_tz": -600
    },
    "id": "66yFo8qGPhdE",
    "outputId": "58639faa-005d-46f5-a3c2-f341637b9283"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>item_name</th>\n",
       "      <th>item_id</th>\n",
       "      <th>item_category_id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>! ВО ВЛАСТИ НАВАЖДЕНИЯ (ПЛАСТ.)         D</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>!ABBYY FineReader 12 Professional Edition Full...</td>\n",
       "      <td>1</td>\n",
       "      <td>76</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>***В ЛУЧАХ СЛАВЫ   (UNV)                    D</td>\n",
       "      <td>2</td>\n",
       "      <td>40</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>***ГОЛУБАЯ ВОЛНА  (Univ)                      D</td>\n",
       "      <td>3</td>\n",
       "      <td>40</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>***КОРОБКА (СТЕКЛО)                       D</td>\n",
       "      <td>4</td>\n",
       "      <td>40</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                           item_name  item_id  \\\n",
       "0          ! ВО ВЛАСТИ НАВАЖДЕНИЯ (ПЛАСТ.)         D        0   \n",
       "1  !ABBYY FineReader 12 Professional Edition Full...        1   \n",
       "2      ***В ЛУЧАХ СЛАВЫ   (UNV)                    D        2   \n",
       "3    ***ГОЛУБАЯ ВОЛНА  (Univ)                      D        3   \n",
       "4        ***КОРОБКА (СТЕКЛО)                       D        4   \n",
       "\n",
       "   item_category_id  \n",
       "0                40  \n",
       "1                76  \n",
       "2                40  \n",
       "3                40  \n",
       "4                40  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "items.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 289
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 9368,
     "status": "ok",
     "timestamp": 1561965845733,
     "user": {
      "displayName": "Shengbing YU",
      "photoUrl": "",
      "userId": "03994843105687918628"
     },
     "user_tz": -600
    },
    "id": "D8v4tjV9PhdX",
    "outputId": "dc533341-f77a-4e27-9326-28d648f300b7"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shape: \n",
      " (22170, 3)\n",
      "\n",
      "\n",
      "dtypes: \n",
      " item_name           object\n",
      "item_id              int64\n",
      "item_category_id     int64\n",
      "dtype: object\n",
      "\n",
      "\n",
      "number of unique value: \n",
      " item_name           22170\n",
      "item_id             22170\n",
      "item_category_id       84\n",
      "dtype: int64\n"
     ]
    }
   ],
   "source": [
    "print('shape: \\n', items.shape)\n",
    "print('\\n')\n",
    "print('dtypes: \\n', items.dtypes)\n",
    "print('\\n')\n",
    "print('number of unique value: \\n', items.nunique())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 204
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 9322,
     "status": "ok",
     "timestamp": 1561965845735,
     "user": {
      "displayName": "Shengbing YU",
      "photoUrl": "",
      "userId": "03994843105687918628"
     },
     "user_tz": -600
    },
    "id": "BjvqLUxRPhda",
    "outputId": "98931770-8985-4100-e577-7068b98077d4"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>item_category_name</th>\n",
       "      <th>item_category_id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>PC - Гарнитуры/Наушники</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Аксессуары - PS2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Аксессуары - PS3</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Аксессуары - PS4</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Аксессуары - PSP</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        item_category_name  item_category_id\n",
       "0  PC - Гарнитуры/Наушники                 0\n",
       "1         Аксессуары - PS2                 1\n",
       "2         Аксессуары - PS3                 2\n",
       "3         Аксессуары - PS4                 3\n",
       "4         Аксессуары - PSP                 4"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "item_categories.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 255
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 9276,
     "status": "ok",
     "timestamp": 1561965845737,
     "user": {
      "displayName": "Shengbing YU",
      "photoUrl": "",
      "userId": "03994843105687918628"
     },
     "user_tz": -600
    },
    "id": "JWdwjEOGPhde",
    "outputId": "60b988ba-c0ed-4051-e8b6-c25e203063b0"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shape: \n",
      " (84, 2)\n",
      "\n",
      "\n",
      "dtypes: \n",
      " item_category_name    object\n",
      "item_category_id       int64\n",
      "dtype: object\n",
      "\n",
      "\n",
      "number of unique value: \n",
      " item_category_name    84\n",
      "item_category_id      84\n",
      "dtype: int64\n"
     ]
    }
   ],
   "source": [
    "print('shape: \\n', item_categories.shape)\n",
    "print('\\n')\n",
    "print('dtypes: \\n', item_categories.dtypes)\n",
    "print('\\n')\n",
    "print('number of unique value: \\n', item_categories.nunique())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 204
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 9224,
     "status": "ok",
     "timestamp": 1561965845738,
     "user": {
      "displayName": "Shengbing YU",
      "photoUrl": "",
      "userId": "03994843105687918628"
     },
     "user_tz": -600
    },
    "id": "xrSg-WQ5Phdj",
    "outputId": "b65b79a0-dcaa-4bee-d8e6-084a18136903"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>shop_name</th>\n",
       "      <th>shop_id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>!Якутск Орджоникидзе, 56 фран</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>!Якутск ТЦ \"Центральный\" фран</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Адыгея ТЦ \"Мега\"</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Балашиха ТРК \"Октябрь-Киномир\"</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Волжский ТЦ \"Волга Молл\"</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                        shop_name  shop_id\n",
       "0   !Якутск Орджоникидзе, 56 фран        0\n",
       "1   !Якутск ТЦ \"Центральный\" фран        1\n",
       "2                Адыгея ТЦ \"Мега\"        2\n",
       "3  Балашиха ТРК \"Октябрь-Киномир\"        3\n",
       "4        Волжский ТЦ \"Волга Молл\"        4"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "shops.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 255
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 9166,
     "status": "ok",
     "timestamp": 1561965845739,
     "user": {
      "displayName": "Shengbing YU",
      "photoUrl": "",
      "userId": "03994843105687918628"
     },
     "user_tz": -600
    },
    "id": "9UyPzzIPPhdo",
    "outputId": "281f422b-cd7a-4983-a32c-9fc752f86289"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shape: \n",
      " (60, 2)\n",
      "\n",
      "\n",
      "dtypes: \n",
      " shop_name    object\n",
      "shop_id       int64\n",
      "dtype: object\n",
      "\n",
      "\n",
      "number of unique value: \n",
      " shop_name    60\n",
      "shop_id      60\n",
      "dtype: int64\n"
     ]
    }
   ],
   "source": [
    "print('shape: \\n', shops.shape)\n",
    "print('\\n')\n",
    "print('dtypes: \\n', shops.dtypes)\n",
    "print('\\n')\n",
    "print('number of unique value: \\n', shops.nunique())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "J4Fuv5-MPhdr"
   },
   "source": [
    "# Part 2/8: Data preprocessing and feature extraction<a name=\"Preprocessing2\"></a>\n",
    "\n",
    "## 2.1. Data cleaning<a name=\"Clear21\"></a>\n",
    "\n",
    "As there are some strange numbers in data set, we need to clear data first."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 296
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 10625,
     "status": "ok",
     "timestamp": 1561965847267,
     "user": {
      "displayName": "Shengbing YU",
      "photoUrl": "",
      "userId": "03994843105687918628"
     },
     "user_tz": -600
    },
    "id": "TtyYjJuFPhdr",
    "outputId": "cbb247f3-8614-45bd-a12b-ce0c1f5ff54e"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x12c6563c8>"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYYAAAEGCAYAAABhMDI9AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAPvElEQVR4nO3df2xV533H8c/XuIkhTQwmoQmZGYaZKYmmbmm6rV1UT9ijTbSpTBVOVbXNtC1ByiIVSPODkTGybCOdEiJNiCmKmsms6QrdFtb90YliVqfbSgxEIY0TZU5MMyflh+rLvMCKifF3f5xzL/d7fa+xAR/bl/dLuvLxOc95nuc8Or6fe865PsfcXQAA5NVMdQcAANMLwQAACAgGAEBAMAAAAoIBABDUVlowODjI15UAoMrV19db6TyOGAAAAcEAAAgyD4be3t6sm6wKjNuFYdwuDON2Yapl3DhiAAAEBAMAICAYAAABwQAACAgGAEBAMAAAAoIBABAQDACAgGAAAAQEAwAgIBgAAAHBAAAICAYAQEAwAAACggEAEBAMAICAYAAABAQDACAgGAAAQW1WDa1cuTL8vmvXrqyaBgBMAEcMAICAYAAABAQDACAgGAAAAcEAAAgIBgBAQDAAAAKCAQAQEAwAgIBgAAAEBAMAICAYAAABwQAACAgGAEBAMAAAAoIBABAQDACAgGAAAAQEAwAgIBgAAAHBAAAICAYAQEAwAAACggEAEBAMAICAYAAABAQDACAgGAAAAcEAAAgIBgBAQDAAAAKCAQAQEAwAgIBgAAAEBAMAICAYAAABwQAACAgGAEBAMAAAAoIBABAQDACAgGAAAAQEAwAgIBgAAAHBAAAICAYAQEAwAAACggEAEBAMAICAYAAABAQDACAgGAAAAcEAAAgIBgBAQDAAAAKCAQAQEAwAgIBgAAAEBAMAICAYAAABwQAACAgGAEBAMAAAAoIBABAQDACAgGAAAAQEAwAgIBgAAAHBAAAICAYAQEAwAAACggEAEBAMAICAYAAABAQDACAgGAAAAcEAAAgIBgBAQDAAAAKCAQAQEAwAgIBgAAAEtZNZeS6X0+bNm3X27NlRy1auXDmZTUuSVq1apUOHDun06dM6fvy45s6dq2PHjmn16tXavn277rvvPm3dulWSdP/992vbtm16+OGHtWPHDj344IOaN2+ecrmcHn/8cR09elSbN2/WyMiINmzYoIULF+rRRx+Vu+vJJ58slC+nr69PGzZs0HXXXae6ujqtX7++Ytm8XC5XqHffvn165plntGDBAs2dO1fr16+Xu+uJJ56QpHHVV1rvPffco2effTb0u7jN8dY3neVyOW3btk0bN26siu0BpGz+Tmdt2rSp7IKhoaHyCyago6ND3d3dOnHixMVWdUFef/11DQwMaHBwUMPDwzp58qQk6eDBg/rggw/U3d2tM2fOaHh4WN3d3RoaGtL+/ft15MgRnT59Wrfddps6Ojp04MABDQ8Pq6enR/v27dPAwIBOnDihoaGhwrx8+XI2btxY6MfAwICGhoYqls3r6Ogo1Ltz505J0qlTpwrr9/T06KWXXhp3faX19vT0qK+vL/S7uM3x1jeddXR06ODBg1WzPVnK5XKaP3/+VHdjxsli3C7132ldXd1jpfMm7VRSLpdTZ2fnZFV/UdxdkjQ8PFyYl58+deqU3F179+7V4cOHtWfPnkKZ/v5+9ff3F37fs2ePOjs7C+XLBWBfX19YR5I6OzvHDMtcLqe9e/fK3bV79+5Cf4vbLe7X+eorV29/f3/od/GyStsyk1Tb9gBSdvv1pAXDzp07wxvvTDMyMqItW7aMuQ3Dw8OF5SMjI9qxY8eoMk8//XTZ9cqVzdu5c6dGRkYK9Y7V7njqK1dvXr7fpW2Op77prNq2B5Cy268nLRi6urpGfdKdSYaHh0d90i+n+Oijq6tr1PJydbh72bJ5XV1dEwrV89U3Vr35fhcvq7QtM0m1bQ8gZbdfT1owtLS0yMwmq/pJV1tbq8bGxvOWy29jbW2tWlpaRi0vV4eZlS2b19LSotra8X8v4Hz1jVVvvt/Fyypty0xSbdsDSNnt15MWDO3t7RN6c5tuampqtG7dujG3oba2trC8pqZGd91116gya9euLbteubJ57e3tqqmpKdQ7Vrvjqa9cvXn5fpe2OZ76prNq2x5Aym6/nrRgaGhoUGtr62RVf1GKP+Xn5aevuuoqmZmWL1+upqYmtbW1Fco0NjaGI4C2tja1trYWypf76tiSJUtGHTW0traO+TWzhoYGLV++XGamFStWjDryamtrC/06X33l6m1sbAz9Ll5WaVtmkmrbHkDKbr+e1H9wa29vV3Nzs5YsWTKZzVS0atUqLVu2TIsWLVJdXZ2uv/56mZlWr16tOXPmaM2aNaqrq1NdXZ3WrFmjOXPm6KGHHtJNN91USOL29nY1NTVp9uzZeuCBB7R27VrNnj1bS5cuLXzSLi5fTn6dRYsWadmyZeP+dJ+v995775UkLViwoLB+e3u7li1bNu76Sutdt27dqH6PZ1tmkvb2di1evLhqtgeQsvk7tUoXiAcHBy/plePSf2jbtWvXpay+6vX29qq5uXmquzHjMG4XhnG7MDNx3Orr60ddDOaWGACAgGAAAAQEAwAgIBgAAAHBAAAICAYAQEAwAAACggEAEBAMAICAYAAABAQDACAgGAAAAcEAAAgIBgBAQDAAAAKCAQAQEAwAgIBgAAAEBAMAICAYAAABwQAACAgGAEBAMAAAAoIBABAQDACAgGAAAAQEAwAgIBgAAAHBAAAICAYAQEAwAAACggEAEBAMAICAYAAABAQDACAgGAAAAcEAAAgIBgBAQDAAAAKCAQAQEAwAgIBgAAAEBAMAICAYAAABwQAACAgGAEBAMAAAAoIBABAQDACAgGAAAAQEAwAgIBgAAAHBAAAICAYAQEAwAAACggEAEBAMAICAYAAABAQDACAgGAAAAcEAAAgIBgBAQDAAAAKCAQAQEAwAgIBgAAAEBAMAICAYAAABwQAACAgGAEBAMAAAAoIBABAQDACAgGAAAAQEAwAgIBgAAAHBAAAICAYAQEAwAAACggEAEBAMAICgNquGdu3aJUnq7e1Vc3NzVs0CACaIIwYAQEAwAAACggEAEBAMAICAYAAABAQDACAgGAAAAcEAAAgIBgBAQDAAAAKCAQAQEAwAgIBgAAAEBAMAICAYAAABwQAACAgGAEBAMAAAAoIBABCYu5ddMDg4WH4BAKBq1NfXW+k8jhgAAAHBAAAIKp5KAgBcnjhiAAAEmQWDmX3GzN40s7fM7JGs2p0pzOzHZvYjM3vFzA6k8xrM7Htm1pv+nFdUfn06lm+a2aenrufZMrPnzOy4mb1WNG/C42RmH0vH+y0z+2szG3UBrppUGLdNZvZeus+9YmZ3Fi1j3CSZWaOZ/ZuZvWFmPWb2lXR+de9z7j7pL0mzJL0taYmkKyQdknRzFm3PlJekH0u6tmTeX0l6JJ1+RNLX0umb0zG8UlJTOrazpnobMhqnT0m6VdJrFzNOkrolfUKSSfqupDumetumYNw2SfpqmbKM27mxuEHSren01ZL+Kx2fqt7nsjpi+FVJb7l7n7ufkfQtSZ/NqO2Z7LOSOtLpDkkri+Z/y92H3P2wpLeUjHHVc/cXJeVKZk9onMzsBknXuPsPPfmL3V60TlWqMG6VMG4pdz/i7i+n0+9LekPSjaryfS6rYLhRUn/R7++m83COS9ptZgfN7N503kfc/YiU7KCSFqTzGc9oouN0YzpdOv9ydL+ZvZqeasqfDmHcyjCzxZJ+RdJLqvJ9LqtgKHcuja9DRb/h7rdKukPSH5nZp8Yoy3iOT6VxYvwSfyNpqaRflnRE0lPpfMathJl9WNI/Slrj7v87VtEy82bc2GUVDO9Kaiz6/eck/SSjtmcEd/9J+vO4pBeUnBo6lh6CKv15PC3OeEYTHad30+nS+ZcVdz/m7mfdfUTSszp3OpJxK2JmH1ISCs+7+z+ls6t6n8sqGPZLajazJjO7QtLnJX0no7anPTO7ysyuzk9LWiHpNSVjdHda7G5J/5xOf0fS583sSjNrktSs5MLW5WpC45Qe+r9vZr+efjPky0XrXDbyb2yp31Wyz0mMW0G6nV+X9Ia7bylaVN37XIZX9+9UckX/bUkbpvqq+3R6Kfm21qH01ZMfH0nzJXVK6k1/NhStsyEdyzc1jb/dMAlj9fdKTnt8oORT2B9cyDhJuk3JG+HbkrYq/WfPan1VGLe/k/QjSa8qeUO7gXEbNW63Kznl86qkV9LXndW+z/GfzwCAgP98BgAEBAMAICAYAAABwQAACAgGAEBAMCBT6R0qf3Oq+5Gl9C6m35jqfgDjRTAgU+5+i7t/f6a9Wc60/gIXg2AAAAQEAzKVPpDotyX9saS7zOykmR1Kl9Wb2dfN7Ej6AJk/N7NZ6bLfM7P/MLOnzex/zKzPzD6Zzu9PH0Jz91htp/XMNrOnzOwdMxs0s39P5y02Mzezu83sv83sp2a2IV3nM+X6O0YbTWbWZWbvm9n3JF1bsvzbZnY0bf9FM7slnf9xMztmZrVFZT9nZq9MaJCBi0QwYCqclvSXkna4+4fd/aPp/A5Jw5J+QcntjVdI+sOi9X5Nya0J5kv6ppLnenw8Lf9FSVvTu2CO5UlJH5P0SUkNkh6SNFK0/HZJvyipVdJGM7vJ3f+1Qn8r+aakg0oC4XGdu6dO3neV3ENngaSXJT0vSe6+X9KApN8qKvtFJbeuADJDMGBaMLOPKLnl+Bp3P+XJXWafVnLDxbzD7v637n5W0g4ld7H8M08eirJb0hklIVGpjRpJvy/pK+7+nid3Fv1Pdx8qKvaYu//M3fP3rjpfCJS2sUhJWP1J2q8XJf1LcRl3f87d30/b3STpo2ZWny7uUBIGMrMGSZ9WEjRAZmrPXwTIxM9L+pCkI0WPwq1RfOjJsaLpn0nJraNL5o11xHCtpDolNzGr5GjR9P+dp75yFko64e6niua9o/RWzOmpsb+QtErSdTp3tHKtpEFJ35D0Rnrk0y7pB54+EAbICkcMmCqld2/slzSk5LnXc9PXNe5+yyVs86dKTmMtvYB1x3u3ySOS5qW3T89bVDT9BSWPf2yTVC9pcTrfJMnd35P0QyW3wf6SOI2EKUAwYKock7Q4Pb2j9FPxbklPmdk1ZlZjZkvNrOVSNejJA2mek7TFzBaa2Swz+4SZXTnR/o7RxjuSDkh6zMyuMLPbJf1OUZGrlQTggKQ5Sq5dlNqu5NrHLyl5aBOQKYIBU+Xb6c8BM3s5nf6ypCskvS7phKR/kHRDmXUvxleVPINgv6ScpK9pfH8H5fpbyReUXCjPSfpTJW/0eduVnFp6T8l27iuz/gtKTq29UHJKCsgEz2MApiEze1vSanffM9V9weWHIwZgmjGzzym5prF3qvuCyxPfSkLVMbMeJadiSq129+cvURsnKyy6w91/cBH1fl/SzZK+lF4TATLHqSQAQMCpJABAQDAAAAKCAQAQEAwAgIBgAAAEBAMAIPh/rILXYwnH5H8AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "sales_copy = sales_train.copy()\n",
    "sns.boxplot(sales_train.item_cnt_day)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 11188,
     "status": "ok",
     "timestamp": 1561965847887,
     "user": {
      "displayName": "Shengbing YU",
      "photoUrl": "",
      "userId": "03994843105687918628"
     },
     "user_tz": -600
    },
    "id": "TTBRY62TPhdv",
    "outputId": "d29c4341-6a41-4239-c2f9-abdce811c876"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[2169.0, 1000.0, 669.0, 637.0, 624.0]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sorted(sales_train.item_cnt_day,reverse=True)[0:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 136
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 11123,
     "status": "ok",
     "timestamp": 1561965847888,
     "user": {
      "displayName": "Shengbing YU",
      "photoUrl": "",
      "userId": "03994843105687918628"
     },
     "user_tz": -600
    },
    "id": "ypUFq6hlPhdy",
    "outputId": "2049c4dd-a938-4ce0-9ad1-aed7d890e16b"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "date              28.10.2015\n",
       "date_block_num            33\n",
       "shop_id                   12\n",
       "item_id                11373\n",
       "item_price          0.908714\n",
       "item_cnt_day            2169\n",
       "Name: 2909818, dtype: object"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sales_train.loc[sales_train.item_cnt_day.idxmax()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "EOdcbgWwPhd1"
   },
   "outputs": [],
   "source": [
    "# Use median value for outliers\n",
    "median = sales_train[(sales_train.shop_id==12)&(\n",
    "    sales_train.item_id==11373)&(sales_train.date_block_num==33)&(\n",
    "    sales_train.item_cnt_day<2000)].item_cnt_day.median()\n",
    "sales_train.loc[sales_train.item_cnt_day > 2000, 'item_cnt_day'] = round(median)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 296
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 12422,
     "status": "ok",
     "timestamp": 1561965849293,
     "user": {
      "displayName": "Shengbing YU",
      "photoUrl": "",
      "userId": "03994843105687918628"
     },
     "user_tz": -600
    },
    "id": "IyXCIsUrPhd4",
    "outputId": "9d4e91c4-a9ee-46dd-a760-916d0ad3371c"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x12c59f588>"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYYAAAEGCAYAAABhMDI9AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAANkUlEQVR4nO3df4xlZ1kH8O+znS4lAmsXkJSyUAolAYwiKAIFlrSJQANh/3FrIhF/m4hJaaCmFaOoRENtrQiJJEbkhyBdRDcmYELT4lKL4YfYQrGhWwqyhaZVp6xgkHXZ1z/u2WWe6eyw0r2zM9PPJ5nMue89557z7Dt7vvc9595zaowRADhqy6neAADWF8EAQCMYAGgEAwCNYACgWTjeEwcPHvRxJYBNbtu2bbW8zYgBgEYwANDMPRj2798/71WcUpu5vs1cW7K569vMtSWbu771UJsRAwCNYACgEQwANIIBgEYwANAIBgAawQBAIxgAaAQDAI1gAKARDAA0ggGARjAA0AgGABrBAEAjGABoBAMAjWAAoBEMADQL83zxXbt2tcd79+6d5+oAOAmMGABoBAMAjWAAoBEMADSCAYBGMADQCAYAGsEAQCMYAGgEAwCNYACgEQwANIIBgEYwANAIBgAawQBAIxgAaAQDAI1gAKARDAA0ggGARjAA0AgGABrBAEAjGABoBAMAjWAAoBEMADSCAYBGMADQCAYAGsEAQCMYAGgEAwCNYACgEQwANIIBgEYwANAIBgAawQBAIxgAaAQDAM3cgmHXrl0rtq3UDsD6YcQAQCMYAGgEAwCNYACgEQwANIIBgEYwANAIBgAawQBAIxgAaAQDAI1gAKARDAA0ggGARjAA0AgGABrBAEAjGABoBAMAjWAAoBEMADSCAYBGMADQCAYAGsEAQCMYAGgEAwCNYACgEQwANIIBgEYwANAIBgAawQBAIxgAaAQDAI1gAKARDAA0ggGARjAA0CycipXu2rXrVKx2VVu3bs2VV16Zc845J3feeWcuv/zyHDp0KEly2WWX5fzzz7/fMouLi3nLW96SM844I1dccUXOPPPMFV97cXExV111VS677LLjzgNwItZif2LEMDl06FCuvvrqJMk111xzLBSOPl7Jnj178uUvfzm33357rr322uO+9p49e3LbbbetOg/AiViL/YlgWOLAgQO58cYbc+DAgdZ++PDh3HTTTa1tcXEx119//bHH119/fe677777vebi4mJuuOGGjDFyww03rDgPwIlYq/2JYFjmzW9+84rty0cNe/bsyeHDh489Pnz48IoJvmfPnhw5ciRJcuTIEaMG4Hu2VvsTwbDM0p39au379u3LGOPY4zFG9u3bd7/l9u3bd2zZw4cPrzgPwIlYq/2JYFhmYWHl8/HL23fu3JmqOva4qrJz5877Lbdz585jyy4sLKw4D8CJWKv9iWBY5pJLLlmx/dJLL22Pd+/e3cJiYWEhF1988f2W2717d7Zsmf0zb9myZcV5AE7EWu1PBMMSO3bsyAte8ILs2LGjtS8sLNzv46rbt2/PhRdeeOzxhRdeuOJHx7Zv354LLrggVZULLrjAx1WB79la7U8Ew2Tr1q157Wtfm2Q2Oti6deux55aPFo7avXt3Hv/4x+cpT3nKqsm9e/fuPPWpTzVaAB6wtdif1NITqEsdPHhw5SdO0GpfYtu7d+8Deel1Zf/+/TnvvPNO9WbMxWauLdnc9W3m2pLNXd9a17Zt27Za3mbEAEAjGABoBAMAjWAAoBEMADSCAYBGMADQCAYAGsEAQCMYAGgEAwCNYACgEQwANIIBgEYwANAIBgAawQBAIxgAaAQDAI1gAKARDAA0ggGARjAA0AgGABrBAEAjGABoBAMAjWAAoBEMADSCAYBGMADQCAYAGsEAQCMYAGgEAwCNYACgEQwANHMLhr17967YtlI7AOuHEQMAjWAAoBEMADSCAYBGMADQCAYAGsEAQCMYAGgEAwCNYACgEQwANIIBgEYwANAIBgAawQBAIxgAaAQDAI1gAKARDAA0ggGARjAA0AgGABrBAEAjGABoBAMAjWAAoBEMADSCAYBGMADQCAYAGsEAQCMYAGgEAwCNYACgEQwANIIBgEYwANAIBgAawQBAIxgAaAQDAI1gAKARDAA0C/N88b1792b//v0577zz5rkaAE4iIwYAGsEAQCMYAGgEAwCNYACgEQwANIIBgEYwANAIBgAawQBAIxgAaAQDAI1gAKARDAA0ggGARjAA0AgGABrBAEAjGABoaoyx4hMHDx5c+QkANo1t27bV8jYjBgAawQBAc9xDSQA8OBkxANDMNRiq6iVV9fmquqOqLp/nuh6oqvpSVX22qm6uqk9Nbdur6rqq2j/9PnPJ/FdMdX2+ql68pP1Z0+vcUVV/UlU1tT+kqq6d2j9eVefMsZa3V9W9VXXrkrY1qaWqXjWtY39VvWoN63tDVX1l6r+bq+qijVhfVe2oqo9U1W1V9bmqumRq3/D9t0ptm6XvzqiqT1TVLVN9vzO1b7y+G2PM5SfJaUm+kOTcJFuT3JLkafNa30nY3i8ledSytiuTXD5NX57kTdP006Z6HpLkiVOdp03PfSLJc5NUkr9P8tKp/VeTvG2a/qkk186xlhcmeWaSW9eyliTbk9w5/T5zmj5zjep7Q5LXrTDvhqovyVlJnjlNPzzJ7VMNG77/Vqlts/RdJXnYNH16ko8nec5G7Lt5jhieneSOMcadY4xDSd6X5BVzXN88vCLJO6fpdybZtaT9fWOMb40xvpjkjiTPrqqzkjxijPFPY9Zb71q2zNHX+uskFx59F3CyjTE+mmTxFNTy4iTXjTEWxxj3JbkuyUvWqL7j2VD1jTHuHmN8epr+epLbkpydTdB/q9R2PBumtqmmMcb4xvTw9OlnZAP23TyD4ewkB5Y8viur/xGcaiPJh6vqn6vql6e2x4wx7k5mf9RJfmBqP15tZ0/Ty9vbMmOMw0kOJnnkHOo4nrWo5VT3+a9V1Wdqdqjp6HB9w9Y3HSb4kczeeW6q/ltWW7JJ+q6qTquqm5Pcm9mOekP23TyDYaV3w+v5I1DnjzGemeSlSV5dVS9cZd7j1bZazev13+Nk1nIqa/zTJE9K8owkdye5emrfkPVV1cOSfCDJa8YY/7XarMfZpnVb3wq1bZq+G2N8e4zxjCSPy+zd/w+uMvu6rW+ewXBXkh1LHj8uyVfnuL4HZIzx1en3vUn+NrNDYfdMw7pMv++dZj9ebXdN08vb2zJVtZBkW078cMjJsBa1nLI+H2PcM/2nPJLkzzLrv7aty7Zp3dZXVadntuN8zxjjb6bmTdF/K9W2mfruqDHG15L8Q2aHczZe353Mky9Lf5IsZHYC5In5zsnnp89rfQ9wW78vycOXTH9s6tA/TD9pdOU0/fT0k0Z35jsnjT6Z2QmnoyeNLpraX51+0mjPnGs6J/3k7NxryezE1xczO/l15jS9fY3qO2vJ9KWZHbvdcPVN2/KuJH+8rH3D998qtW2Wvnt0ku+fph+a5MYkL9uIfTe3HdO0sRdl9smDLyR5/TzX9QC389ypg25J8rmj25rZsbvrk+yffm9fsszrp7o+n+kTA1P7jya5dXrurfnOlwjPSPL+zE4wfSLJuXOs568yG5L/b2bvJH5hrWpJ8vNT+x1Jfm4N63t3ks8m+UySv0vf2WyY+pI8P7NDAJ9JcvP0c9Fm6L9VatssffdDSf5lquPWJL81tW+4vvPNZwAa33wGoBEMADSCAYBGMADQCAYAGsHAujNdmfJFp3o7/j+q6htVde6p3g44GXxclXWrqt6Q5MljjFee6m2BBxMjBngApssSwKYiGFh3anbTpJcl+Y0kF0+HaW6ZnttWVX9eVXdPN3d5Y1WdNj33s1V1U1VdU1Vfq6o7q+p5U/uBmt3c51UnsP53VNXbppuqfL2q9lXVE5Y8P6rq1VW1P7Nvsx5te/I0/dCqurqq/q2qDlbVP1bVQ6fnnlNVH5u275aNdsiMBwfBwHr1P0l+P7MbkTxsjPHDU/s7kxxO8uTMLtv8E0l+cclyP57ZJQkemeS9md0H5Mem+V+Z5K3T1T2/m59O8ntJHpXZpRves+z5XdO6nrbCslcleVaS52V2DZtfT3Kkqs5O8sEkb5zaX5fkA1X16BPYHlgzgoENo6oek9ll0V8zxvjvMbsS7jWZXUzsqC+OMf5ijPHtJNdmdsXJ3x2zm6F8OMmhzELiu/ngGOOjY4xvZXY9m+dW1dKrV/7BmN0U5ZvLtnFLZtesuWSM8ZUxu2rox6bXeWWSD40xPjTGODLGuC7JpzK7XhCsG46PspE8IbO7Yt295OZ3W9JvUHLPkulvJrPLOi9rO5ERw7HXHGN8o6oWkzx2SfuBFZeajTDOyOziZytt/09W1cuXtJ2e5CMnsD2wZgQD69nyj8wdSPKtzO7NfXjO6z42OpgOPW1Pv7798T7O9x+ZHQZ7UmZX613qQJJ3jzF+6SRuJ5x0DiWxnt2T5Jzp8EzG7LaIH05ydVU9oqq2VNWTqmrnHNZ9UVU9v6q2Znau4eNjjOONEo4Zs5vNvD3JH1XVY6dbPT63qh6S5C+TvLyqXjy1n1FVL6qqx63+qrC2BAPr2fun3/9ZVZ+epn8msxs//WuS+zK7IfpZc1j3e5P8dmZ3x3pWZiejT9TrMru/wCen5d+UZMsULK/I7NNW/57ZCOKy+H/IOuMLbrBMVb0jyV1jjN881dsCp4J3KgA0Tj7zoFRVn8vsU0LL/cpabwusNw4lAdA4lARAIxgAaAQDAI1gAKARDAA0ggGA5v8AYLsI1B8vWukAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.boxplot(sales_train.item_price)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 279
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 13154,
     "status": "ok",
     "timestamp": 1561965850083,
     "user": {
      "displayName": "Shengbing YU",
      "photoUrl": "",
      "userId": "03994843105687918628"
     },
     "user_tz": -600
    },
    "id": "1MirypNXPhd9",
    "outputId": "9e8ed3a7-b58f-4ee8-b3c5-944b267f283f"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x12c2a9198>"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAb8AAAD1CAYAAADAp/vMAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deZhc1X3n//e3etXS3WoJBEJgJKC9COzBKJFlcAyGsZHI+Cfin0lEZiyBmWBk+MU/z0NiSOzYHocZbCdxhjEGPxkwkjfAxDZKDAaFxeAELBGbTcKiGglJjTaQ1JtavVTVd/6o263q6ltVt7pvqRd9Xs/Tj6rOPefee2616ttnu9fcHRERkeNJYrxPQERE5FhT8BMRkeOOgp+IiBx3FPxEROS4o+AnIiLHnerxPoHx1tHRoemuIiJTXFNTk+W+V8tPRESOOwp+IiJy3FHwi0kymRzvU6ioqVw/1W3ymsr1m8p1g/GvX6TgZ2bLzGyrmbWa2U0h283Mbgu2v2hm55Uqa2azzWyDmSWDf5tztt0c5N9qZpfmpC82s5eCbbeZmQXpHzSzX5tZysw+npP/XDN7xsw2B+f1R+VfIhERmWpKBj8zqwJuB5YDi4ArzWxRXrblQEvwcy1wR4SyNwGPuXsL8FjwnmD7SuBsYBnwrWA/BPu9NudYy4L0ncBVwA/yzqsHWOXug/v6ezObVarOIiIytUWZ7bkEaHX3bQBmdi+wAtiSk2cFsM6zNwp91sxmmdk8YEGRsiuAi4Lya4Engc8F6fe6ex+w3cxagSVm9jrQ6O7PBPtaB1wOPOzurwdpmdwTd/dXc17vNrP9wIlAe4R6i4jExt3p7u4mk8mUzgzU19fT0dFR4bMaP3HXL5FIMHPmTIIOwZKiBL/5wK6c923A+yLkmV+i7EnuvgfA3feY2dycfT0bsq+B4HV+eiRmtgSoBV6LWkZEJC7d3d3U1dVRW1sbKX9dXR319fUVPqvxE3f9+vv76e7upqGhIVL+KMEvLIzmr40rlCdK2ajHG82+sjvMtkK/C6x294J/do11AHa8B3ArbSrXT3WbvCZL/err6znxxBPp7e2NXKacvJNR3PVrb29n7969Q+9bWloK5o0S/NqA03LenwrsjpintkjZfWY2L2j1zQP2l9hXW/C62HmMYGaNwM+Az7v7s8XyFrtQpSSTyaHy6Yzzzc3dPLuvn48tnMYVZ04f9X4nitz6TTWq2+Q1merX0dFRVkunt7d3Srf8KlG/xsZGTjvttNIZiTbbcxPQYmYLzayW7GSU9Xl51gOrglmfS4GOoEuzWNn1wOrg9WrgwZz0lWZWZ2YLyU5s2Rjsr8vMlgazPFfllAkVHPMnZMcjfxShrrF4YPsRvvhcJw/v6uVPnjrESwcHjtWhRUQkgpLBz91TwA3AI8ArwP3uvtnMrjOz64JsDwHbgFbgH4BPFysblLkV+LCZJYEPB+8Jtt9PdlLMz4Hr3T0dlFkD/J/gOK8BDwOY2e+aWRtwBfBtMxs8xh8CHwSuMrPng59zy7xGZfvUU4eGvf/Cpqk7aC0ik8dHPvIRAHbs2MGPfnTM2gNF7dmzh1WrVh3z40a6t6e7P0Q2wOWm3Znz2oHro5YN0g8AlxQocwtwS0j6c8A5IembGN4lOpj+PeB7Ycc4ltq606UzichxZdZ33oh1f+1Xl57/9+ijjwKwc+dOHnjgAa644opYz6FcqVSKefPmsW7dumN+bN3h5RhIue6dLSLjb/78bID88pe/zDPPPMMHPvABbr/9dtLpNF/4whf40Ic+xPnnn893vvMdAJ5++mkuu+wyrrrqKhYvXsyXvvQl7r//fi6++GLOP/98tm/fXvBYa9as4bOf/SzLly9n8eLF/PznPwfg+9//PqtXr+YTn/gEf/AHf8COHTt4//vfD0A6nebzn/88559/Pueffz7f/va3AXj++ee57LLLuPDCC/nYxz42bFLLaB33T3U4FlLRlvWIiBwTX/ziF/nmN7/JfffdB8A999xDY2MjTzzxBH19fVx66aV86EMfAuDll19m48aNNDc3c+655/KJT3yCxx9/nDvuuINvf/vb3HrrrQWPs3PnTn72s5+xfft2PvrRj3LRRRcBsGnTJh577DHmzZvHjh07hvLfc8897Nixg6eeeorq6moOHTrEwMAAf/7nf84PfvADTjjhBH784x/zla98hdtvv31M10DB7xhIZdTyE5GJ6/HHH2fz5s08+GB2DmFnZyfbtm2jpqaG8847j5NPPhmABQsWcPHFFwOwaNEinn766aL7vfzyy0kkEpx55pmcfvrpvPpq9r4jF110Ec3NzSPyP/nkk3zyk5+kujobmpqbm9myZQuvvPIKl19+OQCZTIaTTjppzHVW8DsGUop9IpKn1BjdsVzq4O587Wtf45JLhk/DePrpp6mrqxt6n0gkht4nEgnS6eLzGfLvtjL4fsaMGQXPI7+Mu/POd76TDRs2RKtMRBrzOwbU8hORiaShoYGurq6h95dccgl33XUXAwPZZVmtra0cPnx4zMd58MEHyWQybN++nR07dpRck3nxxRdz9913k0qlADh06BAtLS289dZbbNy4EYCBgQFeeeWVMZ+bWn7HgMb8RGQiOfvss6muruaCCy7gj//4j1mzZg07d+7kwgsvxN2ZM2cO3//+98d8nLPOOovf//3fZ//+/fzd3/1dyZbsqlWraG1t5YILLqC6uprVq1dz7bXXsnbtWj73uc/R2dlJOp1mzZo1vOtd7xrTuZkf5zMROzo6YrkAuXeayJ/CXF8Fe1dFvg3phDSZ7qRRLtVt8ppM9evo6KCpqSly/sl+h5c1a9awbNkyVqxYEbq9EvUrdo2bmpqG9aeq2/MYUMtPRGRiUbfnMaAJLyIyVf3N3/wNP/3pT4elXX755dxxxx3jdEbRKPhVgBHxcRMiIpPcjTfeyI033jjep1E2dXtWQLWuqojIhKav6QqojvgkYRE5fiQSCfr7+8f7NKas/v5+EonoIU3dnhVQk4Ajupe1iOSYOXMm3d3dHDlyJFL+zs5OGhsbK3xW4yfu+iUSCWbOnBk5v4JfBVSpPS0iecyMhoaGyPn3798f+cGsk9F4109f0xWgbk8RkYlNwa8CNOFFRGRi09d0BVSp5SciMqEp+FVAja6qiMiEpq/pCghr96X1ZAcRkQlDwa8C0iFxrk/BT0RkwlDwq4DQ4Kd1fyIiE4aCXwVkQh4T1RsWEUVEZFwo+FVA2FMc+hT8REQmDAW/CkiHPL9PwU9EZOJQ8KuAdEi3p4KfiMjEoeBXAZrwIiIysUUKfma2zMy2mlmrmd0Ust3M7LZg+4tmdl6psmY228w2mFky+Lc5Z9vNQf6tZnZpTvpiM3sp2HabWfZWKmb2QTP7tZmlzOzjeee2OjhG0sxWl3d5RkdLHUREJraSwc/MqoDbgeXAIuBKM1uUl2050BL8XAvcEaHsTcBj7t4CPBa8J9i+EjgbWAZ8K9gPwX6vzTnWsiB9J3AV8IO8c58NfBF4H7AE+GJukK0UdXuKiExsUVp+S4BWd9/m7v3AvcCKvDwrgHWe9Swwy8zmlSi7AlgbvF4LXJ6Tfq+797n7dqAVWBLsr9Hdn3F3B9YNlnH31939RSB/qsmlwAZ3P+juh4ANHA2YFZPShBcRkQktSvCbD+zKed8WpEXJU6zsSe6+ByD4d26EfbWVOI/RnHvswsf8FPxERCaKKA+zDbtVZf43eaE8UcpGPV6c+wqVTCZL7K64ZDJJdmhv+ohtO3fvJZkKn/XyUmeCAwPGBc3pCX1T7LFen4lMdZu8pnL9pnLdoPL1a2lpKbgtSvBrA3Ift3sqsDtintoiZfeZ2Tx33xN0ae4vsa+24HWx8wg794vyyjxZKHOxC1VKMpmkpaWF/rTDv448rVknnERLy4wR6Wu3HuYzL7YD8Hsn1/JPy08c9TlU0mD9piLVbfKayvWbynWD8a9flHbGJqDFzBaaWS3ZySjr8/KsB1YFsz6XAh1BV2axsuuBwdmXq4EHc9JXmlmdmS0kO7FlY7C/LjNbGszyXJVTppBHgI+YWXMw0eUjQVrFFOrdLNTt+d3k4aHXT+/t5/WuVCVOS0REcpRs+bl7ysxuIBs0qoC73X2zmV0XbL8TeAi4jOzklB7g6mJlg13fCtxvZteQna15RVBms5ndD2wBUsD17j7YX7gGuAeYBjwc/GBmvwv8BGgGPmpmX3b3s939oJl9hWwQBvjv7n5wFNcpsrCZnlA4+LX3DU/v7A+ZLSMiIrGK0u2Juz9ENsDlpt2Z89qB66OWDdIPAJcUKHMLcEtI+nPAOSHpmxjeJZq77W7g7rBtlRA20xOgr0B6f976P82LERGpvAk8vWJyCnuiAxR+qsNAXvArFDxFRCQ+Cn4xK3fMrz9vAmiqQPAUEZH4KPjFLOxxRlA4+KnlJyJy7Cn4xSxd4B6eBVt+I8b81PITEak0Bb+Yld3tmdfSU8tPRKTyFPxiVjj4wab9/ax6/AA3/6qd7oEM6YyT31DUmJ+ISOVFWuog0RXqtmzvz/D/PvoWnQPZ7Q58cXHTiHxq+YmIVJ5afjEr1PJ7YnffUOADuHPL4RHjfcXKi4hIfBT8YlZOy60/JNKl9NBbEZGKU/CLWTmzNcPuZFZoqYSIiMRHwS9m5TTc8tf4gVp+IiLHgoJfzMba7akxPxGRylPwi9mYuz0121NEpOIU/GJWTstN3Z4iIuNDwS9mY57tqdgnIlJxCn4xK/RIozC96ZFphe4NKiIi8VHwi1k53Z7dAyObiWr5iYhUnoJfzMq5N+fhkEinMT8RkcpT8ItZuowxv+4BjfmJiIwHBb+YldPteTik27Oc4CkiIqOj4Bezcnotu8K6PfVIIxGRilPwi1k5wSt0wotafiIiFafgF7Pyuj3V8hMRGQ8KfjErZ51e2IQXjfmJiFSegl/Mypmt2a0xPxGRcaHgF7NyJrxozE9EZHxECn5mtszMtppZq5ndFLLdzOy2YPuLZnZeqbJmNtvMNphZMvi3OWfbzUH+rWZ2aU76YjN7Kdh2m5lZkF5nZvcF6b8yswU5Zb5mZpvN7JXcMpVSzlMdtM5PRGR8lAx+ZlYF3A4sBxYBV5rZorxsy4GW4Oda4I4IZW8CHnP3FuCx4D3B9pXA2cAy4FvBfgj2e23OsZYF6dcAh9z9LOAbwFeDfZ0PXAC8BzgH+F3gwgjXZdTKabmFTXjRvT1FRCovSstvCdDq7tvcvR+4F1iRl2cFsM6zngVmmdm8EmVXAGuD12uBy3PS73X3PnffDrQCS4L9Nbr7M+7uwLq8MoP7egC4JGjhOVAP1AJ1QA2wL0KdR0339hQRmfiiBL/5wK6c921BWpQ8xcqe5O57AIJ/50bYV1uBfQ2VcfcU0AHMcfdngCeAPcHPI+7+Sskaj0E53Z66t6eIyPiojpAnbIws/xu6UJ4oZaMer9i+QreZ2VnAu4BTg7QNZvZBd38q7MDJZLLEqRWXTCbZ92Y12YZmadmW3/BT7+jqJpk8OKbzqJSxXp+JTHWbvKZy/aZy3aDy9WtpaSm4LUrwawNOy3l/KrA7Yp7aImX3mdk8d98TdGnuL7GvNo4Gsfx9DZZpM7NqoAk4CHwSeNbduwHM7GFgKRAa/IpdqFKSySQtLS0093TB652RynhIzK6fPoOWltNHfR6VMli/qUh1m7ymcv2mct1g/OsXpdtzE9BiZgvNrJbsZJT1eXnWA6uCWZ9LgY6gK7NY2fXA6uD1auDBnPSVwQzOhWQntmwM9tdlZkuD8bxVeWUG9/Vx4PFgXHAncKGZVZtZDdnJLhOm2zOMxvxERCqvZMvP3VNmdgPwCFAF3O3um83sumD7ncBDwGVkJ6f0AFcXKxvs+lbgfjO7hmyQuiIos9nM7ge2ACngencffOb5GuAeYBrwcPADcBfwXTNrJdviWxmkPwBcDLxEtov05+7+T2VdoTKNNXhpnZ+ISOVF6fbE3R8iG+By0+7Mee3A9VHLBukHgEsKlLkFuCUk/TmySxby03sJgmdeehr4VNgxKiUzxuClO7yIiFSe7vASs7F2e+reniIilafgF7Mxd3uq5SciUnEKfjEb84QXtfxERCpOwS9mY+221GxPEZHKU/CLWTm3Nwstrzu8iIhUnIJfzLTOT0Rk4lPwi9lYW366t6eISOUp+MVsrBNW1PITEak8Bb+YjX2dn6KfiEilKfjFbKyxSy0/EZHKU/CL2Zi7PbXOT0Sk4hT8Yjb22Z5q+omIVJqCX8zGvs4vnvMQEZHCFPxipnt7iohMfAp+McuMccaLxvxERCpPwS9mY+72dHC1/kREKkrBL2ZxLFUYawAVEZHiFPxiNtbZnqCuTxGRSlPwi1kcN2jRpBcRkcpS8ItZHDemVstPRKSyFPxilj9e9+7ZNUOv3zO7hrqqKPtQy09EpJKqx/sEppr84Pc3S5vY9GY/nQPOp941g/c+sI++EjNa1PITEaksBb+Y5Xd7Tqs2bjinYeh9bZXBQIngp4afiEhFqdszZvlDflVmw97XRrjieqCtiEhlKfjFLL9HsyrvCtdWDQ+GUfYhIiLxUvCLWX6rrTov1tUmSgc/tfxERCorUvAzs2VmttXMWs3sppDtZma3BdtfNLPzSpU1s9lmtsHMksG/zTnbbg7ybzWzS3PSF5vZS8G228yyfYpmVmdm9wXpvzKzBTll3mZmj5rZK2a2JXdbJYxo+eV3e0Zo+WnMT0SkskoGPzOrAm4HlgOLgCvNbFFetuVAS/BzLXBHhLI3AY+5ewvwWPCeYPtK4GxgGfCtYD8E+70251jLgvRrgEPufhbwDeCrOee2Dvi6u78LWALsL1XnscgPfvkNvTqN+YmIjLsoLb8lQKu7b3P3fuBeYEVenhXAOs96FphlZvNKlF0BrA1erwUuz0m/19373H070AosCfbX6O7PePbOz+vyygzu6wHgkqA1ugiodvcNAO7e7e49ka7MKOWv0atOlN/y05ifiEhlRQl+84FdOe/bgrQoeYqVPcnd9wAE/86NsK+2AvsaKuPuKaADmAO8HWg3sx+b2W/M7Os5rciKGNntOfx9tDG/GE9IRERGiLLOL+zbOr9tUihPlLJRj1dsX4W2VQO/B7wX2AncB1wF3BV24GQyWeLUiksmk/QPTBt2Oju2b6Or9miegd46oHj8fX3nLmZ1TLwIONbrM5GpbpPXVK7fVK4bVL5+LS0tBbdFCX5twGk5708FdkfMU1uk7D4zm+fue4IuzcGxuEL7agteh+1rsEybmVUDTcDBIP037r4NwMx+CiylQPArdqFKSSaTtLS04Bt3kxvf337WmTTnDPQ17zwAh3qL7uvk+afSMq9u1OdSCYP1m4pUt8lrKtdvKtcNxr9+Ubo9NwEtZrbQzGrJTkZZn5dnPbAqGGdbCnQEXZnFyq4HVgevVwMP5qSvDGZwLiQ7sWVjsL8uM1sazPJclVdmcF8fBx4PxgU3Ac1mdmKw7WJgS4Q6j1r+XJX8Xs4o3Z66t6eISGWVbPm5e8rMbgAeIdtfd7e7bzaz64LtdwIPAZeRnZzSA1xdrGyw61uB+83sGrJdklcEZTab2f1kg1QKuN7d00GZNcA9wDTg4eAHsi2575pZK9kW38pgX2kzuxF4LAiY/w78Q9lXqQwlx/wijDhqzE9EpLIi3dvT3R8iG+By0+7Mee3A9VHLBukHgEsKlLkFuCUk/TngnJD0XoLgGbJtA/CesG2VMHKR+/DoVxdpnZ9afiIilaQ7vMSs5O3NNNtTRGTcKfjFLI5uT63zExGpLAW/GGXyuisNSIx4qoPu7SkiMt4U/GKU310ZNryne3uKiIw/Bb8YlRrvA7X8REQmAgW/GI24r6eNDHRRbmytMT8RkcpS8ItRXN2eac32FBGpKAW/GOVPeAnr4YzU7al1fiIiFaXgF6P87sr8xxmB7vAiIjIRKPjFKH+WZmi3p1p+IiLjTsEvRum8WZoa8xMRmZgU/GI0cqlDyGzPKN2eaviJiFSUgl+MSt3aDLTOT0RkIlDwi1H+Oj/d4UVEZGJS8IvRiNmeIYvca6MsclfLT0SkohT8YhRpkXuk2Z4xnZCIiIRS8ItRfrdnInSdn8b8RETGm4JfjPJjVnVInKtTy09EZNwp+MUo2r09S+9H6/xERCpLwS9GI2d7hk140R1eRETGm4JfjKI8z68u0phfTCckIiKhFPxiFKXbsybCFVfLT0SkshT8YpT/SKOwbs8oLT+N+YmIVJaCX4xGPtJoZJ4qg1LhTy0/EZHKUvCLUX7QCmvkmVnJm1trzE9EpLIU/GKU312ZCOn2hNIL3bXOT0SksiIFPzNbZmZbzazVzG4K2W5mdluw/UUzO69UWTObbWYbzCwZ/Nucs+3mIP9WM7s0J32xmb0UbLvNLBtdzKzOzO4L0n9lZgvyzq/RzN4ws2+Wc3HKNfLenuH5Si130L09RUQqq2TwM7Mq4HZgObAIuNLMFuVlWw60BD/XAndEKHsT8Ji7twCPBe8Jtq8EzgaWAd8K9kOw32tzjrUsSL8GOOTuZwHfAL6ad35fAX5Rqq5jNWKdX4GrW+ouL2r5iYhUVpSW3xKg1d23uXs/cC+wIi/PCmCdZz0LzDKzeSXKrgDWBq/XApfnpN/r7n3uvh1oBZYE+2t092fc3YF1eWUG9/UAcElOq3AxcBLwaJQLMhYjn+cXHuRqSo75KfqJiFRSlOA3H9iV874tSIuSp1jZk9x9D0Dw79wI+2orsK+hMu6eAjqAOWaWAP4W+LMI9RyzqN2epVp++fsREZF4VUfIE/ZNnf/1XChPlLJRj1dsX4W2fRp4yN13WYFWWK5kMlkyTzFv7NkL1A29P9zdRTJ5YOSJperJ/bujxpwBP3p+3T1HxnwulTARzykuqtvkNZXrN5XrBpWvX0tLS8FtUYJfG3BazvtTgd0R89QWKbvPzOa5+56gS3N/iX21Ba/D9jVYps3MqoEm4CDwfuD3zOzTwEyg1sy63X3EpB0ofqFKSSaTnDD3JEi2D6XNamqkpaV5RN6G3+6HwwND76fXJOjoP/o3QaK2nvU9zfzzziN84OQ6vnBeY6RHIVVSMpkc0/WZyFS3yWsq128q1w3Gv35Ruj03AS1mttDMaslORlmfl2c9sCqY9bkU6Ai6MouVXQ+sDl6vBh7MSV8ZzOBcSHZiy8Zgf11mtjQYz1uVV2ZwXx8HHg/GH/+zu7/N3RcAN5IdlwwNfHGI8kgjGLnUYXpexhcODPCVX3fym7cG+N8vd/OP24/EeZoiIse9ki0/d0+Z2Q3AI0AVcLe7bzaz64LtdwIPAZeRnZzSA1xdrGyw61uB+83sGmAncEVQZrOZ3Q9sAVLA9e6eDsqsAe4BpgEPBz8AdwHfNbNWsi2+laO7HGOTP1Gl0ISX/KUO00q06tY8fYgrz5o+tpMTEZEhUbo9cfeHyAa43LQ7c147cH3UskH6AeCSAmVuAW4JSX8OOCckvZcgeBapwz1kA2fFjJztGZ4v/w4v0wo1EUVEpCJ0h5cYRXmkEUBNoni3p4iIVJaCX4zy78xSqNsz/8kO08LugC0iIhWjb90YRe32rM276lG6PbsHdLdrEZG4KPjFKMojjSBktmeEZQx7etIl84iISDQKfjHKf6RRoac65N/hJUrLb/dhBT8Rkbgo+MUoardn/r09o0x4eUPBT0QkNgp+McrkDcsVimn1Iya8RGj59WjMT0QkLgp+MRrxJPcCN7BeOrdu2PsPnFwXmi+Xuj1FROITaZG7RBO12/M/nlrH/1zSxJO7e/nwqfV86JTSwe8NTXgREYmNgl+M8h9mW+jJRQkz1pw9kzVnzwQg46WfYbT7cJrWjgH+9sVuplUZN7+3gROnlXgwoIiIhFLwi1FqxJhftDu3JMxI2MgbY+dqO5zivzx+kN+2p4be3//hE0Z7qiIixzUFvxjlB69ynkJUbdBfJPgd6nMO9aWG3j/a1oe7E+U5hSIiMpwmvMQov9uz0L09w1SXeLp7mI5i0VJERApS8IvRaLs9s3nLP97+I5oEIyIyGgp+Mcqf7VlOY66cVuKgfUe09k9EZDQU/GI0otuzrDG/8pt++9TyExEZFQW/GOVPeClnHG80TzVSy09EZHQU/GKUP+ZXTsuv0LP/itmvhe8iIqOi4Bejkd2elW75KfiJiIyGgl+MRtzerJylDqNp+anbU0RkVBT8YpQayyJ3tfxERI4ZBb8YZTKj7/YsJ1AOUstPRGR0FPxilN/tWU5rbjR3eHmrN0Oq2A1BRUQklIJfjMbS7Rklb0PN8EwOvNmr1p+ISLl0Y+sYjXykUTm3NwvP+6fnzCRhsK0zxacWzeTPnmlnS/vRG1zv60kzb7oebSQiUg4FvxiNWOQew+3N5tQn+My7G4bez51eNSz4adxPRKR86vaMUf74W7mPNAozPW/D3GnDPzLN+BQRKV+k4Gdmy8xsq5m1mtlNIdvNzG4Ltr9oZueVKmtms81sg5klg3+bc7bdHOTfamaX5qQvNrOXgm23WfAwOzOrM7P7gvRfmdmCIP1cM3vGzDYH5/VHo7lIUY1c51fOIvfwvNPygt9JeU9vV8tPRKR8JYOfmVUBtwPLgUXAlWa2KC/bcqAl+LkWuCNC2ZuAx9y9BXgseE+wfSVwNrAM+FawH4L9XptzrGVB+jXAIXc/C/gG8NUgvQdY5e6D+/p7M5tV+rKMzojZnnG0/Kryg59afiIiYxWl5bcEaHX3be7eD9wLrMjLswJY51nPArPMbF6JsiuAtcHrtcDlOen3unufu28HWoElwf4a3f0Zd3dgXV6ZwX09AFxiZubur7p7EsDddwP7gROjXJjRyO/2LO+RRmr5iYgcK1EmvMwHduW8bwPeFyHP/BJlT3L3PQDuvsfM5ubs69mQfQ0Er/PThx3f3VNm1gHMAd4azGxmS4Ba4LVCFU0mk4U2RXKkr5/cvyfe2LWTaQejrcPr7akl7OM4tG83yb6jAW6gPQHUD71//WA3yeSB0Z5yWcZ6fSYy1W3ymsr1m8p1g8rXr6WlpeC2KMEvrEmS/41eKE+UslGPV2xfRY8TtBq/C6x294JNpWIXqpRkMkl1TS1wdCbmGQtOp5vYTRcAABJ4SURBVGVWTaTys9oOwoEjI8/p9NNomVs79D7dPgAv7x963+m1tLS8bdTnHVUymRzT9ZnIVLfJayrXbyrXDca/flG6PduA03LenwrsjpinWNl9QVAaDE6D3+jF9nVqgX0NlTGzaqAJOBi8bwR+Bnw+6JKtmDHN9izwSajbU0QkflGC3yagxcwWmlkt2cko6/PyrAdWBbM+lwIdQZdmsbLrgdXB69XAgznpK4MZnAvJTmzZGOyvy8yWBrM8V+WVGdzXx4HH3d2DY/6E7Hjkj6JdktEbMdszhnt75i91mFVr1OZ8at0pp3tAAVBEpBwluz2DMbQbgEeAKuBud99sZtcF2+8EHgIuIzs5pQe4uljZYNe3Aveb2TXATuCKoMxmM7sf2EK2D/F6dx+c0rgGuAeYBjwc/ADcBXzXzFrJtvhWBul/CHwQmGNmVwVpV7n785GvUAl/8ouDvN6VYndXPW/0Dp95WdYjjQpMeMkPfmbG3GlVtB0+eqw3j2SYWaMlmyIiUUW6w4u7P0Q2wOWm3Znz2oHro5YN0g8AlxQocwtwS0j6c8A5Iem9BMEzL/17wPfCjhGX597sZ3tXmrBGdFkPsy2QNb/bE7LLHXKD374jaRY26mY9IiJRqbkwRifWF76vZnljftFafgBz88b99mncT0SkLAp+Y3TCtMKXsJxHGoUFymqDmpCgmL/Q/eWDA+zXYncRkcgU/MboxPrCl7Csbs+Q3UyvCS8/N+8pDl9/oYu337uX1U8cION6vp+ISCkKfmNUrNuznDu8hD3SKP/WZoPyW36DHny9l1/t749+UBGR45SC3xgV7fYc4zq/sMkuAEvn1hXcz6s5jzsSEZFwCn5jNDembs+wvIWC39mza7j7wmYuOqWOedOHH393j8b+RERKUfAboxOmFZntWdY6v5FpM4o0HT92xnR+eukJ/Lf3NAxL36PgJyJSkoLfGBWf8BJ9P2FjftMiTBedlzf5Zc9hBT8RkVIU/MboxAJjfgYkxjjbs1C3Z65T8oKfuj1FREpT8Buj5tpE6KzOclp9hfIXmu2Za96MvJZfjxa8i4iUouA3RlUJY07dyMtYzgL3bP6QpQ4F1vnlmlufGBY4D/ZlOJLSWj8RkWIU/GIQNu5XzkxPCF8WEaXlV5WwEev+9qrrU0SkKAW/GITN+Cy32zOs5RdlzA9GTnrRuJ+ITDbbO1P8118c5OonDpLsGKj48fQogBjMDZn0Us4yBwgPluUFv6O/LFruICKTzbVPHWTTm9nvsdbOFE/9PydiZfaglUMtvxicEEe3Z0jLr9g6v1yn5E960XIHEZlEDvamhwIfwEsHByr+tBoFvxiE3d+z7G7PMbT8tNxBRCazzYdG3pZx86HKdn0q+MUgbK1f2KL1Yka7zg+03EFEJrctIYFuy0EFvwkvrNszUfaYX/SnOuQbcZcXtfxEZBIJa+W9rJbfxBdLt2cZz/PLd4pubi0ik8h3Xz3Mf32xjr/c2EEq42wOaeWFdYXGSbM9YxBLt2fYvT0jThnNb/nt7UmTcS/r9moiIsfCb97q50//tR2nihc2d3NCfYJXQh7F9mr7AAMZp6acB6OWQS2/GITP9ixvH6Etv4hjfjNqEjTWHs07kIF7W3v4z48d4K9/3clARnd8EZGJ4f7Xesj9RvrfL3fTE3JXqv4MtHZUrvWnll8MZtaMjFyHy7zFWDnP8wtzyvQqOvuP/qJ8+pftAPxsZy8Jg794b2NZ5yMiMlbtfRk++2/tbOtKccPZM7nizOk82tY7LM/BvsIT9DYfGuBdzTUVOTe1/CrkUJEPNEy5z/PLl9/1meuuVw7Tn1brT0Qqq70vw+GBo999f/ZsOz95/QgvHBjguqcP8c87jvBaZ/Q5CWGzQOOi4BcTY3hwKbflN5Z1flA8+B3oy/DzXb30pDLc+ptO/mpTB2/1alKMiIyOu9OfdtyPfs997flOFvxgD++8by8b2npp607x4+1HhranHW745aGyjhM2ESYu6vaMSVM1hIzZRjaWe3vCyIXu+da9epg7t3Tzb/v6AXjw9SM8vWIu9VXGV5/v5Ok9/Sx/Wz2feffMoYkynf0ZahNGfRnnISJTy8b9fXT0OxefUkdVwnirN81VTxzkl3v7uXBeHd+5qJnnDwzwP37TBUDXgHPNLw7y0dOnkd/h1N5fXqOgkjM+Ffxi0lTjtKdGHyRCn+dXTstvRvFG/L+80Tfs/Y7uNH+xsYO0ww9bewDY+GY/h/oyfGFxI3+xsYO7fnuYplrjG+9v5mygayDD2q2H2dGd5sozp3PeibVA9tZEz+zr591zanjbzKO/UkdSjoGCp8gx4u4j7oe5qztFdcKGeofcnX/d189LBwZYdlo9CxurcXfWvtrD95KHecesGm46t4H5M6q48ZkO7t56GIClc2v54X+cwyefPMQv92b/iP7Fnj4++YtDtHUP70nq7He+n+wp+/yzDwFnKGi2HU7T3pdhVshj48YqUvAzs2XA/wKqgP/j7rfmbbdg+2VAD3CVu/+6WFkzmw3cBywAXgf+0N0PBdtuBq4B0sCfuvsjQfpi4B5gGvAQ8Bl3dzOrA9YBi4EDwB+5++tBmdXA54NT/Wt3Xxv56pShsXpsY2phEzLLmeJbrNuzkO+F/HLe9nI3//JGL1uCv7gO9TlXP3mQ1afW8NQL+9nWlf0lv+u3h7np3AYaaxP89b930p1yEgbXLZrBVW+fwTc3d3Nvaw9m8F9aZnDjf2hg88EBvr2lm1faU3zolDpuOGcmc6dV8YPWHja09XLqjCr++KzpvP+kWn791gAP7+qlL+1ccHItl8yv50jKebStl53daU6bUcVHTqunscb4bXuKjfv7qasyPjivjlNmVNE9kOHZfdlgfs7sGt45K/ur/lpniq3tKeZOq+LcE2qoSRhH0vDMvj4yDu+ZU0NDTQJ35/WuNHt60pzRWM3JwfU9PJAh2ZGisTbBgoYqEma4Ozu70/SknDMaq6kL/pLpHsjQdjjNydOqhv7zZtx543Ca6oRx8rTE0BdVR3+G9r4M82dUDfUCpDPOnp40s+oSwyZVtfdlSLkzp+5o+f60c7Avwwn1iaHy7k77QPY8csv3pLLPfJydUz7jzoHe7JdM7u9dR38GAxprj5bvTzudAxlm1yWGegncnQN9GWZUJ4b1WHQNZBhIO805xxrIOG/1Zs+1Judc3+zN9jTkftF19Gfo6s8wb3oVVYmjdX3jcJrZObOs3Z1dh9O4w2kzq4bO60Bvmn1HMrxtZtXQNehJZXitM01DjXH6zCrMjHTGSXam6Blw3jGrmhnB78Cuw2le70pz6owqFjZk8x7qy/DCgX6qE8a5c2qYWZOgN+VserOfg30Z3jWrmpamahx48cAALx8a4IT6BBecXEdDTYLXOlI8uaeXVAbOP7mOc5qreas3wz/v6GV3T5qFDVX8/unT6E1nZ20/squXOfUJPn7GNH73xFoeaetl3as97D+S5rK3TeOqd0yntSPF377QxdN7+zhndg2feXcDb2+q5q82dfBIW/YP3xUL6vnTcxr4uxe7+NnO7KSTv3qugz//Dw38tj3FPwZdlM+9OcAju3p5/0m1/NOOo5NTnt3fz3n/uJdDfcO/rJ7cPfwP67FY2FBFXZUNW/qw5dAA559cF9sxBllun21oBrMq4FXgw0AbsAm40t235OS5DPj/yAa/9wH/y93fV6ysmX0NOOjut5rZTUCzu3/OzBYBPwSWAKcA/wK83d3TZrYR+AzwLNngd5u7P2xmnwbe4+7XmdlK4A/c/Y+CAPsc8DuAA/8OLB4MsgAdHR2xzAT5Tz/dyS8PDQ9A7VfPj1z+2X19LHvorVGXf/6tfi76pzeHpRkwUaa5FDqX2kR2SnOuplqjI697ZFat0ZPyYXlrE9llJrvzbud2RkMVuw6nyRl3Z970BO6wN+dmuTOqs19+W9sHSJP9skwYvLOpmn1HMhzImbR06owqGmqMrR2poT9UGmuNsxqrea0zNXS+NQl4x6waegYybO9KD9X5tJlVnFCf4NX21NB48Kxa46ymavb2ZIPkYJ1amqoxM5IdA/Sljx7/lOlVbO9K8WZvZug6tTRV09HvbOtMkfbs8c9srGZ6tdHakaJzIHus+dOrOG1mFW/0pNkV/JXeUJMt35d2XutM0ZvOjj0vaKhmdl2CbV0p3gqONXdagoUN1bx5JM2O7jRph2lV2fOvsuyU9O6gpf+2mVXMm17Fju7U0K32moJr1TWQPdeUZ491RmM1DTVGMudcT5qWYEFDNW3dad4IbthQXwUtTTUMZJzWjmx5gJPrMsxvqOPVjhRdQfmZ1cbbZ1Wztyc99LthwBmNVdQlsp/hYMuiqdZY2FBNsuPo55IweEdTNYf6MsN+X06oTzCnLsHWnOn3VQZvb6pmW1dq6LMa/H0byDB0/QZ/N+ZNr2JnXitp3vQE+49khnUR1iagxpzD6eF/AM+othHzCaoNJuPzqxdOz7C9Z3iL7qOn11NXZTyw7ehY4deXNvEn75o55uM1NTUNu5hRgt/7gS+5+6XB+5sB3P1/5uT5NvCku/8weL8VuIhsqy607GAed99jZvOC8u/I37+ZPQJ8iWzr8Al3f2eQfmVQ/lODedz9GTOrBvYCJwIrB/OEnSfEF/xERGTiyg9+UTpS5wO7ct63BWlR8hQre5K77wEI/p0bYV9tBfY1VMbdU0AHMCfiuYuIyHEmSvALG3jKby0VyhOlbNTjFdtXnMcXEZEpLsqElzbgtJz3pwK7I+apLVJ2n5nNy+n23F9iX23B67B9DZZpC7o9m4CDQfpFeWWezD3x/KawiIhMfVFafpuAFjNbaGa1ZMfR1uflWQ+ssqylQEfQlVms7HpgdfB6NfBgTvpKM6szs4VAC7Ax2F+XmS0NZpeuyiszuK+PA497djDzEeAjZtZsZs3AR4I0ERE5nrl7yR+yszhfBV4D/jJIuw64LnhtwO3B9peA3ylWNkifAzwGJIN/Z+ds+8sg/1ZgeU767wAvB9u+ydEJO/XAj4BWYCNwRk6ZTwbprcDVUepb7g+wLDjXVuCmShwjpvN8Pfh8ngeeC9JmAxuCz2ED2Vm3g/lvDuq0Fbg0J31xsJ9W4Lacz6GO7PKVVuBXwIIK1+dusj0GL+ekHZP6kP1jKxn8rD5GdfsS8Ebw+T0PXDYZ6xYc4zTgCeAVYDPZZUtT4vMrUrdJ//mR/a7dCLwQ1O3Lk/Vzi/2X+nj7Ibt+8TXgDLLdvC8Ai8b7vAqc6+vACXlpXyMI2MBNwFeD14uCutQBC4M6VgXbNgLvJ/tHz8MEf6AAnwbuDF6vBO6rcH0+CJzH8ABR8foE/9G3Bf82B6+bj0HdvgTcGJJ3UtUtOM484LzgdQPZP5AXTYXPr0jdJv3nF5zHzOB1DdngtHQyfm66t+fYLQFa3X2bu/cD9wIrxvmcyrECGFz4vxa4PCf9Xnfvc/ftZP8KWxKMzza6+zOe/Y1cl1dmcF8PAJdY/u0mYuTuT5Ed2811LOpzKbDB3Q96ds3oBrKt/0rXrZBJVTfIzvD24EYY7t5FtpU0nynw+RWpWyGTqW7u7t3B25rgx5mEn5uC39hNpuUUDjxqZv9uZtcGacdiycmxNNWX0NxgZi+a2d3BOPaw88w7n0lRNzNbALyXbCtiSn1+eXWDKfD5mVmVmT1Ptlt+g7tPys9NwW/sJtNyigvc/TxgOXC9mX2wSN44l5xMBFNhCc0dwJnAucAe4G+D9ElbNzObCfwj8P+7e2exrCFpE7qOIXWbEp+fu6fd/Vyys+eXmNk5RbJP2Lop+I1dlKUgE4K77w7+3Q/8hGyX7b6gC4IYl5yQt+TkWDoW9RmXz9zd9wVfPBngH8h+fsPOM+98JnTdzKyGbHD4vrv/OEieEp9fWN2m2ufn7u1kl44tYzJ+bnENhB6vP2TXSm4jO5g7OOHl7PE+r5DznAE05Lz+t+CX9usMH6j+WvD6bIYPVG/j6ED1JrKD3IMD1ZcF6dczfKD6/mNQrwUMnxRS8fqQHXDfTnbQvTl4PfsY1G1ezuvPkh1Lmax1M7LjPH+flz7pP78idZv0nx/Z20bOCl5PA54G/tNk/Nwq+sV0vPxQYDnHRPohOxv1BY5OUR5csnJMlpxUqE4/JNt9NED2r8JrjlV9qPASmgJ1+y7ZqeEvkl3bOm8y1i04xgfIdlm9SM7U/6nw+RWp26T//ID3AL8J6vAy8FdB+qT73Ere2FpERGSq0ZifiIgcdxT8RETkuKPgJyIixx0FPxEROe4o+ImIyHFHwU9ERI47Cn4iInLcUfATEZHjzv8FOfymufNt+V4AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.kdeplot(sales_train.item_price)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 13110,
     "status": "ok",
     "timestamp": 1561965850087,
     "user": {
      "displayName": "Shengbing YU",
      "photoUrl": "",
      "userId": "03994843105687918628"
     },
     "user_tz": -600
    },
    "id": "CuMStZuJPheB",
    "outputId": "e304babc-fdd7-4e05-e82f-5885e3dc4921"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[-1.0, 0.07, 0.0875, 0.09, 0.1]\n",
      "[307980.0, 59200.0, 50999.0, 49782.0, 42990.0]\n"
     ]
    }
   ],
   "source": [
    "tmp = sorted(sales_train.item_price.unique())\n",
    "print(tmp[:5])\n",
    "print(tmp[::-1][:5])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 136
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 13063,
     "status": "ok",
     "timestamp": 1561965850088,
     "user": {
      "displayName": "Shengbing YU",
      "photoUrl": "",
      "userId": "03994843105687918628"
     },
     "user_tz": -600
    },
    "id": "0ALQs9cnPheF",
    "outputId": "5072a51e-23ce-4edd-ba11-a881b5547852"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "date              13.12.2013\n",
       "date_block_num            11\n",
       "shop_id                   12\n",
       "item_id                 6066\n",
       "item_price            307980\n",
       "item_cnt_day               1\n",
       "Name: 1163158, dtype: object"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sales_train.loc[sales_train.item_price.idxmax()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 80
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 13013,
     "status": "ok",
     "timestamp": 1561965850089,
     "user": {
      "displayName": "Shengbing YU",
      "photoUrl": "",
      "userId": "03994843105687918628"
     },
     "user_tz": -600
    },
    "id": "4BuOAtoVPheJ",
    "outputId": "9e9f90a0-9ee9-4137-a0f2-f8dbb00a153b"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>date</th>\n",
       "      <th>date_block_num</th>\n",
       "      <th>shop_id</th>\n",
       "      <th>item_id</th>\n",
       "      <th>item_price</th>\n",
       "      <th>item_cnt_day</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1163158</th>\n",
       "      <td>13.12.2013</td>\n",
       "      <td>11</td>\n",
       "      <td>12</td>\n",
       "      <td>6066</td>\n",
       "      <td>307980.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               date  date_block_num  shop_id  item_id  item_price  \\\n",
       "1163158  13.12.2013              11       12     6066    307980.0   \n",
       "\n",
       "         item_cnt_day  \n",
       "1163158           1.0  "
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# only one record for this item_id\n",
    "sales_train[sales_train.item_id==6066]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ke3wfafwPheN"
   },
   "outputs": [],
   "source": [
    "#median = sales_train[(sales_train.shop_id==12)&(sales_train.date_block_num==11)&(\n",
    "#    sales_train.item_price>0)&(sales_train.item_price<60000)].item_price.median()\n",
    "sales_train.loc[sales_train.item_id==6066, 'item_price'] /= 10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 136
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 12955,
     "status": "ok",
     "timestamp": 1561965850104,
     "user": {
      "displayName": "Shengbing YU",
      "photoUrl": "",
      "userId": "03994843105687918628"
     },
     "user_tz": -600
    },
    "id": "5mRZ2fFEPheP",
    "outputId": "4487846b-546e-41fd-e4c9-7016554877d6"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "date              15.05.2013\n",
       "date_block_num             4\n",
       "shop_id                   32\n",
       "item_id                 2973\n",
       "item_price                -1\n",
       "item_cnt_day               1\n",
       "Name: 484683, dtype: object"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sales_train.loc[sales_train.item_price.idxmin()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "OxcaBVxPPheT"
   },
   "outputs": [],
   "source": [
    "median = sales_train[(sales_train.shop_id==32)&(\n",
    "    sales_train.item_id==2973)&(sales_train.date_block_num==4)&(\n",
    "    sales_train.item_price>0)].item_price.median()\n",
    "sales_train.loc[sales_train.item_price<0, 'item_price'] = median"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 136
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 13531,
     "status": "ok",
     "timestamp": 1561965850751,
     "user": {
      "displayName": "Shengbing YU",
      "photoUrl": "",
      "userId": "03994843105687918628"
     },
     "user_tz": -600
    },
    "id": "BDmycTCbPheV",
    "outputId": "6948a05b-1f1c-4320-dfc5-73b66581bf4d"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "date              0\n",
       "date_block_num    0\n",
       "shop_id           0\n",
       "item_id           0\n",
       "item_price        0\n",
       "item_cnt_day      0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sales_train.isnull().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "XX22Eb6APheZ"
   },
   "source": [
    "Let's deal with duplicated shop names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 119
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 13477,
     "status": "ok",
     "timestamp": 1561965850753,
     "user": {
      "displayName": "Shengbing YU",
      "photoUrl": "",
      "userId": "03994843105687918628"
     },
     "user_tz": -600
    },
    "id": "XGfGEDm9Phea",
    "outputId": "d6e12cc9-0879-4fe7-9c6e-185623a89edd"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "!Якутск Орджоникидзе, 56 фран\n",
      "Якутск Орджоникидзе, 56\n",
      "!Якутск ТЦ \"Центральный\" фран\n",
      "Якутск ТЦ \"Центральный\"\n",
      "Жуковский ул. Чкалова 39м?\n",
      "Жуковский ул. Чкалова 39м²\n"
     ]
    }
   ],
   "source": [
    "print(shops.loc[0].shop_name)\n",
    "print(shops.loc[57].shop_name)\n",
    "print(shops.loc[1].shop_name)\n",
    "print(shops.loc[58].shop_name)\n",
    "print(shops.loc[10].shop_name)\n",
    "print(shops.loc[11].shop_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Noi8m9fQPhed"
   },
   "outputs": [],
   "source": [
    "# Якутск Орджоникидзе, 56\n",
    "sales_train.loc[sales_train.shop_id == 0, 'shop_id'] = 57\n",
    "test.loc[test.shop_id == 0, 'shop_id'] = 57\n",
    "# Якутск ТЦ \"Центральный\"\n",
    "sales_train.loc[sales_train.shop_id == 1, 'shop_id'] = 58\n",
    "test.loc[test.shop_id == 1, 'shop_id'] = 58\n",
    "# Жуковский ул. Чкалова 39м²\n",
    "sales_train.loc[sales_train.shop_id == 10, 'shop_id'] = 11\n",
    "test.loc[test.shop_id == 10, 'shop_id'] = 11"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 68
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 15229,
     "status": "ok",
     "timestamp": 1561965852588,
     "user": {
      "displayName": "Shengbing YU",
      "photoUrl": "",
      "userId": "03994843105687918628"
     },
     "user_tz": -600
    },
    "id": "4UumubF1Pheg",
    "outputId": "3ca0b322-b8ee-4157-cb4e-f4ca7dda40ad"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "False    2935825\n",
      "True          24\n",
      "dtype: int64\n"
     ]
    }
   ],
   "source": [
    "# Check for duplicates\n",
    "cols = ['date','date_block_num','shop_id','item_id','item_cnt_day']\n",
    "print(sales_train.duplicated(subset=cols).value_counts())\n",
    "sales_train.drop_duplicates(subset=cols, inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "dwGjci1RPhek"
   },
   "source": [
    "## 2.2. Extracting categorical features from item names<a name=\"Extracting22\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "w3axoffnPhek"
   },
   "outputs": [],
   "source": [
    "# save a copy for text feature processing\n",
    "shops_copy = shops.copy()\n",
    "item_categories_copy = item_categories.copy()\n",
    "items_copy = items.copy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "O42pgIbPPhen"
   },
   "outputs": [],
   "source": [
    "shops.loc[shops.shop_name == 'Сергиев Посад ТЦ \"7Я\"', 'shop_name'] = 'СергиевПосад ТЦ \"7Я\"'\n",
    "shops['city'] = shops['shop_name'].str.split(' ').map(lambda x: x[0])\n",
    "shops.loc[shops.city == '!Якутск', 'city'] = 'Якутск'\n",
    "shops['city_code'] = LabelEncoder().fit_transform(shops['city'])\n",
    "shops = shops[['shop_id','city_code']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 204
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 15126,
     "status": "ok",
     "timestamp": 1561965852599,
     "user": {
      "displayName": "Shengbing YU",
      "photoUrl": "",
      "userId": "03994843105687918628"
     },
     "user_tz": -600
    },
    "id": "Zq79guXaPheo",
    "outputId": "6d4d03ab-0115-4fac-bcab-a9935d302173"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>shop_id</th>\n",
       "      <th>city_code</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>29</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>29</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   shop_id  city_code\n",
       "0        0         29\n",
       "1        1         29\n",
       "2        2          0\n",
       "3        3          1\n",
       "4        4          2"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "shops.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "IMHg7-5QPhes"
   },
   "outputs": [],
   "source": [
    "item_categories['split'] = item_categories['item_category_name'].str.split('-')\n",
    "item_categories['type'] = item_categories['split'].map(lambda x: x[0].strip())\n",
    "item_categories['type_code'] = LabelEncoder().fit_transform(item_categories['type'])\n",
    "item_categories['subtype'] = item_categories['split'].map(\n",
    "    lambda x: x[1].strip() if len(x) > 1 else x[0].strip())\n",
    "item_categories['subtype_code'] = LabelEncoder().fit_transform(item_categories['subtype'])\n",
    "item_categories = item_categories[['item_category_id','type_code', 'subtype_code']]\n",
    "\n",
    "items.drop(['item_name'], axis=1, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 204
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 15058,
     "status": "ok",
     "timestamp": 1561965852603,
     "user": {
      "displayName": "Shengbing YU",
      "photoUrl": "",
      "userId": "03994843105687918628"
     },
     "user_tz": -600
    },
    "id": "--SAOTHFPhet",
    "outputId": "f07e8ae1-61e5-4a19-fefb-cbfb95426db0"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>item_category_id</th>\n",
       "      <th>type_code</th>\n",
       "      <th>subtype_code</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>29</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   item_category_id  type_code  subtype_code\n",
       "0                 0          0            29\n",
       "1                 1          1             9\n",
       "2                 2          1            10\n",
       "3                 3          1            11\n",
       "4                 4          1            13"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "item_categories.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "item_category_id    84\n",
       "type_code           20\n",
       "subtype_code        65\n",
       "dtype: int64"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "item_categories.nunique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 204
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 15018,
     "status": "ok",
     "timestamp": 1561965852604,
     "user": {
      "displayName": "Shengbing YU",
      "photoUrl": "",
      "userId": "03994843105687918628"
     },
     "user_tz": -600
    },
    "id": "lUqOwnwEPhew",
    "outputId": "f564242e-e319-4d11-dc4c-102e0f19fd0c"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>item_id</th>\n",
       "      <th>item_category_id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>76</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>40</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>40</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>40</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   item_id  item_category_id\n",
       "0        0                40\n",
       "1        1                76\n",
       "2        2                40\n",
       "3        3                40\n",
       "4        4                40"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "items.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "xpv2G3TQPhe1"
   },
   "source": [
    "## 2.3. ID values of Train and test data<a name=\"Making23\"></a>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 15216,
     "status": "ok",
     "timestamp": 1561965852854,
     "user": {
      "displayName": "Shengbing YU",
      "photoUrl": "",
      "userId": "03994843105687918628"
     },
     "user_tz": -600
    },
    "id": "AfpwxLcHPhe2",
    "outputId": "8cce8dfb-16f3-4e67-8358-5928c0f5d7bd"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ True]\n"
     ]
    }
   ],
   "source": [
    "# test1 = test.copy()\n",
    "# test1['key'] = test1['shop_id'].astype(str) + '_' + test1['item_id'].astype(str)\n",
    "# print(test1['key'].nunique() == len(test1))\n",
    "# del test1\n",
    "print(test.duplicated(subset=['shop_id','item_id']).value_counts().values == len(test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 68
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 15861,
     "status": "ok",
     "timestamp": 1561965853550,
     "user": {
      "displayName": "Shengbing YU",
      "photoUrl": "",
      "userId": "03994843105687918628"
     },
     "user_tz": -600
    },
    "id": "tmWmXW_IPhe4",
    "outputId": "6236b8ba-e3f7-4ed1-b3f2-0caee9a33397"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "number of item_id in test set 5100\n",
      "number of item_id in train set 21807\n",
      "number of difference between two above -16707\n",
      "number of item_id include in test but not in sales_train 363\n",
      "number of item_id include in sales_train but not in test 17070\n"
     ]
    }
   ],
   "source": [
    "print('number of item_id in test set', test.item_id.nunique())\n",
    "print('number of item_id in train set', sales_train.item_id.nunique())\n",
    "diff = test.item_id.nunique() - sales_train.item_id.nunique()\n",
    "print('number of difference between two above', diff)\n",
    "diff = set(test.item_id) - set(sales_train.item_id)\n",
    "print('number of item_id include in test but not in sales_train', len(diff))\n",
    "diff =   set(sales_train.item_id) - set(test.item_id)\n",
    "print('number of item_id include in sales_train but not in test', len(diff))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "number of shop_id in test set 42\n",
      "number of shop_id in train set 57\n",
      "number of difference between two above -15\n",
      "number of shop_id include in test but not in sales_train 0\n",
      "number of shop_id include in sales_train but not in test 15\n"
     ]
    }
   ],
   "source": [
    "print('number of shop_id in test set', test.shop_id.nunique())\n",
    "print('number of shop_id in train set', sales_train.shop_id.nunique())\n",
    "diff = test.shop_id.nunique() - sales_train.shop_id.nunique()\n",
    "print('number of difference between two above', diff)\n",
    "diff = set(test.shop_id) - set(sales_train.shop_id)\n",
    "print('number of shop_id include in test but not in sales_train', len(diff))\n",
    "diff =   set(sales_train.shop_id) - set(test.shop_id)\n",
    "print('number of shop_id include in sales_train but not in test', len(diff))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "_a86LNA2Phe6"
   },
   "source": [
    "#### Note\n",
    "The difference of unique number of item_id between test and sales_train data set indicates that there are 363 item_ids which are not included in sales_train data set. We need to add these item_ids into our train data set so that train data will be similar to test data.   \n",
    "Tha main idea is to calculate monthly sales and extend it with zero sales for each unique pair within the month. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of unique item_id  22170\n",
      "Number of unique shop_id  57\n",
      "Number of unique shop_id item_id pairs  521638\n"
     ]
    }
   ],
   "source": [
    "cols = ['shop_id','item_id']\n",
    "unique_key = pd.concat([test[cols], sales_train[cols]], ignore_index=True, sort=False)\n",
    "print('Number of unique item_id ',unique_key.item_id.nunique())\n",
    "print('Number of unique shop_id ',unique_key.shop_id.nunique())\n",
    "unique_key.drop_duplicates(subset=cols, inplace=True)\n",
    "print('Number of unique shop_id item_id pairs ',unique_key.shape[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of unique shop_id item_id pairs  441092\n"
     ]
    }
   ],
   "source": [
    "if 1: # delete shop_id which not include in test\n",
    "    unique_key = unique_key.merge(test[['shop_id']].drop_duplicates(), how = 'inner')\n",
    "    print('Number of unique shop_id item_id pairs ',unique_key.shape[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of unique shop_id item_id pairs  214200\n"
     ]
    }
   ],
   "source": [
    "if 1: # delete item_id which not include in test\n",
    "    unique_key = unique_key.merge(test[['item_id']].drop_duplicates(), how = 'inner')\n",
    "    print('Number of unique shop_id item_id pairs ',unique_key.shape[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(214200, 3)"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "shop_id      42\n",
       "item_id    5100\n",
       "dtype: int64"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "unique_key.nunique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 24950,
     "status": "ok",
     "timestamp": 1561965862686,
     "user": {
      "displayName": "Shengbing YU",
      "photoUrl": "",
      "userId": "03994843105687918628"
     },
     "user_tz": -600
    },
    "id": "WeAPQEDbPhe7",
    "outputId": "11206834-48bc-4ced-b1ee-329e04a0d85f"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 34/34 [00:03<00:00,  5.01it/s]\n"
     ]
    }
   ],
   "source": [
    "test1 = test.copy()\n",
    "test1['date_block_num'] = 34\n",
    "cols = ['date_block_num','shop_id','item_id']\n",
    "\n",
    "matrix = pd.DataFrame()\n",
    "# from itertools import product\n",
    "# grid = []\n",
    "# sales1 = sales_train.merge(test[['shop_id']].drop_duplicates(), how = 'inner')\n",
    "for i in tqdm(range(0,34)):\n",
    "    unique_key['date_block_num'] = i\n",
    "    matrix = pd.concat([matrix,unique_key])\n",
    "    \n",
    "#     sales2 = sales1[sales1.date_block_num == i]\n",
    "    \n",
    "#     cur_shops = sales2['shop_id'].unique()\n",
    "\n",
    "#     cur_items = sales2['item_id'].unique()\n",
    "\n",
    "#     grid.append(np.array(list(product(*[cur_shops, cur_items, [i]])),dtype='int32'))\n",
    "# index_cols = ['shop_id', 'item_id', 'date_block_num']\n",
    "# matrix = pd.DataFrame(np.vstack(grid), columns = index_cols,dtype=np.int32)\n",
    "\n",
    "matrix = pd.concat([matrix,test1[cols]], ignore_index=True, sort=False)\n",
    "del test1, unique_key"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shop_id    0.0\n",
      "item_id    0.0\n",
      "dtype: float64\n"
     ]
    }
   ],
   "source": [
    "matrix.sort_values(['date_block_num','shop_id','item_id'],inplace=True)\n",
    "\n",
    "# check id order\n",
    "matrix_tmp = matrix.copy()\n",
    "for m in range(0,35):\n",
    "    matrix_tmp.loc[matrix.date_block_num==m,['shop_id','item_id']] -= test[['shop_id','item_id']]\n",
    "print(matrix_tmp.loc[:,['shop_id','item_id']].sum())\n",
    "del matrix_tmp"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(7497000, 3) (7282800, 3)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>shop_id</th>\n",
       "      <th>item_id</th>\n",
       "      <th>date_block_num</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>7432375</th>\n",
       "      <td>59</td>\n",
       "      <td>22162</td>\n",
       "      <td>34</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7431599</th>\n",
       "      <td>59</td>\n",
       "      <td>22163</td>\n",
       "      <td>34</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7432042</th>\n",
       "      <td>59</td>\n",
       "      <td>22164</td>\n",
       "      <td>34</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7435128</th>\n",
       "      <td>59</td>\n",
       "      <td>22166</td>\n",
       "      <td>34</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7431177</th>\n",
       "      <td>59</td>\n",
       "      <td>22167</td>\n",
       "      <td>34</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         shop_id  item_id  date_block_num\n",
       "7432375       59    22162              34\n",
       "7431599       59    22163              34\n",
       "7432042       59    22164              34\n",
       "7435128       59    22166              34\n",
       "7431177       59    22167              34"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(matrix.shape, matrix[matrix.date_block_num<34].shape)\n",
    "matrix.tail()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "bEuvNhKcPhe-"
   },
   "source": [
    "## 2.4. Text features with TF-IDF<a name=\"Text24\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 414
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 25197,
     "status": "ok",
     "timestamp": 1561965862983,
     "user": {
      "displayName": "Shengbing YU",
      "photoUrl": "",
      "userId": "03994843105687918628"
     },
     "user_tz": -600
    },
    "id": "DEu2yD_UPhe_",
    "outputId": "ecd79db8-0fa9-4010-fc2f-d106a6a27495"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>item_name</th>\n",
       "      <th>item_id</th>\n",
       "      <th>item_category_id</th>\n",
       "      <th>item_name_len</th>\n",
       "      <th>item_name_wc</th>\n",
       "      <th>item_name_tfidf_0</th>\n",
       "      <th>item_name_tfidf_1</th>\n",
       "      <th>item_name_tfidf_2</th>\n",
       "      <th>item_name_tfidf_3</th>\n",
       "      <th>item_name_tfidf_4</th>\n",
       "      <th>item_name_tfidf_5</th>\n",
       "      <th>item_name_tfidf_6</th>\n",
       "      <th>item_name_tfidf_7</th>\n",
       "      <th>item_name_tfidf_8</th>\n",
       "      <th>item_name_tfidf_9</th>\n",
       "      <th>item_name_tfidf_10</th>\n",
       "      <th>item_name_tfidf_11</th>\n",
       "      <th>item_name_tfidf_12</th>\n",
       "      <th>item_name_tfidf_13</th>\n",
       "      <th>item_name_tfidf_14</th>\n",
       "      <th>item_name_tfidf_15</th>\n",
       "      <th>item_name_tfidf_16</th>\n",
       "      <th>item_name_tfidf_17</th>\n",
       "      <th>item_name_tfidf_18</th>\n",
       "      <th>item_name_tfidf_19</th>\n",
       "      <th>item_name_tfidf_20</th>\n",
       "      <th>item_name_tfidf_21</th>\n",
       "      <th>item_name_tfidf_22</th>\n",
       "      <th>item_name_tfidf_23</th>\n",
       "      <th>item_name_tfidf_24</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>! ВО ВЛАСТИ НАВАЖДЕНИЯ (ПЛАСТ.)         D</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>41</td>\n",
       "      <td>14</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>!ABBYY FineReader 12 Professional Edition Full...</td>\n",
       "      <td>1</td>\n",
       "      <td>76</td>\n",
       "      <td>68</td>\n",
       "      <td>9</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.637695</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.442627</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.403809</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.483887</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>***В ЛУЧАХ СЛАВЫ   (UNV)                    D</td>\n",
       "      <td>2</td>\n",
       "      <td>40</td>\n",
       "      <td>45</td>\n",
       "      <td>26</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>***ГОЛУБАЯ ВОЛНА  (Univ)                      D</td>\n",
       "      <td>3</td>\n",
       "      <td>40</td>\n",
       "      <td>47</td>\n",
       "      <td>26</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>***КОРОБКА (СТЕКЛО)                       D</td>\n",
       "      <td>4</td>\n",
       "      <td>40</td>\n",
       "      <td>43</td>\n",
       "      <td>25</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                           item_name  item_id  \\\n",
       "0          ! ВО ВЛАСТИ НАВАЖДЕНИЯ (ПЛАСТ.)         D        0   \n",
       "1  !ABBYY FineReader 12 Professional Edition Full...        1   \n",
       "2      ***В ЛУЧАХ СЛАВЫ   (UNV)                    D        2   \n",
       "3    ***ГОЛУБАЯ ВОЛНА  (Univ)                      D        3   \n",
       "4        ***КОРОБКА (СТЕКЛО)                       D        4   \n",
       "\n",
       "   item_category_id  item_name_len  item_name_wc  item_name_tfidf_0  \\\n",
       "0                40             41            14                0.0   \n",
       "1                76             68             9                0.0   \n",
       "2                40             45            26                0.0   \n",
       "3                40             47            26                0.0   \n",
       "4                40             43            25                0.0   \n",
       "\n",
       "   item_name_tfidf_1  item_name_tfidf_2  item_name_tfidf_3  item_name_tfidf_4  \\\n",
       "0                0.0                0.0                0.0                0.0   \n",
       "1                0.0                0.0                0.0                0.0   \n",
       "2                0.0                0.0                0.0                0.0   \n",
       "3                0.0                0.0                0.0                0.0   \n",
       "4                0.0                0.0                0.0                0.0   \n",
       "\n",
       "   item_name_tfidf_5  item_name_tfidf_6  item_name_tfidf_7  item_name_tfidf_8  \\\n",
       "0                0.0                0.0           0.000000                0.0   \n",
       "1                0.0                0.0           0.637695                0.0   \n",
       "2                0.0                0.0           0.000000                0.0   \n",
       "3                0.0                0.0           0.000000                0.0   \n",
       "4                0.0                0.0           0.000000                0.0   \n",
       "\n",
       "   item_name_tfidf_9  item_name_tfidf_10  item_name_tfidf_11  \\\n",
       "0                0.0                 0.0                 0.0   \n",
       "1                0.0                 0.0                 0.0   \n",
       "2                0.0                 0.0                 0.0   \n",
       "3                0.0                 0.0                 0.0   \n",
       "4                0.0                 0.0                 0.0   \n",
       "\n",
       "   item_name_tfidf_12  item_name_tfidf_13  item_name_tfidf_14  \\\n",
       "0            0.000000                 0.0                 0.0   \n",
       "1            0.442627                 0.0                 0.0   \n",
       "2            0.000000                 0.0                 0.0   \n",
       "3            0.000000                 0.0                 0.0   \n",
       "4            0.000000                 0.0                 0.0   \n",
       "\n",
       "   item_name_tfidf_15  item_name_tfidf_16  item_name_tfidf_17  \\\n",
       "0                 0.0            0.000000                 0.0   \n",
       "1                 0.0            0.403809                 0.0   \n",
       "2                 0.0            0.000000                 0.0   \n",
       "3                 0.0            0.000000                 0.0   \n",
       "4                 0.0            0.000000                 0.0   \n",
       "\n",
       "   item_name_tfidf_18  item_name_tfidf_19  item_name_tfidf_20  \\\n",
       "0                 0.0                 0.0                 0.0   \n",
       "1                 0.0                 0.0                 0.0   \n",
       "2                 0.0                 0.0                 0.0   \n",
       "3                 0.0                 0.0                 0.0   \n",
       "4                 0.0                 0.0                 0.0   \n",
       "\n",
       "   item_name_tfidf_21  item_name_tfidf_22  item_name_tfidf_23  \\\n",
       "0                 0.0                 0.0                 0.0   \n",
       "1                 0.0                 0.0                 0.0   \n",
       "2                 0.0                 0.0                 0.0   \n",
       "3                 0.0                 0.0                 0.0   \n",
       "4                 0.0                 0.0                 0.0   \n",
       "\n",
       "   item_name_tfidf_24  \n",
       "0            0.000000  \n",
       "1            0.483887  \n",
       "2            0.000000  \n",
       "3            0.000000  \n",
       "4            0.000000  "
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nb_features = 25\n",
    "tfidf = text.TfidfVectorizer(max_features=nb_features)\n",
    "items_copy['item_name_len'] = items_copy['item_name'].map(len).astype(np.int8)  # Name lenth\n",
    "items_copy['item_name_wc'] = items_copy['item_name'].map(\n",
    "    lambda x: len(str(x).split(' '))).astype(np.int8)  # Word Count\n",
    "txtFeatures = pd.DataFrame(tfidf.fit_transform(items_copy['item_name']).toarray())\n",
    "cols = txtFeatures.columns\n",
    "for i in range(nb_features):\n",
    "    items_copy['item_name_tfidf_' + str(i)] = txtFeatures[cols[i]].astype(np.float16)\n",
    "items_copy.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['item_name', 'item_id', 'item_category_id', 'item_name_len',\n",
       "       'item_name_wc', 'item_name_tfidf_0', 'item_name_tfidf_1',\n",
       "       'item_name_tfidf_2', 'item_name_tfidf_3', 'item_name_tfidf_4',\n",
       "       'item_name_tfidf_5', 'item_name_tfidf_6', 'item_name_tfidf_7',\n",
       "       'item_name_tfidf_8', 'item_name_tfidf_9', 'item_name_tfidf_10',\n",
       "       'item_name_tfidf_11', 'item_name_tfidf_12', 'item_name_tfidf_13',\n",
       "       'item_name_tfidf_14', 'item_name_tfidf_15', 'item_name_tfidf_16',\n",
       "       'item_name_tfidf_17', 'item_name_tfidf_18', 'item_name_tfidf_19',\n",
       "       'item_name_tfidf_20', 'item_name_tfidf_21', 'item_name_tfidf_22',\n",
       "       'item_name_tfidf_23', 'item_name_tfidf_24'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "items_copy.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 380
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 25157,
     "status": "ok",
     "timestamp": 1561965862986,
     "user": {
      "displayName": "Shengbing YU",
      "photoUrl": "",
      "userId": "03994843105687918628"
     },
     "user_tz": -600
    },
    "id": "HdKsacZOPhfB",
    "outputId": "3f78c871-a1c8-4533-84b4-fe047bc62885"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>shop_name</th>\n",
       "      <th>shop_id</th>\n",
       "      <th>shop_name_len</th>\n",
       "      <th>shop_name_wc</th>\n",
       "      <th>shop_name_tfidf_0</th>\n",
       "      <th>shop_name_tfidf_1</th>\n",
       "      <th>shop_name_tfidf_2</th>\n",
       "      <th>shop_name_tfidf_3</th>\n",
       "      <th>shop_name_tfidf_4</th>\n",
       "      <th>shop_name_tfidf_5</th>\n",
       "      <th>shop_name_tfidf_6</th>\n",
       "      <th>shop_name_tfidf_7</th>\n",
       "      <th>shop_name_tfidf_8</th>\n",
       "      <th>shop_name_tfidf_9</th>\n",
       "      <th>shop_name_tfidf_10</th>\n",
       "      <th>shop_name_tfidf_11</th>\n",
       "      <th>shop_name_tfidf_12</th>\n",
       "      <th>shop_name_tfidf_13</th>\n",
       "      <th>shop_name_tfidf_14</th>\n",
       "      <th>shop_name_tfidf_15</th>\n",
       "      <th>shop_name_tfidf_16</th>\n",
       "      <th>shop_name_tfidf_17</th>\n",
       "      <th>shop_name_tfidf_18</th>\n",
       "      <th>shop_name_tfidf_19</th>\n",
       "      <th>shop_name_tfidf_20</th>\n",
       "      <th>shop_name_tfidf_21</th>\n",
       "      <th>shop_name_tfidf_22</th>\n",
       "      <th>shop_name_tfidf_23</th>\n",
       "      <th>shop_name_tfidf_24</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>!Якутск Орджоникидзе, 56 фран</td>\n",
       "      <td>0</td>\n",
       "      <td>29</td>\n",
       "      <td>4</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>!Якутск ТЦ \"Центральный\" фран</td>\n",
       "      <td>1</td>\n",
       "      <td>29</td>\n",
       "      <td>4</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.322754</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.689453</td>\n",
       "      <td>0.648438</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Адыгея ТЦ \"Мега\"</td>\n",
       "      <td>2</td>\n",
       "      <td>16</td>\n",
       "      <td>3</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.866699</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.498535</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Балашиха ТРК \"Октябрь-Киномир\"</td>\n",
       "      <td>3</td>\n",
       "      <td>30</td>\n",
       "      <td>3</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Волжский ТЦ \"Волга Молл\"</td>\n",
       "      <td>4</td>\n",
       "      <td>24</td>\n",
       "      <td>4</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.905762</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.424072</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                        shop_name  shop_id  shop_name_len  shop_name_wc  \\\n",
       "0   !Якутск Орджоникидзе, 56 фран        0             29             4   \n",
       "1   !Якутск ТЦ \"Центральный\" фран        1             29             4   \n",
       "2                Адыгея ТЦ \"Мега\"        2             16             3   \n",
       "3  Балашиха ТРК \"Октябрь-Киномир\"        3             30             3   \n",
       "4        Волжский ТЦ \"Волга Молл\"        4             24             4   \n",
       "\n",
       "   shop_name_tfidf_0  shop_name_tfidf_1  shop_name_tfidf_2  shop_name_tfidf_3  \\\n",
       "0                0.0                0.0                0.0                0.0   \n",
       "1                0.0                0.0                0.0                0.0   \n",
       "2                0.0                0.0                0.0                0.0   \n",
       "3                0.0                0.0                0.0                0.0   \n",
       "4                0.0                0.0                0.0                0.0   \n",
       "\n",
       "   shop_name_tfidf_4  shop_name_tfidf_5  shop_name_tfidf_6  shop_name_tfidf_7  \\\n",
       "0                0.0                0.0                0.0                0.0   \n",
       "1                0.0                0.0                0.0                0.0   \n",
       "2                0.0                0.0                0.0                0.0   \n",
       "3                0.0                0.0                0.0                0.0   \n",
       "4                0.0                0.0                0.0                0.0   \n",
       "\n",
       "   shop_name_tfidf_8  shop_name_tfidf_9  shop_name_tfidf_10  \\\n",
       "0           0.000000           0.000000                 0.0   \n",
       "1           0.000000           0.000000                 0.0   \n",
       "2           0.866699           0.000000                 0.0   \n",
       "3           0.000000           0.000000                 0.0   \n",
       "4           0.000000           0.905762                 0.0   \n",
       "\n",
       "   shop_name_tfidf_11  shop_name_tfidf_12  shop_name_tfidf_13  \\\n",
       "0                 0.0                 0.0                 0.0   \n",
       "1                 0.0                 0.0                 0.0   \n",
       "2                 0.0                 0.0                 0.0   \n",
       "3                 0.0                 0.0                 0.0   \n",
       "4                 0.0                 0.0                 0.0   \n",
       "\n",
       "   shop_name_tfidf_14  shop_name_tfidf_15  shop_name_tfidf_16  \\\n",
       "0                 0.0                 0.0                 0.0   \n",
       "1                 0.0                 0.0                 0.0   \n",
       "2                 0.0                 0.0                 0.0   \n",
       "3                 0.0                 0.0                 0.0   \n",
       "4                 0.0                 0.0                 0.0   \n",
       "\n",
       "   shop_name_tfidf_17  shop_name_tfidf_18  shop_name_tfidf_19  \\\n",
       "0                 0.0                 0.0                 0.0   \n",
       "1                 0.0                 0.0                 0.0   \n",
       "2                 0.0                 0.0                 0.0   \n",
       "3                 0.0                 0.0                 1.0   \n",
       "4                 0.0                 0.0                 0.0   \n",
       "\n",
       "   shop_name_tfidf_20  shop_name_tfidf_21  shop_name_tfidf_22  \\\n",
       "0                 0.0            0.000000                 0.0   \n",
       "1                 0.0            0.322754                 0.0   \n",
       "2                 0.0            0.498535                 0.0   \n",
       "3                 0.0            0.000000                 0.0   \n",
       "4                 0.0            0.424072                 0.0   \n",
       "\n",
       "   shop_name_tfidf_23  shop_name_tfidf_24  \n",
       "0            0.000000            1.000000  \n",
       "1            0.689453            0.648438  \n",
       "2            0.000000            0.000000  \n",
       "3            0.000000            0.000000  \n",
       "4            0.000000            0.000000  "
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tfidf = text.TfidfVectorizer(max_features=nb_features)\n",
    "shops_copy['shop_name_len'] = shops_copy['shop_name'].map(len).astype(np.int8)  # Name lenth\n",
    "shops_copy['shop_name_wc'] = shops_copy['shop_name'].map(\n",
    "    lambda x: len(str(x).split(' '))).astype(np.int8)  # Word Count\n",
    "txtFeatures = pd.DataFrame(tfidf.fit_transform(shops_copy['shop_name']).toarray())\n",
    "cols = txtFeatures.columns\n",
    "for i in range(nb_features):\n",
    "    shops_copy['shop_name_tfidf_' + str(i)] = txtFeatures[cols[i]].astype(np.float16)\n",
    "shops_copy.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['shop_name', 'shop_id', 'shop_name_len', 'shop_name_wc',\n",
       "       'shop_name_tfidf_0', 'shop_name_tfidf_1', 'shop_name_tfidf_2',\n",
       "       'shop_name_tfidf_3', 'shop_name_tfidf_4', 'shop_name_tfidf_5',\n",
       "       'shop_name_tfidf_6', 'shop_name_tfidf_7', 'shop_name_tfidf_8',\n",
       "       'shop_name_tfidf_9', 'shop_name_tfidf_10', 'shop_name_tfidf_11',\n",
       "       'shop_name_tfidf_12', 'shop_name_tfidf_13', 'shop_name_tfidf_14',\n",
       "       'shop_name_tfidf_15', 'shop_name_tfidf_16', 'shop_name_tfidf_17',\n",
       "       'shop_name_tfidf_18', 'shop_name_tfidf_19', 'shop_name_tfidf_20',\n",
       "       'shop_name_tfidf_21', 'shop_name_tfidf_22', 'shop_name_tfidf_23',\n",
       "       'shop_name_tfidf_24'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "shops_copy.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "yaNNYR1TPhfT"
   },
   "source": [
    "## 2.5. Feature matrix<a name=\"Feature27\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "PMKGmDlxPhfG"
   },
   "source": [
    "#### Group month values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 204
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 26876,
     "status": "ok",
     "timestamp": 1561965864752,
     "user": {
      "displayName": "Shengbing YU",
      "photoUrl": "",
      "userId": "03994843105687918628"
     },
     "user_tz": -600
    },
    "id": "JXPNp0hhPhfH",
    "outputId": "028c084b-028d-4e20-ed51-99b280382016"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3582096 1373797 3261287\n"
     ]
    }
   ],
   "source": [
    "sales_train['item_cnt_day'] = sales_train['item_cnt_day'].clip(0,20).astype(np.int)\n",
    "cols = ['date_block_num','shop_id','item_id']\n",
    "group = sales_train.groupby(cols).agg({'item_cnt_day': ['sum']})\n",
    "group.columns = ['item_cnt_month']\n",
    "group['item_cnt_month'] = group['item_cnt_month'].clip(0,20).astype(np.int)\n",
    "group.reset_index(inplace=True)\n",
    "\n",
    "all_data = pd.merge(matrix, group, on=cols, how='left', suffixes=(False, False))\n",
    "\n",
    "all_data['item_cnt_month'] = all_data['item_cnt_month'].fillna(0).astype(np.int)\n",
    "\n",
    "# Sanity check\n",
    "print(sales_train['item_cnt_day'].sum(), all_data['item_cnt_month'].sum(), group['item_cnt_month'].sum())\n",
    "\n",
    "del group, matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>shop_id</th>\n",
       "      <th>item_id</th>\n",
       "      <th>date_block_num</th>\n",
       "      <th>item_cnt_month</th>\n",
       "      <th>month</th>\n",
       "      <th>days_of_month</th>\n",
       "      <th>year</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2</td>\n",
       "      <td>30</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>31</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>31</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>31</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>32</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>31</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2</td>\n",
       "      <td>33</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>31</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2</td>\n",
       "      <td>38</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>31</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   shop_id  item_id  date_block_num  item_cnt_month  month  days_of_month  \\\n",
       "0        2       30               0               0      1             31   \n",
       "1        2       31               0               0      1             31   \n",
       "2        2       32               0               0      1             31   \n",
       "3        2       33               0               1      1             31   \n",
       "4        2       38               0               0      1             31   \n",
       "\n",
       "   year  \n",
       "0     2  \n",
       "1     2  \n",
       "2     2  \n",
       "3     2  \n",
       "4     2  "
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "all_data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>shop_id</th>\n",
       "      <th>item_id</th>\n",
       "      <th>date_block_num</th>\n",
       "      <th>item_cnt_month</th>\n",
       "      <th>month</th>\n",
       "      <th>days_of_month</th>\n",
       "      <th>year</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2</td>\n",
       "      <td>30</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>31</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>31</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>31</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>32</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>31</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2</td>\n",
       "      <td>33</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>31</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2</td>\n",
       "      <td>38</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>31</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   shop_id  item_id  date_block_num  item_cnt_month  month  days_of_month  \\\n",
       "0        2       30               0               0      1             31   \n",
       "1        2       31               0               0      1             31   \n",
       "2        2       32               0               0      1             31   \n",
       "3        2       33               0               1      1             31   \n",
       "4        2       38               0               0      1             31   \n",
       "\n",
       "   year  \n",
       "0     2  \n",
       "1     2  \n",
       "2     2  \n",
       "3     2  \n",
       "4     2  "
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "all_data['month'] = all_data['date_block_num'] % 12 \n",
    "days = pd.Series([31,28,31,30,31,30,31,31,30,31,30,31])\n",
    "all_data['days_of_month'] = all_data['month'].map(days).astype(np.int8)\n",
    "all_data['month'] = all_data['month'] + 1\n",
    "all_data['year'] = all_data['date_block_num'].apply(lambda x:int((34-x)/12))\n",
    "\n",
    "all_data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "shop_id           int64\n",
       "item_id           int64\n",
       "date_block_num    int64\n",
       "item_cnt_month    int64\n",
       "month             int64\n",
       "days_of_month      int8\n",
       "year              int64\n",
       "dtype: object"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "all_data.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Merge with shops, items and item_categories"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "0E5g8U35PhfT"
   },
   "outputs": [],
   "source": [
    "all_data = pd.merge(all_data, shops, on=['shop_id'], how='left')\n",
    "all_data = pd.merge(all_data, items, on=['item_id'], how='left')\n",
    "all_data = pd.merge(all_data, item_categories, on=['item_category_id'], how='left')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 3573,
     "status": "ok",
     "timestamp": 1561966141442,
     "user": {
      "displayName": "Shengbing YU",
      "photoUrl": "",
      "userId": "03994843105687918628"
     },
     "user_tz": -600
    },
    "id": "3XrGW3TwPhfW",
    "outputId": "17c6c2e2-945c-4b3a-d675-cb5240942da1"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(7497000, 11)"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "all_data.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 5414,
     "status": "ok",
     "timestamp": 1561966154141,
     "user": {
      "displayName": "Shengbing YU",
      "photoUrl": "",
      "userId": "03994843105687918628"
     },
     "user_tz": -600
    },
    "id": "RUM4OaYSPhfZ",
    "outputId": "9c0aad98-1d58-4031-a158-2d8dbaed3431"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(7497000, 67)"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "all_data = pd.merge(all_data, items_copy.drop('item_category_id',axis=1), how='left', on='item_id')\n",
    "all_data = pd.merge(all_data, shops_copy, how='left', on='shop_id')\n",
    "del shops, items, item_categories, items_copy, shops_copy\n",
    "all_data.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "p_tvEHIVPhfP"
   },
   "outputs": [],
   "source": [
    "def downcast_dtypes(df):\n",
    "\n",
    "    float_cols = [c for c in df if df[c].dtype == \"float64\"]\n",
    "    int_cols =   [c for c in df if df[c].dtype in [\"int64\", \"int32\"]]\n",
    "\n",
    "    df[float_cols] = df[float_cols].astype(np.float32)\n",
    "    #df[float_cols] = df[float_cols].astype(np.float16)\n",
    "    df[int_cols]   = df[int_cols].astype(np.int16)\n",
    "    #df[int_cols]   = df[int_cols].astype(np.int8)\n",
    "\n",
    "    return df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 4168,
     "status": "ok",
     "timestamp": 1561966167119,
     "user": {
      "displayName": "Shengbing YU",
      "photoUrl": "",
      "userId": "03994843105687918628"
     },
     "user_tz": -600
    },
    "id": "IQg5IZhrPhfb",
    "outputId": "5d3ab4e7-b333-450e-e65c-d4a8f0fb69d7"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(7497000, 61)\n"
     ]
    }
   ],
   "source": [
    "all_data.drop(['item_name', 'item_name_len', 'item_name_wc', \n",
    "         'shop_name', 'shop_name_len', 'shop_name_wc'], axis=1, inplace=True)\n",
    "all_data = downcast_dtypes(all_data)\n",
    "\n",
    "print(all_data.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 764,
     "status": "ok",
     "timestamp": 1561966171688,
     "user": {
      "displayName": "Shengbing YU",
      "photoUrl": "",
      "userId": "03994843105687918628"
     },
     "user_tz": -600
    },
    "id": "ZOHZVsNMPhfe",
    "outputId": "9e5ea12f-62ec-42d2-9de3-15d943abcf99"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['shop_id', 'item_id', 'date_block_num', 'item_cnt_month', 'month',\n",
       "       'days_of_month', 'year', 'city_code', 'item_category_id',\n",
       "       'type_code', 'subtype_code', 'item_name_tfidf_0',\n",
       "       'item_name_tfidf_1', 'item_name_tfidf_2', 'item_name_tfidf_3',\n",
       "       'item_name_tfidf_4', 'item_name_tfidf_5', 'item_name_tfidf_6',\n",
       "       'item_name_tfidf_7', 'item_name_tfidf_8', 'item_name_tfidf_9',\n",
       "       'item_name_tfidf_10', 'item_name_tfidf_11', 'item_name_tfidf_12',\n",
       "       'item_name_tfidf_13', 'item_name_tfidf_14', 'item_name_tfidf_15',\n",
       "       'item_name_tfidf_16', 'item_name_tfidf_17', 'item_name_tfidf_18',\n",
       "       'item_name_tfidf_19', 'item_name_tfidf_20', 'item_name_tfidf_21',\n",
       "       'item_name_tfidf_22', 'item_name_tfidf_23', 'item_name_tfidf_24',\n",
       "       'shop_name_tfidf_0', 'shop_name_tfidf_1', 'shop_name_tfidf_2',\n",
       "       'shop_name_tfidf_3', 'shop_name_tfidf_4', 'shop_name_tfidf_5',\n",
       "       'shop_name_tfidf_6', 'shop_name_tfidf_7', 'shop_name_tfidf_8',\n",
       "       'shop_name_tfidf_9', 'shop_name_tfidf_10', 'shop_name_tfidf_11',\n",
       "       'shop_name_tfidf_12', 'shop_name_tfidf_13', 'shop_name_tfidf_14',\n",
       "       'shop_name_tfidf_15', 'shop_name_tfidf_16', 'shop_name_tfidf_17',\n",
       "       'shop_name_tfidf_18', 'shop_name_tfidf_19', 'shop_name_tfidf_20',\n",
       "       'shop_name_tfidf_21', 'shop_name_tfidf_22', 'shop_name_tfidf_23',\n",
       "       'shop_name_tfidf_24'], dtype=object)"
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "all_data.columns.values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 11207,
     "status": "ok",
     "timestamp": 1561966237681,
     "user": {
      "displayName": "Shengbing YU",
      "photoUrl": "",
      "userId": "03994843105687918628"
     },
     "user_tz": -600
    },
    "id": "casCDb3zcJQD",
    "outputId": "ea78f8b6-1d0b-42f8-c4a8-cbabff8cbb0d"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Series([], dtype: int64)"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "missing = all_data.isnull().sum()\n",
    "missing[missing > 0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "#all_data.fillna(0, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>shop_id</th>\n",
       "      <th>item_id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>5037</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>5320</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>5</td>\n",
       "      <td>5233</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>5</td>\n",
       "      <td>5232</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>5268</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   ID  shop_id  item_id\n",
       "0   0        5     5037\n",
       "1   1        5     5320\n",
       "2   2        5     5233\n",
       "3   3        5     5232\n",
       "4   4        5     5268"
      ]
     },
     "execution_count": 83,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [],
   "source": [
    "# save temp file\n",
    "outfile = open('test','wb')\n",
    "pickle.dump(test,outfile)\n",
    "outfile.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "FXZvR2tBPhfj"
   },
   "outputs": [],
   "source": [
    "# save temp file\n",
    "outfile = open('all_data2','wb')\n",
    "pickle.dump(all_data,outfile)\n",
    "outfile.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 63
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 30774,
     "status": "ok",
     "timestamp": 1561966343794,
     "user": {
      "displayName": "Shengbing YU",
      "photoUrl": "",
      "userId": "03994843105687918628"
     },
     "user_tz": -600
    },
    "id": "WDHObNTgcdcI",
    "outputId": "7e79f5e6-13a6-45dc-8b9a-ae79c76e9eb1"
   },
   "outputs": [],
   "source": [
    "#!tar -czvf all_data2.tar.gz  'all_data2'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "2gdBGsLUPhfl"
   },
   "source": [
    "infile = open('all_data2','rb')\n",
    "all_data = pickle.load(infile)\n",
    "infile.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "jbUwv8knPhfl"
   },
   "source": [
    "# Part 3/8: Exploratory data analysis<a name=\"Exploratory3\"></a>\n",
    "\n",
    "\n",
    "The **target variable** to be predicted is the `item_cnt_month` (number of items sold)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "7GeC2k5BPhfm"
   },
   "source": [
    "sales_train['date'] = pd.to_datetime(sales_train['date'], format=\"%d.%m.%Y\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 254210,
     "status": "ok",
     "timestamp": 1561964291126,
     "user": {
      "displayName": "Shengbing YU",
      "photoUrl": "",
      "userId": "03994843105687918628"
     },
     "user_tz": -600
    },
    "id": "8Bs6v4k4Phfn",
    "outputId": "2fcdf003-c816-43a8-931e-f229c7f4fc0a"
   },
   "source": [
    "sales_train.groupby('date').agg(\n",
    "    {'item_cnt_day': 'sum'}).plot(kind='area',\n",
    "    figsize=(12,5), title=\"The number of items sold per day\", lw=1.0, color='green')\n",
    "sales_train.groupby('date_block_num').agg(\n",
    "    {'item_cnt_day': 'sum'}).plot(kind='bar',\n",
    "    figsize=(12,5), title=\"The number of items sold per month\",lw=3.0, color='blue')\n",
    "plt.show()\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(10.2,5))\n",
    "ax = sns.kdeplot(all_data['item_cnt_month'].values, shade=True, color=\"red\", ax=ax)\n",
    "ax.set_xlabel('item_cnt_month')\n",
    "ax.set_ylabel('Density')\n",
    "ax.set_title('Target KDE distribution')\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "XsUhRPpJPhfp"
   },
   "source": [
    "#### Note\n",
    "There are sale peaks in every November."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "XarhxNYGPhga"
   },
   "source": [
    "## 3.2. Quick Baseline<a name=\"Quick32\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [],
   "source": [
    "dates = all_data['date_block_num']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "shop_id                 42\n",
       "item_id               5100\n",
       "month                   12\n",
       "days_of_month            3\n",
       "year                     3\n",
       "city_code               28\n",
       "item_category_id        62\n",
       "type_code               16\n",
       "subtype_code            49\n",
       "item_name_tfidf_0       39\n",
       "item_name_tfidf_1       30\n",
       "item_name_tfidf_2       14\n",
       "item_name_tfidf_3       15\n",
       "item_name_tfidf_4       34\n",
       "item_name_tfidf_5       20\n",
       "item_name_tfidf_6       24\n",
       "item_name_tfidf_7       53\n",
       "item_name_tfidf_8       41\n",
       "item_name_tfidf_9       11\n",
       "item_name_tfidf_10      25\n",
       "item_name_tfidf_11      71\n",
       "item_name_tfidf_12      75\n",
       "item_name_tfidf_13      25\n",
       "item_name_tfidf_14      70\n",
       "item_name_tfidf_15      51\n",
       "item_name_tfidf_16     119\n",
       "item_name_tfidf_17      46\n",
       "item_name_tfidf_18      26\n",
       "item_name_tfidf_19      36\n",
       "item_name_tfidf_20       5\n",
       "item_name_tfidf_21      76\n",
       "item_name_tfidf_22       9\n",
       "item_name_tfidf_23      15\n",
       "item_name_tfidf_24      49\n",
       "shop_name_tfidf_0        3\n",
       "shop_name_tfidf_1        2\n",
       "shop_name_tfidf_2        3\n",
       "shop_name_tfidf_3        2\n",
       "shop_name_tfidf_4        2\n",
       "shop_name_tfidf_5        2\n",
       "shop_name_tfidf_6        2\n",
       "shop_name_tfidf_7        3\n",
       "shop_name_tfidf_8        5\n",
       "shop_name_tfidf_9        4\n",
       "shop_name_tfidf_10       6\n",
       "shop_name_tfidf_11       3\n",
       "shop_name_tfidf_12       3\n",
       "shop_name_tfidf_13       3\n",
       "shop_name_tfidf_14       3\n",
       "shop_name_tfidf_15       3\n",
       "shop_name_tfidf_16       2\n",
       "shop_name_tfidf_17       2\n",
       "shop_name_tfidf_18       4\n",
       "shop_name_tfidf_19       4\n",
       "shop_name_tfidf_20       6\n",
       "shop_name_tfidf_21      12\n",
       "shop_name_tfidf_22       3\n",
       "shop_name_tfidf_23       3\n",
       "shop_name_tfidf_24       3\n",
       "dtype: int64"
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Target = 'item_cnt_month'\n",
    "selected_features = all_data.columns.tolist()\n",
    "selected_features.remove(Target)\n",
    "selected_features.remove('date_block_num')\n",
    "all_data[selected_features].nunique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [],
   "source": [
    "def standard_all_data(all_data):\n",
    "    \n",
    "    df = all_data.nunique()\n",
    "    df = df[df > 2]\n",
    "    features_to_scale = df.index.tolist()\n",
    "    features_to_scale.remove('item_cnt_month')\n",
    "\n",
    "    #X_train = all_data[dates < 34]\n",
    "    #X_test = all_data[dates == 34]\n",
    "\n",
    "    scaler = StandardScaler()\n",
    "\n",
    "    all_data.loc[dates < 34,features_to_scale] = scaler.fit_transform(all_data[dates < 34][features_to_scale])\n",
    "    all_data.loc[dates == 34,features_to_scale] = scaler.transform(all_data[dates == 34][features_to_scale])\n",
    "    \n",
    "    #all_data = pd.concat([X_train, X_test], axis = 0)\n",
    "    \n",
    "    #all_data = downcast_dtypes(all_data)\n",
    "\n",
    "    return all_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(7068600, 59)\n",
      "(7068600,)\n",
      "(214200, 59)\n",
      "(214200,)\n",
      "(214200, 59)\n"
     ]
    }
   ],
   "source": [
    "all_data_tmp = all_data.copy()\n",
    "all_data_tmp = standard_all_data(all_data_tmp)\n",
    "\n",
    "X_train = all_data_tmp[dates<33][selected_features]\n",
    "Y_train = all_data_tmp[dates<33][Target]\n",
    "\n",
    "X_valid = all_data_tmp[dates==33][selected_features]\n",
    "Y_valid = all_data_tmp[dates==33][Target]\n",
    "\n",
    "X_test =  all_data_tmp[dates==34][selected_features]\n",
    "\n",
    "\n",
    "\n",
    "print(X_train.shape)\n",
    "print(Y_train.shape)\n",
    "print(X_valid.shape)\n",
    "print(Y_valid.shape)\n",
    "print(X_test.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 54918,
     "status": "ok",
     "timestamp": 1561968508777,
     "user": {
      "displayName": "Shengbing YU",
      "photoUrl": "",
      "userId": "03994843105687918628"
     },
     "user_tz": -600
    },
    "id": "uAelbdz1Phgc",
    "outputId": "8dce608e-386f-4fb8-ca75-9b7040f28c79"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Valid error:  1.0721808294374517\n",
      "preds mean 0.29573819961760944\n"
     ]
    }
   ],
   "source": [
    "model_linreg = LinearRegression(fit_intercept=True)\n",
    "model_linreg.fit(X_train, Y_train)\n",
    "print('Valid error: ',np.sqrt(mean_squared_error(Y_valid, model_linreg.predict(X_valid))))\n",
    "preds = model_linreg.predict(X_test).clip(0,20)\n",
    "print('preds mean',preds.mean())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 6839,
     "status": "ok",
     "timestamp": 1561968536070,
     "user": {
      "displayName": "Shengbing YU",
      "photoUrl": "",
      "userId": "03994843105687918628"
     },
     "user_tz": -600
    },
    "id": "egwtSGkBPhge",
    "outputId": "998a3275-c5e4-4daf-af99-df0631aba135"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Valid error:  1.0721808433646145\n",
      "preds mean 0.29573817708662065\n"
     ]
    }
   ],
   "source": [
    "model_ridge = Ridge()\n",
    "model_ridge.fit(X_train, Y_train)\n",
    "print('Valid error: ',np.sqrt(mean_squared_error(Y_valid, model_ridge.predict(X_valid))))\n",
    "preds = model_ridge.predict(X_test).clip(0,20)\n",
    "print('preds mean',preds.mean())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 9204,
     "status": "ok",
     "timestamp": 1561968549811,
     "user": {
      "displayName": "Shengbing YU",
      "photoUrl": "",
      "userId": "03994843105687918628"
     },
     "user_tz": -600
    },
    "id": "yukmTwMDPhgg",
    "outputId": "291072ac-5a78-419b-a76c-4cd780e23b20"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Valid error:  1.0930244910155278\n",
      "preds mean 0.186589989531166\n"
     ]
    }
   ],
   "source": [
    "model_lasso = Lasso(alpha=10.0, fit_intercept=True, max_iter=1000)\n",
    "model_lasso.fit(X_train, Y_train)\n",
    "print('Valid error: ',np.sqrt(mean_squared_error(Y_valid, model_lasso.predict(X_valid))))\n",
    "preds = model_lasso.predict(X_test).clip(0,20)\n",
    "print('preds mean',preds.mean())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [],
   "source": [
    "del all_data_tmp"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "mw0r1wZ4Phfq"
   },
   "source": [
    "# Part 4/8: Advanced Feature Engineering<a name=\"Advanced4\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "NBtKIKpzPhf2"
   },
   "source": [
    "## 4.1. Matrix factorization of TFIDF processed features<a name=\"Matrix42\"></a>\n",
    "\n",
    "To reduce the dimensionality of the TFIDF-processed features (from 50 dimensions to 10) and extract nonlinear relationships between the text features with **Non-Negative Matrix Factorization (NMF)**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 3232,
     "status": "ok",
     "timestamp": 1561967227089,
     "user": {
      "displayName": "Shengbing YU",
      "photoUrl": "",
      "userId": "03994843105687918628"
     },
     "user_tz": -600
    },
    "id": "mvoL44xhPhf2",
    "outputId": "ce8600bd-29cb-462a-af14-e0743da02c67"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(7282800, 50) (214200, 50)\n"
     ]
    }
   ],
   "source": [
    "tfidf_features = ['item_name_tfidf_0', 'item_name_tfidf_1', 'item_name_tfidf_2',\n",
    "                  'item_name_tfidf_3', 'item_name_tfidf_4', 'item_name_tfidf_5',\n",
    "                  'item_name_tfidf_6', 'item_name_tfidf_7', 'item_name_tfidf_8',\n",
    "                  'item_name_tfidf_9', 'item_name_tfidf_10', 'item_name_tfidf_11',\n",
    "                  'item_name_tfidf_12', 'item_name_tfidf_13', 'item_name_tfidf_14',\n",
    "                  'item_name_tfidf_15', 'item_name_tfidf_16', 'item_name_tfidf_17',\n",
    "                  'item_name_tfidf_18', 'item_name_tfidf_19', 'item_name_tfidf_20',\n",
    "                  'item_name_tfidf_21', 'item_name_tfidf_22', 'item_name_tfidf_23',\n",
    "                  'item_name_tfidf_24', 'shop_name_tfidf_0', 'shop_name_tfidf_1',\n",
    "                  'shop_name_tfidf_2', 'shop_name_tfidf_3', 'shop_name_tfidf_4',\n",
    "                  'shop_name_tfidf_5', 'shop_name_tfidf_6', 'shop_name_tfidf_7',\n",
    "                  'shop_name_tfidf_8', 'shop_name_tfidf_9', 'shop_name_tfidf_10',\n",
    "                  'shop_name_tfidf_11', 'shop_name_tfidf_12', 'shop_name_tfidf_13',\n",
    "                  'shop_name_tfidf_14', 'shop_name_tfidf_15', 'shop_name_tfidf_16',\n",
    "                  'shop_name_tfidf_17', 'shop_name_tfidf_18', 'shop_name_tfidf_19',\n",
    "                  'shop_name_tfidf_20', 'shop_name_tfidf_21', 'shop_name_tfidf_22',\n",
    "                  'shop_name_tfidf_23', 'shop_name_tfidf_24']\n",
    "\n",
    "Xtrain = all_data[all_data['date_block_num']<34][tfidf_features]\n",
    "Xtest = all_data[all_data['date_block_num']==34][tfidf_features]\n",
    "print(Xtrain.shape, Xtest.shape)\n",
    "#X_all = pd.concat([Xtrain, Xtest], ignore_index=True)\n",
    "#X_all.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "8YVzJ25uPhf5"
   },
   "outputs": [],
   "source": [
    "nmf = NMF(n_components=10, \n",
    "          init=None, solver='cd', \n",
    "          beta_loss='frobenius', \n",
    "          tol=0.0001, \n",
    "          max_iter=200)\n",
    "nmf.fit(all_data[tfidf_features]);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "c_ox6vViPhf7"
   },
   "outputs": [],
   "source": [
    "Xtrain = nmf.transform(Xtrain)\n",
    "Xtest = nmf.transform(Xtest)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 729,
     "status": "ok",
     "timestamp": 1561967562135,
     "user": {
      "displayName": "Shengbing YU",
      "photoUrl": "",
      "userId": "03994843105687918628"
     },
     "user_tz": -600
    },
    "id": "g_pjZj5gPhf9",
    "outputId": "be2c9d38-22c2-4988-fbd4-0bdcc989001a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(7282800, 10) (214200, 10)\n"
     ]
    }
   ],
   "source": [
    "print(Xtrain.shape, Xtest.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "VhSq-OiaPhgB"
   },
   "outputs": [],
   "source": [
    "all_data.drop(tfidf_features, axis=1, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 261
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 828,
     "status": "ok",
     "timestamp": 1561967573107,
     "user": {
      "displayName": "Shengbing YU",
      "photoUrl": "",
      "userId": "03994843105687918628"
     },
     "user_tz": -600
    },
    "id": "nr9qYgKJPhgD",
    "outputId": "b43da9d1-6fc7-4365-ed86-0f1a4040d04e"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(7497000, 10)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>tfidf_interaction_1</th>\n",
       "      <th>tfidf_interaction_2</th>\n",
       "      <th>tfidf_interaction_3</th>\n",
       "      <th>tfidf_interaction_4</th>\n",
       "      <th>tfidf_interaction_5</th>\n",
       "      <th>tfidf_interaction_6</th>\n",
       "      <th>tfidf_interaction_7</th>\n",
       "      <th>tfidf_interaction_8</th>\n",
       "      <th>tfidf_interaction_9</th>\n",
       "      <th>tfidf_interaction_10</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.003910</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.029964</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.003729</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.032719</td>\n",
       "      <td>0.029942</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.003910</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.029964</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.003729</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.032719</td>\n",
       "      <td>0.029942</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.003910</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.029964</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   tfidf_interaction_1  tfidf_interaction_2  tfidf_interaction_3  \\\n",
       "0             0.003910                  0.0                  0.0   \n",
       "1             0.003729                  0.0                  0.0   \n",
       "2             0.003910                  0.0                  0.0   \n",
       "3             0.003729                  0.0                  0.0   \n",
       "4             0.003910                  0.0                  0.0   \n",
       "\n",
       "   tfidf_interaction_4  tfidf_interaction_5  tfidf_interaction_6  \\\n",
       "0                  0.0             0.000000             0.029964   \n",
       "1                  0.0             0.032719             0.029942   \n",
       "2                  0.0             0.000000             0.029964   \n",
       "3                  0.0             0.032719             0.029942   \n",
       "4                  0.0             0.000000             0.029964   \n",
       "\n",
       "   tfidf_interaction_7  tfidf_interaction_8  tfidf_interaction_9  \\\n",
       "0                  0.0                  0.0                  0.0   \n",
       "1                  0.0                  0.0                  0.0   \n",
       "2                  0.0                  0.0                  0.0   \n",
       "3                  0.0                  0.0                  0.0   \n",
       "4                  0.0                  0.0                  0.0   \n",
       "\n",
       "   tfidf_interaction_10  \n",
       "0                   0.0  \n",
       "1                   0.0  \n",
       "2                   0.0  \n",
       "3                   0.0  \n",
       "4                   0.0  "
      ]
     },
     "execution_count": 99,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tfidf_reduced_df = pd.concat([pd.DataFrame(Xtrain), pd.DataFrame(Xtest)], axis=0)\n",
    "tfidf_reduced_df.columns = ['tfidf_interaction_1', 'tfidf_interaction_2', 'tfidf_interaction_3',\n",
    "                            'tfidf_interaction_4', 'tfidf_interaction_5', 'tfidf_interaction_6',\n",
    "                            'tfidf_interaction_7', 'tfidf_interaction_8', 'tfidf_interaction_9',\n",
    "                            'tfidf_interaction_10']\n",
    "print(tfidf_reduced_df.shape)\n",
    "tfidf_reduced_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 187
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 1102,
     "status": "ok",
     "timestamp": 1561967578519,
     "user": {
      "displayName": "Shengbing YU",
      "photoUrl": "",
      "userId": "03994843105687918628"
     },
     "user_tz": -600
    },
    "id": "qVrYjBclPhgG",
    "outputId": "77eb38bb-3665-4a4f-a213-7249d5746bf6"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tfidf_interaction_1\n",
      "tfidf_interaction_2\n",
      "tfidf_interaction_3\n",
      "tfidf_interaction_4\n",
      "tfidf_interaction_5\n",
      "tfidf_interaction_6\n",
      "tfidf_interaction_7\n",
      "tfidf_interaction_8\n",
      "tfidf_interaction_9\n",
      "tfidf_interaction_10\n"
     ]
    }
   ],
   "source": [
    "for col in tfidf_reduced_df.columns:\n",
    "    print(col)\n",
    "    all_data[col] = tfidf_reduced_df[col].values\n",
    "     \n",
    "del tfidf_reduced_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [],
   "source": [
    "Target = 'item_cnt_month'\n",
    "selected_features = all_data.columns.tolist()\n",
    "selected_features.remove(Target)\n",
    "selected_features.remove('date_block_num')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(7068600, 19)\n",
      "(7068600,)\n",
      "(214200, 19)\n",
      "(214200,)\n",
      "(214200, 19)\n"
     ]
    }
   ],
   "source": [
    "all_data_tmp = all_data.copy()\n",
    "all_data_tmp = standard_all_data(all_data_tmp)\n",
    "\n",
    "X_train = all_data_tmp[dates<33][selected_features]\n",
    "Y_train = all_data_tmp[dates<33][Target]\n",
    "\n",
    "X_valid = all_data_tmp[dates==33][selected_features]\n",
    "Y_valid = all_data_tmp[dates==33][Target]\n",
    "\n",
    "X_test =  all_data_tmp[dates==34][selected_features]\n",
    "\n",
    "\n",
    "\n",
    "print(X_train.shape)\n",
    "print(Y_train.shape)\n",
    "print(X_valid.shape)\n",
    "print(Y_valid.shape)\n",
    "print(X_test.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 54918,
     "status": "ok",
     "timestamp": 1561968508777,
     "user": {
      "displayName": "Shengbing YU",
      "photoUrl": "",
      "userId": "03994843105687918628"
     },
     "user_tz": -600
    },
    "id": "uAelbdz1Phgc",
    "outputId": "8dce608e-386f-4fb8-ca75-9b7040f28c79"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Valid error:  1.0829878880762345\n",
      "preds mean 0.2944902178390091\n"
     ]
    }
   ],
   "source": [
    "model_linreg = LinearRegression(fit_intercept=True)\n",
    "model_linreg.fit(X_train, Y_train)\n",
    "print('Valid error: ',np.sqrt(mean_squared_error(Y_valid, model_linreg.predict(X_valid))))\n",
    "preds = model_linreg.predict(X_test).clip(0,20)\n",
    "print('preds mean',preds.mean())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 6839,
     "status": "ok",
     "timestamp": 1561968536070,
     "user": {
      "displayName": "Shengbing YU",
      "photoUrl": "",
      "userId": "03994843105687918628"
     },
     "user_tz": -600
    },
    "id": "egwtSGkBPhge",
    "outputId": "998a3275-c5e4-4daf-af99-df0631aba135"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Valid error:  1.0829878899119134\n",
      "preds mean 0.2944902008211132\n"
     ]
    }
   ],
   "source": [
    "model_ridge = Ridge()\n",
    "model_ridge.fit(X_train, Y_train)\n",
    "print('Valid error: ',np.sqrt(mean_squared_error(Y_valid, model_ridge.predict(X_valid))))\n",
    "preds = model_ridge.predict(X_test).clip(0,20)\n",
    "print('preds mean',preds.mean())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 9204,
     "status": "ok",
     "timestamp": 1561968549811,
     "user": {
      "displayName": "Shengbing YU",
      "photoUrl": "",
      "userId": "03994843105687918628"
     },
     "user_tz": -600
    },
    "id": "yukmTwMDPhgg",
    "outputId": "291072ac-5a78-419b-a76c-4cd780e23b20"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Valid error:  1.0930244910155278\n",
      "preds mean 0.186589989531166\n"
     ]
    }
   ],
   "source": [
    "model_lasso = Lasso(alpha=10.0, fit_intercept=True, max_iter=1000)\n",
    "model_lasso.fit(X_train, Y_train)\n",
    "print('Valid error: ',np.sqrt(mean_squared_error(Y_valid, model_lasso.predict(X_valid))))\n",
    "preds = model_lasso.predict(X_test).clip(0,20)\n",
    "print('preds mean',preds.mean())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# save temp file\n",
    "outfile = open('all_data40','wb')\n",
    "pickle.dump(all_data,outfile)\n",
    "outfile.close()\n",
    "all_data.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {},
   "outputs": [],
   "source": [
    "del all_data_tmp"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "infile = open('all_data40','rb')\n",
    "all_data = pickle.load(infile)\n",
    "infile.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.2. Mean encodings on categorical features (not working,overfitting)<a name=\"Mean41\"></a>\n",
    "\n",
    "Mean encodings look to map a high cardinality categorical feature into a 1D array (instead of high numbers of them had we used one-hot encoding) based on how often the target variable appears on average in the categorical feature. It is a convenient approach since it also has a clever way of imputing missing values among categorical variables."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.3. Last sales month<a name=\"Last\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 2/2 [00:40<00:00, 16.25s/it]\n"
     ]
    }
   ],
   "source": [
    "group = sales_train.groupby(['date_block_num', 'shop_id', 'item_id']).agg(\n",
    "    {'item_cnt_day': ['sum']})\n",
    "group.columns = ['item_cnt_month']\n",
    "group['item_cnt_month'] = group['item_cnt_month'].clip(0,20)\n",
    "group.reset_index(inplace=True)\n",
    "\n",
    "for col in tqdm([['item_id'],['shop_id', 'item_id']]):\n",
    "    last_sale = pd.DataFrame()\n",
    "    for month in range(12,35):    \n",
    "        last_month = group.loc[(group['date_block_num']<month)&(\n",
    "            group['item_cnt_month']>0)].groupby(col).agg(\n",
    "            {'date_block_num':['max','min']})\n",
    "        feature_name = ['last_sale_' + ''.join(col),'first_sale_' + ''.join(col)]\n",
    "        last_month.columns = feature_name\n",
    "        last_month.reset_index(inplace=True)\n",
    "        df = pd.DataFrame()\n",
    "        df[col+feature_name] = last_month[col+feature_name]\n",
    "        df[feature_name] = month - df[feature_name]\n",
    "        df['date_block_num'] = month\n",
    "        \n",
    "#         df = pd.DataFrame({'date_block_num':np.ones([last_month.shape[0],])*month,\n",
    "#                            col[0]: last_month.index.values,\n",
    "#                            'last_sale' + '_'.join(col): last_month.values})\n",
    "        last_sale = last_sale.append(df)\n",
    "#     last_sale['date_block_num'] = last_sale['date_block_num'].astype(np.int8)\n",
    "\n",
    "    all_data = all_data.merge(last_sale, on=['date_block_num']+col, \n",
    "                              how='left',suffixes=(False, False))\n",
    "    all_data[feature_name] = all_data[feature_name].fillna(100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['shop_id', 'item_id', 'date_block_num', 'item_cnt_month', 'month',\n",
       "       'days_of_month', 'year', 'city_code', 'item_category_id', 'type_code',\n",
       "       'subtype_code', 'tfidf_interaction_1', 'tfidf_interaction_2',\n",
       "       'tfidf_interaction_3', 'tfidf_interaction_4', 'tfidf_interaction_5',\n",
       "       'tfidf_interaction_6', 'tfidf_interaction_7', 'tfidf_interaction_8',\n",
       "       'tfidf_interaction_9', 'tfidf_interaction_10', 'last_sale_item_id',\n",
       "       'first_sale_item_id', 'last_sale_shop_iditem_id',\n",
       "       'first_sale_shop_iditem_id'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "all_data.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.0"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "all_data['first_sale_shop_iditem_id'].min()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.4. Creating lag features<a name=\"lag\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['date_block_num', 'shop_id', 'item_id', 'item_category_id', 'type_code',\n",
       "       'subtype_code', 'city_code', 'days_of_month', 'item_cnt_month'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Target = 'item_cnt_month'\n",
    "index_cols = ['date_block_num','shop_id', 'item_id',\n",
    "              'item_category_id','type_code','subtype_code','city_code','days_of_month'] \n",
    "train = all_data[all_data['date_block_num']<34][index_cols + [Target]].copy()\n",
    "train.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 9/9 [00:32<00:00,  4.15s/it]\n"
     ]
    }
   ],
   "source": [
    "group_cols = ['date_block_num', 'shop_id', 'item_id']\n",
    "group = sales_train.groupby(group_cols).agg(\n",
    "    {'item_price':['mean']})\n",
    "group.columns = ['price_shop_item_mean']\n",
    "group.reset_index(inplace=True)\n",
    "train = pd.merge(train, group, on=group_cols, how='left', suffixes=(False, False))\n",
    "\n",
    "group_cols = ['item_id', 'date_block_num']\n",
    "group = sales_train.groupby(group_cols).agg(\n",
    "    {'item_price':['mean']})\n",
    "group.columns = ['price_item_mean']\n",
    "group.reset_index(inplace=True)\n",
    "train = pd.merge(train, group, on=group_cols, how='left', suffixes=(False, False))\n",
    "\n",
    "group_cols = ['item_id', 'date_block_num']\n",
    "group = train.groupby(group_cols).agg({'item_cnt_month':'mean'})\n",
    "group.columns = ['item_cnt_month_by_item_id_mean']\n",
    "group.reset_index(inplace=True)\n",
    "train = pd.merge(train, group, on=group_cols, how='left', suffixes=(False, False))\n",
    "\n",
    "cols_group = [\n",
    "['date_block_num', 'shop_id'],\n",
    "['date_block_num', 'item_category_id'],\n",
    "['date_block_num', 'shop_id', 'item_category_id'],\n",
    "['date_block_num', 'shop_id', 'type_code'],\n",
    "['date_block_num', 'shop_id', 'subtype_code'],\n",
    "['date_block_num', 'city_code'],\n",
    "['date_block_num', 'item_id', 'city_code'],\n",
    "['date_block_num', 'type_code'],\n",
    "['date_block_num', 'subtype_code'],\n",
    "]\n",
    "\n",
    "for cols in tqdm(cols_group):\n",
    "    group = train.groupby(cols).agg({'item_cnt_month':'mean'})\n",
    "    group.columns = ['item_cnt_month_by_' + '_'.join(cols)]\n",
    "    group.reset_index(inplace=True)\n",
    "    train = pd.merge(train, group, on=cols, how='left', suffixes=(False, False))\n",
    "\n",
    "lag_cols = ['item_cnt_month',\n",
    "            'item_cnt_month_by_item_id_mean',\n",
    "            'price_shop_item_mean',\n",
    "            'price_item_mean',\n",
    "           ]\n",
    "lag_cols += ['item_cnt_month_by_' + '_'.join(cols) for cols in cols_group]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 11/11 [00:00<00:00, 12.97it/s]\n"
     ]
    }
   ],
   "source": [
    "item_cnt_list = [e for e in lag_cols if e not in ['price_shop_item_mean','price_item_mean']]\n",
    "for col in tqdm(item_cnt_list):\n",
    "    train[col] /= train['days_of_month']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['date_block_num', 'shop_id', 'item_id', 'item_category_id', 'type_code',\n",
       "       'subtype_code', 'city_code', 'days_of_month', 'item_cnt_month',\n",
       "       'price_shop_item_mean', 'price_item_mean',\n",
       "       'item_cnt_month_by_item_id_mean',\n",
       "       'item_cnt_month_by_date_block_num_shop_id',\n",
       "       'item_cnt_month_by_date_block_num_item_category_id',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_item_category_id',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_type_code',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_subtype_code',\n",
       "       'item_cnt_month_by_date_block_num_city_code',\n",
       "       'item_cnt_month_by_date_block_num_item_id_city_code',\n",
       "       'item_cnt_month_by_date_block_num_type_code',\n",
       "       'item_cnt_month_by_date_block_num_subtype_code'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "price_shop_item_mean    6681051\n",
       "price_item_mean         4358256\n",
       "dtype: int64"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "missing = train.isnull().sum()\n",
    "missing[missing > 0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['item_cnt_month',\n",
       " 'item_cnt_month_by_item_id_mean',\n",
       " 'price_shop_item_mean',\n",
       " 'price_item_mean',\n",
       " 'item_cnt_month_by_date_block_num_shop_id',\n",
       " 'item_cnt_month_by_date_block_num_item_category_id',\n",
       " 'item_cnt_month_by_date_block_num_shop_id_item_category_id',\n",
       " 'item_cnt_month_by_date_block_num_shop_id_type_code',\n",
       " 'item_cnt_month_by_date_block_num_shop_id_subtype_code',\n",
       " 'item_cnt_month_by_date_block_num_city_code',\n",
       " 'item_cnt_month_by_date_block_num_item_id_city_code',\n",
       " 'item_cnt_month_by_date_block_num_type_code',\n",
       " 'item_cnt_month_by_date_block_num_subtype_code']"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lag_cols"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['item_cnt_month',\n",
       " 'item_cnt_month_by_item_id_mean',\n",
       " 'item_cnt_month_by_date_block_num_shop_id',\n",
       " 'item_cnt_month_by_date_block_num_item_category_id',\n",
       " 'item_cnt_month_by_date_block_num_shop_id_item_category_id',\n",
       " 'item_cnt_month_by_date_block_num_shop_id_type_code',\n",
       " 'item_cnt_month_by_date_block_num_shop_id_subtype_code',\n",
       " 'item_cnt_month_by_date_block_num_city_code',\n",
       " 'item_cnt_month_by_date_block_num_item_id_city_code',\n",
       " 'item_cnt_month_by_date_block_num_type_code',\n",
       " 'item_cnt_month_by_date_block_num_subtype_code']"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "item_cnt_list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 12/12 [1:14:57<00:00, 785.86s/it]\n"
     ]
    }
   ],
   "source": [
    "# 2.2 Creating item/shop pair lags lag-based features ----------------------------\n",
    "\n",
    "last_lag_names = []\n",
    "\n",
    "lag_month = [1,2,3,4,5,6,7,8,9,10,11,12]\n",
    "all_data = all_data[all_data.date_block_num >= max(lag_month)].copy()\n",
    "i = 0\n",
    "eps = 1e-5\n",
    "index_cols = ['date_block_num','shop_id', 'item_id']\n",
    "train_shift = train[index_cols + lag_cols].copy()\n",
    "for m in tqdm(lag_month):\n",
    "\n",
    "    #train_shift = train[index_cols + cols_to_rename].copy()\n",
    "    train_shift.columns = index_cols + lag_cols\n",
    "\n",
    "    train_shift['date_block_num'] = train['date_block_num'] + m\n",
    "\n",
    "    lag_names = [col + '_lag_' + str(m) for col in lag_cols]\n",
    "\n",
    "    train_shift.columns = index_cols + lag_names\n",
    "\n",
    "    all_data = pd.merge(all_data, train_shift, on=index_cols, how='left', suffixes=(False, False))\n",
    "    \n",
    "    item_cnt_lag_list = [col + '_lag_' + str(m) for col in item_cnt_list]\n",
    "    for col in item_cnt_lag_list:\n",
    "        all_data[col] *= all_data['days_of_month']\n",
    "\n",
    "    feature_name = ['is_sold_shopitem_month_' + str(m),\n",
    "                    'is_sold_item_month_' + str(m),\n",
    "                    'is_sales_down_shopitem_month_' + str(m),\n",
    "                    'is_sales_up_shopitem_month_' + str(m), \n",
    "                    'is_sales_down_item_month_' + str(m),\n",
    "                    'is_sales_up_item_month_' + str(m), \n",
    "                    'is_price_down_shopitem_month_' + str(m),\n",
    "                    'is_price_up_shopitem_month_' + str(m), \n",
    "                    'is_price_down_item_month_' + str(m),\n",
    "                    'is_price_up_item_month_' + str(m)\n",
    "                   ]\n",
    "        \n",
    "    all_data.loc[:,lag_names[:2]].fillna(0, inplace=True)\n",
    "    \n",
    "#     condition = all_data[lag_names[0]] != 0\n",
    "#     all_data.loc[condition,feature_name[0]] = 1\n",
    "#     condition = all_data[lag_names[1]] != 0\n",
    "#     all_data.loc[condition,feature_name[1]] = 1\n",
    "    \n",
    "    if i > 0:\n",
    "        # sales delta\n",
    "        all_data['sales_shopitem_delta'+str(m)] = \\\n",
    "            (all_data[lag_names[0]] - all_data[lag_cols[0]+'_lag_1']) / \\\n",
    "            (all_data[lag_cols[0]+'_lag_1'] + 1e-10)\n",
    "        all_data['sales_item_delta'+str(m)] = \\\n",
    "            (all_data[lag_names[1]] - all_data[lag_cols[1]+'_lag_1']) / \\\n",
    "            (all_data[lag_cols[1]+'_lag_1'] + 1e-10)\n",
    "        all_data['sales_shopitem_delta'+str(m)] = \\\n",
    "            all_data['sales_shopitem_delta'+str(m)].replace([np.inf, -np.inf], np.nan).clip(-10,10)\n",
    "        all_data['sales_item_delta'+str(m)] = \\\n",
    "            all_data['sales_item_delta'+str(m)].replace([np.inf, -np.inf], np.nan).clip(-10,10)\n",
    "        \n",
    "        # sales diff\n",
    "        all_data['sales_shopitem_diff'] = all_data[last_lag_names[0]] - all_data[lag_names[0]]\n",
    "        #  0  price not change\n",
    "        all_data[feature_name[2]] = np.where(all_data['sales_shopitem_diff'] < 0 - eps, 1, 0)\n",
    "        all_data[feature_name[3]] = np.where(all_data['sales_shopitem_diff'] > 0 + eps, 1, 0)\n",
    "        \n",
    "        all_data['sales_item_diff'] = all_data[last_lag_names[1]] - all_data[lag_names[1]]\n",
    "        all_data[feature_name[4]] = np.where(all_data['sales_item_diff'] < 0 - eps, 1, 0)\n",
    "        all_data[feature_name[5]] = np.where(all_data['sales_item_diff'] > 0 + eps, 1, 0)\n",
    "        \n",
    "        # price diff\n",
    "        all_data['price_shopitem_diff'] = all_data[last_lag_names[2]] - all_data[lag_names[2]]\n",
    "        #  0  price not change\n",
    "        all_data[feature_name[6]] = np.where(all_data['price_shopitem_diff'] < 0 - eps, 1, 0)\n",
    "        all_data[feature_name[7]] = np.where(all_data['price_shopitem_diff'] > 0 + eps, 1, 0)\n",
    "        \n",
    "        all_data['price_item_diff'] = all_data[last_lag_names[3]] - all_data[lag_names[3]]\n",
    "        all_data[feature_name[8]] = np.where(all_data['price_item_diff'] < 0 - eps, 1, 0)\n",
    "        all_data[feature_name[9]] = np.where(all_data['price_item_diff'] > 0 + eps, 1, 0)\n",
    "        \n",
    "        cols_drop = ['sales_shopitem_diff','sales_item_diff',\n",
    "                     'price_shopitem_diff','price_item_diff']\n",
    "        all_data.drop(cols_drop, axis=1, inplace=True)\n",
    "    \n",
    "    i += 1\n",
    "    last_lag_names = lag_names\n",
    "\n",
    "all_data.sort_index(axis=1, inplace=True)\n",
    "all_data.fillna(0,inplace=True)\n",
    "del train_shift, train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "  0%|          | 0/1 [00:00<?, ?it/s]\u001b[A\n",
      "100%|██████████| 1/1 [00:19<00:00, 19.92s/it]\u001b[A"
     ]
    }
   ],
   "source": [
    "# delete some lags\n",
    "drop_lag = [4,5,7,8,9,10,11]\n",
    "feature_prefix = [ \n",
    "    'sales_shopitem_delta','sales_item_delta',\n",
    "    'price_shop_item_mean_lag_','price_item_mean_lag_']\n",
    "\n",
    "for feature in tqdm(feature_prefix):\n",
    "    drop_features = [feature+str(lag) for lag in drop_lag]\n",
    "    all_data.drop(drop_features,axis=1,inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(4926600, 277)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array(['city_code', 'date_block_num', 'days_of_month',\n",
       "       'first_sale_item_id', 'first_sale_shop_iditem_id',\n",
       "       'is_price_down_item_month_10', 'is_price_down_item_month_11',\n",
       "       'is_price_down_item_month_12', 'is_price_down_item_month_2',\n",
       "       'is_price_down_item_month_3', 'is_price_down_item_month_4',\n",
       "       'is_price_down_item_month_5', 'is_price_down_item_month_6',\n",
       "       'is_price_down_item_month_7', 'is_price_down_item_month_8',\n",
       "       'is_price_down_item_month_9', 'is_price_down_shopitem_month_10',\n",
       "       'is_price_down_shopitem_month_11',\n",
       "       'is_price_down_shopitem_month_12',\n",
       "       'is_price_down_shopitem_month_2', 'is_price_down_shopitem_month_3',\n",
       "       'is_price_down_shopitem_month_4', 'is_price_down_shopitem_month_5',\n",
       "       'is_price_down_shopitem_month_6', 'is_price_down_shopitem_month_7',\n",
       "       'is_price_down_shopitem_month_8', 'is_price_down_shopitem_month_9',\n",
       "       'is_price_up_item_month_10', 'is_price_up_item_month_11',\n",
       "       'is_price_up_item_month_12', 'is_price_up_item_month_2',\n",
       "       'is_price_up_item_month_3', 'is_price_up_item_month_4',\n",
       "       'is_price_up_item_month_5', 'is_price_up_item_month_6',\n",
       "       'is_price_up_item_month_7', 'is_price_up_item_month_8',\n",
       "       'is_price_up_item_month_9', 'is_price_up_shopitem_month_10',\n",
       "       'is_price_up_shopitem_month_11', 'is_price_up_shopitem_month_12',\n",
       "       'is_price_up_shopitem_month_2', 'is_price_up_shopitem_month_3',\n",
       "       'is_price_up_shopitem_month_4', 'is_price_up_shopitem_month_5',\n",
       "       'is_price_up_shopitem_month_6', 'is_price_up_shopitem_month_7',\n",
       "       'is_price_up_shopitem_month_8', 'is_price_up_shopitem_month_9',\n",
       "       'is_sales_down_item_month_10', 'is_sales_down_item_month_11',\n",
       "       'is_sales_down_item_month_12', 'is_sales_down_item_month_2',\n",
       "       'is_sales_down_item_month_3', 'is_sales_down_item_month_4',\n",
       "       'is_sales_down_item_month_5', 'is_sales_down_item_month_6',\n",
       "       'is_sales_down_item_month_7', 'is_sales_down_item_month_8',\n",
       "       'is_sales_down_item_month_9', 'is_sales_down_shopitem_month_10',\n",
       "       'is_sales_down_shopitem_month_11',\n",
       "       'is_sales_down_shopitem_month_12',\n",
       "       'is_sales_down_shopitem_month_2', 'is_sales_down_shopitem_month_3',\n",
       "       'is_sales_down_shopitem_month_4', 'is_sales_down_shopitem_month_5',\n",
       "       'is_sales_down_shopitem_month_6', 'is_sales_down_shopitem_month_7',\n",
       "       'is_sales_down_shopitem_month_8', 'is_sales_down_shopitem_month_9',\n",
       "       'is_sales_up_item_month_10', 'is_sales_up_item_month_11',\n",
       "       'is_sales_up_item_month_12', 'is_sales_up_item_month_2',\n",
       "       'is_sales_up_item_month_3', 'is_sales_up_item_month_4',\n",
       "       'is_sales_up_item_month_5', 'is_sales_up_item_month_6',\n",
       "       'is_sales_up_item_month_7', 'is_sales_up_item_month_8',\n",
       "       'is_sales_up_item_month_9', 'is_sales_up_shopitem_month_10',\n",
       "       'is_sales_up_shopitem_month_11', 'is_sales_up_shopitem_month_12',\n",
       "       'is_sales_up_shopitem_month_2', 'is_sales_up_shopitem_month_3',\n",
       "       'is_sales_up_shopitem_month_4', 'is_sales_up_shopitem_month_5',\n",
       "       'is_sales_up_shopitem_month_6', 'is_sales_up_shopitem_month_7',\n",
       "       'is_sales_up_shopitem_month_8', 'is_sales_up_shopitem_month_9',\n",
       "       'item_category_id', 'item_cnt_month',\n",
       "       'item_cnt_month_by_date_block_num_city_code_lag_1',\n",
       "       'item_cnt_month_by_date_block_num_city_code_lag_10',\n",
       "       'item_cnt_month_by_date_block_num_city_code_lag_11',\n",
       "       'item_cnt_month_by_date_block_num_city_code_lag_12',\n",
       "       'item_cnt_month_by_date_block_num_city_code_lag_2',\n",
       "       'item_cnt_month_by_date_block_num_city_code_lag_3',\n",
       "       'item_cnt_month_by_date_block_num_city_code_lag_4',\n",
       "       'item_cnt_month_by_date_block_num_city_code_lag_5',\n",
       "       'item_cnt_month_by_date_block_num_city_code_lag_6',\n",
       "       'item_cnt_month_by_date_block_num_city_code_lag_7',\n",
       "       'item_cnt_month_by_date_block_num_city_code_lag_8',\n",
       "       'item_cnt_month_by_date_block_num_city_code_lag_9',\n",
       "       'item_cnt_month_by_date_block_num_item_category_id_lag_1',\n",
       "       'item_cnt_month_by_date_block_num_item_category_id_lag_10',\n",
       "       'item_cnt_month_by_date_block_num_item_category_id_lag_11',\n",
       "       'item_cnt_month_by_date_block_num_item_category_id_lag_12',\n",
       "       'item_cnt_month_by_date_block_num_item_category_id_lag_2',\n",
       "       'item_cnt_month_by_date_block_num_item_category_id_lag_3',\n",
       "       'item_cnt_month_by_date_block_num_item_category_id_lag_4',\n",
       "       'item_cnt_month_by_date_block_num_item_category_id_lag_5',\n",
       "       'item_cnt_month_by_date_block_num_item_category_id_lag_6',\n",
       "       'item_cnt_month_by_date_block_num_item_category_id_lag_7',\n",
       "       'item_cnt_month_by_date_block_num_item_category_id_lag_8',\n",
       "       'item_cnt_month_by_date_block_num_item_category_id_lag_9',\n",
       "       'item_cnt_month_by_date_block_num_item_id_city_code_lag_1',\n",
       "       'item_cnt_month_by_date_block_num_item_id_city_code_lag_10',\n",
       "       'item_cnt_month_by_date_block_num_item_id_city_code_lag_11',\n",
       "       'item_cnt_month_by_date_block_num_item_id_city_code_lag_12',\n",
       "       'item_cnt_month_by_date_block_num_item_id_city_code_lag_2',\n",
       "       'item_cnt_month_by_date_block_num_item_id_city_code_lag_3',\n",
       "       'item_cnt_month_by_date_block_num_item_id_city_code_lag_4',\n",
       "       'item_cnt_month_by_date_block_num_item_id_city_code_lag_5',\n",
       "       'item_cnt_month_by_date_block_num_item_id_city_code_lag_6',\n",
       "       'item_cnt_month_by_date_block_num_item_id_city_code_lag_7',\n",
       "       'item_cnt_month_by_date_block_num_item_id_city_code_lag_8',\n",
       "       'item_cnt_month_by_date_block_num_item_id_city_code_lag_9',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_item_category_id_lag_1',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_item_category_id_lag_10',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_item_category_id_lag_11',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_item_category_id_lag_12',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_item_category_id_lag_2',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_item_category_id_lag_3',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_item_category_id_lag_4',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_item_category_id_lag_5',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_item_category_id_lag_6',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_item_category_id_lag_7',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_item_category_id_lag_8',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_item_category_id_lag_9',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_lag_1',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_lag_10',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_lag_11',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_lag_12',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_lag_2',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_lag_3',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_lag_4',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_lag_5',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_lag_6',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_lag_7',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_lag_8',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_lag_9',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_subtype_code_lag_1',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_subtype_code_lag_10',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_subtype_code_lag_11',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_subtype_code_lag_12',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_subtype_code_lag_2',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_subtype_code_lag_3',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_subtype_code_lag_4',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_subtype_code_lag_5',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_subtype_code_lag_6',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_subtype_code_lag_7',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_subtype_code_lag_8',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_subtype_code_lag_9',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_type_code_lag_1',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_type_code_lag_10',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_type_code_lag_11',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_type_code_lag_12',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_type_code_lag_2',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_type_code_lag_3',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_type_code_lag_4',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_type_code_lag_5',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_type_code_lag_6',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_type_code_lag_7',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_type_code_lag_8',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_type_code_lag_9',\n",
       "       'item_cnt_month_by_date_block_num_subtype_code_lag_1',\n",
       "       'item_cnt_month_by_date_block_num_subtype_code_lag_10',\n",
       "       'item_cnt_month_by_date_block_num_subtype_code_lag_11',\n",
       "       'item_cnt_month_by_date_block_num_subtype_code_lag_12',\n",
       "       'item_cnt_month_by_date_block_num_subtype_code_lag_2',\n",
       "       'item_cnt_month_by_date_block_num_subtype_code_lag_3',\n",
       "       'item_cnt_month_by_date_block_num_subtype_code_lag_4',\n",
       "       'item_cnt_month_by_date_block_num_subtype_code_lag_5',\n",
       "       'item_cnt_month_by_date_block_num_subtype_code_lag_6',\n",
       "       'item_cnt_month_by_date_block_num_subtype_code_lag_7',\n",
       "       'item_cnt_month_by_date_block_num_subtype_code_lag_8',\n",
       "       'item_cnt_month_by_date_block_num_subtype_code_lag_9',\n",
       "       'item_cnt_month_by_date_block_num_type_code_lag_1',\n",
       "       'item_cnt_month_by_date_block_num_type_code_lag_10',\n",
       "       'item_cnt_month_by_date_block_num_type_code_lag_11',\n",
       "       'item_cnt_month_by_date_block_num_type_code_lag_12',\n",
       "       'item_cnt_month_by_date_block_num_type_code_lag_2',\n",
       "       'item_cnt_month_by_date_block_num_type_code_lag_3',\n",
       "       'item_cnt_month_by_date_block_num_type_code_lag_4',\n",
       "       'item_cnt_month_by_date_block_num_type_code_lag_5',\n",
       "       'item_cnt_month_by_date_block_num_type_code_lag_6',\n",
       "       'item_cnt_month_by_date_block_num_type_code_lag_7',\n",
       "       'item_cnt_month_by_date_block_num_type_code_lag_8',\n",
       "       'item_cnt_month_by_date_block_num_type_code_lag_9',\n",
       "       'item_cnt_month_by_item_id_mean_lag_1',\n",
       "       'item_cnt_month_by_item_id_mean_lag_10',\n",
       "       'item_cnt_month_by_item_id_mean_lag_11',\n",
       "       'item_cnt_month_by_item_id_mean_lag_12',\n",
       "       'item_cnt_month_by_item_id_mean_lag_2',\n",
       "       'item_cnt_month_by_item_id_mean_lag_3',\n",
       "       'item_cnt_month_by_item_id_mean_lag_4',\n",
       "       'item_cnt_month_by_item_id_mean_lag_5',\n",
       "       'item_cnt_month_by_item_id_mean_lag_6',\n",
       "       'item_cnt_month_by_item_id_mean_lag_7',\n",
       "       'item_cnt_month_by_item_id_mean_lag_8',\n",
       "       'item_cnt_month_by_item_id_mean_lag_9', 'item_cnt_month_lag_1',\n",
       "       'item_cnt_month_lag_10', 'item_cnt_month_lag_11',\n",
       "       'item_cnt_month_lag_12', 'item_cnt_month_lag_2',\n",
       "       'item_cnt_month_lag_3', 'item_cnt_month_lag_4',\n",
       "       'item_cnt_month_lag_5', 'item_cnt_month_lag_6',\n",
       "       'item_cnt_month_lag_7', 'item_cnt_month_lag_8',\n",
       "       'item_cnt_month_lag_9', 'item_id', 'last_sale_item_id',\n",
       "       'last_sale_shop_iditem_id', 'month', 'price_item_mean_lag_1',\n",
       "       'price_item_mean_lag_10', 'price_item_mean_lag_11',\n",
       "       'price_item_mean_lag_12', 'price_item_mean_lag_2',\n",
       "       'price_item_mean_lag_3', 'price_item_mean_lag_4',\n",
       "       'price_item_mean_lag_5', 'price_item_mean_lag_6',\n",
       "       'price_item_mean_lag_7', 'price_item_mean_lag_8',\n",
       "       'price_item_mean_lag_9', 'price_shop_item_mean_lag_1',\n",
       "       'price_shop_item_mean_lag_10', 'price_shop_item_mean_lag_11',\n",
       "       'price_shop_item_mean_lag_12', 'price_shop_item_mean_lag_2',\n",
       "       'price_shop_item_mean_lag_3', 'price_shop_item_mean_lag_4',\n",
       "       'price_shop_item_mean_lag_5', 'price_shop_item_mean_lag_6',\n",
       "       'price_shop_item_mean_lag_7', 'price_shop_item_mean_lag_8',\n",
       "       'price_shop_item_mean_lag_9', 'sales_item_delta12',\n",
       "       'sales_item_delta2', 'sales_item_delta3', 'sales_item_delta6',\n",
       "       'sales_shopitem_delta12', 'sales_shopitem_delta2',\n",
       "       'sales_shopitem_delta3', 'sales_shopitem_delta6', 'shop_id',\n",
       "       'subtype_code', 'tfidf_interaction_1', 'tfidf_interaction_10',\n",
       "       'tfidf_interaction_2', 'tfidf_interaction_3',\n",
       "       'tfidf_interaction_4', 'tfidf_interaction_5',\n",
       "       'tfidf_interaction_6', 'tfidf_interaction_7',\n",
       "       'tfidf_interaction_8', 'tfidf_interaction_9', 'type_code', 'year'],\n",
       "      dtype=object)"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(all_data.shape)\n",
    "all_data.columns.values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.5. Sale stats in recent<a name=\"stats\"></a> "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 11/11 [45:28<00:00, 214.21s/it]\n"
     ]
    }
   ],
   "source": [
    "sales_features = ['item_cnt_month',\n",
    " 'item_cnt_month_by_item_id_mean',\n",
    " 'item_cnt_month_by_date_block_num_shop_id',\n",
    " 'item_cnt_month_by_date_block_num_item_category_id',\n",
    " 'item_cnt_month_by_date_block_num_shop_id_item_category_id',\n",
    " 'item_cnt_month_by_date_block_num_shop_id_type_code',\n",
    " 'item_cnt_month_by_date_block_num_shop_id_subtype_code',\n",
    " 'item_cnt_month_by_date_block_num_city_code',\n",
    " 'item_cnt_month_by_date_block_num_item_id_city_code',\n",
    " 'item_cnt_month_by_date_block_num_type_code',\n",
    " 'item_cnt_month_by_date_block_num_subtype_code']\n",
    "\n",
    "last_periods = [6,12]\n",
    "for feature in tqdm(sales_features):\n",
    "    for period in last_periods:\n",
    "        #[col + '_lag_' + str(m) for col in lag_cols]\n",
    "            lag_feats = [feature+'_lag_'+str(lag) for lag in np.arange(1,period+1)]\n",
    "            # we do not use mean and svd directly because we want to include months with sales = 0\n",
    "            mean = all_data[lag_feats].sum(axis=1)/float(period)\n",
    "            mean2 = (all_data[lag_feats]**2).sum(axis=1)/float(period)\n",
    "            new_name1 = 'avg_sale_last_'+str(period)+'_'+feature\n",
    "            new_name2 = 'std_sale_last_'+str(period)+'_'+feature\n",
    "            all_data[new_name1] = mean\n",
    "            all_data[new_name2] = (mean2 - mean**2).apply(np.sqrt)\n",
    "            all_data[new_name2].replace(np.inf,0,inplace=True)\n",
    "            # divide by mean, this scales the features for NN\n",
    "            all_data[new_name1] /= (all_data[new_name1].mean() + 1e-10)\n",
    "            all_data[new_name2] /= (all_data[new_name2].mean() + 1e-10)\n",
    "        \n",
    "    # delete\n",
    "    drop_lag = [4,5,7,8,9,10,11]\n",
    "    drop_features = [feature+'_lag_'+str(lag) for lag in drop_lag]\n",
    "    all_data.drop(drop_features,axis=1,inplace=True) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count    4.926600e+06\n",
       "mean     1.000000e+00\n",
       "std      1.853353e+00\n",
       "min      0.000000e+00\n",
       "25%      3.201940e-01\n",
       "50%      6.841172e-01\n",
       "75%      1.041817e+00\n",
       "max      9.365538e+01\n",
       "Name: avg_sale_last_12_item_cnt_month_by_date_block_num_subtype_code, dtype: float64"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "all_data[new_name1].describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.6. Consecutive_month_number<a name=\"consecutive\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "# consecutive number of month for sales down or up before\n",
    "\n",
    "def create_consecutive_month_number(row, feature_name_prefix):\n",
    "    count = 0\n",
    "    for m in lag_month[1:]:\n",
    "        feature_name = feature_name_prefix + str(m)\n",
    "        if row[feature_name] == 1:\n",
    "            count += 1\n",
    "        else:\n",
    "            return count\n",
    "    return count\n",
    "\n",
    "feature_names = [\n",
    "                    'is_sales_down_shopitem_month_',\n",
    "                    'is_sales_up_shopitem_month_', \n",
    "                    'is_sales_down_item_month_',\n",
    "                    'is_sales_up_item_month_', \n",
    "                    'is_price_down_shopitem_month_',\n",
    "                    'is_price_up_shopitem_month_', \n",
    "                    'is_price_down_item_month_',\n",
    "                    'is_price_up_item_month_',\n",
    "                   ]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 8/8 [51:52<00:00, 376.36s/it]\n"
     ]
    }
   ],
   "source": [
    "for feature in tqdm(feature_names):\n",
    "    all_data.loc[:,'consecutive'+feature] = all_data.apply(\n",
    "        lambda row: create_consecutive_month_number(row, feature),axis=1)\n",
    "    drop_cols = [f for f in all_data.columns if f.startswith(feature)]\n",
    "    all_data.drop(drop_cols,axis=1,inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['city_code', 'date_block_num', 'days_of_month',\n",
       "       'first_sale_item_id', 'first_sale_shop_iditem_id',\n",
       "       'item_category_id', 'item_cnt_month',\n",
       "       'item_cnt_month_by_date_block_num_city_code_lag_1',\n",
       "       'item_cnt_month_by_date_block_num_city_code_lag_12',\n",
       "       'item_cnt_month_by_date_block_num_city_code_lag_2',\n",
       "       'item_cnt_month_by_date_block_num_city_code_lag_3',\n",
       "       'item_cnt_month_by_date_block_num_city_code_lag_6',\n",
       "       'item_cnt_month_by_date_block_num_item_category_id_lag_1',\n",
       "       'item_cnt_month_by_date_block_num_item_category_id_lag_12',\n",
       "       'item_cnt_month_by_date_block_num_item_category_id_lag_2',\n",
       "       'item_cnt_month_by_date_block_num_item_category_id_lag_3',\n",
       "       'item_cnt_month_by_date_block_num_item_category_id_lag_6',\n",
       "       'item_cnt_month_by_date_block_num_item_id_city_code_lag_1',\n",
       "       'item_cnt_month_by_date_block_num_item_id_city_code_lag_12',\n",
       "       'item_cnt_month_by_date_block_num_item_id_city_code_lag_2',\n",
       "       'item_cnt_month_by_date_block_num_item_id_city_code_lag_3',\n",
       "       'item_cnt_month_by_date_block_num_item_id_city_code_lag_6',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_item_category_id_lag_1',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_item_category_id_lag_12',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_item_category_id_lag_2',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_item_category_id_lag_3',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_item_category_id_lag_6',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_lag_1',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_lag_12',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_lag_2',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_lag_3',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_lag_6',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_subtype_code_lag_1',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_subtype_code_lag_12',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_subtype_code_lag_2',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_subtype_code_lag_3',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_subtype_code_lag_6',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_type_code_lag_1',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_type_code_lag_12',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_type_code_lag_2',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_type_code_lag_3',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_type_code_lag_6',\n",
       "       'item_cnt_month_by_date_block_num_subtype_code_lag_1',\n",
       "       'item_cnt_month_by_date_block_num_subtype_code_lag_12',\n",
       "       'item_cnt_month_by_date_block_num_subtype_code_lag_2',\n",
       "       'item_cnt_month_by_date_block_num_subtype_code_lag_3',\n",
       "       'item_cnt_month_by_date_block_num_subtype_code_lag_6',\n",
       "       'item_cnt_month_by_date_block_num_type_code_lag_1',\n",
       "       'item_cnt_month_by_date_block_num_type_code_lag_12',\n",
       "       'item_cnt_month_by_date_block_num_type_code_lag_2',\n",
       "       'item_cnt_month_by_date_block_num_type_code_lag_3',\n",
       "       'item_cnt_month_by_date_block_num_type_code_lag_6',\n",
       "       'item_cnt_month_by_item_id_mean_lag_1',\n",
       "       'item_cnt_month_by_item_id_mean_lag_12',\n",
       "       'item_cnt_month_by_item_id_mean_lag_2',\n",
       "       'item_cnt_month_by_item_id_mean_lag_3',\n",
       "       'item_cnt_month_by_item_id_mean_lag_6', 'item_cnt_month_lag_1',\n",
       "       'item_cnt_month_lag_12', 'item_cnt_month_lag_2',\n",
       "       'item_cnt_month_lag_3', 'item_cnt_month_lag_6', 'item_id',\n",
       "       'last_sale_item_id', 'last_sale_shop_iditem_id', 'month',\n",
       "       'price_item_mean_lag_1', 'price_item_mean_lag_12',\n",
       "       'price_item_mean_lag_2', 'price_item_mean_lag_3',\n",
       "       'price_item_mean_lag_6', 'price_shop_item_mean_lag_1',\n",
       "       'price_shop_item_mean_lag_12', 'price_shop_item_mean_lag_2',\n",
       "       'price_shop_item_mean_lag_3', 'price_shop_item_mean_lag_6',\n",
       "       'sales_item_delta12', 'sales_item_delta2', 'sales_item_delta3',\n",
       "       'sales_item_delta6', 'sales_shopitem_delta12',\n",
       "       'sales_shopitem_delta2', 'sales_shopitem_delta3',\n",
       "       'sales_shopitem_delta6', 'shop_id', 'subtype_code',\n",
       "       'tfidf_interaction_1', 'tfidf_interaction_10',\n",
       "       'tfidf_interaction_2', 'tfidf_interaction_3',\n",
       "       'tfidf_interaction_4', 'tfidf_interaction_5',\n",
       "       'tfidf_interaction_6', 'tfidf_interaction_7',\n",
       "       'tfidf_interaction_8', 'tfidf_interaction_9', 'type_code', 'year',\n",
       "       'avg_sale_last_6_item_cnt_month', 'std_sale_last_6_item_cnt_month',\n",
       "       'avg_sale_last_12_item_cnt_month',\n",
       "       'std_sale_last_12_item_cnt_month',\n",
       "       'avg_sale_last_6_item_cnt_month_by_item_id_mean',\n",
       "       'std_sale_last_6_item_cnt_month_by_item_id_mean',\n",
       "       'avg_sale_last_12_item_cnt_month_by_item_id_mean',\n",
       "       'std_sale_last_12_item_cnt_month_by_item_id_mean',\n",
       "       'avg_sale_last_6_item_cnt_month_by_date_block_num_shop_id',\n",
       "       'std_sale_last_6_item_cnt_month_by_date_block_num_shop_id',\n",
       "       'avg_sale_last_12_item_cnt_month_by_date_block_num_shop_id',\n",
       "       'std_sale_last_12_item_cnt_month_by_date_block_num_shop_id',\n",
       "       'avg_sale_last_6_item_cnt_month_by_date_block_num_item_category_id',\n",
       "       'std_sale_last_6_item_cnt_month_by_date_block_num_item_category_id',\n",
       "       'avg_sale_last_12_item_cnt_month_by_date_block_num_item_category_id',\n",
       "       'std_sale_last_12_item_cnt_month_by_date_block_num_item_category_id',\n",
       "       'avg_sale_last_6_item_cnt_month_by_date_block_num_shop_id_item_category_id',\n",
       "       'std_sale_last_6_item_cnt_month_by_date_block_num_shop_id_item_category_id',\n",
       "       'avg_sale_last_12_item_cnt_month_by_date_block_num_shop_id_item_category_id',\n",
       "       'std_sale_last_12_item_cnt_month_by_date_block_num_shop_id_item_category_id',\n",
       "       'avg_sale_last_6_item_cnt_month_by_date_block_num_shop_id_type_code',\n",
       "       'std_sale_last_6_item_cnt_month_by_date_block_num_shop_id_type_code',\n",
       "       'avg_sale_last_12_item_cnt_month_by_date_block_num_shop_id_type_code',\n",
       "       'std_sale_last_12_item_cnt_month_by_date_block_num_shop_id_type_code',\n",
       "       'avg_sale_last_6_item_cnt_month_by_date_block_num_shop_id_subtype_code',\n",
       "       'std_sale_last_6_item_cnt_month_by_date_block_num_shop_id_subtype_code',\n",
       "       'avg_sale_last_12_item_cnt_month_by_date_block_num_shop_id_subtype_code',\n",
       "       'std_sale_last_12_item_cnt_month_by_date_block_num_shop_id_subtype_code',\n",
       "       'avg_sale_last_6_item_cnt_month_by_date_block_num_city_code',\n",
       "       'std_sale_last_6_item_cnt_month_by_date_block_num_city_code',\n",
       "       'avg_sale_last_12_item_cnt_month_by_date_block_num_city_code',\n",
       "       'std_sale_last_12_item_cnt_month_by_date_block_num_city_code',\n",
       "       'avg_sale_last_6_item_cnt_month_by_date_block_num_item_id_city_code',\n",
       "       'std_sale_last_6_item_cnt_month_by_date_block_num_item_id_city_code',\n",
       "       'avg_sale_last_12_item_cnt_month_by_date_block_num_item_id_city_code',\n",
       "       'std_sale_last_12_item_cnt_month_by_date_block_num_item_id_city_code',\n",
       "       'avg_sale_last_6_item_cnt_month_by_date_block_num_type_code',\n",
       "       'std_sale_last_6_item_cnt_month_by_date_block_num_type_code',\n",
       "       'avg_sale_last_12_item_cnt_month_by_date_block_num_type_code',\n",
       "       'std_sale_last_12_item_cnt_month_by_date_block_num_type_code',\n",
       "       'avg_sale_last_6_item_cnt_month_by_date_block_num_subtype_code',\n",
       "       'std_sale_last_6_item_cnt_month_by_date_block_num_subtype_code',\n",
       "       'avg_sale_last_12_item_cnt_month_by_date_block_num_subtype_code',\n",
       "       'std_sale_last_12_item_cnt_month_by_date_block_num_subtype_code',\n",
       "       'consecutiveis_sales_down_shopitem_month_',\n",
       "       'consecutiveis_sales_up_shopitem_month_',\n",
       "       'consecutiveis_sales_down_item_month_',\n",
       "       'consecutiveis_sales_up_item_month_',\n",
       "       'consecutiveis_price_down_shopitem_month_',\n",
       "       'consecutiveis_price_up_shopitem_month_',\n",
       "       'consecutiveis_price_down_item_month_',\n",
       "       'consecutiveis_price_up_item_month_'], dtype=object)"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "all_data.columns.values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "all_data = pd.get_dummies(all_data, prefix=['month'], columns=['month'])\n",
    "all_data.columns.values       "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Series([], dtype: int64)"
      ]
     },
     "execution_count": 84,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "missing = all_data.isnull().sum()\n",
    "missing[missing>0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Series([], dtype: int64)"
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check constant column\n",
    "df = all_data.nunique()\n",
    "df1 = df[df < 2]\n",
    "df1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[]"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check if unique number is [0,1]\n",
    "df1 = df[df == 2]\n",
    "check_list = []\n",
    "for col in df1.index:\n",
    "    if set(all_data[col].unique()) != set([0,1]):\n",
    "        check_list.append(col)\n",
    "check_list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [],
   "source": [
    "def reduce_mem_usage(props):\n",
    "    start_mem_usg = props.memory_usage().sum() / 1024**2 \n",
    "    print(\"Memory usage of properties dataframe is :\",start_mem_usg,\" MB\")\n",
    "    NAlist = [] # Keeps track of columns that have missing values filled in. \n",
    "    \n",
    "    #float_cols = [c for c in df if df[c].dtype == \"float64\"]\n",
    "    #int_cols =   [c for c in df if df[c].dtype in [\"int64\", \"int32\"]]\n",
    "    cols_to_downcast = [c for c in props if props[c].dtype in [\"float64\", \"int64\", \"int32\"]]\n",
    "    for col in cols_to_downcast:\n",
    "        if props[col].dtype != object:  # Exclude strings\n",
    "            \n",
    "            # Print current column type\n",
    "            #print(\"******************************\")\n",
    "            #print(\"Column: \",col)\n",
    "            #print(\"dtype before: \",props[col].dtype)\n",
    "            type_before = props[col].dtype\n",
    "            \n",
    "            # make variables for Int, max and min\n",
    "            IsInt = False\n",
    "            mx = props[col].max()\n",
    "            mn = props[col].min()\n",
    "            \n",
    "            # Integer does not support NA, therefore, NA needs to be filled\n",
    "            if not np.isfinite(props[col]).all(): \n",
    "                NAlist.append(col)\n",
    "                props[col].fillna(mn-1,inplace=True)  \n",
    "                   \n",
    "            # test if column can be converted to an integer\n",
    "            asint = props[col].fillna(0).astype(np.int64)\n",
    "            result = (props[col] - asint)\n",
    "            result = result.sum()\n",
    "            if result > -0.01 and result < 0.01:\n",
    "                IsInt = True\n",
    "\n",
    "    \n",
    "            # Make Integer/unsigned Integer datatypes\n",
    "            if IsInt:\n",
    "                if mn >= 0:\n",
    "                    if mx < 255:\n",
    "                        props[col] = props[col].astype(np.uint8)\n",
    "                    elif mx < 65535:\n",
    "                        props[col] = props[col].astype(np.uint16)\n",
    "                    elif mx < 4294967295:\n",
    "                        props[col] = props[col].astype(np.uint32)\n",
    "                    else:\n",
    "                        props[col] = props[col].astype(np.uint64)\n",
    "                else:\n",
    "                    if mn > np.iinfo(np.int8).min and mx < np.iinfo(np.int8).max:\n",
    "                        props[col] = props[col].astype(np.int8)\n",
    "                    elif mn > np.iinfo(np.int16).min and mx < np.iinfo(np.int16).max:\n",
    "                        props[col] = props[col].astype(np.int16)\n",
    "                    elif mn > np.iinfo(np.int32).min and mx < np.iinfo(np.int32).max:\n",
    "                        props[col] = props[col].astype(np.int32)\n",
    "                    elif mn > np.iinfo(np.int64).min and mx < np.iinfo(np.int64).max:\n",
    "                        props[col] = props[col].astype(np.int64)    \n",
    "            \n",
    "            # Make float datatypes 32 bit\n",
    "            else:\n",
    "                props[col] = props[col].astype(np.float32)\n",
    "            \n",
    "            print(\"Dtype before: {}  after: {}  Column: {}\".format(type_before, props[col].dtype, col))\n",
    "            #print(\"dtype before: \",props[col].dtype)\n",
    "            # Print new column type\n",
    "            #print(\"dtype after: \",props[col].dtype)\n",
    "            #print(\"******************************\")\n",
    "    \n",
    "    # Print final result\n",
    "    print(\"___MEMORY USAGE AFTER COMPLETION:___\")\n",
    "    mem_usg = props.memory_usage().sum() / 1024**2 \n",
    "    print(\"Memory usage is: \",mem_usg,\" MB\")\n",
    "    print(\"This is \",100*mem_usg/start_mem_usg,\"% of the initial size\")\n",
    "    return props, NAlist"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Memory usage of properties dataframe is : 5360.842323303223  MB\n",
      "Dtype before: float64  after: uint8  Column: first_sale_item_id\n",
      "Dtype before: float64  after: uint8  Column: first_sale_shop_iditem_id\n",
      "Dtype before: float64  after: float32  Column: item_cnt_month_by_date_block_num_city_code_lag_1\n",
      "Dtype before: float64  after: float32  Column: item_cnt_month_by_date_block_num_city_code_lag_12\n",
      "Dtype before: float64  after: float32  Column: item_cnt_month_by_date_block_num_city_code_lag_2\n",
      "Dtype before: float64  after: float32  Column: item_cnt_month_by_date_block_num_city_code_lag_3\n",
      "Dtype before: float64  after: float32  Column: item_cnt_month_by_date_block_num_city_code_lag_6\n",
      "Dtype before: float64  after: float32  Column: item_cnt_month_by_date_block_num_item_category_id_lag_1\n",
      "Dtype before: float64  after: float32  Column: item_cnt_month_by_date_block_num_item_category_id_lag_12\n",
      "Dtype before: float64  after: float32  Column: item_cnt_month_by_date_block_num_item_category_id_lag_2\n",
      "Dtype before: float64  after: float32  Column: item_cnt_month_by_date_block_num_item_category_id_lag_3\n",
      "Dtype before: float64  after: float32  Column: item_cnt_month_by_date_block_num_item_category_id_lag_6\n",
      "Dtype before: float64  after: float32  Column: item_cnt_month_by_date_block_num_item_id_city_code_lag_1\n",
      "Dtype before: float64  after: float32  Column: item_cnt_month_by_date_block_num_item_id_city_code_lag_12\n",
      "Dtype before: float64  after: float32  Column: item_cnt_month_by_date_block_num_item_id_city_code_lag_2\n",
      "Dtype before: float64  after: float32  Column: item_cnt_month_by_date_block_num_item_id_city_code_lag_3\n",
      "Dtype before: float64  after: float32  Column: item_cnt_month_by_date_block_num_item_id_city_code_lag_6\n",
      "Dtype before: float64  after: float32  Column: item_cnt_month_by_date_block_num_shop_id_item_category_id_lag_1\n",
      "Dtype before: float64  after: float32  Column: item_cnt_month_by_date_block_num_shop_id_item_category_id_lag_12\n",
      "Dtype before: float64  after: float32  Column: item_cnt_month_by_date_block_num_shop_id_item_category_id_lag_2\n",
      "Dtype before: float64  after: float32  Column: item_cnt_month_by_date_block_num_shop_id_item_category_id_lag_3\n",
      "Dtype before: float64  after: float32  Column: item_cnt_month_by_date_block_num_shop_id_item_category_id_lag_6\n",
      "Dtype before: float64  after: float32  Column: item_cnt_month_by_date_block_num_shop_id_lag_1\n",
      "Dtype before: float64  after: float32  Column: item_cnt_month_by_date_block_num_shop_id_lag_12\n",
      "Dtype before: float64  after: float32  Column: item_cnt_month_by_date_block_num_shop_id_lag_2\n",
      "Dtype before: float64  after: float32  Column: item_cnt_month_by_date_block_num_shop_id_lag_3\n",
      "Dtype before: float64  after: float32  Column: item_cnt_month_by_date_block_num_shop_id_lag_6\n",
      "Dtype before: float64  after: float32  Column: item_cnt_month_by_date_block_num_shop_id_subtype_code_lag_1\n",
      "Dtype before: float64  after: float32  Column: item_cnt_month_by_date_block_num_shop_id_subtype_code_lag_12\n",
      "Dtype before: float64  after: float32  Column: item_cnt_month_by_date_block_num_shop_id_subtype_code_lag_2\n",
      "Dtype before: float64  after: float32  Column: item_cnt_month_by_date_block_num_shop_id_subtype_code_lag_3\n",
      "Dtype before: float64  after: float32  Column: item_cnt_month_by_date_block_num_shop_id_subtype_code_lag_6\n",
      "Dtype before: float64  after: float32  Column: item_cnt_month_by_date_block_num_shop_id_type_code_lag_1\n",
      "Dtype before: float64  after: float32  Column: item_cnt_month_by_date_block_num_shop_id_type_code_lag_12\n",
      "Dtype before: float64  after: float32  Column: item_cnt_month_by_date_block_num_shop_id_type_code_lag_2\n",
      "Dtype before: float64  after: float32  Column: item_cnt_month_by_date_block_num_shop_id_type_code_lag_3\n",
      "Dtype before: float64  after: float32  Column: item_cnt_month_by_date_block_num_shop_id_type_code_lag_6\n",
      "Dtype before: float64  after: float32  Column: item_cnt_month_by_date_block_num_subtype_code_lag_1\n",
      "Dtype before: float64  after: float32  Column: item_cnt_month_by_date_block_num_subtype_code_lag_12\n",
      "Dtype before: float64  after: float32  Column: item_cnt_month_by_date_block_num_subtype_code_lag_2\n",
      "Dtype before: float64  after: float32  Column: item_cnt_month_by_date_block_num_subtype_code_lag_3\n",
      "Dtype before: float64  after: float32  Column: item_cnt_month_by_date_block_num_subtype_code_lag_6\n",
      "Dtype before: float64  after: float32  Column: item_cnt_month_by_date_block_num_type_code_lag_1\n",
      "Dtype before: float64  after: float32  Column: item_cnt_month_by_date_block_num_type_code_lag_12\n",
      "Dtype before: float64  after: float32  Column: item_cnt_month_by_date_block_num_type_code_lag_2\n",
      "Dtype before: float64  after: float32  Column: item_cnt_month_by_date_block_num_type_code_lag_3\n",
      "Dtype before: float64  after: float32  Column: item_cnt_month_by_date_block_num_type_code_lag_6\n",
      "Dtype before: float64  after: float32  Column: item_cnt_month_by_item_id_mean_lag_1\n",
      "Dtype before: float64  after: float32  Column: item_cnt_month_by_item_id_mean_lag_12\n",
      "Dtype before: float64  after: float32  Column: item_cnt_month_by_item_id_mean_lag_2\n",
      "Dtype before: float64  after: float32  Column: item_cnt_month_by_item_id_mean_lag_3\n",
      "Dtype before: float64  after: float32  Column: item_cnt_month_by_item_id_mean_lag_6\n",
      "Dtype before: float64  after: float32  Column: item_cnt_month_lag_1\n",
      "Dtype before: float64  after: uint8  Column: item_cnt_month_lag_12\n",
      "Dtype before: float64  after: float32  Column: item_cnt_month_lag_2\n",
      "Dtype before: float64  after: float32  Column: item_cnt_month_lag_3\n",
      "Dtype before: float64  after: float32  Column: item_cnt_month_lag_6\n",
      "Dtype before: float64  after: uint8  Column: last_sale_item_id\n",
      "Dtype before: float64  after: uint8  Column: last_sale_shop_iditem_id\n",
      "Dtype before: float64  after: float32  Column: price_item_mean_lag_1\n",
      "Dtype before: float64  after: float32  Column: price_item_mean_lag_12\n",
      "Dtype before: float64  after: float32  Column: price_item_mean_lag_2\n",
      "Dtype before: float64  after: float32  Column: price_item_mean_lag_3\n",
      "Dtype before: float64  after: float32  Column: price_item_mean_lag_6\n",
      "Dtype before: float64  after: float32  Column: price_shop_item_mean_lag_1\n",
      "Dtype before: float64  after: float32  Column: price_shop_item_mean_lag_12\n",
      "Dtype before: float64  after: float32  Column: price_shop_item_mean_lag_2\n",
      "Dtype before: float64  after: float32  Column: price_shop_item_mean_lag_3\n",
      "Dtype before: float64  after: float32  Column: price_shop_item_mean_lag_6\n",
      "Dtype before: float64  after: float32  Column: sales_item_delta12\n",
      "Dtype before: float64  after: float32  Column: sales_item_delta2\n",
      "Dtype before: float64  after: float32  Column: sales_item_delta3\n",
      "Dtype before: float64  after: float32  Column: sales_item_delta6\n",
      "Dtype before: float64  after: float32  Column: sales_shopitem_delta12\n",
      "Dtype before: float64  after: float32  Column: sales_shopitem_delta2\n",
      "Dtype before: float64  after: float32  Column: sales_shopitem_delta3\n",
      "Dtype before: float64  after: float32  Column: sales_shopitem_delta6\n",
      "Dtype before: float64  after: float32  Column: tfidf_interaction_1\n",
      "Dtype before: float64  after: float32  Column: tfidf_interaction_10\n",
      "Dtype before: float64  after: float32  Column: tfidf_interaction_2\n",
      "Dtype before: float64  after: float32  Column: tfidf_interaction_3\n",
      "Dtype before: float64  after: float32  Column: tfidf_interaction_4\n",
      "Dtype before: float64  after: float32  Column: tfidf_interaction_5\n",
      "Dtype before: float64  after: float32  Column: tfidf_interaction_6\n",
      "Dtype before: float64  after: float32  Column: tfidf_interaction_7\n",
      "Dtype before: float64  after: float32  Column: tfidf_interaction_8\n",
      "Dtype before: float64  after: float32  Column: tfidf_interaction_9\n",
      "Dtype before: float64  after: float32  Column: avg_sale_last_6_item_cnt_month\n",
      "Dtype before: float64  after: float32  Column: std_sale_last_6_item_cnt_month\n",
      "Dtype before: float64  after: float32  Column: avg_sale_last_12_item_cnt_month\n",
      "Dtype before: float64  after: float32  Column: std_sale_last_12_item_cnt_month\n",
      "Dtype before: float64  after: float32  Column: avg_sale_last_6_item_cnt_month_by_item_id_mean\n",
      "Dtype before: float64  after: float32  Column: std_sale_last_6_item_cnt_month_by_item_id_mean\n",
      "Dtype before: float64  after: float32  Column: avg_sale_last_12_item_cnt_month_by_item_id_mean\n",
      "Dtype before: float64  after: float32  Column: std_sale_last_12_item_cnt_month_by_item_id_mean\n",
      "Dtype before: float64  after: float32  Column: avg_sale_last_6_item_cnt_month_by_date_block_num_shop_id\n",
      "Dtype before: float64  after: float32  Column: std_sale_last_6_item_cnt_month_by_date_block_num_shop_id\n",
      "Dtype before: float64  after: float32  Column: avg_sale_last_12_item_cnt_month_by_date_block_num_shop_id\n",
      "Dtype before: float64  after: float32  Column: std_sale_last_12_item_cnt_month_by_date_block_num_shop_id\n",
      "Dtype before: float64  after: float32  Column: avg_sale_last_6_item_cnt_month_by_date_block_num_item_category_id\n",
      "Dtype before: float64  after: float32  Column: std_sale_last_6_item_cnt_month_by_date_block_num_item_category_id\n",
      "Dtype before: float64  after: float32  Column: avg_sale_last_12_item_cnt_month_by_date_block_num_item_category_id\n",
      "Dtype before: float64  after: float32  Column: std_sale_last_12_item_cnt_month_by_date_block_num_item_category_id\n",
      "Dtype before: float64  after: float32  Column: avg_sale_last_6_item_cnt_month_by_date_block_num_shop_id_item_category_id\n",
      "Dtype before: float64  after: float32  Column: std_sale_last_6_item_cnt_month_by_date_block_num_shop_id_item_category_id\n",
      "Dtype before: float64  after: float32  Column: avg_sale_last_12_item_cnt_month_by_date_block_num_shop_id_item_category_id\n",
      "Dtype before: float64  after: float32  Column: std_sale_last_12_item_cnt_month_by_date_block_num_shop_id_item_category_id\n",
      "Dtype before: float64  after: float32  Column: avg_sale_last_6_item_cnt_month_by_date_block_num_shop_id_type_code\n",
      "Dtype before: float64  after: float32  Column: std_sale_last_6_item_cnt_month_by_date_block_num_shop_id_type_code\n",
      "Dtype before: float64  after: float32  Column: avg_sale_last_12_item_cnt_month_by_date_block_num_shop_id_type_code\n",
      "Dtype before: float64  after: float32  Column: std_sale_last_12_item_cnt_month_by_date_block_num_shop_id_type_code\n",
      "Dtype before: float64  after: float32  Column: avg_sale_last_6_item_cnt_month_by_date_block_num_shop_id_subtype_code\n",
      "Dtype before: float64  after: float32  Column: std_sale_last_6_item_cnt_month_by_date_block_num_shop_id_subtype_code\n",
      "Dtype before: float64  after: float32  Column: avg_sale_last_12_item_cnt_month_by_date_block_num_shop_id_subtype_code\n",
      "Dtype before: float64  after: float32  Column: std_sale_last_12_item_cnt_month_by_date_block_num_shop_id_subtype_code\n",
      "Dtype before: float64  after: float32  Column: avg_sale_last_6_item_cnt_month_by_date_block_num_city_code\n",
      "Dtype before: float64  after: float32  Column: std_sale_last_6_item_cnt_month_by_date_block_num_city_code\n",
      "Dtype before: float64  after: float32  Column: avg_sale_last_12_item_cnt_month_by_date_block_num_city_code\n",
      "Dtype before: float64  after: float32  Column: std_sale_last_12_item_cnt_month_by_date_block_num_city_code\n",
      "Dtype before: float64  after: float32  Column: avg_sale_last_6_item_cnt_month_by_date_block_num_item_id_city_code\n",
      "Dtype before: float64  after: float32  Column: std_sale_last_6_item_cnt_month_by_date_block_num_item_id_city_code\n",
      "Dtype before: float64  after: float32  Column: avg_sale_last_12_item_cnt_month_by_date_block_num_item_id_city_code\n",
      "Dtype before: float64  after: float32  Column: std_sale_last_12_item_cnt_month_by_date_block_num_item_id_city_code\n",
      "Dtype before: float64  after: float32  Column: avg_sale_last_6_item_cnt_month_by_date_block_num_type_code\n",
      "Dtype before: float64  after: float32  Column: std_sale_last_6_item_cnt_month_by_date_block_num_type_code\n",
      "Dtype before: float64  after: float32  Column: avg_sale_last_12_item_cnt_month_by_date_block_num_type_code\n",
      "Dtype before: float64  after: float32  Column: std_sale_last_12_item_cnt_month_by_date_block_num_type_code\n",
      "Dtype before: float64  after: float32  Column: avg_sale_last_6_item_cnt_month_by_date_block_num_subtype_code\n",
      "Dtype before: float64  after: float32  Column: std_sale_last_6_item_cnt_month_by_date_block_num_subtype_code\n",
      "Dtype before: float64  after: float32  Column: avg_sale_last_12_item_cnt_month_by_date_block_num_subtype_code\n",
      "Dtype before: float64  after: float32  Column: std_sale_last_12_item_cnt_month_by_date_block_num_subtype_code\n",
      "Dtype before: int64  after: uint8  Column: consecutiveis_sales_down_shopitem_month_\n",
      "Dtype before: int64  after: uint8  Column: consecutiveis_sales_up_shopitem_month_\n",
      "Dtype before: int64  after: uint8  Column: consecutiveis_sales_down_item_month_\n",
      "Dtype before: int64  after: uint8  Column: consecutiveis_sales_up_item_month_\n",
      "Dtype before: int64  after: uint8  Column: consecutiveis_price_down_shopitem_month_\n",
      "Dtype before: int64  after: uint8  Column: consecutiveis_price_up_shopitem_month_\n",
      "Dtype before: int64  after: uint8  Column: consecutiveis_price_down_item_month_\n",
      "Dtype before: int64  after: uint8  Column: consecutiveis_price_up_item_month_\n",
      "___MEMORY USAGE AFTER COMPLETION:___\n",
      "Memory usage is:  2565.311050415039  MB\n",
      "This is  47.852760736196316 % of the initial size\n"
     ]
    }
   ],
   "source": [
    "all_data, NAlist = reduce_mem_usage(all_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[]"
      ]
     },
     "execution_count": 89,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "NAlist"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(4926600, 150)"
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "all_data.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [],
   "source": [
    "# save processed data to disk in case the kernel restart because of out of memory\n",
    "outfile = open('all_data42','wb')\n",
    "pickle.dump(all_data,outfile,protocol=4)\n",
    "outfile.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "metadata": {},
   "outputs": [],
   "source": [
    "infile = open('all_data42','rb')\n",
    "all_data = pickle.load(infile)\n",
    "infile.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# save processed data to disk in case the kernel restart because of out of memory\n",
    "outfile = open('all_data_final','wb')\n",
    "pickle.dump(all_data,outfile,protocol=4)\n",
    "outfile.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "infile = open('all_data_final','rb')\n",
    "all_data = pickle.load(infile)\n",
    "infile.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.7. StandardScaler<a name=\"Simple43\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [],
   "source": [
    "infile = open('test','rb')\n",
    "test = pickle.load(infile)\n",
    "infile.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shop_id    0.0\n",
      "item_id    0.0\n",
      "dtype: float64\n"
     ]
    }
   ],
   "source": [
    "# check id order\n",
    "all_data_tmp = all_data[['date_block_num','shop_id','item_id']].copy()\n",
    "for m in range(12,35):\n",
    "    all_data_tmp.loc[all_data_tmp.date_block_num==m,['shop_id','item_id']] -= test[['shop_id','item_id']]\n",
    "print(all_data_tmp.loc[:,['shop_id','item_id']].sum())\n",
    "del all_data_tmp\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 244
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 3133,
     "status": "ok",
     "timestamp": 1561967621516,
     "user": {
      "displayName": "Shengbing YU",
      "photoUrl": "",
      "userId": "03994843105687918628"
     },
     "user_tz": -600
    },
    "id": "2s-tEO3qPhgI",
    "outputId": "cef0df39-72da-4ee5-b3b0-7e92e79da433"
   },
   "outputs": [],
   "source": [
    "# Save `date_block_num`, as we can't use them as features, but will need them to split the dataset into parts\n",
    "dates = all_data['date_block_num']\n",
    "all_data.drop(['date_block_num'], axis=1, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "metadata": {},
   "outputs": [],
   "source": [
    "#all_data.fillna(0, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [],
   "source": [
    "def standard_all_data(all_data):\n",
    "    \n",
    "    df = all_data.nunique()\n",
    "    df = df[df > 2]\n",
    "    features_to_scale = df.index.tolist()\n",
    "    features_to_scale.remove('item_cnt_month')\n",
    "\n",
    "    #X_train = all_data[dates < 34]\n",
    "    #X_test = all_data[dates == 34]\n",
    "\n",
    "    scaler = StandardScaler()\n",
    "\n",
    "    all_data.loc[dates < 34,features_to_scale] = scaler.fit_transform(all_data[dates < 34][features_to_scale])\n",
    "    all_data.loc[dates == 34,features_to_scale] = scaler.transform(all_data[dates == 34][features_to_scale])\n",
    "    \n",
    "    #all_data = pd.concat([X_train, X_test], axis = 0)\n",
    "    \n",
    "    #all_data = downcast_dtypes(all_data)\n",
    "\n",
    "    return all_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [],
   "source": [
    "all_data = standard_all_data(all_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['city_code', 'days_of_month', 'first_sale_item_id',\n",
       "       'first_sale_shop_iditem_id', 'item_category_id', 'item_cnt_month',\n",
       "       'item_cnt_month_by_date_block_num_city_code_lag_1',\n",
       "       'item_cnt_month_by_date_block_num_city_code_lag_12',\n",
       "       'item_cnt_month_by_date_block_num_city_code_lag_2',\n",
       "       'item_cnt_month_by_date_block_num_city_code_lag_3',\n",
       "       'item_cnt_month_by_date_block_num_city_code_lag_6',\n",
       "       'item_cnt_month_by_date_block_num_item_category_id_lag_1',\n",
       "       'item_cnt_month_by_date_block_num_item_category_id_lag_12',\n",
       "       'item_cnt_month_by_date_block_num_item_category_id_lag_2',\n",
       "       'item_cnt_month_by_date_block_num_item_category_id_lag_3',\n",
       "       'item_cnt_month_by_date_block_num_item_category_id_lag_6',\n",
       "       'item_cnt_month_by_date_block_num_item_id_city_code_lag_1',\n",
       "       'item_cnt_month_by_date_block_num_item_id_city_code_lag_12',\n",
       "       'item_cnt_month_by_date_block_num_item_id_city_code_lag_2',\n",
       "       'item_cnt_month_by_date_block_num_item_id_city_code_lag_3',\n",
       "       'item_cnt_month_by_date_block_num_item_id_city_code_lag_6',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_item_category_id_lag_1',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_item_category_id_lag_12',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_item_category_id_lag_2',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_item_category_id_lag_3',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_item_category_id_lag_6',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_lag_1',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_lag_12',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_lag_2',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_lag_3',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_lag_6',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_subtype_code_lag_1',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_subtype_code_lag_12',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_subtype_code_lag_2',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_subtype_code_lag_3',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_subtype_code_lag_6',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_type_code_lag_1',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_type_code_lag_12',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_type_code_lag_2',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_type_code_lag_3',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_type_code_lag_6',\n",
       "       'item_cnt_month_by_date_block_num_subtype_code_lag_1',\n",
       "       'item_cnt_month_by_date_block_num_subtype_code_lag_12',\n",
       "       'item_cnt_month_by_date_block_num_subtype_code_lag_2',\n",
       "       'item_cnt_month_by_date_block_num_subtype_code_lag_3',\n",
       "       'item_cnt_month_by_date_block_num_subtype_code_lag_6',\n",
       "       'item_cnt_month_by_date_block_num_type_code_lag_1',\n",
       "       'item_cnt_month_by_date_block_num_type_code_lag_12',\n",
       "       'item_cnt_month_by_date_block_num_type_code_lag_2',\n",
       "       'item_cnt_month_by_date_block_num_type_code_lag_3',\n",
       "       'item_cnt_month_by_date_block_num_type_code_lag_6',\n",
       "       'item_cnt_month_by_item_id_mean_lag_1',\n",
       "       'item_cnt_month_by_item_id_mean_lag_12',\n",
       "       'item_cnt_month_by_item_id_mean_lag_2',\n",
       "       'item_cnt_month_by_item_id_mean_lag_3',\n",
       "       'item_cnt_month_by_item_id_mean_lag_6', 'item_cnt_month_lag_1',\n",
       "       'item_cnt_month_lag_12', 'item_cnt_month_lag_2',\n",
       "       'item_cnt_month_lag_3', 'item_cnt_month_lag_6', 'item_id',\n",
       "       'last_sale_item_id', 'last_sale_shop_iditem_id', 'month',\n",
       "       'price_item_mean_lag_1', 'price_item_mean_lag_12',\n",
       "       'price_item_mean_lag_2', 'price_item_mean_lag_3',\n",
       "       'price_item_mean_lag_6', 'price_shop_item_mean_lag_1',\n",
       "       'price_shop_item_mean_lag_12', 'price_shop_item_mean_lag_2',\n",
       "       'price_shop_item_mean_lag_3', 'price_shop_item_mean_lag_6',\n",
       "       'sales_item_delta12', 'sales_item_delta2', 'sales_item_delta3',\n",
       "       'sales_item_delta6', 'sales_shopitem_delta12',\n",
       "       'sales_shopitem_delta2', 'sales_shopitem_delta3',\n",
       "       'sales_shopitem_delta6', 'shop_id', 'subtype_code',\n",
       "       'tfidf_interaction_1', 'tfidf_interaction_10',\n",
       "       'tfidf_interaction_2', 'tfidf_interaction_3',\n",
       "       'tfidf_interaction_4', 'tfidf_interaction_5',\n",
       "       'tfidf_interaction_6', 'tfidf_interaction_7',\n",
       "       'tfidf_interaction_8', 'tfidf_interaction_9', 'type_code', 'year',\n",
       "       'avg_sale_last_6_item_cnt_month', 'std_sale_last_6_item_cnt_month',\n",
       "       'avg_sale_last_12_item_cnt_month',\n",
       "       'std_sale_last_12_item_cnt_month',\n",
       "       'avg_sale_last_6_item_cnt_month_by_item_id_mean',\n",
       "       'std_sale_last_6_item_cnt_month_by_item_id_mean',\n",
       "       'avg_sale_last_12_item_cnt_month_by_item_id_mean',\n",
       "       'std_sale_last_12_item_cnt_month_by_item_id_mean',\n",
       "       'avg_sale_last_6_item_cnt_month_by_date_block_num_shop_id',\n",
       "       'std_sale_last_6_item_cnt_month_by_date_block_num_shop_id',\n",
       "       'avg_sale_last_12_item_cnt_month_by_date_block_num_shop_id',\n",
       "       'std_sale_last_12_item_cnt_month_by_date_block_num_shop_id',\n",
       "       'avg_sale_last_6_item_cnt_month_by_date_block_num_item_category_id',\n",
       "       'std_sale_last_6_item_cnt_month_by_date_block_num_item_category_id',\n",
       "       'avg_sale_last_12_item_cnt_month_by_date_block_num_item_category_id',\n",
       "       'std_sale_last_12_item_cnt_month_by_date_block_num_item_category_id',\n",
       "       'avg_sale_last_6_item_cnt_month_by_date_block_num_shop_id_item_category_id',\n",
       "       'std_sale_last_6_item_cnt_month_by_date_block_num_shop_id_item_category_id',\n",
       "       'avg_sale_last_12_item_cnt_month_by_date_block_num_shop_id_item_category_id',\n",
       "       'std_sale_last_12_item_cnt_month_by_date_block_num_shop_id_item_category_id',\n",
       "       'avg_sale_last_6_item_cnt_month_by_date_block_num_shop_id_type_code',\n",
       "       'std_sale_last_6_item_cnt_month_by_date_block_num_shop_id_type_code',\n",
       "       'avg_sale_last_12_item_cnt_month_by_date_block_num_shop_id_type_code',\n",
       "       'std_sale_last_12_item_cnt_month_by_date_block_num_shop_id_type_code',\n",
       "       'avg_sale_last_6_item_cnt_month_by_date_block_num_shop_id_subtype_code',\n",
       "       'std_sale_last_6_item_cnt_month_by_date_block_num_shop_id_subtype_code',\n",
       "       'avg_sale_last_12_item_cnt_month_by_date_block_num_shop_id_subtype_code',\n",
       "       'std_sale_last_12_item_cnt_month_by_date_block_num_shop_id_subtype_code',\n",
       "       'avg_sale_last_6_item_cnt_month_by_date_block_num_city_code',\n",
       "       'std_sale_last_6_item_cnt_month_by_date_block_num_city_code',\n",
       "       'avg_sale_last_12_item_cnt_month_by_date_block_num_city_code',\n",
       "       'std_sale_last_12_item_cnt_month_by_date_block_num_city_code',\n",
       "       'avg_sale_last_6_item_cnt_month_by_date_block_num_item_id_city_code',\n",
       "       'std_sale_last_6_item_cnt_month_by_date_block_num_item_id_city_code',\n",
       "       'avg_sale_last_12_item_cnt_month_by_date_block_num_item_id_city_code',\n",
       "       'std_sale_last_12_item_cnt_month_by_date_block_num_item_id_city_code',\n",
       "       'avg_sale_last_6_item_cnt_month_by_date_block_num_type_code',\n",
       "       'std_sale_last_6_item_cnt_month_by_date_block_num_type_code',\n",
       "       'avg_sale_last_12_item_cnt_month_by_date_block_num_type_code',\n",
       "       'std_sale_last_12_item_cnt_month_by_date_block_num_type_code',\n",
       "       'avg_sale_last_6_item_cnt_month_by_date_block_num_subtype_code',\n",
       "       'std_sale_last_6_item_cnt_month_by_date_block_num_subtype_code',\n",
       "       'avg_sale_last_12_item_cnt_month_by_date_block_num_subtype_code',\n",
       "       'std_sale_last_12_item_cnt_month_by_date_block_num_subtype_code',\n",
       "       'consecutiveis_sales_down_shopitem_month_',\n",
       "       'consecutiveis_sales_up_shopitem_month_',\n",
       "       'consecutiveis_sales_down_item_month_',\n",
       "       'consecutiveis_sales_up_item_month_',\n",
       "       'consecutiveis_price_down_shopitem_month_',\n",
       "       'consecutiveis_price_up_shopitem_month_',\n",
       "       'consecutiveis_price_down_item_month_',\n",
       "       'consecutiveis_price_up_item_month_'], dtype=object)"
      ]
     },
     "execution_count": 122,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "all_data.columns.values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [],
   "source": [
    "outfile = open('all_data_scaled','wb')\n",
    "pickle.dump(all_data,outfile, protocol=4)\n",
    "outfile.close()\n",
    "\n",
    "outfile = open('dates','wb')\n",
    "pickle.dump(dates,outfile)\n",
    "outfile.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "infile = open('all_data_scaled','rb')\n",
    "all_data = pickle.load(infile)\n",
    "infile.close()\n",
    "\n",
    "infile = open('dates','rb')\n",
    "dates = pickle.load(infile)\n",
    "infile.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [],
   "source": [
    "Target = 'item_cnt_month'\n",
    "selected_features = all_data.columns.tolist()\n",
    "selected_features.remove(Target)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(4498200, 148)\n",
      "(4498200,)\n",
      "(214200, 148)\n",
      "(214200,)\n",
      "(214200, 148)\n"
     ]
    }
   ],
   "source": [
    "\n",
    "X_train = all_data[dates<33][selected_features]\n",
    "Y_train = all_data[dates<33][Target]\n",
    "\n",
    "X_valid = all_data[dates==33][selected_features]\n",
    "Y_valid = all_data[dates==33][Target]\n",
    "\n",
    "X_test =  all_data[dates==34][selected_features]\n",
    "\n",
    "print(X_train.shape)\n",
    "print(Y_train.shape)\n",
    "print(X_valid.shape)\n",
    "print(Y_valid.shape)\n",
    "print(X_test.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 54918,
     "status": "ok",
     "timestamp": 1561968508777,
     "user": {
      "displayName": "Shengbing YU",
      "photoUrl": "",
      "userId": "03994843105687918628"
     },
     "user_tz": -600
    },
    "id": "uAelbdz1Phgc",
    "outputId": "8dce608e-386f-4fb8-ca75-9b7040f28c79"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Valid error:  0.8826219320254746\n",
      "preds mean 0.3229976229264609\n"
     ]
    }
   ],
   "source": [
    "model_linreg = LinearRegression(fit_intercept=True)\n",
    "model_linreg.fit(X_train, Y_train)\n",
    "print('Valid error: ',np.sqrt(mean_squared_error(Y_valid, model_linreg.predict(X_valid))))\n",
    "preds = model_linreg.predict(X_test).clip(0,20)\n",
    "print('preds mean',preds.mean())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 6839,
     "status": "ok",
     "timestamp": 1561968536070,
     "user": {
      "displayName": "Shengbing YU",
      "photoUrl": "",
      "userId": "03994843105687918628"
     },
     "user_tz": -600
    },
    "id": "egwtSGkBPhge",
    "outputId": "998a3275-c5e4-4daf-af99-df0631aba135"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Valid error:  0.8826218483580077\n",
      "preds mean 0.32299793713761005\n"
     ]
    }
   ],
   "source": [
    "model_ridge = Ridge()\n",
    "model_ridge.fit(X_train, Y_train)\n",
    "print('Valid error: ',np.sqrt(mean_squared_error(Y_valid, model_ridge.predict(X_valid))))\n",
    "preds = model_ridge.predict(X_test).clip(0,20)\n",
    "print('preds mean',preds.mean())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 9204,
     "status": "ok",
     "timestamp": 1561968549811,
     "user": {
      "displayName": "Shengbing YU",
      "photoUrl": "",
      "userId": "03994843105687918628"
     },
     "user_tz": -600
    },
    "id": "yukmTwMDPhgg",
    "outputId": "291072ac-5a78-419b-a76c-4cd780e23b20"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Valid error:  1.0913896101211986\n",
      "preds mean 0.2205517762660619\n"
     ]
    }
   ],
   "source": [
    "model_lasso = Lasso(alpha=10.0, fit_intercept=True, max_iter=1000)\n",
    "model_lasso.fit(X_train, Y_train)\n",
    "print('Valid error: ',np.sqrt(mean_squared_error(Y_valid, model_lasso.predict(X_valid))))\n",
    "preds = model_lasso.predict(X_test).clip(0,20)\n",
    "print('preds mean',preds.mean())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Great! Feature engineering works as expected. Linear models give lower valid error. So happy!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.8. Simple feature selection<a name=\"Simple48\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_selection import VarianceThreshold\n",
    "\n",
    "from sklearn.feature_selection import SelectFromModel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {},
   "outputs": [],
   "source": [
    "Target = 'item_cnt_month'\n",
    "selected_features = all_data.columns.tolist()\n",
    "selected_features.remove(Target)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(4498200, 148)\n",
      "(4498200,)\n",
      "(214200, 148)\n"
     ]
    }
   ],
   "source": [
    "X_train = all_data[dates<33][selected_features]\n",
    "Y_train = all_data[dates<33][Target]\n",
    "\n",
    "X_test =  all_data[dates==33][selected_features]\n",
    "Y_test = all_data[dates==33][Target]\n",
    "\n",
    "print(X_train.shape)\n",
    "print(Y_train.shape)\n",
    "print(X_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Remove constant features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "constant_features []\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "((4498200, 148), (214200, 148))"
      ]
     },
     "execution_count": 106,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# remove constant features\n",
    "constant_features = [\n",
    "    feat for feat in X_train.columns if X_train[feat].std() == 0\n",
    "]\n",
    "print('constant_features',constant_features)\n",
    "X_train.drop(labels=constant_features, axis=1, inplace=True)\n",
    "X_test.drop(labels=constant_features, axis=1, inplace=True)\n",
    "\n",
    "X_train.shape, X_test.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Remove quasi-constant features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "148"
      ]
     },
     "execution_count": 107,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# remove quasi-constant features\n",
    "sel = VarianceThreshold(\n",
    "    threshold=0.01)  # 0.1 indicates 99% of observations approximately\n",
    "\n",
    "sel.fit(X_train)  # fit finds the features with low variance\n",
    "\n",
    "sum(sel.get_support()) # how many not quasi-constant?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index([], dtype='object')"
      ]
     },
     "execution_count": 108,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train.columns[~sel.get_support()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "features_to_keep = X_train.columns[sel.get_support()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((4498200, 148), (214200, 148))"
      ]
     },
     "execution_count": 110,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# we can then remove the features like this\n",
    "X_train = sel.transform(X_train)\n",
    "X_test = sel.transform(X_test)\n",
    "\n",
    "X_train.shape, X_test.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "# sklearn transformations lead to numpy arrays\n",
    "# here I transform the arrays back to dataframes\n",
    "# please be mindful of getting the columns assigned\n",
    "# correctly\n",
    "\n",
    "X_train= pd.DataFrame(X_train)\n",
    "X_train.columns = features_to_keep\n",
    "\n",
    "X_test= pd.DataFrame(X_test)\n",
    "X_test.columns = features_to_keep"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Remove duplicated features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "  0%|          | 0/148 [00:00<?, ?it/s]\u001b[A\n",
      "  1%|          | 1/148 [01:40<4:06:53, 100.77s/it]\u001b[A\n",
      "  1%|▏         | 2/148 [03:37<4:16:42, 105.50s/it]\u001b[A\n",
      "  2%|▏         | 3/148 [05:33<4:22:35, 108.66s/it]\u001b[A\n",
      "  3%|▎         | 4/148 [07:38<4:32:17, 113.45s/it]\u001b[A\n",
      "  3%|▎         | 5/148 [09:36<4:33:48, 114.89s/it]\u001b[A\n",
      "  4%|▍         | 6/148 [11:30<4:31:16, 114.62s/it]\u001b[A\n",
      "  5%|▍         | 7/148 [13:21<4:26:57, 113.60s/it]\u001b[A\n",
      "  5%|▌         | 8/148 [15:12<4:23:02, 112.73s/it]\u001b[A\n",
      "  6%|▌         | 9/148 [17:01<4:19:01, 111.81s/it]\u001b[A\n",
      "  7%|▋         | 10/148 [18:50<4:14:41, 110.73s/it]\u001b[A\n",
      "  7%|▋         | 11/148 [20:37<4:10:32, 109.73s/it]\u001b[A\n",
      "  8%|▊         | 12/148 [22:25<4:07:27, 109.17s/it]\u001b[A\n",
      "  9%|▉         | 13/148 [24:11<4:03:32, 108.24s/it]\u001b[A\n",
      "  9%|▉         | 14/148 [25:56<3:59:51, 107.40s/it]\u001b[A\n",
      " 10%|█         | 15/148 [27:41<3:55:56, 106.44s/it]\u001b[A\n",
      " 11%|█         | 16/148 [29:24<3:52:07, 105.51s/it]\u001b[A\n",
      " 11%|█▏        | 17/148 [31:07<3:48:39, 104.73s/it]\u001b[A\n",
      " 12%|█▏        | 18/148 [32:50<3:45:51, 104.24s/it]\u001b[A\n",
      " 13%|█▎        | 19/148 [34:31<3:42:19, 103.40s/it]\u001b[A\n",
      " 14%|█▎        | 20/148 [36:12<3:38:52, 102.60s/it]\u001b[A\n",
      " 14%|█▍        | 21/148 [37:52<3:35:22, 101.75s/it]\u001b[A\n",
      " 15%|█▍        | 22/148 [39:31<3:31:52, 100.90s/it]\u001b[A\n",
      " 16%|█▌        | 23/148 [41:09<3:28:29, 100.07s/it]\u001b[A\n",
      " 16%|█▌        | 24/148 [42:46<3:25:03, 99.22s/it] \u001b[A\n",
      " 17%|█▋        | 25/148 [44:24<3:22:27, 98.76s/it]\u001b[A\n",
      " 18%|█▊        | 26/148 [45:59<3:18:54, 97.83s/it]\u001b[A\n",
      " 18%|█▊        | 27/148 [47:34<3:15:10, 96.78s/it]\u001b[A\n",
      " 19%|█▉        | 28/148 [49:07<3:11:38, 95.82s/it]\u001b[A\n",
      " 20%|█▉        | 29/148 [50:40<3:08:27, 95.02s/it]\u001b[A\n",
      " 20%|██        | 30/148 [52:13<3:05:13, 94.19s/it]\u001b[A\n",
      " 21%|██        | 31/148 [53:45<3:02:33, 93.62s/it]\u001b[A\n",
      " 22%|██▏       | 32/148 [55:16<2:59:27, 92.82s/it]\u001b[A\n",
      " 22%|██▏       | 33/148 [56:48<2:57:43, 92.73s/it]\u001b[A\n",
      " 23%|██▎       | 34/148 [58:18<2:54:20, 91.76s/it]\u001b[A\n",
      " 24%|██▎       | 35/148 [59:46<2:50:54, 90.74s/it]\u001b[A\n",
      " 24%|██▍       | 36/148 [1:01:14<2:47:49, 89.91s/it]\u001b[A\n",
      " 25%|██▌       | 37/148 [1:02:41<2:44:45, 89.06s/it]\u001b[A\n",
      " 26%|██▌       | 38/148 [1:04:09<2:42:28, 88.62s/it]\u001b[A\n",
      " 26%|██▋       | 39/148 [1:05:36<2:40:13, 88.20s/it]\u001b[A\n",
      " 27%|██▋       | 40/148 [1:07:07<2:40:11, 89.00s/it]\u001b[A\n",
      " 28%|██▊       | 41/148 [1:08:32<2:36:18, 87.65s/it]\u001b[A\n",
      " 28%|██▊       | 42/148 [1:09:55<2:32:32, 86.35s/it]\u001b[A\n",
      " 29%|██▉       | 43/148 [1:11:17<2:29:03, 85.17s/it]\u001b[A\n",
      " 30%|██▉       | 44/148 [1:12:40<2:26:34, 84.57s/it]\u001b[A\n",
      " 30%|███       | 45/148 [1:14:02<2:23:28, 83.58s/it]\u001b[A\n",
      " 31%|███       | 46/148 [1:15:22<2:20:23, 82.58s/it]\u001b[A\n",
      " 32%|███▏      | 47/148 [1:16:44<2:18:31, 82.29s/it]\u001b[A\n",
      " 32%|███▏      | 48/148 [1:18:02<2:15:21, 81.21s/it]\u001b[A\n",
      " 33%|███▎      | 49/148 [1:19:20<2:12:20, 80.21s/it]\u001b[A\n",
      " 34%|███▍      | 50/148 [1:20:39<2:10:20, 79.80s/it]\u001b[A\n",
      " 34%|███▍      | 51/148 [1:22:00<2:09:22, 80.02s/it]\u001b[A\n",
      " 35%|███▌      | 52/148 [1:23:16<2:06:10, 78.86s/it]\u001b[A\n",
      " 36%|███▌      | 53/148 [1:24:30<2:02:34, 77.41s/it]\u001b[A\n",
      " 36%|███▋      | 54/148 [1:25:45<2:00:22, 76.83s/it]\u001b[A\n",
      " 37%|███▋      | 55/148 [1:26:59<1:57:52, 76.05s/it]\u001b[A\n",
      " 38%|███▊      | 56/148 [1:28:11<1:54:23, 74.60s/it]\u001b[A\n",
      " 39%|███▊      | 57/148 [1:29:20<1:50:42, 72.99s/it]\u001b[A\n",
      " 39%|███▉      | 58/148 [1:30:36<1:50:39, 73.78s/it]\u001b[A\n",
      " 40%|███▉      | 59/148 [1:31:51<1:50:00, 74.16s/it]\u001b[A\n",
      " 41%|████      | 60/148 [1:32:58<1:45:52, 72.19s/it]\u001b[A\n",
      " 41%|████      | 61/148 [1:34:02<1:41:08, 69.76s/it]\u001b[A\n",
      " 42%|████▏     | 62/148 [1:35:14<1:40:49, 70.34s/it]\u001b[A\n",
      " 43%|████▎     | 63/148 [1:36:23<1:39:03, 69.93s/it]\u001b[A\n",
      " 43%|████▎     | 64/148 [1:37:29<1:36:07, 68.66s/it]\u001b[A\n",
      " 44%|████▍     | 65/148 [1:38:30<1:32:03, 66.55s/it]\u001b[A\n",
      " 45%|████▍     | 66/148 [1:39:27<1:26:54, 63.59s/it]\u001b[A\n",
      " 45%|████▌     | 67/148 [1:40:40<1:29:51, 66.56s/it]\u001b[A\n",
      " 46%|████▌     | 68/148 [1:41:41<1:26:15, 64.69s/it]\u001b[A\n",
      " 47%|████▋     | 69/148 [1:42:32<1:20:01, 60.78s/it]\u001b[A\n",
      " 47%|████▋     | 70/148 [1:43:25<1:15:46, 58.29s/it]\u001b[A\n",
      " 48%|████▊     | 71/148 [1:44:23<1:14:35, 58.13s/it]\u001b[A\n",
      " 49%|████▊     | 72/148 [1:45:11<1:09:44, 55.05s/it]\u001b[A\n",
      " 49%|████▉     | 73/148 [1:46:01<1:07:05, 53.68s/it]\u001b[A\n",
      " 50%|█████     | 74/148 [1:46:48<1:03:34, 51.55s/it]\u001b[A\n",
      " 51%|█████     | 75/148 [1:47:34<1:00:46, 49.95s/it]\u001b[A\n",
      " 51%|█████▏    | 76/148 [1:48:19<58:13, 48.53s/it]  \u001b[A\n",
      " 52%|█████▏    | 77/148 [1:49:12<59:08, 49.98s/it]\u001b[A\n",
      " 53%|█████▎    | 78/148 [1:49:56<56:10, 48.16s/it]\u001b[A\n",
      " 53%|█████▎    | 79/148 [1:50:47<56:08, 48.82s/it]\u001b[A\n",
      " 54%|█████▍    | 80/148 [1:51:24<51:29, 45.44s/it]\u001b[A\n",
      " 55%|█████▍    | 81/148 [1:51:59<47:07, 42.20s/it]\u001b[A\n",
      " 55%|█████▌    | 82/148 [1:52:34<43:58, 39.98s/it]\u001b[A\n",
      " 56%|█████▌    | 83/148 [1:53:07<41:02, 37.89s/it]\u001b[A\n",
      " 57%|█████▋    | 84/148 [1:53:39<38:43, 36.31s/it]\u001b[A\n",
      " 57%|█████▋    | 85/148 [1:54:11<36:43, 34.98s/it]\u001b[A\n",
      " 58%|█████▊    | 86/148 [1:54:42<34:56, 33.82s/it]\u001b[A\n",
      " 59%|█████▉    | 87/148 [1:55:13<33:24, 32.87s/it]\u001b[A\n",
      " 59%|█████▉    | 88/148 [1:55:43<32:03, 32.05s/it]\u001b[A\n",
      " 60%|██████    | 89/148 [1:56:12<30:44, 31.27s/it]\u001b[A\n",
      " 61%|██████    | 90/148 [1:56:41<29:32, 30.55s/it]\u001b[A\n",
      " 61%|██████▏   | 91/148 [1:57:10<28:26, 29.93s/it]\u001b[A\n",
      " 62%|██████▏   | 92/148 [1:57:38<27:26, 29.39s/it]\u001b[A\n",
      " 63%|██████▎   | 93/148 [1:58:05<26:23, 28.80s/it]\u001b[A\n",
      " 64%|██████▎   | 94/148 [1:58:32<25:26, 28.26s/it]\u001b[A\n",
      " 64%|██████▍   | 95/148 [1:58:59<24:28, 27.71s/it]\u001b[A\n",
      " 65%|██████▍   | 96/148 [1:59:25<23:34, 27.20s/it]\u001b[A\n",
      " 66%|██████▌   | 97/148 [1:59:50<22:41, 26.70s/it]\u001b[A\n",
      " 66%|██████▌   | 98/148 [2:00:16<21:54, 26.29s/it]\u001b[A\n",
      " 67%|██████▋   | 99/148 [2:00:40<21:03, 25.78s/it]\u001b[A\n",
      " 68%|██████▊   | 100/148 [2:01:04<20:14, 25.30s/it]\u001b[A\n",
      " 68%|██████▊   | 101/148 [2:01:28<19:29, 24.87s/it]\u001b[A\n",
      " 69%|██████▉   | 102/148 [2:01:51<18:37, 24.30s/it]\u001b[A\n",
      " 70%|██████▉   | 103/148 [2:02:14<17:47, 23.72s/it]\u001b[A\n",
      " 70%|███████   | 104/148 [2:02:36<16:59, 23.18s/it]\u001b[A\n",
      " 71%|███████   | 105/148 [2:02:57<16:14, 22.66s/it]\u001b[A\n",
      " 72%|███████▏  | 106/148 [2:03:18<15:27, 22.09s/it]\u001b[A\n",
      " 72%|███████▏  | 107/148 [2:03:38<14:44, 21.57s/it]\u001b[A\n",
      " 73%|███████▎  | 108/148 [2:03:58<14:01, 21.05s/it]\u001b[A\n",
      " 74%|███████▎  | 109/148 [2:04:18<13:23, 20.60s/it]\u001b[A\n",
      " 74%|███████▍  | 110/148 [2:04:36<12:44, 20.11s/it]\u001b[A\n",
      " 75%|███████▌  | 111/148 [2:04:55<12:05, 19.62s/it]\u001b[A\n",
      " 76%|███████▌  | 112/148 [2:05:13<11:28, 19.13s/it]\u001b[A\n",
      " 76%|███████▋  | 113/148 [2:05:31<10:56, 18.77s/it]\u001b[A\n",
      " 77%|███████▋  | 114/148 [2:05:48<10:24, 18.37s/it]\u001b[A\n",
      " 78%|███████▊  | 115/148 [2:06:05<09:48, 17.82s/it]\u001b[A\n",
      " 78%|███████▊  | 116/148 [2:06:21<09:13, 17.30s/it]\u001b[A\n",
      " 79%|███████▉  | 117/148 [2:06:36<08:39, 16.77s/it]\u001b[A\n",
      " 80%|███████▉  | 118/148 [2:06:52<08:08, 16.27s/it]\u001b[A\n",
      " 80%|████████  | 119/148 [2:07:06<07:35, 15.72s/it]\u001b[A\n",
      " 81%|████████  | 120/148 [2:07:20<07:05, 15.18s/it]\u001b[A\n",
      " 82%|████████▏ | 121/148 [2:07:33<06:35, 14.67s/it]\u001b[A\n",
      " 82%|████████▏ | 122/148 [2:07:46<06:08, 14.18s/it]\u001b[A\n",
      " 83%|████████▎ | 123/148 [2:07:59<05:41, 13.67s/it]\u001b[A\n",
      " 84%|████████▍ | 124/148 [2:08:11<05:16, 13.18s/it]\u001b[A\n",
      " 84%|████████▍ | 125/148 [2:08:22<04:51, 12.67s/it]\u001b[A\n",
      " 85%|████████▌ | 126/148 [2:08:33<04:27, 12.17s/it]\u001b[A\n",
      " 86%|████████▌ | 127/148 [2:08:44<04:04, 11.65s/it]\u001b[A\n",
      " 86%|████████▋ | 128/148 [2:08:54<03:42, 11.15s/it]\u001b[A\n",
      " 87%|████████▋ | 129/148 [2:09:03<03:22, 10.65s/it]\u001b[A\n",
      " 88%|████████▊ | 130/148 [2:09:12<03:02, 10.12s/it]\u001b[A\n",
      " 89%|████████▊ | 131/148 [2:09:21<02:43,  9.61s/it]\u001b[A\n",
      " 89%|████████▉ | 132/148 [2:09:29<02:25,  9.10s/it]\u001b[A\n",
      " 90%|████████▉ | 133/148 [2:09:36<02:09,  8.60s/it]\u001b[A\n",
      " 91%|█████████ | 134/148 [2:09:43<01:53,  8.10s/it]\u001b[A\n",
      " 91%|█████████ | 135/148 [2:09:49<01:38,  7.61s/it]\u001b[A\n",
      " 92%|█████████▏| 136/148 [2:09:55<01:25,  7.14s/it]\u001b[A\n",
      " 93%|█████████▎| 137/148 [2:10:01<01:12,  6.64s/it]\u001b[A\n",
      " 93%|█████████▎| 138/148 [2:10:06<01:01,  6.13s/it]\u001b[A\n",
      " 94%|█████████▍| 139/148 [2:10:10<00:50,  5.62s/it]\u001b[A\n",
      " 95%|█████████▍| 140/148 [2:10:14<00:41,  5.13s/it]\u001b[A\n",
      " 95%|█████████▌| 141/148 [2:10:18<00:32,  4.62s/it]\u001b[A\n",
      " 96%|█████████▌| 142/148 [2:10:21<00:24,  4.12s/it]\u001b[A\n",
      " 97%|█████████▋| 143/148 [2:10:23<00:18,  3.62s/it]\u001b[A\n",
      " 97%|█████████▋| 144/148 [2:10:25<00:12,  3.12s/it]\u001b[A\n",
      " 98%|█████████▊| 145/148 [2:10:27<00:07,  2.64s/it]\u001b[A\n",
      " 99%|█████████▊| 146/148 [2:10:28<00:04,  2.14s/it]\u001b[A\n",
      " 99%|█████████▉| 147/148 [2:10:28<00:01,  1.65s/it]\u001b[A\n",
      "100%|██████████| 148/148 [2:10:28<00:00, 52.90s/it]\u001b[A"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[]"
      ]
     },
     "execution_count": 112,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check for duplicated features in the training set\n",
    "duplicated_feat = []\n",
    "for i in tqdm(range(0, len(X_train.columns))):\n",
    "#     if i % 10 == 0:  # this helps me understand how the loop is going\n",
    "#         print(i)\n",
    "\n",
    "    col_1 = X_train.columns[i]\n",
    "\n",
    "    for col_2 in X_train.columns[i + 1:]:\n",
    "        if X_train[col_1].equals(X_train[col_2]):\n",
    "            duplicated_feat.append(col_2)\n",
    "            \n",
    "print(len(duplicated_feat))\n",
    "duplicated_feat"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((4498200, 148), (214200, 148))"
      ]
     },
     "execution_count": 113,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# remove duplicated features\n",
    "X_train.drop(labels=duplicated_feat, axis=1, inplace=True)\n",
    "X_test.drop(labels=duplicated_feat, axis=1, inplace=True)\n",
    "\n",
    "X_train.shape, X_test.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## find and remove correlated features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Using Pearson Correlation\n",
    "tmp_df = pd.concat([X_train,Y_train], axis=1, ignore_index=True, sort=False)\n",
    "cor = tmp_df.corr().abs()\n",
    "outfile = open('cor','wb')\n",
    "pickle.dump(cor,outfile)\n",
    "outfile.close()\n",
    "\n",
    "# plt.figure(figsize=(12,10))\n",
    "# sns.heatmap(cor, annot=True, cmap=plt.cm.Reds)\n",
    "# plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "infile = open('cor','rb')\n",
    "cor = pickle.load(infile)\n",
    "infile.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {},
   "outputs": [],
   "source": [
    "cor.columns = X_train.columns.tolist() + ['item_cnt_month']\n",
    "cor.index = X_train.columns.tolist() + ['item_cnt_month']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>level_0</th>\n",
       "      <th>level_1</th>\n",
       "      <th>0</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>last_sale_shop_iditem_id</td>\n",
       "      <td>first_sale_shop_iditem_id</td>\n",
       "      <td>0.994481</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>563</th>\n",
       "      <td>avg_sale_last_12_item_cnt_month_by_date_block_...</td>\n",
       "      <td>avg_sale_last_12_item_cnt_month_by_date_block_...</td>\n",
       "      <td>0.994006</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>544</th>\n",
       "      <td>avg_sale_last_6_item_cnt_month_by_date_block_n...</td>\n",
       "      <td>avg_sale_last_6_item_cnt_month_by_date_block_n...</td>\n",
       "      <td>0.993876</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>46</th>\n",
       "      <td>item_cnt_month_by_date_block_num_item_category...</td>\n",
       "      <td>item_cnt_month_by_date_block_num_subtype_code_...</td>\n",
       "      <td>0.992255</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>76</th>\n",
       "      <td>item_cnt_month_by_date_block_num_item_category...</td>\n",
       "      <td>item_cnt_month_by_date_block_num_subtype_code_...</td>\n",
       "      <td>0.992081</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>91</th>\n",
       "      <td>item_cnt_month_by_date_block_num_item_category...</td>\n",
       "      <td>item_cnt_month_by_date_block_num_subtype_code_...</td>\n",
       "      <td>0.991453</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>106</th>\n",
       "      <td>item_cnt_month_by_date_block_num_subtype_code_...</td>\n",
       "      <td>item_cnt_month_by_date_block_num_item_category...</td>\n",
       "      <td>0.991103</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>599</th>\n",
       "      <td>avg_sale_last_12_item_cnt_month_by_date_block_...</td>\n",
       "      <td>avg_sale_last_12_item_cnt_month_by_date_block_...</td>\n",
       "      <td>0.989973</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>580</th>\n",
       "      <td>avg_sale_last_6_item_cnt_month_by_date_block_n...</td>\n",
       "      <td>avg_sale_last_6_item_cnt_month_by_date_block_n...</td>\n",
       "      <td>0.989629</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>first_sale_item_id</td>\n",
       "      <td>last_sale_item_id</td>\n",
       "      <td>0.988886</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                               level_0  \\\n",
       "6                             last_sale_shop_iditem_id   \n",
       "563  avg_sale_last_12_item_cnt_month_by_date_block_...   \n",
       "544  avg_sale_last_6_item_cnt_month_by_date_block_n...   \n",
       "46   item_cnt_month_by_date_block_num_item_category...   \n",
       "76   item_cnt_month_by_date_block_num_item_category...   \n",
       "91   item_cnt_month_by_date_block_num_item_category...   \n",
       "106  item_cnt_month_by_date_block_num_subtype_code_...   \n",
       "599  avg_sale_last_12_item_cnt_month_by_date_block_...   \n",
       "580  avg_sale_last_6_item_cnt_month_by_date_block_n...   \n",
       "4                                   first_sale_item_id   \n",
       "\n",
       "                                               level_1         0  \n",
       "6                            first_sale_shop_iditem_id  0.994481  \n",
       "563  avg_sale_last_12_item_cnt_month_by_date_block_...  0.994006  \n",
       "544  avg_sale_last_6_item_cnt_month_by_date_block_n...  0.993876  \n",
       "46   item_cnt_month_by_date_block_num_subtype_code_...  0.992255  \n",
       "76   item_cnt_month_by_date_block_num_subtype_code_...  0.992081  \n",
       "91   item_cnt_month_by_date_block_num_subtype_code_...  0.991453  \n",
       "106  item_cnt_month_by_date_block_num_item_category...  0.991103  \n",
       "599  avg_sale_last_12_item_cnt_month_by_date_block_...  0.989973  \n",
       "580  avg_sale_last_6_item_cnt_month_by_date_block_n...  0.989629  \n",
       "4                                    last_sale_item_id  0.988886  "
      ]
     },
     "execution_count": 116,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataCorr = cor[cor >= 0.8].stack().reset_index()\n",
    "dataCorr = dataCorr[dataCorr['level_0'].astype(str)!=dataCorr['level_1'].astype(str)]\n",
    "\n",
    "# filtering out lower/upper triangular duplicates \n",
    "dataCorr['ordered-cols'] = dataCorr.apply(lambda x: '-'.join(sorted([x['level_0'],x['level_1']])),axis=1)\n",
    "dataCorr = dataCorr.drop_duplicates(['ordered-cols'])\n",
    "dataCorr.drop(['ordered-cols'], axis=1, inplace=True)\n",
    "dataCorr.sort_values(0, axis = 0, ascending = False, inplace = True) \n",
    "\n",
    "corr_features = []\n",
    "#Level_0 has higher cor with target\n",
    "for i in range(len(dataCorr)):\n",
    "    if cor.loc[dataCorr.iloc[i,0],'item_cnt_month'] < cor.loc[dataCorr.iloc[i,1],'item_cnt_month']:\n",
    "        tmp = dataCorr.iloc[i,0]\n",
    "        dataCorr.iloc[i,0] = dataCorr.iloc[i,1]\n",
    "        dataCorr.iloc[i,1] = tmp\n",
    "    if  (dataCorr.iloc[i,1] not in dataCorr.iloc[0:i,0]) & ('item_cnt_month' not in dataCorr.iloc[i,0:2]):\n",
    "        corr_features.append(dataCorr.iloc[i,1])\n",
    "dataCorr.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "90"
      ]
     },
     "execution_count": 117,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(set(corr_features))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((4498200, 151), (214200, 151))"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# removed correlated  features\n",
    "X_train.drop(labels=corr_features, axis=1, inplace=True)\n",
    "X_test.drop(labels=corr_features, axis=1, inplace=True)\n",
    "\n",
    "X_train.shape, X_test.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [],
   "source": [
    "selected_features = X_train.columns.tolist()\n",
    "pickle.dump(selected_features, open('selected_features.pkl', 'wb'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=2)]: Using backend ThreadingBackend with 2 concurrent workers.\n",
      "[Parallel(n_jobs=2)]: Done  46 tasks      | elapsed: 235.9min\n",
      "[Parallel(n_jobs=2)]: Done 100 out of 100 | elapsed: 508.8min finished\n",
      "[Parallel(n_jobs=2)]: Using backend ThreadingBackend with 2 concurrent workers.\n",
      "[Parallel(n_jobs=2)]: Done  46 tasks      | elapsed:    5.6s\n",
      "[Parallel(n_jobs=2)]: Done 100 out of 100 | elapsed:   10.5s finished\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test error:  0.8990913555582597\n",
      "Took 509.75 minutes\n"
     ]
    }
   ],
   "source": [
    "start_time = time.time()\n",
    "\n",
    "model_rf = RandomForestRegressor(n_estimators=100, n_jobs=2, verbose=1)\n",
    "# Fit model\n",
    "model_rf.fit(X_train, Y_train)\n",
    "#print('Valid error: ',np.sqrt(mean_squared_error(Y_valid, model_rf.predict(X_valid))))\n",
    "print('Test error: ', np.sqrt(mean_squared_error(Y_test, model_rf.predict(X_test))))\n",
    "\n",
    "pickle.dump(model_rf, open('model_rf.pkl', \"wb\"))\n",
    "\n",
    "print('Took %0.2f minutes'%((time.time() - start_time)/60))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['item_cnt_month_lag_1', 'item_id',\n",
       "       'avg_sale_last_6_item_cnt_month',\n",
       "       'item_cnt_month_by_item_id_mean_lag_1', 'last_sale_shop_iditem_id',\n",
       "       'tfidf_interaction_3', 'avg_sale_last_12_item_cnt_month', 'month',\n",
       "       'price_item_mean_lag_1',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_subtype_code_lag_1',\n",
       "       'first_sale_item_id',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_lag_12',\n",
       "       'avg_sale_last_12_item_cnt_month_by_date_block_num_shop_id_subtype_code',\n",
       "       'sales_item_delta2',\n",
       "       'std_sale_last_6_item_cnt_month_by_item_id_mean',\n",
       "       'std_sale_last_12_item_cnt_month_by_item_id_mean',\n",
       "       'avg_sale_last_12_item_cnt_month_by_item_id_mean',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_item_category_id_lag_2',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_type_code_lag_12',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_item_category_id_lag_3',\n",
       "       'std_sale_last_12_item_cnt_month',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_item_category_id_lag_1',\n",
       "       'price_item_mean_lag_2',\n",
       "       'avg_sale_last_6_item_cnt_month_by_item_id_mean',\n",
       "       'std_sale_last_6_item_cnt_month', 'sales_item_delta3',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_subtype_code_lag_2',\n",
       "       'item_cnt_month_by_date_block_num_item_id_city_code_lag_1',\n",
       "       'tfidf_interaction_1', 'item_cnt_month_by_item_id_mean_lag_2',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_item_category_id_lag_12',\n",
       "       'tfidf_interaction_4',\n",
       "       'std_sale_last_12_item_cnt_month_by_date_block_num_item_id_city_code',\n",
       "       'avg_sale_last_12_item_cnt_month_by_date_block_num_item_id_city_code',\n",
       "       'std_sale_last_6_item_cnt_month_by_date_block_num_item_id_city_code',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_type_code_lag_2',\n",
       "       'std_sale_last_6_item_cnt_month_by_date_block_num_shop_id_type_code',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_subtype_code_lag_12',\n",
       "       'tfidf_interaction_8',\n",
       "       'avg_sale_last_6_item_cnt_month_by_date_block_num_shop_id_item_category_id',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_lag_2',\n",
       "       'item_cnt_month_lag_2', 'tfidf_interaction_10',\n",
       "       'item_cnt_month_by_date_block_num_city_code_lag_12',\n",
       "       'std_sale_last_6_item_cnt_month_by_date_block_num_shop_id_subtype_code',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_type_code_lag_1',\n",
       "       'item_cnt_month_by_date_block_num_type_code_lag_1',\n",
       "       'avg_sale_last_6_item_cnt_month_by_date_block_num_item_id_city_code',\n",
       "       'std_sale_last_6_item_cnt_month_by_date_block_num_shop_id_item_category_id',\n",
       "       'tfidf_interaction_2',\n",
       "       'std_sale_last_12_item_cnt_month_by_date_block_num_shop_id_subtype_code',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_type_code_lag_6',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_subtype_code_lag_3',\n",
       "       'tfidf_interaction_6',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_type_code_lag_3',\n",
       "       'tfidf_interaction_7',\n",
       "       'std_sale_last_12_item_cnt_month_by_date_block_num_shop_id_type_code',\n",
       "       'avg_sale_last_12_item_cnt_month_by_date_block_num_shop_id_item_category_id',\n",
       "       'item_cnt_month_by_date_block_num_type_code_lag_12',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_lag_6',\n",
       "       'std_sale_last_12_item_cnt_month_by_date_block_num_shop_id_item_category_id',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_item_category_id_lag_6',\n",
       "       'std_sale_last_6_item_cnt_month_by_date_block_num_shop_id',\n",
       "       'std_sale_last_12_item_cnt_month_by_date_block_num_subtype_code',\n",
       "       'item_cnt_month_by_date_block_num_type_code_lag_2',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_subtype_code_lag_6',\n",
       "       'item_cnt_month_by_date_block_num_item_id_city_code_lag_2',\n",
       "       'item_cnt_month_by_date_block_num_type_code_lag_6',\n",
       "       'std_sale_last_6_item_cnt_month_by_date_block_num_type_code',\n",
       "       'tfidf_interaction_5', 'sales_item_delta6',\n",
       "       'avg_sale_last_12_item_cnt_month_by_date_block_num_subtype_code',\n",
       "       'std_sale_last_12_item_cnt_month_by_date_block_num_type_code',\n",
       "       'std_sale_last_12_item_cnt_month_by_date_block_num_item_category_id',\n",
       "       'item_cnt_month_lag_3',\n",
       "       'item_cnt_month_by_date_block_num_type_code_lag_3',\n",
       "       'item_cnt_month_by_date_block_num_item_id_city_code_lag_3',\n",
       "       'item_cnt_month_by_item_id_mean_lag_3',\n",
       "       'std_sale_last_6_item_cnt_month_by_date_block_num_item_category_id',\n",
       "       'avg_sale_last_12_item_cnt_month_by_date_block_num_shop_id_type_code',\n",
       "       'avg_sale_last_6_item_cnt_month_by_date_block_num_shop_id_subtype_code',\n",
       "       'item_cnt_month_by_date_block_num_subtype_code_lag_6',\n",
       "       'price_shop_item_mean_lag_1',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_lag_1',\n",
       "       'std_sale_last_6_item_cnt_month_by_date_block_num_city_code',\n",
       "       'item_cnt_month_by_date_block_num_item_category_id_lag_1',\n",
       "       'avg_sale_last_12_item_cnt_month_by_date_block_num_type_code',\n",
       "       'std_sale_last_12_item_cnt_month_by_date_block_num_shop_id',\n",
       "       'item_cnt_month_by_date_block_num_item_category_id_lag_2',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_lag_3',\n",
       "       'item_cnt_month_by_date_block_num_subtype_code_lag_1',\n",
       "       'avg_sale_last_12_item_cnt_month_by_date_block_num_item_category_id',\n",
       "       'item_cnt_month_by_date_block_num_subtype_code_lag_2',\n",
       "       'avg_sale_last_6_item_cnt_month_by_date_block_num_shop_id_type_code',\n",
       "       'item_cnt_month_by_date_block_num_city_code_lag_6',\n",
       "       'std_sale_last_6_item_cnt_month_by_date_block_num_subtype_code',\n",
       "       'avg_sale_last_12_item_cnt_month_by_date_block_num_shop_id',\n",
       "       'item_cnt_month_by_date_block_num_city_code_lag_2',\n",
       "       'avg_sale_last_6_item_cnt_month_by_date_block_num_type_code',\n",
       "       'item_cnt_month_by_date_block_num_subtype_code_lag_3',\n",
       "       'item_cnt_month_by_date_block_num_item_category_id_lag_3',\n",
       "       'price_item_mean_lag_3',\n",
       "       'std_sale_last_12_item_cnt_month_by_date_block_num_city_code',\n",
       "       'item_cnt_month_by_date_block_num_item_category_id_lag_6',\n",
       "       'shop_id', 'tfidf_interaction_9',\n",
       "       'item_cnt_month_by_date_block_num_subtype_code_lag_12',\n",
       "       'item_cnt_month_by_date_block_num_city_code_lag_1',\n",
       "       'item_cnt_month_by_date_block_num_city_code_lag_3',\n",
       "       'item_cnt_month_by_date_block_num_item_category_id_lag_12',\n",
       "       'avg_sale_last_6_item_cnt_month_by_date_block_num_shop_id',\n",
       "       'sales_shopitem_delta2',\n",
       "       'avg_sale_last_12_item_cnt_month_by_date_block_num_city_code',\n",
       "       'sales_shopitem_delta3',\n",
       "       'avg_sale_last_6_item_cnt_month_by_date_block_num_subtype_code',\n",
       "       'avg_sale_last_6_item_cnt_month_by_date_block_num_item_category_id',\n",
       "       'item_cnt_month_by_item_id_mean_lag_6',\n",
       "       'first_sale_shop_iditem_id',\n",
       "       'avg_sale_last_6_item_cnt_month_by_date_block_num_city_code',\n",
       "       'price_item_mean_lag_6', 'price_shop_item_mean_lag_3',\n",
       "       'price_shop_item_mean_lag_2',\n",
       "       'item_cnt_month_by_date_block_num_item_id_city_code_lag_6',\n",
       "       'subtype_code', 'city_code', 'item_category_id',\n",
       "       'sales_item_delta12', 'price_item_mean_lag_12',\n",
       "       'item_cnt_month_by_item_id_mean_lag_12', 'item_cnt_month_lag_6',\n",
       "       'consecutiveis_price_down_item_month_',\n",
       "       'price_shop_item_mean_lag_6',\n",
       "       'item_cnt_month_by_date_block_num_item_id_city_code_lag_12',\n",
       "       'consecutiveis_sales_up_item_month_', 'sales_shopitem_delta6',\n",
       "       'days_of_month', 'consecutiveis_price_up_item_month_',\n",
       "       'sales_shopitem_delta12', 'price_shop_item_mean_lag_12',\n",
       "       'consecutiveis_sales_down_item_month_', 'item_cnt_month_lag_12',\n",
       "       'last_sale_item_id', 'consecutiveis_sales_up_shopitem_month_',\n",
       "       'consecutiveis_price_up_shopitem_month_',\n",
       "       'consecutiveis_price_down_shopitem_month_',\n",
       "       'consecutiveis_sales_down_shopitem_month_', 'type_code', 'year'],\n",
       "      dtype='<U74')"
      ]
     },
     "execution_count": 121,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "idx = np.argsort(model_rf.feature_importances_)\n",
    "cols = np.array(X_train.columns.tolist())[idx][::-1]\n",
    "cols"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['item_cnt_month_lag_1', 'item_id',\n",
       "       'avg_sale_last_6_item_cnt_month',\n",
       "       'item_cnt_month_by_item_id_mean_lag_1', 'last_sale_shop_iditem_id',\n",
       "       'tfidf_interaction_3', 'avg_sale_last_12_item_cnt_month', 'month',\n",
       "       'price_item_mean_lag_1',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_subtype_code_lag_1',\n",
       "       'first_sale_item_id',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_lag_12',\n",
       "       'avg_sale_last_12_item_cnt_month_by_date_block_num_shop_id_subtype_code',\n",
       "       'sales_item_delta2',\n",
       "       'std_sale_last_6_item_cnt_month_by_item_id_mean',\n",
       "       'std_sale_last_12_item_cnt_month_by_item_id_mean',\n",
       "       'avg_sale_last_12_item_cnt_month_by_item_id_mean',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_item_category_id_lag_2',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_type_code_lag_12',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_item_category_id_lag_3',\n",
       "       'std_sale_last_12_item_cnt_month',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_item_category_id_lag_1',\n",
       "       'price_item_mean_lag_2',\n",
       "       'avg_sale_last_6_item_cnt_month_by_item_id_mean',\n",
       "       'std_sale_last_6_item_cnt_month', 'sales_item_delta3',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_subtype_code_lag_2',\n",
       "       'item_cnt_month_by_date_block_num_item_id_city_code_lag_1',\n",
       "       'tfidf_interaction_1', 'item_cnt_month_by_item_id_mean_lag_2',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_item_category_id_lag_12',\n",
       "       'tfidf_interaction_4',\n",
       "       'std_sale_last_12_item_cnt_month_by_date_block_num_item_id_city_code',\n",
       "       'avg_sale_last_12_item_cnt_month_by_date_block_num_item_id_city_code',\n",
       "       'std_sale_last_6_item_cnt_month_by_date_block_num_item_id_city_code',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_type_code_lag_2',\n",
       "       'std_sale_last_6_item_cnt_month_by_date_block_num_shop_id_type_code',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_subtype_code_lag_12',\n",
       "       'tfidf_interaction_8',\n",
       "       'avg_sale_last_6_item_cnt_month_by_date_block_num_shop_id_item_category_id',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_lag_2',\n",
       "       'item_cnt_month_lag_2', 'tfidf_interaction_10',\n",
       "       'item_cnt_month_by_date_block_num_city_code_lag_12',\n",
       "       'std_sale_last_6_item_cnt_month_by_date_block_num_shop_id_subtype_code',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_type_code_lag_1',\n",
       "       'item_cnt_month_by_date_block_num_type_code_lag_1',\n",
       "       'avg_sale_last_6_item_cnt_month_by_date_block_num_item_id_city_code',\n",
       "       'std_sale_last_6_item_cnt_month_by_date_block_num_shop_id_item_category_id',\n",
       "       'tfidf_interaction_2',\n",
       "       'std_sale_last_12_item_cnt_month_by_date_block_num_shop_id_subtype_code',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_type_code_lag_6',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_subtype_code_lag_3',\n",
       "       'tfidf_interaction_6',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_type_code_lag_3',\n",
       "       'tfidf_interaction_7',\n",
       "       'std_sale_last_12_item_cnt_month_by_date_block_num_shop_id_type_code',\n",
       "       'avg_sale_last_12_item_cnt_month_by_date_block_num_shop_id_item_category_id',\n",
       "       'item_cnt_month_by_date_block_num_type_code_lag_12',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_lag_6',\n",
       "       'std_sale_last_12_item_cnt_month_by_date_block_num_shop_id_item_category_id',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_item_category_id_lag_6',\n",
       "       'std_sale_last_6_item_cnt_month_by_date_block_num_shop_id',\n",
       "       'std_sale_last_12_item_cnt_month_by_date_block_num_subtype_code',\n",
       "       'item_cnt_month_by_date_block_num_type_code_lag_2',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_subtype_code_lag_6',\n",
       "       'item_cnt_month_by_date_block_num_item_id_city_code_lag_2',\n",
       "       'item_cnt_month_by_date_block_num_type_code_lag_6',\n",
       "       'std_sale_last_6_item_cnt_month_by_date_block_num_type_code',\n",
       "       'tfidf_interaction_5', 'sales_item_delta6',\n",
       "       'avg_sale_last_12_item_cnt_month_by_date_block_num_subtype_code',\n",
       "       'std_sale_last_12_item_cnt_month_by_date_block_num_type_code',\n",
       "       'std_sale_last_12_item_cnt_month_by_date_block_num_item_category_id',\n",
       "       'item_cnt_month_lag_3',\n",
       "       'item_cnt_month_by_date_block_num_type_code_lag_3',\n",
       "       'item_cnt_month_by_date_block_num_item_id_city_code_lag_3',\n",
       "       'item_cnt_month_by_item_id_mean_lag_3',\n",
       "       'std_sale_last_6_item_cnt_month_by_date_block_num_item_category_id',\n",
       "       'avg_sale_last_12_item_cnt_month_by_date_block_num_shop_id_type_code',\n",
       "       'avg_sale_last_6_item_cnt_month_by_date_block_num_shop_id_subtype_code',\n",
       "       'item_cnt_month_by_date_block_num_subtype_code_lag_6',\n",
       "       'price_shop_item_mean_lag_1',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_lag_1',\n",
       "       'std_sale_last_6_item_cnt_month_by_date_block_num_city_code',\n",
       "       'item_cnt_month_by_date_block_num_item_category_id_lag_1',\n",
       "       'avg_sale_last_12_item_cnt_month_by_date_block_num_type_code',\n",
       "       'std_sale_last_12_item_cnt_month_by_date_block_num_shop_id',\n",
       "       'item_cnt_month_by_date_block_num_item_category_id_lag_2',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_lag_3',\n",
       "       'item_cnt_month_by_date_block_num_subtype_code_lag_1',\n",
       "       'avg_sale_last_12_item_cnt_month_by_date_block_num_item_category_id',\n",
       "       'item_cnt_month_by_date_block_num_subtype_code_lag_2',\n",
       "       'avg_sale_last_6_item_cnt_month_by_date_block_num_shop_id_type_code'],\n",
       "      dtype='<U74')"
      ]
     },
     "execution_count": 123,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "selected_features = cols[:94]\n",
    "selected_features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {},
   "outputs": [],
   "source": [
    "Target = 'item_cnt_month'\n",
    "selected_features = all_data.columns.tolist()\n",
    "selected_features.remove(Target)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(4498200, 148)\n",
      "(4498200,)\n",
      "(214200, 148)\n",
      "(214200,)\n",
      "(214200, 148)\n"
     ]
    }
   ],
   "source": [
    "\n",
    "X_train = all_data[dates<33][selected_features]\n",
    "Y_train = all_data[dates<33][Target]\n",
    "\n",
    "X_valid = all_data[dates==33][selected_features]\n",
    "Y_valid = all_data[dates==33][Target]\n",
    "\n",
    "X_test =  all_data[dates==34][selected_features]\n",
    "\n",
    "\n",
    "\n",
    "print(X_train.shape)\n",
    "print(Y_train.shape)\n",
    "print(X_valid.shape)\n",
    "print(Y_valid.shape)\n",
    "print(X_test.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 54918,
     "status": "ok",
     "timestamp": 1561968508777,
     "user": {
      "displayName": "Shengbing YU",
      "photoUrl": "",
      "userId": "03994843105687918628"
     },
     "user_tz": -600
    },
    "id": "uAelbdz1Phgc",
    "outputId": "8dce608e-386f-4fb8-ca75-9b7040f28c79"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Valid error:  0.8826219320254746\n",
      "preds mean 0.3229976229264609\n"
     ]
    }
   ],
   "source": [
    "model_linreg = LinearRegression(fit_intercept=True)\n",
    "model_linreg.fit(X_train, Y_train)\n",
    "print('Valid error: ',np.sqrt(mean_squared_error(Y_valid, model_linreg.predict(X_valid))))\n",
    "preds = model_linreg.predict(X_test).clip(0,20)\n",
    "print('preds mean',preds.mean())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 6839,
     "status": "ok",
     "timestamp": 1561968536070,
     "user": {
      "displayName": "Shengbing YU",
      "photoUrl": "",
      "userId": "03994843105687918628"
     },
     "user_tz": -600
    },
    "id": "egwtSGkBPhge",
    "outputId": "998a3275-c5e4-4daf-af99-df0631aba135"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Valid error:  0.8826218483580077\n",
      "preds mean 0.32299793713761005\n"
     ]
    }
   ],
   "source": [
    "model_ridge = Ridge()\n",
    "model_ridge.fit(X_train, Y_train)\n",
    "print('Valid error: ',np.sqrt(mean_squared_error(Y_valid, model_ridge.predict(X_valid))))\n",
    "preds = model_ridge.predict(X_test).clip(0,20)\n",
    "print('preds mean',preds.mean())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 9204,
     "status": "ok",
     "timestamp": 1561968549811,
     "user": {
      "displayName": "Shengbing YU",
      "photoUrl": "",
      "userId": "03994843105687918628"
     },
     "user_tz": -600
    },
    "id": "yukmTwMDPhgg",
    "outputId": "291072ac-5a78-419b-a76c-4cd780e23b20"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Valid error:  1.0913896101211986\n",
      "preds mean 0.2205517762660619\n"
     ]
    }
   ],
   "source": [
    "model_lasso = Lasso(alpha=10.0, fit_intercept=True, max_iter=1000)\n",
    "model_lasso.fit(X_train, Y_train)\n",
    "print('Valid error: ',np.sqrt(mean_squared_error(Y_valid, model_lasso.predict(X_valid))))\n",
    "preds = model_lasso.predict(X_test).clip(0,20)\n",
    "print('preds mean',preds.mean())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(4926600, 95)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# all_data = all_data[list(selected_features)+['item_cnt_month']]\n",
    "# all_data.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "outfile = open('all_data_ready','wb')\n",
    "pickle.dump(all_data,outfile, protocol=4)\n",
    "outfile.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 252,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['item_cnt_month_lag_1', 'is_price_down_shopitem_month_2',\n",
       "       'item_cnt_month_by_item_id_mean_lag_1',\n",
       "       'is_sales_down_shopitem_month_2', 'item_cnt_month_lag_2',\n",
       "       'is_price_down_shopitem_month_7', 'is_price_down_shopitem_month_3',\n",
       "       'item_cnt_month_lag_3', 'is_sales_down_item_month_2',\n",
       "       'is_price_down_shopitem_month_9', 'is_sold_item_month_1',\n",
       "       'item_category_id_cnt_month_mean_Expanding',\n",
       "       'is_price_down_shopitem_month_11',\n",
       "       'shop_id_cnt_month_mean_Expanding', 'item_cnt_month_lag_4',\n",
       "       'is_price_down_shopitem_month_4', 'month',\n",
       "       'is_sold_shopitem_month_7', 'is_sales_up_shopitem_month_7',\n",
       "       'item_cnt_month_lag_5', 'item_cnt_month_lag_12',\n",
       "       'item_cnt_month_lag_7', 'item_cnt_month_lag_6',\n",
       "       'is_sold_shopitem_month_4', 'item_cnt_month_lag_8',\n",
       "       'is_sold_shopitem_month_5', 'consecutiveis_sales_up_item_month_',\n",
       "       'price_item_mean_lag_3', 'is_sold_shopitem_month_9',\n",
       "       'is_sold_shopitem_month_11', 'is_price_up_shopitem_month_7',\n",
       "       'is_price_down_shopitem_month_8', 'is_price_down_item_month_3',\n",
       "       'is_sold_shopitem_month_12', 'is_price_up_item_month_3',\n",
       "       'is_price_up_shopitem_month_5', 'is_price_up_shopitem_month_9',\n",
       "       'item_cnt_month_lag_11', 'item_cnt_month_lag_9',\n",
       "       'is_price_up_item_month_9', 'item_id_cnt_month_mean_Expanding',\n",
       "       'is_price_up_item_month_5', 'is_price_down_item_month_2',\n",
       "       'price_shop_item_mean_lag_3', 'is_price_down_item_month_7',\n",
       "       'item_cnt_month_lag_10', 'is_price_up_item_month_7',\n",
       "       'is_sales_up_shopitem_month_11',\n",
       "       'type_code_cnt_month_mean_Expanding',\n",
       "       'is_sales_up_shopitem_month_6', 'is_price_up_item_month_8',\n",
       "       'is_sold_item_month_4', 'is_sales_up_shopitem_month_5',\n",
       "       'item_cnt_month_by_item_id_mean_lag_5',\n",
       "       'is_price_down_item_month_4', 'is_sales_up_item_month_7',\n",
       "       'price_item_mean_lag_2', 'is_price_up_item_month_10',\n",
       "       'is_sales_up_item_month_2', 'is_sold_item_month_10',\n",
       "       'price_shop_item_mean_lag_2', 'tfidf_interaction_2',\n",
       "       'is_sales_up_item_month_5', 'item_cnt_month_by_item_id_mean_lag_8',\n",
       "       'tfidf_interaction_4', 'is_price_up_item_month_2',\n",
       "       'is_price_down_item_month_5', 'is_price_down_item_month_9',\n",
       "       'is_sales_up_shopitem_month_9', 'is_price_up_item_month_4',\n",
       "       'is_sold_item_month_9', 'is_sold_shopitem_month_10',\n",
       "       'is_price_up_item_month_11', 'is_price_up_item_month_12',\n",
       "       'tfidf_interaction_8', 'consecutiveis_price_up_shopitem_month_',\n",
       "       'tfidf_interaction_10', 'is_sold_shopitem_month_6',\n",
       "       'price_item_mean_lag_9', 'is_sales_up_item_month_8',\n",
       "       'tfidf_interaction_7', 'is_sales_up_item_month_4', 'shop_id',\n",
       "       'price_shop_item_mean_lag_8', 'price_item_mean_lag_6',\n",
       "       'is_sold_item_month_11', 'tfidf_interaction_3',\n",
       "       'is_sales_up_shopitem_month_12', 'is_price_down_item_month_8',\n",
       "       'is_sales_down_item_month_12', 'price_shop_item_mean_lag_9',\n",
       "       'is_sold_shopitem_month_8', 'is_sales_up_shopitem_month_4',\n",
       "       'price_item_mean_lag_7', 'tfidf_interaction_6',\n",
       "       'is_price_down_item_month_12', 'is_sales_up_item_month_9',\n",
       "       'price_item_mean_lag_12', 'price_item_mean_lag_8',\n",
       "       'is_sold_item_month_8', 'item_id', 'price_shop_item_mean_lag_7',\n",
       "       'tfidf_interaction_5', 'price_shop_item_mean_lag_11',\n",
       "       'price_shop_item_mean_lag_6', 'item_category_id',\n",
       "       'consecutiveis_price_down_item_month_', 'is_sold_item_month_5',\n",
       "       'price_shop_item_mean_lag_12',\n",
       "       'item_cnt_month_by_item_id_mean_lag_3', 'tfidf_interaction_1',\n",
       "       'is_price_up_item_month_6', 'is_price_down_shopitem_month_12',\n",
       "       'price_item_mean_lag_5', 'is_price_down_item_month_11',\n",
       "       'tfidf_interaction_9', 'year_cnt_month_mean_Expanding',\n",
       "       'is_sales_up_item_month_6', 'is_sold_item_month_7',\n",
       "       'consecutiveis_price_up_item_month_',\n",
       "       'is_sales_down_item_month_11', 'price_shop_item_mean_lag_10',\n",
       "       'month_cnt_month_mean_Expanding', 'price_item_mean_lag_11',\n",
       "       'price_shop_item_mean_lag_5', 'is_price_down_item_month_10',\n",
       "       'is_sales_up_shopitem_month_8', 'is_price_down_shopitem_month_6',\n",
       "       'item_cnt_month_by_item_id_mean_lag_11', 'subtype_code',\n",
       "       'is_price_down_shopitem_month_5',\n",
       "       'city_code_cnt_month_mean_Expanding',\n",
       "       'item_cnt_month_by_item_id_mean_lag_7', 'price_item_mean_lag_10',\n",
       "       'is_sold_item_month_6', 'is_sales_down_item_month_10',\n",
       "       'is_sales_up_item_month_10',\n",
       "       'item_cnt_month_by_item_id_mean_lag_9',\n",
       "       'is_sales_up_item_month_11', 'price_shop_item_mean_lag_4',\n",
       "       'price_shop_item_mean_lag_1',\n",
       "       'item_cnt_month_by_item_id_mean_lag_10',\n",
       "       'is_sales_up_item_month_12', 'days_of_month',\n",
       "       'is_sold_shopitem_month_3',\n",
       "       'item_cnt_month_by_item_id_mean_lag_12',\n",
       "       'is_sales_up_shopitem_month_10', 'is_price_down_item_month_6',\n",
       "       'is_sold_item_month_12', 'price_item_mean_lag_1',\n",
       "       'consecutiveis_sales_down_shopitem_month_',\n",
       "       'is_sales_down_item_month_9',\n",
       "       'item_cnt_month_by_item_id_mean_lag_6',\n",
       "       'consecutiveis_sales_down_item_month_', 'price_item_mean_lag_4',\n",
       "       'is_price_up_shopitem_month_3', 'is_sales_down_shopitem_month_10',\n",
       "       'is_sales_up_shopitem_month_3', 'is_sales_down_item_month_8',\n",
       "       'is_sales_down_shopitem_month_9',\n",
       "       'item_cnt_month_by_item_id_mean_lag_4',\n",
       "       'is_sales_down_item_month_7', 'is_sales_down_item_month_5',\n",
       "       'is_sales_down_item_month_6',\n",
       "       'consecutiveis_sales_up_shopitem_month_',\n",
       "       'is_sales_down_shopitem_month_12', 'is_price_up_shopitem_month_10',\n",
       "       'is_sales_down_shopitem_month_11', 'is_sales_up_item_month_3',\n",
       "       'consecutiveis_price_down_shopitem_month_', 'is_sold_item_month_3',\n",
       "       'is_price_up_shopitem_month_11', 'last_sale_item_month',\n",
       "       'is_price_up_shopitem_month_6', 'is_sales_down_item_month_4',\n",
       "       'is_sales_down_shopitem_month_8',\n",
       "       'is_price_down_shopitem_month_10', 'is_price_up_shopitem_month_12',\n",
       "       'is_sales_up_shopitem_month_2', 'last_sale_shopitem_month',\n",
       "       'is_sales_down_item_month_3',\n",
       "       'days_of_month_cnt_month_mean_Expanding',\n",
       "       'is_sales_down_shopitem_month_5', 'is_sales_down_shopitem_month_4',\n",
       "       'is_sales_down_shopitem_month_7', 'is_sales_down_shopitem_month_6',\n",
       "       'is_price_up_shopitem_month_8',\n",
       "       'item_cnt_month_by_item_id_mean_lag_2',\n",
       "       'is_price_up_shopitem_month_4', 'is_sold_item_month_2',\n",
       "       'is_sales_down_shopitem_month_3', 'year',\n",
       "       'is_sold_shopitem_month_1', 'is_sold_shopitem_month_2',\n",
       "       'is_price_up_shopitem_month_2'], dtype='<U41')"
      ]
     },
     "execution_count": 252,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "idx = np.argsort(model_linreg.coef_)[::-1]\n",
    "cols = X_train.columns.tolist()\n",
    "cols = np.array(cols)[idx]\n",
    "cols"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0:\tlearn: 1.0485272\ttotal: 18.3s\tremaining: 3h 33m 46s\n",
      "1:\tlearn: 0.9899452\ttotal: 32.8s\tremaining: 3h 10m 59s\n",
      "2:\tlearn: 0.9440494\ttotal: 46.9s\tremaining: 3h 1m 44s\n",
      "3:\tlearn: 0.9084047\ttotal: 59.9s\tremaining: 2h 53m 37s\n",
      "4:\tlearn: 0.8798759\ttotal: 1m 14s\tremaining: 2h 52m 21s\n",
      "5:\tlearn: 0.8581389\ttotal: 1m 27s\tremaining: 2h 49m 22s\n",
      "6:\tlearn: 0.8409671\ttotal: 1m 42s\tremaining: 2h 48m 57s\n",
      "7:\tlearn: 0.8277260\ttotal: 1m 57s\tremaining: 2h 49m 40s\n",
      "8:\tlearn: 0.8170442\ttotal: 2m 10s\tremaining: 2h 46m 35s\n",
      "9:\tlearn: 0.8090707\ttotal: 2m 25s\tremaining: 2h 47m 12s\n",
      "10:\tlearn: 0.8022831\ttotal: 2m 40s\tremaining: 2h 47m 59s\n",
      "11:\tlearn: 0.7967013\ttotal: 2m 57s\tremaining: 2h 49m 11s\n",
      "12:\tlearn: 0.7921533\ttotal: 3m 13s\tremaining: 2h 50m 36s\n",
      "13:\tlearn: 0.7882261\ttotal: 3m 32s\tremaining: 2h 53m 14s\n",
      "14:\tlearn: 0.7851062\ttotal: 3m 45s\tremaining: 2h 51m 34s\n",
      "15:\tlearn: 0.7823303\ttotal: 4m\tremaining: 2h 51m 3s\n",
      "16:\tlearn: 0.7793622\ttotal: 4m 13s\tremaining: 2h 49m 57s\n",
      "17:\tlearn: 0.7772327\ttotal: 4m 27s\tremaining: 2h 49m 6s\n",
      "18:\tlearn: 0.7751111\ttotal: 4m 43s\tremaining: 2h 49m 34s\n",
      "19:\tlearn: 0.7733739\ttotal: 5m 3s\tremaining: 2h 51m 46s\n",
      "20:\tlearn: 0.7719666\ttotal: 5m 17s\tremaining: 2h 51m 2s\n",
      "21:\tlearn: 0.7707022\ttotal: 5m 31s\tremaining: 2h 50m 8s\n",
      "22:\tlearn: 0.7691787\ttotal: 5m 43s\tremaining: 2h 48m 30s\n",
      "23:\tlearn: 0.7680944\ttotal: 5m 58s\tremaining: 2h 48m 17s\n",
      "24:\tlearn: 0.7669423\ttotal: 6m 15s\tremaining: 2h 49m\n",
      "25:\tlearn: 0.7653688\ttotal: 6m 35s\tremaining: 2h 50m 39s\n",
      "26:\tlearn: 0.7636558\ttotal: 6m 49s\tremaining: 2h 50m 3s\n",
      "27:\tlearn: 0.7627420\ttotal: 7m 2s\tremaining: 2h 49m 5s\n",
      "28:\tlearn: 0.7613203\ttotal: 7m 16s\tremaining: 2h 48m 19s\n",
      "29:\tlearn: 0.7603739\ttotal: 7m 30s\tremaining: 2h 47m 34s\n",
      "30:\tlearn: 0.7591484\ttotal: 7m 47s\tremaining: 2h 48m 16s\n",
      "31:\tlearn: 0.7582792\ttotal: 8m 1s\tremaining: 2h 47m 36s\n",
      "32:\tlearn: 0.7577394\ttotal: 8m 15s\tremaining: 2h 46m 50s\n",
      "33:\tlearn: 0.7572351\ttotal: 8m 27s\tremaining: 2h 45m 50s\n",
      "34:\tlearn: 0.7566699\ttotal: 8m 45s\tremaining: 2h 46m 26s\n",
      "35:\tlearn: 0.7556295\ttotal: 8m 59s\tremaining: 2h 45m 42s\n",
      "36:\tlearn: 0.7549528\ttotal: 9m 12s\tremaining: 2h 44m 53s\n",
      "37:\tlearn: 0.7537557\ttotal: 9m 26s\tremaining: 2h 44m 22s\n",
      "38:\tlearn: 0.7533003\ttotal: 9m 39s\tremaining: 2h 43m 45s\n",
      "39:\tlearn: 0.7526729\ttotal: 9m 52s\tremaining: 2h 42m 57s\n",
      "40:\tlearn: 0.7515806\ttotal: 10m 6s\tremaining: 2h 42m 28s\n",
      "41:\tlearn: 0.7510820\ttotal: 10m 20s\tremaining: 2h 42m 4s\n",
      "42:\tlearn: 0.7501123\ttotal: 10m 34s\tremaining: 2h 41m 29s\n",
      "43:\tlearn: 0.7495277\ttotal: 10m 48s\tremaining: 2h 41m 7s\n",
      "44:\tlearn: 0.7487276\ttotal: 11m 2s\tremaining: 2h 40m 45s\n",
      "45:\tlearn: 0.7483087\ttotal: 11m 16s\tremaining: 2h 40m 19s\n",
      "46:\tlearn: 0.7478784\ttotal: 11m 31s\tremaining: 2h 40m 7s\n",
      "47:\tlearn: 0.7475768\ttotal: 11m 44s\tremaining: 2h 39m 34s\n",
      "48:\tlearn: 0.7472174\ttotal: 11m 57s\tremaining: 2h 38m 55s\n",
      "49:\tlearn: 0.7468520\ttotal: 12m 17s\tremaining: 2h 39m 41s\n",
      "50:\tlearn: 0.7465738\ttotal: 12m 32s\tremaining: 2h 39m 40s\n",
      "51:\tlearn: 0.7461047\ttotal: 12m 49s\tremaining: 2h 39m 46s\n",
      "52:\tlearn: 0.7457644\ttotal: 13m 6s\tremaining: 2h 39m 57s\n",
      "53:\tlearn: 0.7451296\ttotal: 13m 21s\tremaining: 2h 39m 49s\n",
      "54:\tlearn: 0.7448707\ttotal: 13m 39s\tremaining: 2h 40m 11s\n",
      "55:\tlearn: 0.7445055\ttotal: 13m 55s\tremaining: 2h 40m 13s\n",
      "56:\tlearn: 0.7439195\ttotal: 14m 10s\tremaining: 2h 39m 48s\n",
      "57:\tlearn: 0.7433093\ttotal: 14m 24s\tremaining: 2h 39m 23s\n",
      "58:\tlearn: 0.7428624\ttotal: 14m 40s\tremaining: 2h 39m 27s\n",
      "59:\tlearn: 0.7420439\ttotal: 14m 57s\tremaining: 2h 39m 33s\n",
      "60:\tlearn: 0.7415708\ttotal: 15m 12s\tremaining: 2h 39m 15s\n",
      "61:\tlearn: 0.7400197\ttotal: 15m 36s\tremaining: 2h 40m 33s\n",
      "62:\tlearn: 0.7397123\ttotal: 15m 49s\tremaining: 2h 40m 3s\n",
      "63:\tlearn: 0.7393575\ttotal: 16m 5s\tremaining: 2h 39m 58s\n",
      "64:\tlearn: 0.7386939\ttotal: 16m 20s\tremaining: 2h 39m 42s\n",
      "65:\tlearn: 0.7381484\ttotal: 16m 33s\tremaining: 2h 39m 2s\n",
      "66:\tlearn: 0.7377985\ttotal: 16m 47s\tremaining: 2h 38m 38s\n",
      "67:\tlearn: 0.7369971\ttotal: 17m 1s\tremaining: 2h 38m 12s\n",
      "68:\tlearn: 0.7368098\ttotal: 17m 13s\tremaining: 2h 37m 27s\n",
      "69:\tlearn: 0.7365332\ttotal: 17m 29s\tremaining: 2h 37m 25s\n",
      "70:\tlearn: 0.7362425\ttotal: 17m 42s\tremaining: 2h 36m 50s\n",
      "71:\tlearn: 0.7360578\ttotal: 17m 54s\tremaining: 2h 36m 14s\n",
      "72:\tlearn: 0.7355553\ttotal: 18m 7s\tremaining: 2h 35m 44s\n",
      "73:\tlearn: 0.7354069\ttotal: 18m 19s\tremaining: 2h 35m 4s\n",
      "74:\tlearn: 0.7351035\ttotal: 18m 32s\tremaining: 2h 34m 32s\n",
      "75:\tlearn: 0.7346916\ttotal: 18m 45s\tremaining: 2h 33m 59s\n",
      "76:\tlearn: 0.7344087\ttotal: 18m 58s\tremaining: 2h 33m 29s\n",
      "77:\tlearn: 0.7341141\ttotal: 19m 10s\tremaining: 2h 32m 57s\n",
      "78:\tlearn: 0.7334936\ttotal: 19m 24s\tremaining: 2h 32m 37s\n",
      "79:\tlearn: 0.7331323\ttotal: 19m 39s\tremaining: 2h 32m 23s\n",
      "80:\tlearn: 0.7326628\ttotal: 19m 55s\tremaining: 2h 32m 14s\n",
      "81:\tlearn: 0.7324777\ttotal: 20m 10s\tremaining: 2h 32m 2s\n",
      "82:\tlearn: 0.7310961\ttotal: 20m 29s\tremaining: 2h 32m 21s\n",
      "83:\tlearn: 0.7299486\ttotal: 20m 46s\tremaining: 2h 32m 18s\n",
      "84:\tlearn: 0.7297844\ttotal: 21m 2s\tremaining: 2h 32m 16s\n",
      "85:\tlearn: 0.7296033\ttotal: 21m 16s\tremaining: 2h 31m 55s\n",
      "86:\tlearn: 0.7292296\ttotal: 21m 29s\tremaining: 2h 31m 25s\n",
      "87:\tlearn: 0.7289206\ttotal: 21m 44s\tremaining: 2h 31m 10s\n",
      "88:\tlearn: 0.7286979\ttotal: 22m\tremaining: 2h 31m 3s\n",
      "89:\tlearn: 0.7284475\ttotal: 22m 14s\tremaining: 2h 30m 44s\n",
      "90:\tlearn: 0.7282456\ttotal: 22m 26s\tremaining: 2h 30m 11s\n",
      "91:\tlearn: 0.7280373\ttotal: 22m 39s\tremaining: 2h 29m 45s\n",
      "92:\tlearn: 0.7277735\ttotal: 22m 53s\tremaining: 2h 29m 23s\n",
      "93:\tlearn: 0.7275251\ttotal: 23m 6s\tremaining: 2h 29m 1s\n",
      "94:\tlearn: 0.7265740\ttotal: 23m 20s\tremaining: 2h 28m 38s\n",
      "95:\tlearn: 0.7263260\ttotal: 23m 35s\tremaining: 2h 28m 24s\n",
      "96:\tlearn: 0.7254579\ttotal: 23m 48s\tremaining: 2h 28m 2s\n",
      "97:\tlearn: 0.7250596\ttotal: 24m 1s\tremaining: 2h 27m 32s\n",
      "98:\tlearn: 0.7245396\ttotal: 24m 14s\tremaining: 2h 27m 10s\n",
      "99:\tlearn: 0.7244042\ttotal: 24m 26s\tremaining: 2h 26m 38s\n",
      "100:\tlearn: 0.7242135\ttotal: 24m 39s\tremaining: 2h 26m 15s\n",
      "101:\tlearn: 0.7238551\ttotal: 24m 51s\tremaining: 2h 25m 45s\n",
      "102:\tlearn: 0.7233632\ttotal: 25m 4s\tremaining: 2h 25m 19s\n",
      "103:\tlearn: 0.7231180\ttotal: 25m 18s\tremaining: 2h 25m 3s\n",
      "104:\tlearn: 0.7228894\ttotal: 25m 33s\tremaining: 2h 24m 47s\n",
      "105:\tlearn: 0.7225853\ttotal: 25m 44s\tremaining: 2h 24m 17s\n",
      "106:\tlearn: 0.7224335\ttotal: 25m 58s\tremaining: 2h 23m 56s\n",
      "107:\tlearn: 0.7222913\ttotal: 26m 11s\tremaining: 2h 23m 34s\n",
      "108:\tlearn: 0.7218970\ttotal: 26m 26s\tremaining: 2h 23m 22s\n",
      "109:\tlearn: 0.7214871\ttotal: 26m 39s\tremaining: 2h 22m 59s\n",
      "110:\tlearn: 0.7212913\ttotal: 26m 53s\tremaining: 2h 22m 41s\n",
      "111:\tlearn: 0.7210052\ttotal: 27m 10s\tremaining: 2h 22m 38s\n",
      "112:\tlearn: 0.7205356\ttotal: 27m 25s\tremaining: 2h 22m 27s\n",
      "113:\tlearn: 0.7204256\ttotal: 27m 37s\tremaining: 2h 22m 2s\n",
      "114:\tlearn: 0.7202154\ttotal: 27m 53s\tremaining: 2h 21m 51s\n",
      "115:\tlearn: 0.7201457\ttotal: 28m 6s\tremaining: 2h 21m 28s\n",
      "116:\tlearn: 0.7200146\ttotal: 28m 19s\tremaining: 2h 21m 7s\n",
      "117:\tlearn: 0.7187377\ttotal: 28m 33s\tremaining: 2h 20m 50s\n",
      "118:\tlearn: 0.7182852\ttotal: 28m 47s\tremaining: 2h 20m 33s\n",
      "119:\tlearn: 0.7180711\ttotal: 29m\tremaining: 2h 20m 12s\n",
      "120:\tlearn: 0.7179377\ttotal: 29m 14s\tremaining: 2h 19m 57s\n",
      "121:\tlearn: 0.7178016\ttotal: 29m 26s\tremaining: 2h 19m 30s\n",
      "122:\tlearn: 0.7177028\ttotal: 29m 38s\tremaining: 2h 19m 1s\n",
      "123:\tlearn: 0.7174539\ttotal: 29m 50s\tremaining: 2h 18m 38s\n",
      "124:\tlearn: 0.7172639\ttotal: 30m 3s\tremaining: 2h 18m 15s\n",
      "125:\tlearn: 0.7170940\ttotal: 30m 16s\tremaining: 2h 17m 53s\n",
      "126:\tlearn: 0.7168491\ttotal: 30m 29s\tremaining: 2h 17m 33s\n",
      "127:\tlearn: 0.7167205\ttotal: 30m 41s\tremaining: 2h 17m 7s\n",
      "128:\tlearn: 0.7165397\ttotal: 30m 56s\tremaining: 2h 16m 58s\n",
      "129:\tlearn: 0.7163374\ttotal: 31m 11s\tremaining: 2h 16m 45s\n",
      "130:\tlearn: 0.7161497\ttotal: 31m 24s\tremaining: 2h 16m 26s\n",
      "131:\tlearn: 0.7160377\ttotal: 31m 41s\tremaining: 2h 16m 20s\n",
      "132:\tlearn: 0.7159119\ttotal: 31m 57s\tremaining: 2h 16m 16s\n",
      "133:\tlearn: 0.7158126\ttotal: 32m 17s\tremaining: 2h 16m 22s\n",
      "134:\tlearn: 0.7154288\ttotal: 32m 34s\tremaining: 2h 16m 19s\n",
      "135:\tlearn: 0.7152765\ttotal: 32m 51s\tremaining: 2h 16m 14s\n",
      "136:\tlearn: 0.7151012\ttotal: 33m 4s\tremaining: 2h 15m 56s\n",
      "137:\tlearn: 0.7149253\ttotal: 33m 17s\tremaining: 2h 15m 35s\n",
      "138:\tlearn: 0.7147810\ttotal: 33m 31s\tremaining: 2h 15m 17s\n",
      "139:\tlearn: 0.7145673\ttotal: 33m 47s\tremaining: 2h 15m 9s\n",
      "140:\tlearn: 0.7144312\ttotal: 34m\tremaining: 2h 14m 49s\n",
      "141:\tlearn: 0.7141087\ttotal: 34m 14s\tremaining: 2h 14m 32s\n",
      "142:\tlearn: 0.7139306\ttotal: 34m 27s\tremaining: 2h 14m 14s\n",
      "143:\tlearn: 0.7136832\ttotal: 34m 43s\tremaining: 2h 14m 5s\n",
      "144:\tlearn: 0.7135480\ttotal: 34m 57s\tremaining: 2h 13m 48s\n",
      "145:\tlearn: 0.7130932\ttotal: 35m 11s\tremaining: 2h 13m 33s\n",
      "146:\tlearn: 0.7127936\ttotal: 35m 26s\tremaining: 2h 13m 20s\n",
      "147:\tlearn: 0.7126328\ttotal: 35m 42s\tremaining: 2h 13m 10s\n",
      "148:\tlearn: 0.7124327\ttotal: 35m 58s\tremaining: 2h 13m 1s\n",
      "149:\tlearn: 0.7123153\ttotal: 36m 13s\tremaining: 2h 12m 48s\n",
      "150:\tlearn: 0.7122203\ttotal: 36m 28s\tremaining: 2h 12m 36s\n",
      "151:\tlearn: 0.7121025\ttotal: 36m 43s\tremaining: 2h 12m 23s\n",
      "152:\tlearn: 0.7119563\ttotal: 36m 58s\tremaining: 2h 12m 12s\n",
      "153:\tlearn: 0.7117532\ttotal: 37m 13s\tremaining: 2h 11m 58s\n",
      "154:\tlearn: 0.7104031\ttotal: 37m 28s\tremaining: 2h 11m 45s\n",
      "155:\tlearn: 0.7102226\ttotal: 37m 46s\tremaining: 2h 11m 42s\n",
      "156:\tlearn: 0.7100088\ttotal: 38m 1s\tremaining: 2h 11m 29s\n",
      "157:\tlearn: 0.7098224\ttotal: 38m 15s\tremaining: 2h 11m 15s\n",
      "158:\tlearn: 0.7095084\ttotal: 38m 30s\tremaining: 2h 11m 1s\n",
      "159:\tlearn: 0.7094136\ttotal: 38m 46s\tremaining: 2h 10m 50s\n",
      "160:\tlearn: 0.7088613\ttotal: 39m\tremaining: 2h 10m 34s\n",
      "161:\tlearn: 0.7087203\ttotal: 39m 13s\tremaining: 2h 10m 14s\n",
      "162:\tlearn: 0.7086120\ttotal: 39m 28s\tremaining: 2h 10m 1s\n",
      "163:\tlearn: 0.7085212\ttotal: 39m 43s\tremaining: 2h 9m 50s\n",
      "164:\tlearn: 0.7083804\ttotal: 40m 1s\tremaining: 2h 9m 46s\n",
      "165:\tlearn: 0.7082128\ttotal: 40m 17s\tremaining: 2h 9m 36s\n",
      "166:\tlearn: 0.7080404\ttotal: 40m 31s\tremaining: 2h 9m 21s\n",
      "167:\tlearn: 0.7077897\ttotal: 40m 46s\tremaining: 2h 9m 8s\n",
      "168:\tlearn: 0.7075829\ttotal: 41m 1s\tremaining: 2h 8m 52s\n",
      "169:\tlearn: 0.7073149\ttotal: 41m 13s\tremaining: 2h 8m 31s\n",
      "170:\tlearn: 0.7070670\ttotal: 41m 26s\tremaining: 2h 8m 11s\n",
      "171:\tlearn: 0.7068259\ttotal: 41m 39s\tremaining: 2h 7m 51s\n",
      "172:\tlearn: 0.7067324\ttotal: 41m 50s\tremaining: 2h 7m 28s\n",
      "173:\tlearn: 0.7066228\ttotal: 42m 2s\tremaining: 2h 7m 4s\n",
      "174:\tlearn: 0.7065298\ttotal: 42m 14s\tremaining: 2h 6m 44s\n",
      "175:\tlearn: 0.7063492\ttotal: 42m 30s\tremaining: 2h 6m 34s\n",
      "176:\tlearn: 0.7061563\ttotal: 42m 43s\tremaining: 2h 6m 14s\n",
      "177:\tlearn: 0.7058994\ttotal: 42m 55s\tremaining: 2h 5m 53s\n",
      "178:\tlearn: 0.7057206\ttotal: 43m 10s\tremaining: 2h 5m 41s\n",
      "179:\tlearn: 0.7055978\ttotal: 43m 25s\tremaining: 2h 5m 26s\n",
      "180:\tlearn: 0.7055038\ttotal: 43m 40s\tremaining: 2h 5m 14s\n",
      "181:\tlearn: 0.7053990\ttotal: 43m 54s\tremaining: 2h 4m 56s\n",
      "182:\tlearn: 0.7053063\ttotal: 44m 7s\tremaining: 2h 4m 40s\n",
      "183:\tlearn: 0.7051339\ttotal: 44m 21s\tremaining: 2h 4m 24s\n",
      "184:\tlearn: 0.7047370\ttotal: 44m 36s\tremaining: 2h 4m 11s\n",
      "185:\tlearn: 0.7046478\ttotal: 44m 50s\tremaining: 2h 3m 54s\n",
      "186:\tlearn: 0.7043337\ttotal: 45m 4s\tremaining: 2h 3m 38s\n",
      "187:\tlearn: 0.7041430\ttotal: 45m 17s\tremaining: 2h 3m 21s\n",
      "188:\tlearn: 0.7040542\ttotal: 45m 32s\tremaining: 2h 3m 8s\n",
      "189:\tlearn: 0.7038923\ttotal: 45m 46s\tremaining: 2h 2m 52s\n",
      "190:\tlearn: 0.7038236\ttotal: 46m 1s\tremaining: 2h 2m 37s\n",
      "191:\tlearn: 0.7037022\ttotal: 46m 15s\tremaining: 2h 2m 22s\n",
      "192:\tlearn: 0.7035866\ttotal: 46m 28s\tremaining: 2h 2m 4s\n",
      "193:\tlearn: 0.7034202\ttotal: 46m 42s\tremaining: 2h 1m 49s\n",
      "194:\tlearn: 0.7032438\ttotal: 46m 56s\tremaining: 2h 1m 34s\n",
      "195:\tlearn: 0.7031424\ttotal: 47m 10s\tremaining: 2h 1m 18s\n",
      "196:\tlearn: 0.7028710\ttotal: 47m 24s\tremaining: 2h 1m 2s\n",
      "197:\tlearn: 0.7025953\ttotal: 47m 39s\tremaining: 2h 50s\n",
      "198:\tlearn: 0.7023331\ttotal: 47m 53s\tremaining: 2h 33s\n",
      "199:\tlearn: 0.7022175\ttotal: 48m 5s\tremaining: 2h 13s\n",
      "200:\tlearn: 0.7020732\ttotal: 48m 18s\tremaining: 1h 59m 55s\n",
      "201:\tlearn: 0.7019507\ttotal: 48m 34s\tremaining: 1h 59m 44s\n",
      "202:\tlearn: 0.7018136\ttotal: 48m 48s\tremaining: 1h 59m 30s\n",
      "203:\tlearn: 0.7016803\ttotal: 49m 4s\tremaining: 1h 59m 18s\n",
      "204:\tlearn: 0.7015954\ttotal: 49m 18s\tremaining: 1h 59m 3s\n",
      "205:\tlearn: 0.7014293\ttotal: 49m 33s\tremaining: 1h 58m 51s\n",
      "206:\tlearn: 0.7012990\ttotal: 49m 49s\tremaining: 1h 58m 40s\n",
      "207:\tlearn: 0.7012183\ttotal: 50m 3s\tremaining: 1h 58m 23s\n",
      "208:\tlearn: 0.7011074\ttotal: 50m 16s\tremaining: 1h 58m 7s\n",
      "209:\tlearn: 0.7009870\ttotal: 50m 30s\tremaining: 1h 57m 51s\n",
      "210:\tlearn: 0.7008497\ttotal: 50m 44s\tremaining: 1h 57m 36s\n",
      "211:\tlearn: 0.7003636\ttotal: 50m 59s\tremaining: 1h 57m 21s\n",
      "212:\tlearn: 0.6998580\ttotal: 51m 13s\tremaining: 1h 57m 7s\n",
      "213:\tlearn: 0.6997595\ttotal: 51m 27s\tremaining: 1h 56m 51s\n",
      "214:\tlearn: 0.6987646\ttotal: 51m 41s\tremaining: 1h 56m 35s\n",
      "215:\tlearn: 0.6987128\ttotal: 51m 54s\tremaining: 1h 56m 18s\n",
      "216:\tlearn: 0.6986292\ttotal: 52m 8s\tremaining: 1h 56m 2s\n",
      "217:\tlearn: 0.6981701\ttotal: 52m 22s\tremaining: 1h 55m 47s\n",
      "218:\tlearn: 0.6980415\ttotal: 52m 34s\tremaining: 1h 55m 27s\n",
      "219:\tlearn: 0.6978011\ttotal: 52m 47s\tremaining: 1h 55m 11s\n",
      "220:\tlearn: 0.6977151\ttotal: 52m 59s\tremaining: 1h 54m 52s\n",
      "221:\tlearn: 0.6975923\ttotal: 53m 13s\tremaining: 1h 54m 35s\n",
      "222:\tlearn: 0.6974856\ttotal: 53m 25s\tremaining: 1h 54m 17s\n",
      "223:\tlearn: 0.6972823\ttotal: 53m 39s\tremaining: 1h 54m 1s\n",
      "224:\tlearn: 0.6962873\ttotal: 53m 54s\tremaining: 1h 53m 48s\n",
      "225:\tlearn: 0.6960964\ttotal: 54m 8s\tremaining: 1h 53m 33s\n",
      "226:\tlearn: 0.6960334\ttotal: 54m 22s\tremaining: 1h 53m 17s\n",
      "227:\tlearn: 0.6959436\ttotal: 54m 36s\tremaining: 1h 53m 2s\n",
      "228:\tlearn: 0.6957578\ttotal: 54m 50s\tremaining: 1h 52m 47s\n",
      "229:\tlearn: 0.6956053\ttotal: 55m 4s\tremaining: 1h 52m 33s\n",
      "230:\tlearn: 0.6954513\ttotal: 55m 18s\tremaining: 1h 52m 17s\n",
      "231:\tlearn: 0.6953672\ttotal: 55m 32s\tremaining: 1h 52m 2s\n",
      "232:\tlearn: 0.6949258\ttotal: 55m 49s\tremaining: 1h 51m 52s\n",
      "233:\tlearn: 0.6948042\ttotal: 56m 4s\tremaining: 1h 51m 40s\n",
      "234:\tlearn: 0.6946941\ttotal: 56m 18s\tremaining: 1h 51m 24s\n",
      "235:\tlearn: 0.6944822\ttotal: 56m 32s\tremaining: 1h 51m 10s\n",
      "236:\tlearn: 0.6943751\ttotal: 56m 46s\tremaining: 1h 50m 54s\n",
      "237:\tlearn: 0.6943291\ttotal: 56m 59s\tremaining: 1h 50m 38s\n",
      "238:\tlearn: 0.6941708\ttotal: 57m 14s\tremaining: 1h 50m 23s\n",
      "239:\tlearn: 0.6940869\ttotal: 57m 27s\tremaining: 1h 50m 7s\n",
      "240:\tlearn: 0.6939204\ttotal: 57m 41s\tremaining: 1h 49m 53s\n",
      "241:\tlearn: 0.6937692\ttotal: 57m 55s\tremaining: 1h 49m 37s\n",
      "242:\tlearn: 0.6937033\ttotal: 58m 8s\tremaining: 1h 49m 21s\n",
      "243:\tlearn: 0.6936569\ttotal: 58m 22s\tremaining: 1h 49m 5s\n",
      "244:\tlearn: 0.6935559\ttotal: 58m 36s\tremaining: 1h 48m 49s\n",
      "245:\tlearn: 0.6934012\ttotal: 58m 50s\tremaining: 1h 48m 36s\n",
      "246:\tlearn: 0.6933294\ttotal: 59m 3s\tremaining: 1h 48m 19s\n",
      "247:\tlearn: 0.6930824\ttotal: 59m 16s\tremaining: 1h 48m 2s\n",
      "248:\tlearn: 0.6923575\ttotal: 59m 29s\tremaining: 1h 47m 45s\n",
      "249:\tlearn: 0.6923168\ttotal: 59m 41s\tremaining: 1h 47m 27s\n",
      "250:\tlearn: 0.6922117\ttotal: 59m 54s\tremaining: 1h 47m 10s\n",
      "251:\tlearn: 0.6921460\ttotal: 1h 5s\tremaining: 1h 46m 50s\n",
      "252:\tlearn: 0.6914526\ttotal: 1h 18s\tremaining: 1h 46m 32s\n",
      "253:\tlearn: 0.6913760\ttotal: 1h 30s\tremaining: 1h 46m 13s\n",
      "254:\tlearn: 0.6912770\ttotal: 1h 41s\tremaining: 1h 45m 54s\n",
      "255:\tlearn: 0.6912015\ttotal: 1h 53s\tremaining: 1h 45m 36s\n",
      "256:\tlearn: 0.6910759\ttotal: 1h 1m 7s\tremaining: 1h 45m 21s\n",
      "257:\tlearn: 0.6909543\ttotal: 1h 1m 21s\tremaining: 1h 45m 7s\n",
      "258:\tlearn: 0.6908262\ttotal: 1h 1m 38s\tremaining: 1h 44m 56s\n",
      "259:\tlearn: 0.6906627\ttotal: 1h 1m 54s\tremaining: 1h 44m 46s\n",
      "260:\tlearn: 0.6904831\ttotal: 1h 2m 8s\tremaining: 1h 44m 31s\n",
      "261:\tlearn: 0.6903748\ttotal: 1h 2m 22s\tremaining: 1h 44m 16s\n",
      "262:\tlearn: 0.6902727\ttotal: 1h 2m 35s\tremaining: 1h 44m\n",
      "263:\tlearn: 0.6888444\ttotal: 1h 2m 51s\tremaining: 1h 43m 47s\n",
      "264:\tlearn: 0.6886871\ttotal: 1h 3m 3s\tremaining: 1h 43m 30s\n",
      "265:\tlearn: 0.6885642\ttotal: 1h 3m 16s\tremaining: 1h 43m 13s\n",
      "266:\tlearn: 0.6883717\ttotal: 1h 3m 28s\tremaining: 1h 42m 57s\n",
      "267:\tlearn: 0.6883170\ttotal: 1h 3m 41s\tremaining: 1h 42m 39s\n",
      "268:\tlearn: 0.6882352\ttotal: 1h 3m 53s\tremaining: 1h 42m 22s\n",
      "269:\tlearn: 0.6881751\ttotal: 1h 4m 6s\tremaining: 1h 42m 5s\n",
      "270:\tlearn: 0.6880502\ttotal: 1h 4m 18s\tremaining: 1h 41m 47s\n",
      "271:\tlearn: 0.6879802\ttotal: 1h 4m 30s\tremaining: 1h 41m 30s\n",
      "272:\tlearn: 0.6878752\ttotal: 1h 4m 42s\tremaining: 1h 41m 12s\n",
      "273:\tlearn: 0.6878224\ttotal: 1h 4m 54s\tremaining: 1h 40m 55s\n",
      "274:\tlearn: 0.6877367\ttotal: 1h 5m 7s\tremaining: 1h 40m 38s\n",
      "275:\tlearn: 0.6871505\ttotal: 1h 5m 19s\tremaining: 1h 40m 21s\n",
      "276:\tlearn: 0.6870783\ttotal: 1h 5m 32s\tremaining: 1h 40m 4s\n",
      "277:\tlearn: 0.6869428\ttotal: 1h 5m 44s\tremaining: 1h 39m 46s\n",
      "278:\tlearn: 0.6865217\ttotal: 1h 5m 57s\tremaining: 1h 39m 31s\n",
      "279:\tlearn: 0.6864967\ttotal: 1h 6m 10s\tremaining: 1h 39m 15s\n",
      "280:\tlearn: 0.6859120\ttotal: 1h 6m 23s\tremaining: 1h 38m 59s\n",
      "281:\tlearn: 0.6858523\ttotal: 1h 6m 35s\tremaining: 1h 38m 42s\n",
      "282:\tlearn: 0.6857159\ttotal: 1h 6m 49s\tremaining: 1h 38m 27s\n",
      "283:\tlearn: 0.6850526\ttotal: 1h 7m 3s\tremaining: 1h 38m 13s\n",
      "284:\tlearn: 0.6847710\ttotal: 1h 7m 16s\tremaining: 1h 37m 58s\n",
      "285:\tlearn: 0.6846617\ttotal: 1h 7m 30s\tremaining: 1h 37m 43s\n",
      "286:\tlearn: 0.6845974\ttotal: 1h 7m 43s\tremaining: 1h 37m 27s\n",
      "287:\tlearn: 0.6842700\ttotal: 1h 7m 56s\tremaining: 1h 37m 11s\n",
      "288:\tlearn: 0.6841958\ttotal: 1h 8m 8s\tremaining: 1h 36m 54s\n",
      "289:\tlearn: 0.6841433\ttotal: 1h 8m 20s\tremaining: 1h 36m 37s\n",
      "290:\tlearn: 0.6840494\ttotal: 1h 8m 34s\tremaining: 1h 36m 22s\n",
      "291:\tlearn: 0.6836992\ttotal: 1h 8m 47s\tremaining: 1h 36m 7s\n",
      "292:\tlearn: 0.6836217\ttotal: 1h 9m\tremaining: 1h 35m 50s\n",
      "293:\tlearn: 0.6835411\ttotal: 1h 9m 12s\tremaining: 1h 35m 33s\n",
      "294:\tlearn: 0.6833955\ttotal: 1h 9m 25s\tremaining: 1h 35m 19s\n",
      "295:\tlearn: 0.6832930\ttotal: 1h 9m 40s\tremaining: 1h 35m 5s\n",
      "296:\tlearn: 0.6831770\ttotal: 1h 9m 54s\tremaining: 1h 34m 51s\n",
      "297:\tlearn: 0.6830936\ttotal: 1h 10m 10s\tremaining: 1h 34m 40s\n",
      "298:\tlearn: 0.6829954\ttotal: 1h 10m 23s\tremaining: 1h 34m 23s\n",
      "299:\tlearn: 0.6829187\ttotal: 1h 10m 36s\tremaining: 1h 34m 8s\n",
      "300:\tlearn: 0.6827416\ttotal: 1h 10m 48s\tremaining: 1h 33m 51s\n",
      "301:\tlearn: 0.6826849\ttotal: 1h 11m\tremaining: 1h 33m 34s\n",
      "302:\tlearn: 0.6825936\ttotal: 1h 11m 15s\tremaining: 1h 33m 22s\n",
      "303:\tlearn: 0.6825198\ttotal: 1h 11m 29s\tremaining: 1h 33m 7s\n",
      "304:\tlearn: 0.6824554\ttotal: 1h 11m 42s\tremaining: 1h 32m 51s\n",
      "305:\tlearn: 0.6821815\ttotal: 1h 11m 55s\tremaining: 1h 32m 37s\n",
      "306:\tlearn: 0.6820720\ttotal: 1h 12m 9s\tremaining: 1h 32m 22s\n",
      "307:\tlearn: 0.6819952\ttotal: 1h 12m 21s\tremaining: 1h 32m 5s\n",
      "308:\tlearn: 0.6819659\ttotal: 1h 12m 34s\tremaining: 1h 31m 49s\n",
      "309:\tlearn: 0.6818617\ttotal: 1h 12m 47s\tremaining: 1h 31m 34s\n",
      "310:\tlearn: 0.6817802\ttotal: 1h 13m\tremaining: 1h 31m 18s\n",
      "311:\tlearn: 0.6817080\ttotal: 1h 13m 11s\tremaining: 1h 31m 1s\n",
      "312:\tlearn: 0.6816003\ttotal: 1h 13m 24s\tremaining: 1h 30m 46s\n",
      "313:\tlearn: 0.6815500\ttotal: 1h 13m 36s\tremaining: 1h 30m 28s\n",
      "314:\tlearn: 0.6814546\ttotal: 1h 13m 48s\tremaining: 1h 30m 12s\n",
      "315:\tlearn: 0.6813497\ttotal: 1h 14m 1s\tremaining: 1h 29m 56s\n",
      "316:\tlearn: 0.6812780\ttotal: 1h 14m 13s\tremaining: 1h 29m 40s\n",
      "317:\tlearn: 0.6811481\ttotal: 1h 14m 28s\tremaining: 1h 29m 27s\n",
      "318:\tlearn: 0.6809637\ttotal: 1h 14m 42s\tremaining: 1h 29m 13s\n",
      "319:\tlearn: 0.6808655\ttotal: 1h 14m 56s\tremaining: 1h 28m 59s\n",
      "320:\tlearn: 0.6807306\ttotal: 1h 15m 8s\tremaining: 1h 28m 42s\n",
      "321:\tlearn: 0.6805097\ttotal: 1h 15m 20s\tremaining: 1h 28m 27s\n",
      "322:\tlearn: 0.6804805\ttotal: 1h 15m 32s\tremaining: 1h 28m 10s\n",
      "323:\tlearn: 0.6803836\ttotal: 1h 15m 43s\tremaining: 1h 27m 53s\n",
      "324:\tlearn: 0.6801822\ttotal: 1h 15m 55s\tremaining: 1h 27m 36s\n",
      "325:\tlearn: 0.6801000\ttotal: 1h 16m 7s\tremaining: 1h 27m 19s\n",
      "326:\tlearn: 0.6799842\ttotal: 1h 16m 18s\tremaining: 1h 27m 2s\n",
      "327:\tlearn: 0.6798513\ttotal: 1h 16m 29s\tremaining: 1h 26m 45s\n",
      "328:\tlearn: 0.6796000\ttotal: 1h 16m 41s\tremaining: 1h 26m 28s\n",
      "329:\tlearn: 0.6794586\ttotal: 1h 16m 53s\tremaining: 1h 26m 13s\n",
      "330:\tlearn: 0.6793478\ttotal: 1h 17m 5s\tremaining: 1h 25m 56s\n",
      "331:\tlearn: 0.6793097\ttotal: 1h 17m 15s\tremaining: 1h 25m 38s\n",
      "332:\tlearn: 0.6789589\ttotal: 1h 17m 27s\tremaining: 1h 25m 22s\n",
      "333:\tlearn: 0.6788605\ttotal: 1h 17m 39s\tremaining: 1h 25m 5s\n",
      "334:\tlearn: 0.6787509\ttotal: 1h 17m 50s\tremaining: 1h 24m 49s\n",
      "335:\tlearn: 0.6786739\ttotal: 1h 18m 2s\tremaining: 1h 24m 32s\n",
      "336:\tlearn: 0.6786042\ttotal: 1h 18m 13s\tremaining: 1h 24m 15s\n",
      "337:\tlearn: 0.6785810\ttotal: 1h 18m 24s\tremaining: 1h 23m 58s\n",
      "338:\tlearn: 0.6785366\ttotal: 1h 18m 36s\tremaining: 1h 23m 42s\n",
      "339:\tlearn: 0.6784848\ttotal: 1h 18m 47s\tremaining: 1h 23m 26s\n",
      "340:\tlearn: 0.6784217\ttotal: 1h 18m 59s\tremaining: 1h 23m 9s\n",
      "341:\tlearn: 0.6783314\ttotal: 1h 19m 10s\tremaining: 1h 22m 52s\n",
      "342:\tlearn: 0.6782700\ttotal: 1h 19m 21s\tremaining: 1h 22m 36s\n",
      "343:\tlearn: 0.6782030\ttotal: 1h 19m 33s\tremaining: 1h 22m 19s\n",
      "344:\tlearn: 0.6781523\ttotal: 1h 19m 44s\tremaining: 1h 22m 3s\n",
      "345:\tlearn: 0.6773801\ttotal: 1h 19m 56s\tremaining: 1h 21m 47s\n",
      "346:\tlearn: 0.6773009\ttotal: 1h 20m 7s\tremaining: 1h 21m 30s\n",
      "347:\tlearn: 0.6772388\ttotal: 1h 20m 18s\tremaining: 1h 21m 14s\n",
      "348:\tlearn: 0.6771533\ttotal: 1h 20m 30s\tremaining: 1h 20m 58s\n",
      "349:\tlearn: 0.6769680\ttotal: 1h 20m 42s\tremaining: 1h 20m 42s\n",
      "350:\tlearn: 0.6768221\ttotal: 1h 20m 55s\tremaining: 1h 20m 27s\n",
      "351:\tlearn: 0.6767517\ttotal: 1h 21m 6s\tremaining: 1h 20m 11s\n",
      "352:\tlearn: 0.6764943\ttotal: 1h 21m 18s\tremaining: 1h 19m 55s\n",
      "353:\tlearn: 0.6764179\ttotal: 1h 21m 30s\tremaining: 1h 19m 39s\n",
      "354:\tlearn: 0.6763549\ttotal: 1h 21m 41s\tremaining: 1h 19m 23s\n",
      "355:\tlearn: 0.6756500\ttotal: 1h 21m 52s\tremaining: 1h 19m 7s\n",
      "356:\tlearn: 0.6751425\ttotal: 1h 22m 4s\tremaining: 1h 18m 51s\n",
      "357:\tlearn: 0.6750876\ttotal: 1h 22m 15s\tremaining: 1h 18m 35s\n",
      "358:\tlearn: 0.6749964\ttotal: 1h 22m 27s\tremaining: 1h 18m 19s\n",
      "359:\tlearn: 0.6741106\ttotal: 1h 22m 39s\tremaining: 1h 18m 4s\n",
      "360:\tlearn: 0.6740404\ttotal: 1h 22m 50s\tremaining: 1h 17m 48s\n",
      "361:\tlearn: 0.6739649\ttotal: 1h 23m 2s\tremaining: 1h 17m 32s\n",
      "362:\tlearn: 0.6738920\ttotal: 1h 23m 13s\tremaining: 1h 17m 15s\n",
      "363:\tlearn: 0.6738057\ttotal: 1h 23m 25s\tremaining: 1h 17m\n",
      "364:\tlearn: 0.6736688\ttotal: 1h 23m 37s\tremaining: 1h 16m 45s\n",
      "365:\tlearn: 0.6735856\ttotal: 1h 23m 50s\tremaining: 1h 16m 30s\n",
      "366:\tlearn: 0.6729451\ttotal: 1h 24m 2s\tremaining: 1h 16m 14s\n",
      "367:\tlearn: 0.6728598\ttotal: 1h 24m 13s\tremaining: 1h 15m 59s\n",
      "368:\tlearn: 0.6722480\ttotal: 1h 24m 25s\tremaining: 1h 15m 44s\n",
      "369:\tlearn: 0.6721692\ttotal: 1h 24m 37s\tremaining: 1h 15m 28s\n",
      "370:\tlearn: 0.6715273\ttotal: 1h 24m 48s\tremaining: 1h 15m 12s\n",
      "371:\tlearn: 0.6714724\ttotal: 1h 24m 59s\tremaining: 1h 14m 56s\n",
      "372:\tlearn: 0.6711955\ttotal: 1h 25m 11s\tremaining: 1h 14m 41s\n",
      "373:\tlearn: 0.6711074\ttotal: 1h 25m 22s\tremaining: 1h 14m 25s\n",
      "374:\tlearn: 0.6710492\ttotal: 1h 25m 34s\tremaining: 1h 14m 9s\n",
      "375:\tlearn: 0.6709702\ttotal: 1h 25m 45s\tremaining: 1h 13m 53s\n",
      "376:\tlearn: 0.6708855\ttotal: 1h 25m 56s\tremaining: 1h 13m 38s\n",
      "377:\tlearn: 0.6707989\ttotal: 1h 26m 8s\tremaining: 1h 13m 23s\n",
      "378:\tlearn: 0.6706701\ttotal: 1h 26m 21s\tremaining: 1h 13m 8s\n",
      "379:\tlearn: 0.6703003\ttotal: 1h 26m 34s\tremaining: 1h 12m 54s\n",
      "380:\tlearn: 0.6702097\ttotal: 1h 26m 46s\tremaining: 1h 12m 39s\n",
      "381:\tlearn: 0.6698801\ttotal: 1h 26m 58s\tremaining: 1h 12m 24s\n",
      "382:\tlearn: 0.6698213\ttotal: 1h 27m 10s\tremaining: 1h 12m 9s\n",
      "383:\tlearn: 0.6696857\ttotal: 1h 27m 22s\tremaining: 1h 11m 53s\n",
      "384:\tlearn: 0.6695895\ttotal: 1h 27m 34s\tremaining: 1h 11m 39s\n",
      "385:\tlearn: 0.6695317\ttotal: 1h 27m 48s\tremaining: 1h 11m 25s\n",
      "386:\tlearn: 0.6694735\ttotal: 1h 28m 1s\tremaining: 1h 11m 11s\n",
      "387:\tlearn: 0.6693372\ttotal: 1h 28m 14s\tremaining: 1h 10m 57s\n",
      "388:\tlearn: 0.6693004\ttotal: 1h 28m 27s\tremaining: 1h 10m 43s\n",
      "389:\tlearn: 0.6692203\ttotal: 1h 28m 40s\tremaining: 1h 10m 29s\n",
      "390:\tlearn: 0.6689984\ttotal: 1h 28m 56s\tremaining: 1h 10m 17s\n",
      "391:\tlearn: 0.6688863\ttotal: 1h 29m 12s\tremaining: 1h 10m 5s\n",
      "392:\tlearn: 0.6687890\ttotal: 1h 29m 24s\tremaining: 1h 9m 50s\n",
      "393:\tlearn: 0.6687166\ttotal: 1h 29m 37s\tremaining: 1h 9m 36s\n",
      "394:\tlearn: 0.6686687\ttotal: 1h 29m 49s\tremaining: 1h 9m 21s\n",
      "395:\tlearn: 0.6685919\ttotal: 1h 30m 2s\tremaining: 1h 9m 7s\n",
      "396:\tlearn: 0.6685066\ttotal: 1h 30m 16s\tremaining: 1h 8m 53s\n",
      "397:\tlearn: 0.6682918\ttotal: 1h 30m 28s\tremaining: 1h 8m 39s\n",
      "398:\tlearn: 0.6681041\ttotal: 1h 30m 43s\tremaining: 1h 8m 26s\n",
      "399:\tlearn: 0.6680763\ttotal: 1h 30m 56s\tremaining: 1h 8m 12s\n",
      "400:\tlearn: 0.6680431\ttotal: 1h 31m 8s\tremaining: 1h 7m 57s\n",
      "401:\tlearn: 0.6679721\ttotal: 1h 31m 21s\tremaining: 1h 7m 43s\n",
      "402:\tlearn: 0.6679213\ttotal: 1h 31m 34s\tremaining: 1h 7m 29s\n",
      "403:\tlearn: 0.6678357\ttotal: 1h 31m 47s\tremaining: 1h 7m 15s\n",
      "404:\tlearn: 0.6677343\ttotal: 1h 32m 6s\tremaining: 1h 7m 5s\n",
      "405:\tlearn: 0.6676778\ttotal: 1h 32m 23s\tremaining: 1h 6m 54s\n",
      "406:\tlearn: 0.6675240\ttotal: 1h 32m 40s\tremaining: 1h 6m 43s\n",
      "407:\tlearn: 0.6674430\ttotal: 1h 32m 55s\tremaining: 1h 6m 30s\n",
      "408:\tlearn: 0.6673404\ttotal: 1h 33m 10s\tremaining: 1h 6m 17s\n",
      "409:\tlearn: 0.6671432\ttotal: 1h 33m 24s\tremaining: 1h 6m 4s\n",
      "410:\tlearn: 0.6670891\ttotal: 1h 33m 37s\tremaining: 1h 5m 49s\n",
      "411:\tlearn: 0.6670004\ttotal: 1h 33m 53s\tremaining: 1h 5m 37s\n",
      "412:\tlearn: 0.6669187\ttotal: 1h 34m 5s\tremaining: 1h 5m 23s\n",
      "413:\tlearn: 0.6668444\ttotal: 1h 34m 17s\tremaining: 1h 5m 8s\n",
      "414:\tlearn: 0.6668011\ttotal: 1h 34m 33s\tremaining: 1h 4m 56s\n",
      "415:\tlearn: 0.6666912\ttotal: 1h 34m 45s\tremaining: 1h 4m 41s\n",
      "416:\tlearn: 0.6666193\ttotal: 1h 34m 58s\tremaining: 1h 4m 27s\n",
      "417:\tlearn: 0.6665537\ttotal: 1h 35m 11s\tremaining: 1h 4m 12s\n",
      "418:\tlearn: 0.6664420\ttotal: 1h 35m 24s\tremaining: 1h 3m 58s\n",
      "419:\tlearn: 0.6663874\ttotal: 1h 35m 35s\tremaining: 1h 3m 43s\n",
      "420:\tlearn: 0.6663345\ttotal: 1h 35m 49s\tremaining: 1h 3m 30s\n",
      "421:\tlearn: 0.6662400\ttotal: 1h 36m 2s\tremaining: 1h 3m 16s\n",
      "422:\tlearn: 0.6661740\ttotal: 1h 36m 15s\tremaining: 1h 3m 1s\n",
      "423:\tlearn: 0.6661353\ttotal: 1h 36m 29s\tremaining: 1h 2m 48s\n",
      "424:\tlearn: 0.6660333\ttotal: 1h 36m 43s\tremaining: 1h 2m 34s\n",
      "425:\tlearn: 0.6658455\ttotal: 1h 36m 55s\tremaining: 1h 2m 20s\n",
      "426:\tlearn: 0.6657729\ttotal: 1h 37m 9s\tremaining: 1h 2m 6s\n",
      "427:\tlearn: 0.6657080\ttotal: 1h 37m 22s\tremaining: 1h 1m 52s\n",
      "428:\tlearn: 0.6654796\ttotal: 1h 37m 36s\tremaining: 1h 1m 39s\n",
      "429:\tlearn: 0.6654068\ttotal: 1h 37m 49s\tremaining: 1h 1m 25s\n",
      "430:\tlearn: 0.6653110\ttotal: 1h 38m 4s\tremaining: 1h 1m 12s\n",
      "431:\tlearn: 0.6652785\ttotal: 1h 38m 18s\tremaining: 1h 59s\n",
      "432:\tlearn: 0.6651098\ttotal: 1h 38m 33s\tremaining: 1h 46s\n",
      "433:\tlearn: 0.6650294\ttotal: 1h 38m 47s\tremaining: 1h 33s\n",
      "434:\tlearn: 0.6649457\ttotal: 1h 39m 1s\tremaining: 1h 19s\n",
      "435:\tlearn: 0.6648908\ttotal: 1h 39m 16s\tremaining: 1h 6s\n",
      "436:\tlearn: 0.6648225\ttotal: 1h 39m 30s\tremaining: 59m 53s\n",
      "437:\tlearn: 0.6647375\ttotal: 1h 39m 43s\tremaining: 59m 39s\n",
      "438:\tlearn: 0.6646665\ttotal: 1h 39m 56s\tremaining: 59m 25s\n",
      "439:\tlearn: 0.6645546\ttotal: 1h 40m 9s\tremaining: 59m 11s\n",
      "440:\tlearn: 0.6641728\ttotal: 1h 40m 22s\tremaining: 58m 56s\n",
      "441:\tlearn: 0.6640991\ttotal: 1h 40m 35s\tremaining: 58m 42s\n",
      "442:\tlearn: 0.6640084\ttotal: 1h 40m 49s\tremaining: 58m 29s\n",
      "443:\tlearn: 0.6638997\ttotal: 1h 41m 3s\tremaining: 58m 15s\n",
      "444:\tlearn: 0.6638227\ttotal: 1h 41m 18s\tremaining: 58m 3s\n",
      "445:\tlearn: 0.6636277\ttotal: 1h 41m 30s\tremaining: 57m 48s\n",
      "446:\tlearn: 0.6635824\ttotal: 1h 41m 42s\tremaining: 57m 34s\n",
      "447:\tlearn: 0.6634861\ttotal: 1h 41m 55s\tremaining: 57m 19s\n",
      "448:\tlearn: 0.6634084\ttotal: 1h 42m 6s\tremaining: 57m 5s\n",
      "449:\tlearn: 0.6632997\ttotal: 1h 42m 19s\tremaining: 56m 50s\n",
      "450:\tlearn: 0.6632086\ttotal: 1h 42m 30s\tremaining: 56m 35s\n",
      "451:\tlearn: 0.6631565\ttotal: 1h 42m 42s\tremaining: 56m 21s\n",
      "452:\tlearn: 0.6629370\ttotal: 1h 42m 53s\tremaining: 56m 6s\n",
      "453:\tlearn: 0.6628561\ttotal: 1h 43m 5s\tremaining: 55m 51s\n",
      "454:\tlearn: 0.6627907\ttotal: 1h 43m 17s\tremaining: 55m 37s\n",
      "455:\tlearn: 0.6627302\ttotal: 1h 43m 30s\tremaining: 55m 22s\n",
      "456:\tlearn: 0.6626669\ttotal: 1h 43m 44s\tremaining: 55m 9s\n",
      "457:\tlearn: 0.6626084\ttotal: 1h 43m 58s\tremaining: 54m 56s\n",
      "458:\tlearn: 0.6624820\ttotal: 1h 44m 13s\tremaining: 54m 43s\n",
      "459:\tlearn: 0.6624468\ttotal: 1h 44m 27s\tremaining: 54m 30s\n",
      "460:\tlearn: 0.6624063\ttotal: 1h 44m 41s\tremaining: 54m 16s\n",
      "461:\tlearn: 0.6622781\ttotal: 1h 44m 56s\tremaining: 54m 3s\n",
      "462:\tlearn: 0.6622289\ttotal: 1h 45m 10s\tremaining: 53m 50s\n",
      "463:\tlearn: 0.6621650\ttotal: 1h 45m 24s\tremaining: 53m 36s\n",
      "464:\tlearn: 0.6620787\ttotal: 1h 45m 38s\tremaining: 53m 23s\n",
      "465:\tlearn: 0.6619823\ttotal: 1h 45m 52s\tremaining: 53m 9s\n",
      "466:\tlearn: 0.6619244\ttotal: 1h 46m 5s\tremaining: 52m 55s\n",
      "467:\tlearn: 0.6618310\ttotal: 1h 46m 19s\tremaining: 52m 42s\n",
      "468:\tlearn: 0.6617769\ttotal: 1h 46m 33s\tremaining: 52m 29s\n",
      "469:\tlearn: 0.6617043\ttotal: 1h 46m 47s\tremaining: 52m 15s\n",
      "470:\tlearn: 0.6616107\ttotal: 1h 47m 2s\tremaining: 52m 2s\n",
      "471:\tlearn: 0.6615700\ttotal: 1h 47m 16s\tremaining: 51m 48s\n",
      "472:\tlearn: 0.6615145\ttotal: 1h 47m 30s\tremaining: 51m 35s\n",
      "473:\tlearn: 0.6614471\ttotal: 1h 47m 44s\tremaining: 51m 22s\n",
      "474:\tlearn: 0.6613191\ttotal: 1h 47m 58s\tremaining: 51m 8s\n",
      "475:\tlearn: 0.6612453\ttotal: 1h 48m 13s\tremaining: 50m 55s\n",
      "476:\tlearn: 0.6610900\ttotal: 1h 48m 27s\tremaining: 50m 42s\n",
      "477:\tlearn: 0.6609951\ttotal: 1h 48m 42s\tremaining: 50m 29s\n",
      "478:\tlearn: 0.6609479\ttotal: 1h 48m 56s\tremaining: 50m 15s\n",
      "479:\tlearn: 0.6608532\ttotal: 1h 49m 11s\tremaining: 50m 2s\n",
      "480:\tlearn: 0.6607744\ttotal: 1h 49m 24s\tremaining: 49m 48s\n",
      "481:\tlearn: 0.6607065\ttotal: 1h 49m 38s\tremaining: 49m 35s\n",
      "482:\tlearn: 0.6606513\ttotal: 1h 49m 51s\tremaining: 49m 21s\n",
      "483:\tlearn: 0.6604556\ttotal: 1h 50m 5s\tremaining: 49m 7s\n",
      "484:\tlearn: 0.6604097\ttotal: 1h 50m 19s\tremaining: 48m 54s\n",
      "485:\tlearn: 0.6603583\ttotal: 1h 50m 32s\tremaining: 48m 40s\n",
      "486:\tlearn: 0.6603340\ttotal: 1h 50m 46s\tremaining: 48m 26s\n",
      "487:\tlearn: 0.6603103\ttotal: 1h 50m 59s\tremaining: 48m 13s\n",
      "488:\tlearn: 0.6601695\ttotal: 1h 51m 13s\tremaining: 47m 59s\n",
      "489:\tlearn: 0.6601108\ttotal: 1h 51m 28s\tremaining: 47m 46s\n",
      "490:\tlearn: 0.6600861\ttotal: 1h 51m 44s\tremaining: 47m 33s\n",
      "491:\tlearn: 0.6600143\ttotal: 1h 51m 59s\tremaining: 47m 20s\n",
      "492:\tlearn: 0.6599004\ttotal: 1h 52m 14s\tremaining: 47m 7s\n",
      "493:\tlearn: 0.6598648\ttotal: 1h 52m 27s\tremaining: 46m 53s\n",
      "494:\tlearn: 0.6598020\ttotal: 1h 52m 41s\tremaining: 46m 40s\n",
      "495:\tlearn: 0.6596778\ttotal: 1h 52m 55s\tremaining: 46m 26s\n",
      "496:\tlearn: 0.6596425\ttotal: 1h 53m 8s\tremaining: 46m 12s\n",
      "497:\tlearn: 0.6595729\ttotal: 1h 53m 22s\tremaining: 45m 59s\n",
      "498:\tlearn: 0.6595239\ttotal: 1h 53m 35s\tremaining: 45m 45s\n",
      "499:\tlearn: 0.6594413\ttotal: 1h 53m 47s\tremaining: 45m 31s\n",
      "500:\tlearn: 0.6593880\ttotal: 1h 53m 59s\tremaining: 45m 16s\n",
      "501:\tlearn: 0.6593033\ttotal: 1h 54m 12s\tremaining: 45m 2s\n",
      "502:\tlearn: 0.6592296\ttotal: 1h 54m 24s\tremaining: 44m 48s\n",
      "503:\tlearn: 0.6591438\ttotal: 1h 54m 38s\tremaining: 44m 35s\n",
      "504:\tlearn: 0.6591163\ttotal: 1h 54m 52s\tremaining: 44m 21s\n",
      "505:\tlearn: 0.6590152\ttotal: 1h 55m 6s\tremaining: 44m 8s\n",
      "506:\tlearn: 0.6588897\ttotal: 1h 55m 21s\tremaining: 43m 54s\n",
      "507:\tlearn: 0.6587719\ttotal: 1h 55m 35s\tremaining: 43m 41s\n",
      "508:\tlearn: 0.6587108\ttotal: 1h 55m 48s\tremaining: 43m 27s\n",
      "509:\tlearn: 0.6582197\ttotal: 1h 56m 3s\tremaining: 43m 14s\n",
      "510:\tlearn: 0.6580611\ttotal: 1h 56m 17s\tremaining: 43m\n",
      "511:\tlearn: 0.6579713\ttotal: 1h 56m 30s\tremaining: 42m 46s\n",
      "512:\tlearn: 0.6578800\ttotal: 1h 56m 45s\tremaining: 42m 33s\n",
      "513:\tlearn: 0.6577311\ttotal: 1h 57m\tremaining: 42m 20s\n",
      "514:\tlearn: 0.6576626\ttotal: 1h 57m 15s\tremaining: 42m 7s\n",
      "515:\tlearn: 0.6576092\ttotal: 1h 57m 28s\tremaining: 41m 53s\n",
      "516:\tlearn: 0.6570790\ttotal: 1h 57m 41s\tremaining: 41m 39s\n",
      "517:\tlearn: 0.6570295\ttotal: 1h 57m 52s\tremaining: 41m 24s\n",
      "518:\tlearn: 0.6569774\ttotal: 1h 58m 4s\tremaining: 41m 10s\n",
      "519:\tlearn: 0.6569205\ttotal: 1h 58m 18s\tremaining: 40m 57s\n",
      "520:\tlearn: 0.6568684\ttotal: 1h 58m 31s\tremaining: 40m 43s\n",
      "521:\tlearn: 0.6560267\ttotal: 1h 58m 44s\tremaining: 40m 29s\n",
      "522:\tlearn: 0.6559625\ttotal: 1h 58m 56s\tremaining: 40m 15s\n",
      "523:\tlearn: 0.6559463\ttotal: 1h 59m 8s\tremaining: 40m\n",
      "524:\tlearn: 0.6558857\ttotal: 1h 59m 21s\tremaining: 39m 47s\n",
      "525:\tlearn: 0.6558198\ttotal: 1h 59m 35s\tremaining: 39m 33s\n",
      "526:\tlearn: 0.6557613\ttotal: 1h 59m 51s\tremaining: 39m 20s\n",
      "527:\tlearn: 0.6556763\ttotal: 2h 5s\tremaining: 39m 7s\n",
      "528:\tlearn: 0.6552751\ttotal: 2h 17s\tremaining: 38m 53s\n",
      "529:\tlearn: 0.6552029\ttotal: 2h 29s\tremaining: 38m 38s\n",
      "530:\tlearn: 0.6551174\ttotal: 2h 41s\tremaining: 38m 24s\n",
      "531:\tlearn: 0.6550630\ttotal: 2h 52s\tremaining: 38m 10s\n",
      "532:\tlearn: 0.6550223\ttotal: 2h 1m 4s\tremaining: 37m 56s\n",
      "533:\tlearn: 0.6548923\ttotal: 2h 1m 16s\tremaining: 37m 41s\n",
      "534:\tlearn: 0.6547837\ttotal: 2h 1m 28s\tremaining: 37m 27s\n",
      "535:\tlearn: 0.6546446\ttotal: 2h 1m 40s\tremaining: 37m 13s\n",
      "536:\tlearn: 0.6541746\ttotal: 2h 1m 51s\tremaining: 36m 59s\n",
      "537:\tlearn: 0.6540996\ttotal: 2h 2m 3s\tremaining: 36m 45s\n",
      "538:\tlearn: 0.6540338\ttotal: 2h 2m 15s\tremaining: 36m 31s\n",
      "539:\tlearn: 0.6539674\ttotal: 2h 2m 27s\tremaining: 36m 16s\n",
      "540:\tlearn: 0.6539255\ttotal: 2h 2m 39s\tremaining: 36m 2s\n",
      "541:\tlearn: 0.6538765\ttotal: 2h 2m 50s\tremaining: 35m 48s\n",
      "542:\tlearn: 0.6538225\ttotal: 2h 3m 1s\tremaining: 35m 34s\n",
      "543:\tlearn: 0.6537907\ttotal: 2h 3m 12s\tremaining: 35m 20s\n",
      "544:\tlearn: 0.6537013\ttotal: 2h 3m 23s\tremaining: 35m 5s\n",
      "545:\tlearn: 0.6536701\ttotal: 2h 3m 35s\tremaining: 34m 51s\n",
      "546:\tlearn: 0.6536119\ttotal: 2h 3m 46s\tremaining: 34m 37s\n",
      "547:\tlearn: 0.6535352\ttotal: 2h 3m 57s\tremaining: 34m 23s\n",
      "548:\tlearn: 0.6534755\ttotal: 2h 4m 8s\tremaining: 34m 8s\n",
      "549:\tlearn: 0.6533660\ttotal: 2h 4m 20s\tremaining: 33m 54s\n",
      "550:\tlearn: 0.6532612\ttotal: 2h 4m 32s\tremaining: 33m 40s\n",
      "551:\tlearn: 0.6531666\ttotal: 2h 4m 43s\tremaining: 33m 26s\n",
      "552:\tlearn: 0.6531260\ttotal: 2h 4m 54s\tremaining: 33m 12s\n",
      "553:\tlearn: 0.6530931\ttotal: 2h 5m 6s\tremaining: 32m 58s\n",
      "554:\tlearn: 0.6530258\ttotal: 2h 5m 17s\tremaining: 32m 44s\n",
      "555:\tlearn: 0.6529766\ttotal: 2h 5m 29s\tremaining: 32m 29s\n",
      "556:\tlearn: 0.6527954\ttotal: 2h 5m 41s\tremaining: 32m 16s\n",
      "557:\tlearn: 0.6527074\ttotal: 2h 5m 53s\tremaining: 32m 2s\n",
      "558:\tlearn: 0.6526010\ttotal: 2h 6m 5s\tremaining: 31m 48s\n",
      "559:\tlearn: 0.6525303\ttotal: 2h 6m 16s\tremaining: 31m 34s\n",
      "560:\tlearn: 0.6524448\ttotal: 2h 6m 27s\tremaining: 31m 20s\n",
      "561:\tlearn: 0.6524160\ttotal: 2h 6m 39s\tremaining: 31m 6s\n",
      "562:\tlearn: 0.6523243\ttotal: 2h 6m 50s\tremaining: 30m 51s\n",
      "563:\tlearn: 0.6522507\ttotal: 2h 7m 2s\tremaining: 30m 37s\n",
      "564:\tlearn: 0.6522167\ttotal: 2h 7m 13s\tremaining: 30m 23s\n",
      "565:\tlearn: 0.6521870\ttotal: 2h 7m 24s\tremaining: 30m 9s\n",
      "566:\tlearn: 0.6521188\ttotal: 2h 7m 36s\tremaining: 29m 55s\n",
      "567:\tlearn: 0.6520516\ttotal: 2h 7m 48s\tremaining: 29m 42s\n",
      "568:\tlearn: 0.6519769\ttotal: 2h 7m 59s\tremaining: 29m 28s\n",
      "569:\tlearn: 0.6518918\ttotal: 2h 8m 10s\tremaining: 29m 14s\n",
      "570:\tlearn: 0.6518277\ttotal: 2h 8m 22s\tremaining: 29m\n",
      "571:\tlearn: 0.6517726\ttotal: 2h 8m 33s\tremaining: 28m 46s\n",
      "572:\tlearn: 0.6516864\ttotal: 2h 8m 45s\tremaining: 28m 32s\n",
      "573:\tlearn: 0.6516268\ttotal: 2h 8m 56s\tremaining: 28m 18s\n",
      "574:\tlearn: 0.6515631\ttotal: 2h 9m 8s\tremaining: 28m 4s\n",
      "575:\tlearn: 0.6512806\ttotal: 2h 9m 22s\tremaining: 27m 51s\n",
      "576:\tlearn: 0.6512217\ttotal: 2h 9m 34s\tremaining: 27m 37s\n",
      "577:\tlearn: 0.6511150\ttotal: 2h 9m 46s\tremaining: 27m 23s\n",
      "578:\tlearn: 0.6510236\ttotal: 2h 9m 57s\tremaining: 27m 9s\n",
      "579:\tlearn: 0.6509349\ttotal: 2h 10m 9s\tremaining: 26m 55s\n",
      "580:\tlearn: 0.6508682\ttotal: 2h 10m 21s\tremaining: 26m 41s\n",
      "581:\tlearn: 0.6508452\ttotal: 2h 10m 32s\tremaining: 26m 28s\n",
      "582:\tlearn: 0.6508164\ttotal: 2h 10m 43s\tremaining: 26m 14s\n",
      "583:\tlearn: 0.6507217\ttotal: 2h 10m 56s\tremaining: 26m\n",
      "584:\tlearn: 0.6506591\ttotal: 2h 11m 7s\tremaining: 25m 46s\n",
      "585:\tlearn: 0.6506242\ttotal: 2h 11m 19s\tremaining: 25m 32s\n",
      "586:\tlearn: 0.6505710\ttotal: 2h 11m 30s\tremaining: 25m 19s\n",
      "587:\tlearn: 0.6503678\ttotal: 2h 11m 42s\tremaining: 25m 5s\n",
      "588:\tlearn: 0.6503216\ttotal: 2h 11m 53s\tremaining: 24m 51s\n",
      "589:\tlearn: 0.6502913\ttotal: 2h 12m 5s\tremaining: 24m 37s\n",
      "590:\tlearn: 0.6502409\ttotal: 2h 12m 16s\tremaining: 24m 23s\n",
      "591:\tlearn: 0.6501633\ttotal: 2h 12m 28s\tremaining: 24m 10s\n",
      "592:\tlearn: 0.6501382\ttotal: 2h 12m 39s\tremaining: 23m 56s\n",
      "593:\tlearn: 0.6500614\ttotal: 2h 12m 51s\tremaining: 23m 42s\n",
      "594:\tlearn: 0.6497350\ttotal: 2h 13m 2s\tremaining: 23m 28s\n",
      "595:\tlearn: 0.6496849\ttotal: 2h 13m 14s\tremaining: 23m 14s\n",
      "596:\tlearn: 0.6496671\ttotal: 2h 13m 25s\tremaining: 23m 1s\n",
      "597:\tlearn: 0.6496358\ttotal: 2h 13m 36s\tremaining: 22m 47s\n",
      "598:\tlearn: 0.6495534\ttotal: 2h 13m 47s\tremaining: 22m 33s\n",
      "599:\tlearn: 0.6495041\ttotal: 2h 13m 59s\tremaining: 22m 19s\n",
      "600:\tlearn: 0.6494537\ttotal: 2h 14m 10s\tremaining: 22m 6s\n",
      "601:\tlearn: 0.6493729\ttotal: 2h 14m 22s\tremaining: 21m 52s\n",
      "602:\tlearn: 0.6493407\ttotal: 2h 14m 33s\tremaining: 21m 38s\n",
      "603:\tlearn: 0.6493008\ttotal: 2h 14m 45s\tremaining: 21m 25s\n",
      "604:\tlearn: 0.6492502\ttotal: 2h 14m 57s\tremaining: 21m 11s\n",
      "605:\tlearn: 0.6492006\ttotal: 2h 15m 9s\tremaining: 20m 57s\n",
      "606:\tlearn: 0.6491455\ttotal: 2h 15m 20s\tremaining: 20m 44s\n",
      "607:\tlearn: 0.6487340\ttotal: 2h 15m 31s\tremaining: 20m 30s\n",
      "608:\tlearn: 0.6487192\ttotal: 2h 15m 43s\tremaining: 20m 16s\n",
      "609:\tlearn: 0.6486198\ttotal: 2h 15m 57s\tremaining: 20m 3s\n",
      "610:\tlearn: 0.6485461\ttotal: 2h 16m 9s\tremaining: 19m 49s\n",
      "611:\tlearn: 0.6480825\ttotal: 2h 16m 21s\tremaining: 19m 36s\n",
      "612:\tlearn: 0.6479895\ttotal: 2h 16m 34s\tremaining: 19m 22s\n",
      "613:\tlearn: 0.6477786\ttotal: 2h 16m 45s\tremaining: 19m 9s\n",
      "614:\tlearn: 0.6477316\ttotal: 2h 16m 56s\tremaining: 18m 55s\n",
      "615:\tlearn: 0.6476639\ttotal: 2h 17m 8s\tremaining: 18m 42s\n",
      "616:\tlearn: 0.6475982\ttotal: 2h 17m 20s\tremaining: 18m 28s\n",
      "617:\tlearn: 0.6475656\ttotal: 2h 17m 31s\tremaining: 18m 14s\n",
      "618:\tlearn: 0.6475485\ttotal: 2h 17m 42s\tremaining: 18m 1s\n",
      "619:\tlearn: 0.6474717\ttotal: 2h 17m 54s\tremaining: 17m 47s\n",
      "620:\tlearn: 0.6474170\ttotal: 2h 18m 5s\tremaining: 17m 34s\n",
      "621:\tlearn: 0.6472811\ttotal: 2h 18m 17s\tremaining: 17m 20s\n",
      "622:\tlearn: 0.6472656\ttotal: 2h 18m 28s\tremaining: 17m 6s\n",
      "623:\tlearn: 0.6471941\ttotal: 2h 18m 40s\tremaining: 16m 53s\n",
      "624:\tlearn: 0.6471530\ttotal: 2h 18m 51s\tremaining: 16m 39s\n",
      "625:\tlearn: 0.6470979\ttotal: 2h 19m 3s\tremaining: 16m 26s\n",
      "626:\tlearn: 0.6469979\ttotal: 2h 19m 15s\tremaining: 16m 12s\n",
      "627:\tlearn: 0.6469201\ttotal: 2h 19m 27s\tremaining: 15m 59s\n",
      "628:\tlearn: 0.6468875\ttotal: 2h 19m 38s\tremaining: 15m 45s\n",
      "629:\tlearn: 0.6468224\ttotal: 2h 19m 51s\tremaining: 15m 32s\n",
      "630:\tlearn: 0.6467726\ttotal: 2h 20m 2s\tremaining: 15m 18s\n",
      "631:\tlearn: 0.6466746\ttotal: 2h 20m 14s\tremaining: 15m 5s\n",
      "632:\tlearn: 0.6466606\ttotal: 2h 20m 24s\tremaining: 14m 51s\n",
      "633:\tlearn: 0.6465782\ttotal: 2h 20m 34s\tremaining: 14m 38s\n",
      "634:\tlearn: 0.6461255\ttotal: 2h 20m 44s\tremaining: 14m 24s\n",
      "635:\tlearn: 0.6460370\ttotal: 2h 20m 55s\tremaining: 14m 10s\n",
      "636:\tlearn: 0.6459420\ttotal: 2h 21m 5s\tremaining: 13m 57s\n",
      "637:\tlearn: 0.6457347\ttotal: 2h 21m 15s\tremaining: 13m 43s\n",
      "638:\tlearn: 0.6456784\ttotal: 2h 21m 25s\tremaining: 13m 30s\n",
      "639:\tlearn: 0.6456326\ttotal: 2h 21m 35s\tremaining: 13m 16s\n",
      "640:\tlearn: 0.6455753\ttotal: 2h 21m 45s\tremaining: 13m 2s\n",
      "641:\tlearn: 0.6455070\ttotal: 2h 21m 55s\tremaining: 12m 49s\n",
      "642:\tlearn: 0.6454836\ttotal: 2h 22m 5s\tremaining: 12m 35s\n",
      "643:\tlearn: 0.6454188\ttotal: 2h 22m 14s\tremaining: 12m 22s\n",
      "644:\tlearn: 0.6453481\ttotal: 2h 22m 24s\tremaining: 12m 8s\n",
      "645:\tlearn: 0.6452978\ttotal: 2h 22m 35s\tremaining: 11m 55s\n",
      "646:\tlearn: 0.6452688\ttotal: 2h 22m 45s\tremaining: 11m 41s\n",
      "647:\tlearn: 0.6452099\ttotal: 2h 22m 55s\tremaining: 11m 28s\n",
      "648:\tlearn: 0.6451388\ttotal: 2h 23m 5s\tremaining: 11m 14s\n",
      "649:\tlearn: 0.6450483\ttotal: 2h 23m 15s\tremaining: 11m 1s\n",
      "650:\tlearn: 0.6450161\ttotal: 2h 23m 24s\tremaining: 10m 47s\n",
      "651:\tlearn: 0.6449305\ttotal: 2h 23m 35s\tremaining: 10m 34s\n",
      "652:\tlearn: 0.6448246\ttotal: 2h 23m 44s\tremaining: 10m 20s\n",
      "653:\tlearn: 0.6447629\ttotal: 2h 23m 54s\tremaining: 10m 7s\n",
      "654:\tlearn: 0.6447099\ttotal: 2h 24m 4s\tremaining: 9m 53s\n",
      "655:\tlearn: 0.6446466\ttotal: 2h 24m 14s\tremaining: 9m 40s\n",
      "656:\tlearn: 0.6445984\ttotal: 2h 24m 24s\tremaining: 9m 27s\n",
      "657:\tlearn: 0.6445277\ttotal: 2h 24m 34s\tremaining: 9m 13s\n",
      "658:\tlearn: 0.6444942\ttotal: 2h 24m 44s\tremaining: 9m\n",
      "659:\tlearn: 0.6444723\ttotal: 2h 24m 54s\tremaining: 8m 46s\n",
      "660:\tlearn: 0.6443780\ttotal: 2h 25m 4s\tremaining: 8m 33s\n",
      "661:\tlearn: 0.6443341\ttotal: 2h 25m 14s\tremaining: 8m 20s\n",
      "662:\tlearn: 0.6443059\ttotal: 2h 25m 24s\tremaining: 8m 6s\n",
      "663:\tlearn: 0.6442372\ttotal: 2h 25m 34s\tremaining: 7m 53s\n",
      "664:\tlearn: 0.6441789\ttotal: 2h 25m 44s\tremaining: 7m 40s\n",
      "665:\tlearn: 0.6440640\ttotal: 2h 25m 54s\tremaining: 7m 26s\n",
      "666:\tlearn: 0.6440019\ttotal: 2h 26m 4s\tremaining: 7m 13s\n",
      "667:\tlearn: 0.6438765\ttotal: 2h 26m 14s\tremaining: 7m\n",
      "668:\tlearn: 0.6437132\ttotal: 2h 26m 25s\tremaining: 6m 47s\n",
      "669:\tlearn: 0.6436686\ttotal: 2h 26m 35s\tremaining: 6m 33s\n",
      "670:\tlearn: 0.6436245\ttotal: 2h 26m 45s\tremaining: 6m 20s\n",
      "671:\tlearn: 0.6435314\ttotal: 2h 26m 55s\tremaining: 6m 7s\n",
      "672:\tlearn: 0.6434619\ttotal: 2h 27m 5s\tremaining: 5m 54s\n",
      "673:\tlearn: 0.6434187\ttotal: 2h 27m 15s\tremaining: 5m 40s\n",
      "674:\tlearn: 0.6433646\ttotal: 2h 27m 24s\tremaining: 5m 27s\n",
      "675:\tlearn: 0.6433091\ttotal: 2h 27m 35s\tremaining: 5m 14s\n",
      "676:\tlearn: 0.6430933\ttotal: 2h 27m 45s\tremaining: 5m 1s\n",
      "677:\tlearn: 0.6430172\ttotal: 2h 27m 55s\tremaining: 4m 47s\n",
      "678:\tlearn: 0.6429652\ttotal: 2h 28m 5s\tremaining: 4m 34s\n",
      "679:\tlearn: 0.6428374\ttotal: 2h 28m 15s\tremaining: 4m 21s\n",
      "680:\tlearn: 0.6426767\ttotal: 2h 28m 24s\tremaining: 4m 8s\n",
      "681:\tlearn: 0.6426331\ttotal: 2h 28m 34s\tremaining: 3m 55s\n",
      "682:\tlearn: 0.6425785\ttotal: 2h 28m 44s\tremaining: 3m 42s\n",
      "683:\tlearn: 0.6425148\ttotal: 2h 28m 54s\tremaining: 3m 29s\n",
      "684:\tlearn: 0.6424363\ttotal: 2h 29m 5s\tremaining: 3m 15s\n",
      "685:\tlearn: 0.6423835\ttotal: 2h 29m 15s\tremaining: 3m 2s\n",
      "686:\tlearn: 0.6423364\ttotal: 2h 29m 25s\tremaining: 2m 49s\n",
      "687:\tlearn: 0.6422784\ttotal: 2h 29m 35s\tremaining: 2m 36s\n",
      "688:\tlearn: 0.6422306\ttotal: 2h 29m 45s\tremaining: 2m 23s\n",
      "689:\tlearn: 0.6418982\ttotal: 2h 29m 55s\tremaining: 2m 10s\n",
      "690:\tlearn: 0.6418300\ttotal: 2h 30m 5s\tremaining: 1m 57s\n",
      "691:\tlearn: 0.6417801\ttotal: 2h 30m 15s\tremaining: 1m 44s\n",
      "692:\tlearn: 0.6417149\ttotal: 2h 30m 25s\tremaining: 1m 31s\n",
      "693:\tlearn: 0.6415696\ttotal: 2h 30m 35s\tremaining: 1m 18s\n",
      "694:\tlearn: 0.6415006\ttotal: 2h 30m 46s\tremaining: 1m 5s\n",
      "695:\tlearn: 0.6414420\ttotal: 2h 30m 56s\tremaining: 52s\n",
      "696:\tlearn: 0.6414114\ttotal: 2h 31m 5s\tremaining: 39s\n",
      "697:\tlearn: 0.6413941\ttotal: 2h 31m 15s\tremaining: 26s\n",
      "698:\tlearn: 0.6413740\ttotal: 2h 31m 24s\tremaining: 13s\n",
      "699:\tlearn: 0.6413431\ttotal: 2h 31m 34s\tremaining: 0us\n",
      "Valid error:  0.8700649558717015\n",
      "preds mean 0.33820575494716737\n",
      "Took 153.39 minutes\n"
     ]
    }
   ],
   "source": [
    "start_time = time.time()\n",
    "# Initialize CatBoostRegressor rsm=0.8, depth=5, learning_rate=0.037, eval_metric='MAE\n",
    "model_cat = CatBoostRegressor(iterations=700, l2_leaf_reg=9,\n",
    "                              learning_rate=0.15,\n",
    "                          depth=10, eval_metric='RMSE',#task_type='GPU'\n",
    "                             )\n",
    "# Fit model\n",
    "model_cat.fit(X_train, Y_train)\n",
    "print('Valid error: ',np.sqrt(mean_squared_error(Y_valid, model_cat.predict(X_valid))))\n",
    "preds = model_cat.predict(X_test).clip(0,20)\n",
    "print('preds mean',preds.mean())\n",
    "\n",
    "pickle.dump(model_cat, open('model_cat1.pkl', \"wb\"))\n",
    "\n",
    "print('Took %0.2f minutes'%((time.time() - start_time)/60))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['item_cnt_month_lag_1', 'item_id',\n",
       "       'item_cnt_month_by_item_id_mean_lag_1', 'month',\n",
       "       'tfidf_interaction_3', 'first_sale_item_id',\n",
       "       'avg_sale_last_6_item_cnt_month',\n",
       "       'avg_sale_last_12_item_cnt_month', 'item_cnt_month_lag_2',\n",
       "       'price_item_mean_lag_2',\n",
       "       'item_cnt_month_by_date_block_num_item_id_city_code_lag_1',\n",
       "       'std_sale_last_6_item_cnt_month', 'price_item_mean_lag_1',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_lag_12',\n",
       "       'item_cnt_month_by_date_block_num_subtype_code_lag_6',\n",
       "       'item_cnt_month_by_date_block_num_item_category_id_lag_1',\n",
       "       'avg_sale_last_12_item_cnt_month_by_date_block_num_subtype_code',\n",
       "       'sales_item_delta2',\n",
       "       'item_cnt_month_by_date_block_num_type_code_lag_3',\n",
       "       'avg_sale_last_12_item_cnt_month_by_item_id_mean',\n",
       "       'price_shop_item_mean_lag_1', 'sales_item_delta3',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_item_category_id_lag_1',\n",
       "       'item_cnt_month_by_date_block_num_type_code_lag_1',\n",
       "       'item_cnt_month_by_date_block_num_type_code_lag_12',\n",
       "       'std_sale_last_6_item_cnt_month_by_item_id_mean',\n",
       "       'item_cnt_month_lag_3',\n",
       "       'item_cnt_month_by_date_block_num_type_code_lag_2',\n",
       "       'item_cnt_month_by_date_block_num_type_code_lag_6',\n",
       "       'item_category_id',\n",
       "       'std_sale_last_12_item_cnt_month_by_date_block_num_item_category_id',\n",
       "       'item_cnt_month_by_item_id_mean_lag_2',\n",
       "       'item_cnt_month_by_date_block_num_subtype_code_lag_2',\n",
       "       'avg_sale_last_6_item_cnt_month_by_date_block_num_type_code',\n",
       "       'avg_sale_last_6_item_cnt_month_by_item_id_mean',\n",
       "       'avg_sale_last_6_item_cnt_month_by_date_block_num_subtype_code',\n",
       "       'item_cnt_month_by_date_block_num_subtype_code_lag_1',\n",
       "       'std_sale_last_12_item_cnt_month_by_date_block_num_subtype_code',\n",
       "       'avg_sale_last_12_item_cnt_month_by_date_block_num_type_code',\n",
       "       'std_sale_last_12_item_cnt_month_by_item_id_mean',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_item_category_id_lag_2',\n",
       "       'item_cnt_month_by_date_block_num_subtype_code_lag_3',\n",
       "       'avg_sale_last_6_item_cnt_month_by_date_block_num_shop_id_subtype_code',\n",
       "       'std_sale_last_6_item_cnt_month_by_date_block_num_subtype_code',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_item_category_id_lag_3',\n",
       "       'days_of_month',\n",
       "       'avg_sale_last_12_item_cnt_month_by_date_block_num_shop_id_subtype_code',\n",
       "       'std_sale_last_6_item_cnt_month_by_date_block_num_item_category_id',\n",
       "       'last_sale_shop_iditem_id',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_subtype_code_lag_1',\n",
       "       'std_sale_last_6_item_cnt_month_by_date_block_num_type_code',\n",
       "       'avg_sale_last_12_item_cnt_month_by_date_block_num_item_category_id',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_subtype_code_lag_2',\n",
       "       'item_cnt_month_by_date_block_num_city_code_lag_12',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_subtype_code_lag_6',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_item_category_id_lag_6',\n",
       "       'item_cnt_month_by_date_block_num_item_category_id_lag_12',\n",
       "       'subtype_code', 'sales_item_delta6',\n",
       "       'item_cnt_month_by_date_block_num_subtype_code_lag_12',\n",
       "       'item_cnt_month_by_date_block_num_item_id_city_code_lag_2',\n",
       "       'sales_shopitem_delta2',\n",
       "       'item_cnt_month_by_date_block_num_item_category_id_lag_3',\n",
       "       'std_sale_last_12_item_cnt_month',\n",
       "       'std_sale_last_12_item_cnt_month_by_date_block_num_type_code',\n",
       "       'item_cnt_month_by_date_block_num_item_category_id_lag_2', 'year',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_lag_2',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_type_code_lag_12',\n",
       "       'sales_shopitem_delta3', 'last_sale_item_id',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_item_category_id_lag_12',\n",
       "       'tfidf_interaction_6', 'item_cnt_month_by_item_id_mean_lag_3',\n",
       "       'avg_sale_last_6_item_cnt_month_by_date_block_num_item_category_id',\n",
       "       'std_sale_last_6_item_cnt_month_by_date_block_num_shop_id_subtype_code',\n",
       "       'avg_sale_last_12_item_cnt_month_by_date_block_num_shop_id',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_subtype_code_lag_12',\n",
       "       'price_item_mean_lag_3', 'tfidf_interaction_5',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_lag_1',\n",
       "       'tfidf_interaction_1',\n",
       "       'avg_sale_last_12_item_cnt_month_by_date_block_num_item_id_city_code',\n",
       "       'tfidf_interaction_10',\n",
       "       'std_sale_last_6_item_cnt_month_by_date_block_num_city_code',\n",
       "       'item_cnt_month_by_date_block_num_item_id_city_code_lag_3',\n",
       "       'tfidf_interaction_8',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_type_code_lag_6',\n",
       "       'consecutiveis_sales_up_item_month_', 'first_sale_shop_iditem_id',\n",
       "       'avg_sale_last_6_item_cnt_month_by_date_block_num_item_id_city_code',\n",
       "       'item_cnt_month_by_date_block_num_item_category_id_lag_6',\n",
       "       'shop_id',\n",
       "       'std_sale_last_12_item_cnt_month_by_date_block_num_shop_id_item_category_id',\n",
       "       'std_sale_last_6_item_cnt_month_by_date_block_num_item_id_city_code',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_lag_6',\n",
       "       'item_cnt_month_by_item_id_mean_lag_6', 'item_cnt_month_lag_6',\n",
       "       'tfidf_interaction_4', 'consecutiveis_price_up_shopitem_month_',\n",
       "       'price_shop_item_mean_lag_2',\n",
       "       'item_cnt_month_by_date_block_num_city_code_lag_6',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_lag_3',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_type_code_lag_3',\n",
       "       'std_sale_last_6_item_cnt_month_by_date_block_num_shop_id_item_category_id',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_type_code_lag_2',\n",
       "       'sales_shopitem_delta12',\n",
       "       'avg_sale_last_6_item_cnt_month_by_date_block_num_shop_id_item_category_id',\n",
       "       'price_item_mean_lag_6',\n",
       "       'std_sale_last_6_item_cnt_month_by_date_block_num_shop_id',\n",
       "       'std_sale_last_12_item_cnt_month_by_date_block_num_item_id_city_code',\n",
       "       'std_sale_last_12_item_cnt_month_by_date_block_num_shop_id_subtype_code',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_subtype_code_lag_3',\n",
       "       'item_cnt_month_by_date_block_num_city_code_lag_3',\n",
       "       'tfidf_interaction_7', 'tfidf_interaction_2',\n",
       "       'item_cnt_month_by_item_id_mean_lag_12',\n",
       "       'item_cnt_month_by_date_block_num_city_code_lag_2',\n",
       "       'std_sale_last_6_item_cnt_month_by_date_block_num_shop_id_type_code',\n",
       "       'avg_sale_last_12_item_cnt_month_by_date_block_num_shop_id_item_category_id',\n",
       "       'std_sale_last_12_item_cnt_month_by_date_block_num_shop_id_type_code',\n",
       "       'avg_sale_last_12_item_cnt_month_by_date_block_num_city_code',\n",
       "       'consecutiveis_price_down_shopitem_month_', 'sales_item_delta12',\n",
       "       'sales_shopitem_delta6', 'item_cnt_month_lag_12',\n",
       "       'item_cnt_month_by_date_block_num_city_code_lag_1',\n",
       "       'std_sale_last_12_item_cnt_month_by_date_block_num_city_code',\n",
       "       'consecutiveis_price_down_item_month_',\n",
       "       'avg_sale_last_6_item_cnt_month_by_date_block_num_shop_id',\n",
       "       'price_shop_item_mean_lag_3', 'city_code',\n",
       "       'item_cnt_month_by_date_block_num_shop_id_type_code_lag_1',\n",
       "       'avg_sale_last_6_item_cnt_month_by_date_block_num_shop_id_type_code',\n",
       "       'item_cnt_month_by_date_block_num_item_id_city_code_lag_6',\n",
       "       'std_sale_last_12_item_cnt_month_by_date_block_num_shop_id',\n",
       "       'consecutiveis_price_up_item_month_', 'price_item_mean_lag_12',\n",
       "       'tfidf_interaction_9', 'price_shop_item_mean_lag_6',\n",
       "       'avg_sale_last_6_item_cnt_month_by_date_block_num_city_code',\n",
       "       'type_code',\n",
       "       'avg_sale_last_12_item_cnt_month_by_date_block_num_shop_id_type_code',\n",
       "       'item_cnt_month_by_date_block_num_item_id_city_code_lag_12',\n",
       "       'consecutiveis_sales_down_item_month_',\n",
       "       'consecutiveis_sales_up_shopitem_month_',\n",
       "       'price_shop_item_mean_lag_12',\n",
       "       'consecutiveis_sales_down_shopitem_month_'], dtype='<U74')"
      ]
     },
     "execution_count": 130,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "idx = np.argsort(model_cat.feature_importances_)\n",
    "cols = np.array(model_cat.feature_names_)[idx][::-1]\n",
    "cols"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 159,
   "metadata": {},
   "outputs": [],
   "source": [
    "#all_data.drop(cols[40:], axis=1, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 257,
   "metadata": {},
   "outputs": [],
   "source": [
    "all_data.drop(cols[145:],axis=1,inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 258,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "xFqC2DnQPhgU"
   },
   "outputs": [],
   "source": [
    "# save processed data to disk in case the kernel restart because of out of memory\n",
    "outfile = open('all_data_ready','wb')\n",
    "pickle.dump(all_data, outfile)\n",
    "outfile.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "vfkHoQODkL9n"
   },
   "source": [
    "infile = open('all_data6','rb')\n",
    "all_data = pickle.load(infile)\n",
    "infile.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ApKrp641Phg-"
   },
   "source": [
    "# Part 7/8. Conclusion<a name=\"Conclusion\"></a>\n",
    "\n",
    "#### Garbage in, garbage out\n",
    "Data cleaning is a vital step. No matter how good the machine learning algorithm you are ready to use, the final prediction of the model you build will not be good if the data you feed in are full of misleading noise. So first thing to do is to identify those misleading data and correct it if possible. If not, maybe drop it is a good choice.\n",
    "\n",
    "#### Dealing with missing data is very tricky. \n",
    "If you fill in with values you think most reasonable for you, then the final model prediction will also prefer to your reasonable guessing. In another words, the data you filled in for missing values will probably mislead following machine learning analysis. So do not fill in any arbitrary values for missing data only because you think or guess it should be. Check it very carefully.\n",
    " \n",
    "#### Feature engineering\n",
    "Feature engineering is a key step to organise your data. Correct choice will maximise the power of machine learning algorithms.\n",
    "\n",
    "#### It should be? or you are just rationalizing\n",
    "\"It should be\" usually is a fashion way to say \"I don't know, but I guess\". If dig deep enough, it should be is just a way to express what you expect things to be, not necessarily the things really should be. Most times this is just a way to rationalize your guess. Data analysis contain a lot of different methods and skills. Different approaches have different usage. Try different approaches if possible, but never say it should be without verfication.\n",
    "\n",
    "#### Time effect\n",
    "Time is the best cure for any problem. Just like you lost your iPhone or computer, the first several days may be harsh. Weeks later it will mellow down. And months later it will be forgotten as nothing happened. Feature engineering may produce a lot of new features, thus fitting model will become terribly computationally expensive. Feature selection becomes necessary, even though selecting itself is also hard and time-consuming. However, if take enough time learn, wait and try, fitting model will be finished one way or another.\n",
    "\n",
    "#### New skills\n",
    "Kaggle dataset is a great way to implement new skills you learn. Don't worry about the final outcome for this dataset. Try new skills as many as possible. Even though it doesn't work out for this particular dataset and imporove your position on leaderboard, the implementation of new skills itself is a great way to learn new things. These skills may be used in other dataset in future. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "AxnC4FslPhg_"
   },
   "source": [
    "# Part 8/8. Reference<a name=\"Reference\"></a>\n",
    "Thanks for sharing below two notebooks. These notebooks are really helpful, especially for extracting features part.\n",
    "\n",
    "https://www.kaggle.com/dlarionov/feature-engineering-xgboost/output\n",
    "\n",
    "https://github.com/SJD1882/MOOC-Coursera-Advanced-Machine-Learning/blob/master/2%20-%20How%20to%20Win%20a%20Data%20Science%20Competition/Final%20Project/Final%20Project%20(Predict%20Future%20Sales%20Kaggle%20Competition).ipynb\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "K4ItT-dEPhg_"
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [
    "XsUhRPpJPhfp",
    "isP3yRHFPhgl",
    "bVL4Q5akPhgn",
    "GNTAQx4UPhg5",
    "YNFbT8UDPhg8",
    "ApKrp641Phg-",
    "AxnC4FslPhg_"
   ],
   "name": "Final Project (Predict Future Sales)_redo.ipynb",
   "provenance": [],
   "toc_visible": true,
   "version": "0.3.2"
  },
  "kernelspec": {
   "display_name": "Python [conda env:data_science_env]",
   "language": "python",
   "name": "conda-env-data_science_env-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}