Module-03f-Model-Boosting.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Gradient Boosting Machines"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Boosting** \n",
    "\n",
    "Combine weak learners to build a strong model\n",
    "\n",
    "How does this work?\n",
    "\n",
    "Build a base model:\n",
    "\n",
    "`Y = Model1(x) + Error1`\n",
    "\n",
    "Models are abstractions. There will be error between predictions and actual\n",
    "\n",
    "What is this error can be modeled ? Say:\n",
    "\n",
    "`Error1 = Model2(x) + Error2`\n",
    "\n",
    "If modeled right, this will improve the accuracy of the predictions.\n",
    "\n",
    "And we can continue:\n",
    "\n",
    "`Error2 = Model3(x) + Error3`\n",
    "\n",
    "Combining these three steps, we have:\n",
    "\n",
    "`Y = Model1(x) + Model2(x) + Model3(x) + Error3`\n",
    "\n",
    "And if we find weights(parameters) for these models?\n",
    "\n",
    "$$ Y = \\alpha Model1(x) + \\beta Model2(x) + \\gamma Model3(x) + Error3$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Intuition\n",
    "\n",
    "![](img/boost1.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### AdaBoost\n",
    "\n",
    "AdaBoost is done using decision stump"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise**\n",
    "\n",
    "Run `sklearn.ensemble.AdaBoostClassifier`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`xgboost`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "import xgboost as xgb\n",
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Read the data\n",
    "df = pd.read_csv(\"data/historical_loan.csv\")\n",
    "\n",
    "# refine the data\n",
    "df.years = df.years.fillna(np.mean(df.years))\n",
    "\n",
    "#Load the preprocessing module\n",
    "from sklearn import preprocessing\n",
    "categorical_variables = df.dtypes[df.dtypes==\"object\"].index.tolist()\n",
    "for i in categorical_variables:\n",
    "    lbl = preprocessing.LabelEncoder()\n",
    "    lbl.fit(list(df[i]))\n",
    "    df[i] = lbl.transform(df[i])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>default</th>\n",
       "      <th>amount</th>\n",
       "      <th>grade</th>\n",
       "      <th>years</th>\n",
       "      <th>ownership</th>\n",
       "      <th>income</th>\n",
       "      <th>age</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>1000</td>\n",
       "      <td>1</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3</td>\n",
       "      <td>19200.0</td>\n",
       "      <td>24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>6500</td>\n",
       "      <td>0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0</td>\n",
       "      <td>66000.0</td>\n",
       "      <td>28</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>2400</td>\n",
       "      <td>0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3</td>\n",
       "      <td>60000.0</td>\n",
       "      <td>36</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>10000</td>\n",
       "      <td>2</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3</td>\n",
       "      <td>62000.0</td>\n",
       "      <td>24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>4000</td>\n",
       "      <td>2</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3</td>\n",
       "      <td>20000.0</td>\n",
       "      <td>28</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   default  amount  grade  years  ownership   income  age\n",
       "0        0    1000      1    2.0          3  19200.0   24\n",
       "1        1    6500      0    2.0          0  66000.0   28\n",
       "2        0    2400      0    2.0          3  60000.0   36\n",
       "3        0   10000      2    3.0          3  62000.0   24\n",
       "4        1    4000      2    2.0          3  20000.0   28"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Setup the features and target\n",
    "X = df.iloc[:,1:]\n",
    "y = df.iloc[:,0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Details on the various parameters for xgboost can be found here:\n",
    "https://github.com/dmlc/xgboost/blob/master/doc/parameter.md\n",
    "\n",
    "\n",
    "  General Parameters\n",
    "------------------\n",
    "* booster [default=gbtree]\n",
    "  - which booster to use, can be gbtree, gblinear or dart. gbtree and dart use tree based model while gblinear uses linear function.\n",
    "* silent [default=0]\n",
    "  - 0 means printing running messages, 1 means silent mode.\n",
    "* nthread [default to maximum number of threads available if not set]\n",
    "  - number of parallel threads used to run xgboost\n",
    "* num_pbuffer [set automatically by xgboost, no need to be set by user]\n",
    "  - size of prediction buffer, normally set to number of training instances. The buffers are used to save the prediction results of last boosting step.\n",
    "* num_feature [set automatically by xgboost, no need to be set by user]\n",
    "  - feature dimension used in boosting, set to maximum dimension of the feature\n",
    "\n",
    "Parameters for Tree Booster\n",
    "---------------------------\n",
    "* eta [default=0.3]\n",
    "  - step size shrinkage used in update to prevents overfitting. After each boosting step, we can directly get the weights of new features. and eta actually shrinks the feature weights to make the boosting process more conservative.\n",
    "  - range: [0,1]\n",
    "* gamma [default=0]\n",
    "  - minimum loss reduction required to make a further partition on a leaf node of the tree. The larger, the more conservative the algorithm will be.\n",
    "  - range: [0,∞]\n",
    "* max_depth [default=6]\n",
    "  - maximum depth of a tree, increase this value will make the model more complex / likely to be overfitting.\n",
    "  - range: [1,∞]\n",
    "* min_child_weight [default=1]\n",
    "  - minimum sum of instance weight (hessian) needed in a child. If the tree partition step results in a leaf node with the sum of instance weight less than min_child_weight, then the building process will give up further partitioning. In linear regression mode, this simply corresponds to minimum number of instances needed to be in each node. The larger, the more conservative the algorithm will be.\n",
    "  - range: [0,∞]\n",
    "* max_delta_step [default=0]\n",
    "  - Maximum delta step we allow each tree's weight estimation to be. If the value is set to 0, it means there is no constraint. If it is set to a positive value, it can help making the update step more conservative. Usually this parameter is not needed, but it might help in logistic regression when class is extremely imbalanced. Set it to value of 1-10 might help control the update\n",
    "  - range: [0,∞]\n",
    "* subsample [default=1]\n",
    "  - subsample ratio of the training instance. Setting it to 0.5 means that XGBoost randomly collected half of the data instances to grow trees and this will prevent overfitting.\n",
    "  - range: (0,1]\n",
    "* colsample_bytree [default=1]\n",
    "  - subsample ratio of columns when constructing each tree.\n",
    "  - range: (0,1]\n",
    "* colsample_bylevel [default=1]\n",
    "  - subsample ratio of columns for each split, in each level.\n",
    "  - range: (0,1]\n",
    "* lambda [default=1]\n",
    "  - L2 regularization term on weights, increase this value will make model more conservative.\n",
    "* alpha [default=0]\n",
    "  - L1 regularization term on weights, increase this value will make model more conservative.\n",
    "* tree_method, string [default='auto']\n",
    "  - The tree construction algorithm used in XGBoost(see description in the [reference paper](http://arxiv.org/abs/1603.02754))\n",
    "  - Distributed and external memory version only support approximate algorithm.\n",
    "  - Choices: {'auto', 'exact', 'approx'}\n",
    "    - 'auto': Use heuristic to choose faster one.\n",
    "      - For small to medium dataset, exact greedy will be used.\n",
    "      - For very large-dataset, approximate algorithm will be chosen.\n",
    "      - Because old behavior is always use exact greedy in single machine,\n",
    "        user will get a message when approximate algorithm is chosen to notify this choice.\n",
    "    - 'exact': Exact greedy algorithm.\n",
    "    - 'approx': Approximate greedy algorithm using sketching and histogram.\n",
    "* sketch_eps, [default=0.03]\n",
    "  - This is only used for approximate greedy algorithm.\n",
    "  - This roughly translated into ```O(1 / sketch_eps)``` number of bins.\n",
    "    Compared to directly select number of bins, this comes with theoretical guarantee with sketch accuracy.\n",
    "  - Usually user does not have to tune this.\n",
    "    but consider setting to a lower number for more accurate enumeration.\n",
    "  - range: (0, 1)\n",
    "* scale_pos_weight, [default=1]\n",
    "  - Control the balance of positive and negative weights, useful for unbalanced classes. A typical value to consider: sum(negative  cases) / sum(positive cases) See [Parameters Tuning](how_to/param_tuning.md) for more discussion. Also see Higgs Kaggle competition demo for examples: [R](../demo/kaggle-higgs/higgs-train.R ), [py1](../demo/kaggle-higgs/higgs-numpy.py ), [py2](../demo/kaggle-higgs/higgs-cv.py ), [py3](../demo/guide-python/cross_validation.py)\n",
    "* updater_seq, [default=\"grow_colmaker,prune\"]\n",
    "  - A comma separated string mentioning tThe sequence of Tree updaters that should be run. A tree updater is a pluggable operation performed on the tree at every step using the gradient information. Tree updaters can be registered using the plugin system provided.\n",
    "\n",
    "\n",
    "Learning Task Parameters\n",
    "------------------------\n",
    "Specify the learning task and the corresponding learning objective. The objective options are below:\n",
    "* objective [ default=reg:linear ]\n",
    " - \"reg:linear\" --linear regression\n",
    " - \"reg:logistic\" --logistic regression\n",
    " - \"binary:logistic\" --logistic regression for binary classification, output probability\n",
    " - \"binary:logitraw\" --logistic regression for binary classification, output score before logistic transformation\n",
    " - \"count:poisson\" --poisson regression for count data, output mean of poisson distribution\n",
    "   - max_delta_step is set to 0.7 by default in poisson regression (used to safeguard optimization)\n",
    " - \"multi:softmax\" --set XGBoost to do multiclass classification using the softmax objective, you also need to set num_class(number of classes)\n",
    " - \"multi:softprob\" --same as softmax, but output a vector of ndata * nclass, which can be further reshaped to ndata, nclass matrix. The result contains predicted probability of each data point belonging to each class.\n",
    " - \"rank:pairwise\" --set XGBoost to do ranking task by minimizing the pairwise loss\n",
    " - \"reg:gamma\" --gamma regression for severity data, output mean of gamma distribution\n",
    " - \"reg:tweedie\" --tweedie regression for insurance data\n",
    "   - tweedie_variance_power is set to 1.5 by default in tweedie regression and must be in the range [1, 2)\n",
    "* base_score [ default=0.5 ]\n",
    "  - the initial prediction score of all instances, global bias\n",
    "  - for sufficient number of iterations, changing this value will not have too much effect.\n",
    "* eval_metric [ default according to objective ]\n",
    "  - evaluation metrics for validation data, a default metric will be assigned according to objective (rmse for regression, and error for classification, mean average precision for ranking )\n",
    "  - User can add multiple evaluation metrics, for python user, remember to pass the metrics in as list of parameters pairs instead of map, so that latter 'eval_metric' won't override previous one\n",
    "  - The choices are listed below:\n",
    "  - \"rmse\": [root mean square error](http://en.wikipedia.org/wiki/Root_mean_square_error)\n",
    "  - \"mae\": [mean absolute error](https://en.wikipedia.org/wiki/Mean_absolute_error)\n",
    "  - \"logloss\": negative [log-likelihood](http://en.wikipedia.org/wiki/Log-likelihood)\n",
    "  - \"error\": Binary classification error rate. It is calculated as #(wrong cases)/#(all cases). For the predictions, the evaluation will regard the instances with prediction value larger than 0.5 as positive instances, and the others as negative instances.\n",
    "  - \"merror\": Multiclass classification error rate. It is calculated as #(wrong cases)/#(all cases).\n",
    "  - \"mlogloss\": [Multiclass logloss](https://www.kaggle.com/wiki/MultiClassLogLoss)\n",
    "  - \"auc\": [Area under the curve](http://en.wikipedia.org/wiki/Receiver_operating_characteristic#Area_under_curve) for ranking evaluation.\n",
    "  - \"ndcg\":[Normalized Discounted Cumulative Gain](http://en.wikipedia.org/wiki/NDCG)\n",
    "  - \"map\":[Mean average precision](http://en.wikipedia.org/wiki/Mean_average_precision#Mean_average_precision)\n",
    "  - \"ndcg@n\",\"map@n\": n can be assigned as an integer to cut off the top positions in the lists for evaluation.\n",
    "  - \"ndcg-\",\"map-\",\"ndcg@n-\",\"map@n-\": In XGBoost, NDCG and MAP will evaluate the score of a list without any positive samples as 1. By adding \"-\" in the evaluation metric XGBoost will evaluate these score as 0 to be consistent under some conditions.\n",
    "training repeatedly\n",
    "  - \"gamma-deviance\": [residual deviance for gamma regression]\n",
    "* seed [ default=0 ]\n",
    " - random number seed.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Parameters\n",
    "\n",
    "params = {}\n",
    "params[\"min_child_weight\"] = 3\n",
    "params[\"subsample\"] = 0.7\n",
    "params[\"colsample_bytree\"] = 0.7\n",
    "params[\"scale_pos_weight\"] = 1\n",
    "params[\"silent\"] = 0\n",
    "params[\"max_depth\"] = 4\n",
    "params[\"nthread\"] = 6\n",
    "params[\"gamma\"] = 1\n",
    "params[\"objective\"] = \"binary:logistic\"\n",
    "params[\"eta\"] = 0.005\n",
    "params[\"base_score\"] = 0.1\n",
    "params[\"eval_metric\"] = \"auc\"\n",
    "params[\"seed\"] = 123\n",
    "\n",
    "plst = list(params.items())\n",
    "num_rounds = 40"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "xgtrain = xgb.DMatrix(X_train, label=y_train)\n",
    "watchlist = [(xgtrain, 'train')]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model_xgboost = xgb.train(plst, xgtrain, num_rounds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x11943bfd0>"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAhwAAAFrCAYAAACaMVCYAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmYnWV9//H3J0RkDQQECWu0QEFZIqASXIqCFgRNXFqK\nWAFRS1sKuED5uQZaFalKg7hUQKBCQQFlECyFIhSqBAgYVkEQYkERREDCTuD7++M8iYcwITPDPDkz\nk/frus41z7mf5XzPzQnzmfu+zzmpKiRJkto0rtcFSJKksc/AIUmSWmfgkCRJrTNwSJKk1hk4JElS\n6wwckiSpdQYOSSNekm8m+XSv65A0dPFzOKSxK8lc4KXA013Nm1bVb17ANXcETqmq9V9YdaNTkpOA\nu6rqU72uRRpNHOGQxr63V9UqXbchh43hkGR8Lx//hUiyXK9rkEYrA4e0jEqyfZKfJnkwybXNyMWC\nffsm+XmSeUluT/I3TfvKwH8C6yZ5uLmtm+SkJP/cdf6OSe7quj83yT8muQ54JMn45ryzkvwuyR1J\nDnyeWhdef8G1kxya5N4kdyeZnuRtSX6R5P4kn+g6d0aSM5N8t3k+1yTZumv/5kkuafrhxiTvWORx\nv5HkR0keAfYD9gIObZ77D5vjDkvyy+b6NyV5Z9c19knyv0m+lOSB5rnu2rV/jSQnJvlNs//srn27\nJ5nT1PbTJFsN+D+wNMIYOKRlUJL1gPOAfwbWAD4OnJVkreaQe4HdgQnAvsDRSbapqkeAXYHfDGHE\nZE9gN2B14Bngh8C1wHrATsDBSf58gNdaB1ihOfczwHHA+4BtgTcAn0ny8q7jpwFnNM/1P4Czk7wo\nyYuaOi4A1gb+ATg1yZ92nfte4HPAqsC/A6cCRzXP/e3NMb9sHnc14HDglCSTuq7xWuAW4CXAUcAJ\nSdLs+w6wEvDKpoajAZJsA3wb+BtgTeDfgHOSvHiAfSSNKAYOaew7u/kL+cGuv57fB/yoqn5UVc9U\n1YXAbOBtAFV1XlX9sjr+h84v5De8wDqOqao7q+ox4NXAWlV1RFU9WVW30wkNfzXAaz0FfK6qngJO\np/OLfGZVzauqG4Ebge7RgKur6szm+K/QCSvbN7dVgCObOn4MnEsnHC3QV1U/afrp8f6Kqaozquo3\nzTHfBW4FXtN1yK+q6riqeho4GZgEvLQJJbsC+1fVA1X1VNPfAB8C/q2qrqiqp6vqZOCJpmZp1Bm1\nc6mSBmx6Vf33Im0bAX+R5O1dbS8CLgZohvw/C2xK5w+TlYDrX2Addy7y+OsmebCrbTngsgFe6/fN\nL2+Ax5qf93Ttf4xOkHjOY1fVM810z7oL9lXVM13H/orOyEl/dfcryfuBjwKTm6ZV6ISgBX7b9fiP\nNoMbq9AZcbm/qh7o57IbAXsn+YeutuW76pZGFQOHtGy6E/hOVX1o0R3NkP1ZwPvp/HX/VDMysmAK\noL+3tj1CJ5QssE4/x3SfdydwR1VtMpTih2CDBRtJxgHrAwumgjZIMq4rdGwI/KLr3EWf77PuJ9mI\nzujMTsDlVfV0kjn8sb+ez53AGklWr6oH+9n3uar63ACuI414TqlIy6ZTgLcn+fMkyyVZoVmMuT6d\nv6JfDPwOmN+Mdry169x7gDWTrNbVNgd4W7MAch3g4CU8/pXAQ81C0hWbGrZI8uphe4bPtm2SdzXv\nkDmYztTELOAKOmHp0GZNx47A2+lM0yzOPUD3+pCV6YSQ30FnwS2wxUCKqqq76SzC/XqSiU0Nb2x2\nHwfsn+S16Vg5yW5JVh3gc5ZGFAOHtAyqqjvpLKT8BJ1flHcChwDjqmoecCDwPeABOosmz+k692bg\nNOD2Zl3IunQWPl4LzKWz3uO7S3j8p+n8Yp8C3AHcBxxPZ9FlG/qAPeg8n78G3tWsl3gSeAeddRT3\nAV8H3t88x8U5AXjFgjUxVXUT8GXgcjphZEvgJ4Oo7a/prEm5mc5i3YMBqmo2nXUcxzZ13wbsM4jr\nSiOKH/wlaUxLMgPYuKre1+tapGWZIxySJKl1Bg5JktQ6p1QkSVLrHOGQJEmt83M4htGECRNq0003\n7XUZo8qDDz7I6quv3usyRh37bfDss8Gzz4ZmWeu3q6+++r6qWmtJxxk4htHaa6/N7Nmze13GqNLX\n18e0adN6XcaoY78Nnn02ePbZ0Cxr/ZbkVwM5zikVSZLUOgOHJElqnYFDkiS1zsAhSZJaZ+CQJEmt\nM3BIkqTWGTgkSVLrDBySJKl1Bg5JktQ6A4ckSWqdgUOSJLXOwCFJklpn4JAkSa0zcEiSpNYZOCRJ\nUusMHJIkqXUGDkmS1DoDhyRJap2BQ5Iktc7AIUmSWmfgkCRJrTNwSJKk1hk4JElS6wwckiSpdQYO\nSZLUOgOHJElqnYFDkiS1zsAhSZJaZ+CQJEmtM3BIkqTWGTgkSVLrDBySJKl1Bg5JktQ6A4ckSWqd\ngUOSJLXOwCFJklpn4JAkSa1LVfW6hjFj4403rvnvmdnrMkaVmVPnc9Dl43tdxqhjvw2efTZ49tnQ\nDGe/zT1yt+e0feADH+Dcc89l7bXX5oYbbgDg/vvvZ4899mDu3LlMnjyZ733ve0ycOJFLLrmEadOm\n8bKXvQyAd73rXXzmM58ZltoWSHJ1VW23pONG5AhHkp/2ugZJkkaiffbZh/PPP/9ZbUceeSQ77bQT\nt956KzvttBNHHnnkwn1veMMbmDNnDnPmzBn2sDEYIzJwVNUOva5BkqSR6I1vfCNrrLHGs9r6+vrY\ne++9Adh77705++yze1Ha8xqRgSPJw83PHZNckuTMJDcnOTVJmn2vTvLTJNcmuTLJqklWSHJikuuT\n/CzJm5pj90lydpIfJrkjyQFJPtocMyvJGs1xf5Lk/CRXJ7ksyWa96wVJkgbmnnvuYdKkSQBMmjSJ\ne++9d+G+yy+/nK233ppdd92VG2+8sVclMhom514FvBL4DfAT4HVJrgS+C+xRVVclmQA8BhwEUFVb\nNmHhgiSbNtfZornWCsBtwD9W1auSHA28H/hX4FvA/lV1a5LXAl8H3vx8xSWZAXwWYOLEiZw4df7w\nPfNlxEz7bEjst8GzzwbPPhua4eq3vr6+ftvvuece5s2bt3D/U0899axjF9x/9NFH+drXvsaKK67I\n7Nmzectb3sI3vvGNYaltsEZD4Liyqu4CSDIHmAz8Abi7qq4CqKqHmv2vB77atN2c5FfAgsBxcVXN\nA+Yl+QPww6b9emCrJKsAOwBnNIMoAC9eUnFVNQOYAZ1Foy6wGhwXpQ2N/TZ49tng2WdD0/aiUYC5\nc+dyzDHHMG3aNADWX399XvOa1zBp0iTuvvtu1ltvvYX7Fpg2bRrf+c53eN3rXsdLXvKSYalvMEbk\nlMoinujafppOSArQ39tr0k9bf9d5puv+M801xwEPVtWUrtvmQy9bkqSl4x3veAcnn3wyACeffPLC\nsPHb3/6WBe9GvfLKK3nmmWdYc801e1LjaAgc/bkZWDfJqwGa9RvjgUuBvZq2TYENgVsGcsFmlOSO\nJH/RnJ8kW7dRvCRJQ7XnnnsydepUbrnlFtZff31OOOEEDjvsMC688EI22WQTLrzwQg477DAAzjzz\nTLbYYgu23nprDjzwQE4//XS6RvGXqlE5VlZVTybZA/hqkhXprN/Ymc6ai28muR6YD+xTVU8MonP3\nAr6R5FPAi4DTgWsHU9vihr/Uv76+PvtsCOy3wbPPBs8+G5q2++20007rt/2iiy56TtsBBxzAAQcc\n0FotgzEiA0dVrdL8vAS4pKv9gK7tq4Dt+zl9n36udxJwUtf9yf3tq6o7gF2GXrkkSerPaJ1SkSRJ\no4iBQ5Iktc7AIUmSWmfgkCRJrTNwSJKk1hk4JElS6wwckiSpdQYOSZLUOgOHJElqnYFDkiS1zsAh\nSZJaZ+CQJEmtM3BIkqTWGTgkSVLrDBySJKl1Bg5JktQ6A4ckSWqdgUOSJLXOwCFJklpn4JAkSa0z\ncEiSpNYZOCRJUusMHJIkqXUGDkmS1DoDhyRJap2BQ5Iktc7AIUmSWmfgkCRJrTNwSJKk1hk4JElS\n6wwckiSpdQYOSZLUuvG9LmCsmXzYeb0uYVSZOdU+Gwr7bfDss8Fbmn0298jdlsrjqHcc4ZAkjVhH\nH300r3zlK9liiy3Yc889efzxx3nDG97AlClTmDJlCuuuuy7Tp0/vdZkaAEc4+pFkOvCLqrqp17VI\n0rLq17/+Nccccww33XQTK664In/5l3/J6aefzmWXXbbwmHe/+91Mmzath1VqoBzh6N904BW9LkKS\nlnXz58/nscceY/78+Tz66KOsu+66C/fNmzePH//4x45wjBIjLnAkOTvJ1UluTPLhpu3hJF9s2v87\nyWuSXJLk9iTvaI5ZIcmJSa5P8rMkb2ra90lybNf1z02yY9d1P5fk2iSzkrw0yQ7AO4B/STInyZ8s\n9U6QJLHeeuvx8Y9/nA033JBJkyax2mqr8da3vnXh/h/84AfstNNOTJgwoYdVaqBSVb2u4VmSrFFV\n9ydZEbgK+DPgPuBtVfWfSX4ArAzsRmcU4uSqmpLkY8AWVbVvks2AC4BNgb8CtquqA5rrnwt8qaou\nSVLAO6rqh0mOAh6qqn9OchJwblWdOYB6ZwCfBZg4cSInnnjicHaHJC2zHn74Yb74xS/y8Y9/nJVX\nXpmjjjqKHXbYgR133BGAI444gp133pkddtiht4Uu46ZPn351VW23pONG4hqOA5O8s9neANgEeBI4\nv2m7Hniiqp5Kcj0wuWl/PfBVgKq6Ocmv6ASO5/MkcG6zfTXwlsEWW1UzgBkAG2+8cR10+Ujs0pFr\n5tT52GeDZ78Nnn02eEuzz/p7l8oZZ5zBq1/9at7//vcD8MgjjzBr1iymTZvG73//e+bOncvhhx/O\nCiussFRqHKi+vj7XlfRjRE2pNFMdOwNTq2pr4GfACsBT9cehmGeAJwCq6hn+GJqymMvO59nPs/uV\n2X3dpxmZAUySlkkbbrghs2bN4tFHH6WquOiii9h8882BThjZfffdR1zY0OKNqMABrAY8UFWPNtMi\n2w/i3EuBvQCSbApsCNwCzAWmJBmXZAPgNQO41jxg1cEULkkaXq997Wt5z3vewzbbbMOWW27JM888\nw4c//GEATj/9dPbcc88eV6jBGGl/0Z8P7J/kOjphYdYgzv068M1mmmU+sE9VPZHkJ8AddKZibgCu\nGcC1TgeOS3Ig8J6q+uVAi/DDawanr6/PPhsC+23w7LPBGwl9dvjhh3P44Yc/p/2SSy5Z+sXoBRlR\ngaOqngB27WfXKl3HzFjknFWan48D+/RzzaIZ+ehnX/d1zwTObLZ/gm+LlSRp2Iy0KRVJkjQGGTgk\nSVLrDBySJKl1Bg5JktQ6A4ckSWqdgUOSJLXOwCFJklpn4JAkSa0zcEiSpNYZOCRJUusMHJIkqXUG\nDkmS1DoDhyRJap2BQ5Iktc7AIUmSWmfgkCRJrTNwSJKk1hk4JElS6wwckiSpdQYOSZLUOgOHJElq\nnYFDkiS1zsAhSZJaZ+CQJEmtM3BIkqTWGTgkSVLrDBySJKl1Bg5JktQ6A4ckSWqdgUOSJLXOwCFJ\nklpn4JAkSa0b3+sCxprJh53X6xJGlZlT7bOhsN8G74X22dwjd3tO24MPPsgHP/hBbrjhBpLw7W9/\nm6lTpwLwpS99iUMOOYTf/e53vOQlLxny40pjxTIfOJLMBbarqvt6XYuk0eWggw5il1124cwzz+TJ\nJ5/k0UcfBeDOO+/kwgsvZMMNN+xxhdLIMSanVJIs80FKUrseeughLr30Uvbbbz8All9+eVZffXUA\nPvKRj3DUUUeRpJclSiPKqPzFnOTTwF7AncB9wNXA7sBPgdcB5yT5BfApYHng98BeVXVPkjWB04C1\ngCuBdF33fcCBzTlXAH9XVU8vreclafS4/fbbWWuttdh333259tpr2XbbbZk5cyYXXXQR6623Hltv\nvXWvS5RGlFRVr2sYlCTbAccDU+kEpmuAf6MTOG6qqr9rjpsIPFhVleSDwOZV9bEkxwD3VdURSXYD\nzqUTPtYCjgLeVVVPJfk6MKuq/n0J9cwAPgswceJETjzxxOF/0pJGnNtuu41DDz2UI488kk033ZTj\njz+e8ePHc+ONNzJjxgxWXnllPvShD/HlL3+ZCRMm9LpcqTXTp0+/uqq2W9JxozFwHAxMrKrPNve/\nAvyGTuD4bFX9T9O+JfBlYBKdEYs7qmqXJHPohIrbm+PuBzYF/gr4BHBv81ArAqdV1YyB1rbxxhvX\n/PfMfOFPchkyc+p8Drp8VA609ZT9NngvtM8WXTT629/+lu233565c+cCcNlllzFjxgyuv/56Vlpp\nJQDuuusu1l13Xa688krWWWedIT92r/T19TFt2rRelzHqLGv9lmRAgWNAaziSfDTJas32d5LcnOSt\nL7TIIXq+SdFHura/ChxbVVsCfwOs0LWvv5QV4OSqmtLc/nQwYUPSsmWdddZhgw024JZbbgHgoosu\nYptttuHee+9l7ty5zJ07l/XXX59rrrlmVIYNabgNdNHoPlX1hyRvAtYGPgB8vr2yntf/Am9PskKS\nVYDnvletYzXg18323l3tl9JZ/0GSXYGJTftFwHuSrN3sWyPJRsNdvKSx46tf/Sp77bUXW221FXPm\nzOETn/hEr0uSRqyBji8uWDj5JuDUqvppkp68w6WqrkpyDnAt8CtgNvCHfg6dAZyR5NfALOBlTfvh\nwGlJrgH+B/i/5ro3JfkUcEHz3J4C/r55DEl6jilTpjB79uzF7l8w3SJp4IHjsSSfpDMy8Lp03uu1\nfHtlLdGXqmpGkpXojFh8uaqO6z6gqvqAvkVPrKrfA93TQR/p2vdd4LsvpLD+PhxIi9fX12efDYH9\nNnj2mdRbA55SofMujkOq6rfAy4FT2ypqAL7VLP68Bjirqq7pYS2SJGkJBjTCUVW/AA5OslZz/5fA\nF9osbAn1vLdXjy1JkgZvoO9SeW2SX9EZUSDJdkm+1WplkiRpzBjolMpXgF3pfKonVTWbzid6SpIk\nLdFAA8fyVXXTIm1PDncxkiRpbBpo4Hii+cyLAkjyCuDx1qqSJEljykDfFvt54AJg3SQnAbsA72ur\nKEmSNLYM9F0qP0pyM/DndD4C/J+r6rZWK5MkSWPGEgNHkuWAs6vq7cA32i9JkiSNNUtcw1FVTwMr\n9uqjzCVJ0ug30DUcVwDfT/IfwMMLGqvqR61UJUmSxpSBBo4dmp9/29VWgIFDkiQt0UAXjb6p7UIk\nSdLYNaDAkeRt/bU7pSJJkgZioFMqh3RtrwBMofO9KgYOSZK0REOaUmk+afSjrVQkSZLGnCG91bX5\nXpWthrkWSZI0Rg1lDcc44NUMMaxIkqRlz1DWcMwHfgn8xfCXI0mSxqKBBo7pVfWH7oYkE1qoR5Ik\njUEDnRa5uJ+2S4axDkmSNIY97whHkvHA8sC4JCvS+aZYgNWAlVquTZIkjRFLGuH4JJ3vTtkSeKTZ\nfhj4OXBqu6VJkqSx4nkDR1UdXlXjgG9U1biu2+pV9U9LqUZJkjTKDWgNR1Ud0HYhkiRp7BpQ4Eiy\nVZLLkzya5OkFt7aLkyRJY8NA3xb7DeBTwFeAXYC/B+a1VZQkSRpbBvq22BWq6iJgXFXdXVWfAnZt\nsS5JkjSGDDRwzG9+3p9k6yRrAhu1VJMkSRpjBjql8t0mZHwB+F9gOeAzrVUlSZLGlIF+Pf1Xms3z\nk6xBZ4rFNRySJGlABvoulSTZL8kXq+opYM0kO7RcmyRJGiMGuobjK8BOwLTm/jzgX1upSJIkjTkD\nXcPxJuBVwDUAVfX7JCu0VtUoNvmw83pdwqgyc6p9NhRjpd/mHrlbv+2TJ09m1VVXZbnllmP8+PHM\nnj2bT3/60/T19TFu3DjWXnttTjrpJNZdd92lXLGkoRroCMfjVVUL7iQZxx+/yE2Sht3FF1/MnDlz\nmD17NgCHHHII1113HXPmzGH33XfniCOO6HGFkgZjoIHj+iR70VnOMZnOB4Fd1lZRvZBkuV7XIGnx\nJkyYsHD7kUceIfFvHmk0GWjg+CiwIzAJuKI575CWalqiJP+U5KCu+59LcmCSQ5JcleS6JId37T87\nydVJbkzy4a72h5MckeQKYGqSI5Pc1Jz/paX8tCQ1kvDWt76Vbbfdlm9961sL2z/5yU+ywQYbcOqp\npzrCIY0y6Zopee7O5MtV9bFm+y1VdeFSq+x5NKMs36+qbZrpnVuBT9BZ2Po3dKZ7zgGOqqpLk6xR\nVfcnWRG4CvizZh1KAXtU1feat/teDmxWVZVk9ap6cAC1zAA+CzBx4kROPPHEYX++0rLm/vvvZ401\n1uDBBx9kxowZfOhDH+KVr3zlwv1nnnkmTz31FHvuuWcPq5QEMH369KurarslHbekRaNv6tr+IjAi\nAkdVzU3y+ySvAl4K/Ax4NfDWZhtgFWAT4FLgwCTvbNo3aNp/DzwNnNW0PwQ8Dhyf5Dzg3AHWMgOY\nAbDxxhvXQZcPdB2uAGZOnY99Nnhjpd8Wt2i02x133MHyyy/PtGnTFrZNmTKF3Xbb7VltS9LX1zeo\n42WfDZX91r8lTalkMdsjwfHAPsC+wLfp1PeFqprS3DauqhOS7AjsDEytqq3pBJIF77B5vKqeBqiq\n+cBr6ASQ6cD5S/PJSOp45JFHmDdv3sLtCy64gC222IJbb7114THnnHMOm222Wa9KlDQES/oT6cVJ\nNqfzy7x7G4CquqnN4pbgB8ARwIuA99L5vpd/SnJqVT2cZD3gKWA14IGqejTJZsD2/V0sySrASlX1\noySzgNuWyrOQ9Cz33HMP73xnZ0By/vz5vPe972WXXXbh3e9+N7fccgvjxo1jo4024pvf/GaPK5U0\nGEsKHCsBP+q6371dwMuHvaIBqqonk1wMPNiMUlzQBKLLm9XrDwPvozNSsX+S64BbgFmLueSqQF/z\n+SIBPjKUugYyRKw/6uvrs8+GYCz328tf/nKuvfba57SfddZZ/RwtabR43sBRVZOXUh2D1iwW3R74\niwVtVTUTmNnP4bv2d42qWqVr+246UyqSJGmYDfRtsSNKklfQmfK4qKpuXdLxkiSpt0blMvdm7UjP\npnMkSdLgjMoRDkmSNLoYOCRJUusMHJIkqXUGDkmS1DoDhyRJap2BQ5Iktc7AIUmSWmfgkCRJrTNw\nSJKk1hk4JElS6wwckiSpdQYOSZLUOgOHJElqnYFDkiS1zsAhSZJaZ+CQJEmtM3BIkqTWGTgkSVLr\nDBySJKl1Bg5JktQ6A4ckSWqdgUOSJLXOwCFJklpn4JAkSa0zcEiSpNYZOCRJUusMHJIkqXUGDkmS\n1DoDhyRJap2BQ5Iktc7AIUmSWje+1wWMNZMPO6/XJYwqM6faZ0MxmH6be+Rui9339NNPs91227He\neutx7rnnst9++zF79myqik033ZSTTjqJVVZZZbjKlrQMc4RDWobNnDmTzTfffOH9o48+mmuvvZbr\nrruODTfckGOPPbaH1UkaSwwc0jLqrrvu4rzzzuODH/zgwrYJEyYAUFU89thjJOlVeZLGmDEfOJKc\nneTqJDcm+XDTtl+SXyS5JMlxSY5t2tdKclaSq5rb63pbvdSegw8+mKOOOopx4579v4F9992XddZZ\nh5tvvpl/+Id/6FF1ksaaVFWva2hVkjWq6v4kKwJXAX8O/ATYBpgH/Bi4tqoOSPIfwNer6n+TbAj8\nV1VtvtiLd64/A/gswMSJEznxxBNbfDbS8Ljqqqu4+uqr2X///bn++uvp6+vjU5/61ML9Tz/9NMcd\ndxybbLIJO+20Uw8rlTTSTZ8+/eqq2m5Jxy0Li0YPTPLOZnsD4K+B/6mq+wGSnAFs2uzfGXhF1zDy\nhCSrVtW8xV28qmYAMwA23njjOujyZaFLh8/MqfOxzwZvMP3W36LRWbNmcf3113PQQQfx+OOP89BD\nD3HGGWdwyimnLDxmzTXX5F/+5V845phjhq3uXurr62PatGm9LmNUsc+Gxn7r35ieUkmyI50QMbWq\ntgZ+BtzyPKeMa46d0tzWe76wIY1WX/jCF7jrrruYO3cup59+Om9+85v5zne+w2233QZ01nD88Ic/\nZLPNNutxpZLGijEdOIDVgAeq6tEkmwHbAysBf5ZkYpLxwLu7jr8AOGDBnSRTlmq1Ug9VFXvvvTdb\nbrklW265JXfffTef+cxnel2WpDFirI9lnw/sn+Q6OiMbs4BfA58HrgB+A9wE/KE5/kDga83x44FL\ngf0H84DP95kHeq6+vj77bAiGs9923HFHdtxxRwB+8pOfDMs1JWlRYzpwVNUTwK6LtieZXVXfakY4\nfkBnZIOqug/YY+lWKUnS2DfWp1QWZ0aSOcANwB3A2T2uR5KkMW1Mj3AsTlV9vNc1SJK0LFlWRzgk\nSdJSZOCQJEmtM3BIkqTWGTgkSVLrDBySJKl1Bg5JktQ6A4ckSWqdgUOSJLXOwCFJklpn4JAkSa0z\ncEiSpNYZOCRJUusMHJIkqXUGDkmS1DoDhyRJap2BQ5Iktc7AIUmSWmfgkCRJrTNwSJKk1hk4JElS\n6wwckiSpdQYOSZLUOgOHJElqnYFDkiS1zsAhSZJaZ+CQJEmtM3BIkqTWGTgkSVLrDBySJKl1Bg5J\nktS68b0uYKyZfNh5vS6hdXOP3O05bR/4wAc499xzWXvttbnhhhsAmDFjBscddxxrrbUWAJ///Od5\n29vetlRrlSSNDMvECEeSyUluWMy+I5LsvLRrGmv22Wcfzj///Oe0f+QjH2HOnDnMmTPHsCFJy7Ax\nNcKRZHxVzR/MOVX1mbbqWZa88Y1vZO7cub0uQ5I0QrU6wpHko0luaG4HJzk0yYHNvqOT/LjZ3inJ\nKc32w0k+l+TaJLOSvLRpXyvJWUmuam6va9pnJPlWkguAf0/yyiRXJpmT5LokmzTlLJfkuCQ3Jrkg\nyYrN+ScleU+zPTfJF5vzr0yycZv9syw49thj2WqrrfjABz7AAw880OtyJEk90lrgSLItsC/wWmB7\n4EPAZcAbmkO2A1ZJ8iLg9c0+gJWBWVW1NXBpcx7ATODoqno18G7g+K6H2xaYVlXvBfYHZlbVlOYx\n7mqO2QT3BSkUAAAJdklEQVT4WlW9EniwuUZ/Hqqq1wDHAv86xKcv4G//9m/55S9/yZw5c5g0aRIf\n+9jHel2SJKlH2pxSeT3wg6p6BCDJ94HXANsmWRV4AriGTih4A3Bgc96TwLnN9tXAW5rtnYFXJFlw\n/QnNdQDOqarHmu3LgU8mWR/4flXd2pxzR1XN6bru5MXUfVrXz6OX9CSTzAA+CzBx4kROnDqoGZ1R\nqa+vr9/2e+65h3nz5vW7f6ONNuKUU07pd9/irqfnZ78Nnn02ePbZ0Nhvz9Vm4Eg/bQXMpTPy8VPg\nOuBNwJ8AP2+Oeaqqqtl+uqvGccDUrmDReZBOmHhk4QNU/UeSK4DdgP9K8kHgdjoBZ4GngRUXU3ct\nZrv/g6tmADMANt544zro8jG1LKZf/b1LBWDu3Lkcc8wxTJs2DYC7776bSZMmAXD00UczderUhfsW\n6Ovre06blsx+Gzz7bPDss6Gx3/rX5hqOS4HpSVZKsjLwTjrTJpcCH29+XkZnCmROV8hYnAuAAxbc\nSTKlv4OSvBy4vaqOAc4Bthpk3Xt0/bx8kOcus/bcc0+mTp3KLbfcwvrrr88JJ5zAoYceypZbbslW\nW23FxRdfzNFHL3HASJI0RrX253hVXZPkJODKpun4qvpZkjWATwKXV9UjSR7nj+s3ns+BwNeSXEen\n7kvphJVF7QG8L8lTwG+BI4AJgyj9xc0IyThgz0Gct0w77bTTntO233779aASSdJI1Or4f1V9BfjK\nIm0XAS/qur/pIvtX6do+Eziz2b6PP44+dB8/Y5H7XwC+sMhh9wNbdB3zpa7tfRY59mtVdfjin9Xz\nW9x0gyRJy7Jl4oO/JElSb439FY6DUFWTe12DJEljkSMckiSpdQYOSZLUOgOHJElqnYFDkiS1zsAh\nSZJaZ+CQJEmtM3BIkqTWGTgkSVLrDBySJKl1Bg5JktQ6A4ckSWqdgUOSJLXOwCFJklpn4JAkSa0z\ncEiSpNYZOCRJUusMHJIkqXUGDkmS1DoDhyRJap2BQ5Iktc7AIUmSWmfgkCRJrTNwSJKk1hk4JElS\n6wwckiSpdQYOSZLUOgOHJElqnYFDkiS1zsAhSZJaZ+CQJEmtM3BIkqTWGTgkSVLrDBySJKl1Bg5J\nktQ6A4ckSWpdqqrXNYwZSR4Bft7rOkaZdYHf9LqIUch+Gzz7bPDss6FZ1vpto6paa0kHGTiGUZKq\nqvS6jtHEPhsa+23w7LPBs8+Gxn7rn1MqkiSpdQYOSZLUOgPH8Dq81wWMQvbZ0Nhvg2efDZ59NjT2\nWz9cwyFJklrnCIckSWqdgUOSJLXOwCFJklpn4JAkSa0zcEiSpNYZOCRJUusMHMMkyS5JbklyW5LD\nel3PSJRkgyQXJ/l5khuTHNS0r5HkwiS3Nj8n9rrWkSbJckl+luTc5v7LklzR9Nl3kyzf6xpHmiSr\nJzkzyc3Na26qr7Xnl+Qjzb/NG5KclmQFX2vPluTbSe5NckNXW7+vq3Qc0/xeuC7JNr2rvPcMHMMg\nyXLA14BdgVcAeyZ5RW+rGpHmAx+rqs2B7YG/b/rpMOCiqtoEuKi5r2c7iGd/MeAXgaObPnsA2K8n\nVY1sM4Hzq2ozYGs6/edrbTGSrAccCGxXVVsAywF/ha+1RZ0E7LJI2+JeV7sCmzS3DwPfWEo1jkgG\njuHxGuC2qrq9qp4ETgem9bimEaeq7q6qa5rteXR+AaxHp69Obg47GZjemwpHpiTrA7sBxzf3A7wZ\nOLM5xD5bRJIJwBuBEwCq6smqehBfa0syHlgxyXhgJeBufK09S1VdCty/SPPiXlfTgH+vjlnA6kkm\nLZ1KRx4Dx/BYD7iz6/5dTZsWI8lk4FXAFcBLq+pu6IQSYO3eVTYi/StwKPBMc39N4MGqmt/c9/X2\nXC8Hfgec2ExFHZ9kZXytLVZV/Rr4EvB/dILGH4Cr8bU2EIt7Xfm7oYuBY3j09zXEfmb8YiRZBTgL\nOLiqHup1PSNZkt2Be6vq6u7mfg719fZs44FtgG9U1auAR3D65Hk16w6mAS8D1gVWpjMlsChfawPn\nv9UuBo7hcRewQdf99YHf9KiWES3Ji+iEjVOr6vtN8z0Lhhmbn/f2qr4R6HXAO5LMpTNV92Y6Ix6r\nN8Pe4OutP3cBd1XVFc39M+kEEF9ri7czcEdV/a6qngK+D+yAr7WBWNzryt8NXQwcw+MqYJNmNffy\ndBZandPjmkacZu3BCcDPq+orXbvOAfZutvcG+pZ2bSNVVf2/qlq/qibTeV39uKr2Ai4G3tMcZp8t\noqp+C9yZ5E+bpp2Am/C19nz+D9g+yUrNv9UFfeZrbckW97o6B3h/826V7YE/LJh6WRb5bbHDJMnb\n6PzluRzw7ar6XI9LGnGSvB64DLieP65H+ASddRzfAzak8z+9v6iqRRdlLfOS7Ah8vKp2T/JyOiMe\nawA/A95XVU/0sr6RJskUOgttlwduB/al80eWr7XFSHI4sAedd5T9DPggnTUHvtYaSU4DdgReAtwD\nfBY4m35eV01wO5bOu1oeBfatqtm9qHskMHBIkqTWOaUiSZJaZ+CQJEmtM3BIkqTWGTgkSVLrDByS\nJKl145d8iCQNv+bDzB5vbgAXV9VHeleRpDYZOCT10nuq6oYlHza8kowDqvxcAGmpcUpF0oiVZO0k\n/53k+uZ2dNe+/9e0XZvkp02IIMk/JrmhuZ3YfHcPSWYkOSXJ2cC1dD6y+0+T/GeSq5rr7NubZyqN\nfY5wSOqlM5MsmFL5x6r6r0X27wX8qqp2hoVfMEaSvYF3AK+rqoeSrFlVzyTZFfhrOt8BMo/OV4V/\nGvjH5npvBLapqvua7wf5b2Cvqro5yarA7CSXV9XN7T1ladlk4JDUS0uaUpkFfDTJvwD/AywIJLvT\n+SbYhwCq6vdN+87A6Qvak3wLmNl1vR9V1X3N9qbA5sDpnU+gBuDFTZuBQxpmBg5JI1ZVXd58J8pb\n6IxcHAa8nv6/9pumfdF1Gd33H17k2PuqasowlSvpebiGQ9KIleRlwENVdTrwUWDbZq3GD4G/baZB\nSLJmc8qFwF8lWbX54qwP0pk26c8twKNJ/rrr8TZLMqGlpyMt0xzhkDSS7Qh8LMl8On8g7d+s1fh3\nOt9iOqvZNy/JG6vqP5NsBVzenD8b+Of+LlxV85O8HfjXJIfQ+abne4C/bPcpScsmvy1WkiS1zikV\nSZLUOgOHJElqnYFDkiS1zsAhSZJaZ+CQJEmtM3BIkqTWGTgkSVLr/j/4hads5pETUQAAAABJRU5E\nrkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x11853b748>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#Variable Importance Plot\n",
    "plt.style.use('seaborn-notebook')\n",
    "xgb.plot_importance(model_xgboost)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x11b6ce898>"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAe0AAAELCAYAAAD0qd5kAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xd4FNX6B/DvphECIRCIcAOhhSIgoKAIiohIR0FALgZR\nAUGU+NOgKMWCKFVBQJBy6eK9AQSkd0OPSA9NOqFITyC9bfL+/tjssJvsZktm9szsvp/nyZPd2Zlz\n3p2dnXdn5sw5OiICY4wxxtTPS3QAjDHGGLMPJ23GGGNMIzhpM8YYYxrBSZsxxhjTCE7ajDHGmEZw\n0maMMcY0gpM2Y4wxphGctBljjDGN4KTNGGOMaYSP6ACs4G7aGGOqdPXqVYSHh6NmzZro168fSpUq\nhX379mHlypUYM2YMvv76a9EhMu3S2ZxBpd2YqjIoxphnIiJ4eXnB3v3lF198gZSUFPz0008KR8bc\nDCdtxhgrDn9/f2RmZjq17IkTJ9CoUSOZI2JujJM2Y4w5q0yZMkhOTi5WGffv30eFChVkioi5OZtJ\nmxuiMcaYBV26dCl2wgYgJewhQ4YUuyzG+EibMcYKOHPmDOrXry97udevX0dYWJjs5TK3wafHGWPM\nEX5+fsjOzhYdBvNMfHqcMcYcoXTC7tChg6LlM/fGR9qMMZZPp9PZfVsXYwrgI23GGLOXqxL2mjVr\nXFIPcz98pM0YYwAePnyIsmXLuqSu+vXr48yZMy6pi2kKH2kzxpg9unXr5rK6tm/f7rK6mHvhpM0Y\nYwCOHz8ue5kHDx60OL1y5cqy18U8AydtxhgD8Pbbb1t9bevWrQCAO3fuAAB+//13AMCoUaMAADt2\n7EBCQoI0v/E0+/37962WGRMTU7yAmUfipM0YYwACAgIsTg8NDcWWLVsAABUrVgQALFu2DFOnTsX4\n8eMBAG3btkX58uVRrlw5AIbr4wDw8ssvW60vIyNDttiZ5+CkzRhjAP7++2+L0318fPDpp59Kz3U6\nHfbu3Yt+/foBMAwK0qJFC4wePRoPHjxAeno6Jk2ahM6dO2PevHlW63v88cdljZ95Bm49zhhjsH6P\n9gsvvICxY8fixRdfFBAV8zDcepwxxuyxZMkSi9P37t0re8IeM2aMrOUxz8FH2owx5mLe3t7Izc0V\nHQZTHz7SZowxe7mqRTcnbOYsTtqMMZbPWmM0Of3f//2f4nUw98WnxxljzMSqVavQs2dP0WEwz8Sn\nxxljzBFBQUGKlV23bl3FymaegY+0GWOsgNKlSyM1NVXWMuPi4tC4cWNZy2Ruh4+0GWPMUampqXj6\n6cuylVe1YkVO2EwWfKTNGGMF5OUBXl6GrkZ9fHzg6+vrdFmhoaG4efMm0Lw5cOCAjFEyN8RH2owx\n5oghQwwJGwBKliwJHx8fzJ0716myXn31VUPCBjhhM1nwkTZjjOUbPhyYNMn66zqdDtnZ2UUeeXfo\n0AFjxoxB8+bNFYiQuTmbR9qctBljDMDYscCXXzq2zPnz55GamoomTZrYv9DDh0D+0J2MFcBJmzHG\nVGfPHqBVK9FRMPXha9qMMVaUIkbPVE6rVsDkyQIqZlrHR9qMMY8VHQ1ERAgM4KWXgJ07BQbAVIZP\njzPGWEFEwPr1QNeuoiMBEBwMJCaKjoKpAydtxhgzlZsLeHuLjqIALy/DzeHM0/E1bcYYMxUbKzoC\nC/LyAAX7PGfug5M2Y8xjHD4MvPCCY8vMmjWr0LQff/xRpohMJCUBLVvKXy5zK5y0GWNuLyUFOH0a\nOHVqMQ4ePAgAGDduHABgUoHeVJYtW4Y1a9bgzp07AAB/f38QEWbMmAEiwvbt29G3b18AQHp6OgBD\nX+ULFy4sfqD79gHjx0tP9Xo9li9fnv8eUnD9+vXi18E0jZM2Y8ytPXgABAYCDRoARITRo0dDp9Ph\niy++APAoeRtFREQgIyMDFStWxLlz5zBq1CjExcVh7dq1iIuLQ6NGjRAVFQUACAgIwM8//4zAwEB0\nlatV26hRhlMCAHx8fNC7d28AwPHjx6HX6+Wpg2kWJ23GmFtLTn70uH///ti8eTNmzZqFXbt25b9u\nmKFz587SfGlpaQCArKwsJCUlITIyEtu3b0dkZCQaNmyIrKwsad6MjAw0b94cDx48kC/omjWBxESz\n4UG7dOmCzZs3y1cH0yRuPc4Yc1u3bwOVKj16HhUVhWnTplmcNyMjAyVLlnSqnrlz5+Lll19GrVq1\nnFqesXx8yxdjzPNcvmxojF2+vOhIGHMI3/LFGPMsf/9tOLvsNgn74UPRETAV4SNtxphbSUsDSpUS\nHYXMypbl5O0Z+EibMeY5srLcMGEDhoTNA4wwcNJmjLmBmBhArwdKlBAdiYKGDTMMMMI8Gp8eZ4xp\n2oYNwCuviI7ChXiAEXfGp8cZY+q0dOlSVKxY0WIvX2+++SYCAgKKXN44vkaXLkpEp2KJicAPPxge\n8yAjHoePtBljLhUWFuZQd5zR0dFo0aIFqlevLk3z9zdcv87LA3Q2j03clJeXYYxRC/vwunXrIjw8\nHJs2bSr0WuvWrdGpUycMHz7cFVEyx/B92owxdWjfvj22bdvm9PKrV69Gjx49AJgnanXuwhRWuTJw\n86bhsckKqF69OuLj4+0uJjg4GIl8ql1N+PQ4Y0y8pUuXFithA5ASdtmyhG++sXqQ6Rn++cfw5iMi\nAJ0OJfJb4DmSsAFICXvdunVyR8gUwkfajDFFlSxZEhkZGbKVd/nyZdSsWVO28rTu6aefxuH8AUaK\no0ePHli9erUMEbFi4CNtxpg4devWlTVhA0DNmjUxePBgWcvUqh49esiSsAHD5YdRo0bJUhZTDh9p\nM8Y0qV27dti+fbvoMIQhIug8thWe2+IjbcaYGEofDW/fvh23bt1StA41W7ZsmSLlfvXVV4qUy+TB\nR9qMMdlt2rTJbHxqpQQFBSEpKUnxetSmbdu22LFjh+gwmPz4SJsx5nqrVq1yST2emLABYOXKlYqW\nz6fd1YuPtBljmnb9+nWEhYWJDsOt8PVyYfhImzHm3t58803RIbjU8uXLFa9Dp9Ph5MmTitfDHMdJ\nmzEmq7i4OKeWi4qKsvpaUUd9f/75p1P1aVX//v2LtXzDhg2xceNGACiyf/d+/foVqx6mDE7ajDFZ\nFXUbVlZWFjp27AhfX1/odDoYL88NGjQIR44cAQD89ttv0vyBgYE262vXrl0xI9YWa/e9T5s2DTVq\n1AAA7N69Gx07dsSyZctw5swZAMD8+fNx//59nDx5El3yR1kp6h76o0ePyhw5k4OP6AAYY+7Fx8f6\nbuXLL7/Eli1bpISdkZGBkiVLYt68eTh37hw+/fRTTJkyBQBQvnx5pKSk2KxPr9fLFruWJSQkoEWL\nFgCAW7duYcuWLdJrY8eOhbe3NypUqAAAqFy5Mv755x8hcbLi4aTNGJPVM888Y/W1yZMno169etLz\nRo0aoWrVqjhw4AC+/fZbTJkyRUroCQkJyMzMhL+/f5H1/fXXX7LFrgXWLhXcvHkTPXv2hF6vR0RE\nBNLT0zF48GBkZWXhyy+/BADUqFEDiYmJdrW6r1WrlqxxM3lw63HGmMt4eXkhT+YxoOvXry+dAvYE\nM2bMwP/93/9Zff3UqVPo1KmTQ8OfWhITE4M2bdoUqwzmMB6akzHm3rKzs+Hn5yc6DLeSkJCA8uXL\niw7DE/EtX4wx12vatKnL6vLEhL17925Fy69ataqi5TPncdJmjMmuZ8+eLqnn6aefdkk9atO7d29F\ny09LS1O0fOY8Pj3OGJOdry9QtWodXLp0XnQobkup8a8bNmzIHauIw9e0GWOu4+MDuOoOrNKlSyM1\nNdU1lXkSnQ5QZ17wBHxNmzGmrLg4YPZsw+OCCfu7775TpM5KlSpxwgawf/9+Wcu7cuWKIWHv3m34\nY6rDR9qMMad5eQG27uBq1aoV9uzZ45qAPJCPj48sHcwEBgYW7sxmzx6gVatil83sxkfajDF5XbgA\n/PCD4bE9t1zv2bMHDRo0kKVuProuTK/X45133nE6cScnJyM3N9dy73OtWhkaKDDV4CNtxphdFi8G\n6tUDnn3WueUPHjyIZs2aOV1/gwYNcPr0aaeX9wSlSpVyqOX3Y489hrt379qe8e23gSVLDNe7mZL4\nSJsxVjzG26D79XM+YQOQEnbp0qUdWq569eoAwAnbDsaEvX79euh0OkyePFk6As/Ozsa4cePg5eWF\n3NxcALAvYQPAL78YEvbPPysSN7MfH2kzxixatQoICgLatlWm/MaNG6Nv37747LPPCr12/PhxtG3b\nFtu2bUOTJk2UCcCDREbKmG///ttwyoUpgW/5Yow5plQpQEjfGvfuASEhAip2f7ImbQDo3x9YtEjG\nAlk+Pj3OGLOP8XIld4bFbFq0CGjdWnQUHomTNmMeLD4eGDfO8FidJ92Yau3aBUyaJDoKj8PjaTPm\noYwdX33xhehImGYNH25oqZidLToSj8FH2ox5kDt3gBEjDI/5yJrJIjsb8PcXHYXH4KTNmAdYvx7Y\ntAmoWBGYOFF0NMztZGYCCxeKjsIjcNJmzI2VKWP4/+qrQOfOYmNhbm7AAKB7d9FRuD1O2oy5oV27\ngOXLgeTk4pfVvHnz4hfC3MbmzZutv/j77/zrUGGctBlzIxUqGP63bg0888xl1KtXD4MGDUKvXr3w\n8OFD9OnTx2z+H3/8Ea+//joiIiKkXrLmz59v9j8+Ph46nQ5RUVEAgL179xarO9Ki9O3bF6Ghofjk\nk09w7NgxRerwRGfP/gIA6N+/P9577z1ERkYWuT2Y9ppm7O/duD1cvXrVbHvIyspCx44dHxWyaRNg\n0nvdtGnTMGHCBP5M5UJEavxjjDkAFr41lStXJiKicePGmcxnPuPZs2cLTUtOTqaEhATpf8WKFUmn\n00mv16lTR8bITdy9SwMHDiQAFBgYSERE58+fV6YuDzNkyKPPPjs7W5pubXtYtmyZNG3btm1E9Gi7\nmD17ttn2MGzYMMuVLl5MREQ3btygChUq8GdqH5v5kXtEY0yj7t8Hxo8HfvzR9rzp6enw8/ODj4/h\nLs9169aha9euFufNzs6Gn5+f9L+gixcvYvTo0fjvf/9brPgLKdAj2uHDh/H000/LW4eHiowEunff\ngbYmfdLq9XppeyjK3bt38dhjj1ndHo4cOYKePXsiPj7eZln8mdrE3Zgy5o7sGcf6t99+Q69evWSv\n29/fH8nJyRZ34MXC3ZgqpnbtN3DkyH9QxtgyUUZeXl7Is7YxPvkkcPy47HW6Me7GlDF3kZICDBpk\neGzPONZKJGwAyMzMlD9hM0W1b79MkYQNwHrCBgwJW6H2D56KkzZjKte1K7B0KRAYCMybJzoaxhx0\n8CD35CMjPj3OmEpVqgTcvi06Chfi0+OK8PIy5ExfX4G9jV68CNSqJahyTeHT44xpTZ8+wOzZHpaw\nmWKMg3FFRgoMolatR9d2WLHwkTZjKlGjBnDliugoBOIjbcUYB4cRbvVqoEcP0VGoGR9pM6Z2xrOG\nHp2wmSLi4uLwww8/IDT0GiZPnoxTp06JDYgTdrHxkTZzCxcuXMDUqVMRGxuL7OxshIWFISIiAv36\n9RMdmkXp6cAHHwBLloiOREX4SLtYUlJSEBQUhIyMDJQoUcLm/KmpqQgMDERWVpZr7wYoWxZ4+NB1\n9WkL36fN3FeLFi3QunVrTJgwocj5VqxYgTfeeAPXrl1DlSpVXBSddVWqADduiI5ChThpO+Xvv//G\nggULMHnyZKfLKF26NM6dO4fKlSvLGBlzAidt5l4iIyMxcuTIYiXfChUq4P79+zJGZVt2NhARAaxa\n5dJqtYWTtkOysrKwevVqREREyFbm1KlTERUVBZ3OZu4onrffBn75Rdk6tImTNnMPJ0+eROXKlREc\nHCxbmTqdDkpv//37AzNnAqVKKVqNe+Ckbbdu3bph7dq1ipVfqlQppKWlKVY+AMO1oXfeUbYO7eGG\naEz7/P390bBhQ1kTNmAYLOfxxx+XtUyjvDygfXtg0SJO2Hbx8gIee8zQzJkVqVevXoombABIS0tD\nixYtFK2DOYeTNlO1qKgoZGZmKlb+2bNnkZKSgt9++02W8v7v/4AHDww5aNs2WYr0DHfvio5AEy5e\nvCjbtmrLn3/+ib179ypXwTvvADL/EPcEfHqcqdahQ4fwzDPPuKy+wYMHY+7cuU4ty+MiyMDXF8jJ\nER2FKqWnp+P+/fuoWrWqy+s+cOAAmjdvrlwFej1gx2hjHoJPjzNtWrVqlUsTNgDMnTsXGzdudGiZ\nihUN/zlhF8+GDRuwYsAALFiwoOgBKDxUQECAkIQNAM2bN8cbb7yhXAV836ND+Eibqc5zzz2H2NhY\nIXXfv38fFSpUsDnf888D+/e7ICA31qJFCwwYMACDLHRvef36dTzzzDM4dOgQwsLCBESnHiVKlEBW\nVpboMJRtuEnE7RkMuPU405ZmzZrh4MGDQmP417/+hVu3bknPw8KAs2cNDcq4X4jia968OQ4cOGD3\n/KdOnUJwcDBCQ0MVjEqd4uLi0LhxY9FhSOz9Ueuwr74CvvtO/nK1h0+PM20RnbAB4NatW/D19QUA\nVKhg6AglMNDwGids582ZMwcAHErYAPDEE08gNDQU73jg7UGiTolbc1upUWy++84whOc33yhTvhvh\nI22mGuXKlcODBw9EhyEJDARSUw2PvbyA3Fyx8WiZXB3aXL16FdWqVZMhIvUbNGgQ5qlwAPWpU6di\n6NCh8hZavz7w99+Gx+rMSa7CR9pMO9SUsAHgiSdagMiwD+GE7bxp06bJ1gNdtWrVULp0aVnKUju1\nXst39EyJXc6ckb9MN8VH2kwV+vXrh8WLF4sOw0xGRgZKliwpOgxmwccff4zp06eLDkMxQ4cOxdSp\nU0WH4XpeXsCdO57cM57NI22+OY6pQrt27USHUEjJkiVBRMr3w+zGOnbsiC1btshe7vTp05GcnIwy\nZcrIXrYayN37n9zatm2LHTt2WHwtMjISTzzxhHMF//wzvKOjkZvfpkROH3zwgexlisBJmwn3559/\n4s033xQdhkW+vr7Q6/Wiw9AsJRK20TfffIMff/xRsfJF+uqrr0SHUCRbfSi4S4JUI76mzYTr0qWL\n6BCsevfdd0WHoFlKt3x214StBbaGw2XK4aTNhOvYsaPoEKxytltTBsyfP1/xOqydotWyGTNmiA7B\nLjExMaJD8EjcEI0Jd/v2bVSqVEl0GEyDXDG8qqsFBQUhKSlJdBg2Weu5MDIyEj///LOAiNwC3/LF\n1E/phG3rdqP93B+p7KZMmVKs5U+cOCF1b1rUuM4dOnQoVj1qlJyc7JJ6+vTpAwC4du0aAPP2BydO\nnLC5/J9//mlXPfaecTl27FiheStUqIBr165Jt4OG5LcqHzp0KMqXLw/A0C87AAwfPtxsWXc9E8BJ\nm6laXFyclAByc3PRvXt3DBo0CBEREYiKigJg+LJHRkZCp9OhR48eyM2/qfrDDz/EuHHjEBISghs3\nbuCVV17BqlWrAABvvfUWAMOY2s8//7yAd+be/jZ2lGFBfHw8AGDv3r1o1qwZBg0aJO2Mz549iy1b\ntqBRo0ZSxyKlihiQvEGDBvIFrRJeXkXvlo1nFnQ6HVq1aoVhw4YBABo1aoTAwEA88cQTSElJAQD0\n7dsXoaGhaNmyJY7nj2oTmN+9X/Xq1QE8anvQqVMnqXw5x5nX6XR47733EBkZiWnTpqFGjRo4fPgw\nDh8+jH379mHfvn24fv06nnrqKQwcOBDAo54RGzZsaNY24s0338TRo0cxdepUjB8/HgDQtWtX5OTk\nmDUYDQ8PR5s2bWR7D6pCRGr8Y4yIiAybKFGfPn0IAE2cONHmvHXq1JGmBQYGStON87Rt25ZiY2Ol\naRUqVJA7bI8XFRVlcx7j5zRu3DhpWrt27czm+eOPP6hy5cpUuXJli2UMHDiwGFGqk+n2akl6ejoR\nEYWFhZlNnzRpEuXm5tLrr78uTRs4cCABoNdee42IiIKDg6XXRo4cabZ8qVKlpMdZWVlOxzlkyBCz\n5/Pnz6fs7GwiIvryyy8pIiJCem38+PE0fvx42r9/v9kymZmZRPRoG0lMTKRbt24REZGvr6/VWC5f\nvixN+/zzz22+BxWymR/5mjZTvRs3bqBKlSpWXy/YtWVSUhKCgoIKzZeeni6dSgOA7Oxs6PV6s2lM\nHqdPn7Z5FHzx4kWMHj0a//3vf52ux8fHx+1uyRN5nT4nJ0fqd9+W0NBQ3Lx5s9B0ua5p2zs4SXZ2\nNvz8/HD58mXUrFkTgO19horxNW2mfrNnzy7ydVtfvoJ9UVtK2NnZ2cjOzjab5ufnxwlbIfactq5W\nrVqxEjYAvP3228VaXo1EDoxib8IGIH//4wXYO5qYn58fAEgJG7C9z9AyPtJmwoWEhODevXuiw7Do\n7t27eOyxx0SHoUmvvPIKNmzYIDoMzTl79qys15SVotfr4eNTuH8ubj1eLHykzdSvV69eokOwqk6d\nOqJD0Kx+/fopWn5OTo6i5YuihYQdFxdnMWEz5XHSZsLNmjVLdAhWGceAZo57/fXXkZWVpVj57ni7\nl1FiYqLoEIr05JNPig7BY/HpcaYK7thJBgPWrl2Lbt26yV5uqVKlirx/W+vU3sAuLy/P5q1pzCl8\nepxpg9rG0gaAjz76SHQImtetWzcsW7ZM9nLdOWEDkO6zVitO2OLwkTZTjX379qFly5aiw5Bo+LYR\n1cnNzYW3t7csZW3dutWtT40bqfXsk9rPAmgcH2kz7VDT2MgjRozghC0jb29vNGnSpNjlbNq0ySMS\nNgBVJmwAnLAF4yNtpipNmzbFkSNHhMaQmJiI4OBgoTG4q8aNGyMuLs7h5YKDgw2Ns3Q6QJ37LEX4\n+/sjMzNTdBiSsLAwXL9+XXQY7oyPtJm2HD58WHQIGD16tOgQ3FZcXBxiY2OxevVqu5dp3Ljxo9bU\nRICb3upliZoSNgBO2CrAR9pMdWrVWomLF18XUnfJkiWRkZEhpG5PNHHiRIwcuRfDhtXH448/jsTE\nRMyZMwfVqlXD1q1bLffQlZcHeFBDqCVLlgjtJc0oNjYWzz33nOgw3J3NI21O2kxVrl8HwsLENEp7\n/vnneZhOAf7zH+C99xxcaPBgYO5cReJRI9FjbHPPgC7Dp8eZNixZYvgfFmb437JlS7vH65VDeHg4\nJ2xBHE7YgCFhT5okeyxqlZSUhKlTpwqp+3bz5pywVYSPtJlwlSoBt29bfu2tt97C0qVLFa3fy8sL\neXl5itbBFPLFF8C4caKjcBlXN0yrUKEC7t+/D6SlAUWMa85kw0faTN2eftp6wgYgJeyqVavKXjcR\nYc+ePZywtcyDEjZgaJj21VdfKV7PP//8g4ULFxoSNmBI2EOGKF4vs42PtJkwd+8Cjpx1u3fvHn7/\n/Xe859T51EeSkpJQo0YN1ffv7Ak++ACwMTKrbVevAgWGZ/UESp0hKrJTl9WrgR49ZK+TSbghGlMf\nPz+gwNDWDgsKCsKpU6cQZrwIboennnoK0dHRmhhFyVPIdtt1+fJAQoIMBWnPk08+iePHjxe7nOrV\nqyM+Pt72jLdvG65pMSXw6XGmLsHBxU/YgOFoOSwsDCtXroROp0OXLl0QExODrKwsEBGuXbuGCRMm\nwMvLC1988QUA4NixY5ywVWb4cJkKSkgAmjaVqTBtOX78OJYsWeJ0D37BwcHIzc21L2EDgPGUOROC\nj7SZy7RqBezZIzoK5tZycwGZ+jjXsldffRUbNmxA79690b59ewQHByMhIQGbN2/GqlWrMGbMGHz9\n9dfOVzByJDBhgnwBMyM+Pc7Emz0biIgAypYVHQlze/fuASEhoqNQnchI4OefZS70n3+AypVlLtTj\n8elxJta2bYbGRpywmUuEhADca5drVK5saJTAXIqTNlNMYCDQvr3oKJhaTZ6sUMGxsQoVzAohAlq3\nFh2FR+GkzRTRqROQkiI6CqZmn32mYOHceMJ1du0CTp8WHYXH4KTNZGXsWXLzZrFxMPX79lsFC2/V\nSuEKmJkGDTyqL3iRuCEakw33u8AcQeSCS6LnzgF16ypcifop0hCtoORkoEwZhStxe9wQjbmGvz8n\nbGa/O3dc1IYpPNwFlTAAhoTdoYPoKNweJ21WbKmpgAvHMGBu4JVXnFvO0ljn94vq7MPHx6PG3hZu\n61agQgXRUbg13pqZ07780vB/5871ZtM7duyI1q1bIzo6utAyq1evhr+/v9m0gpdo2rRpg/Xrzct0\nhK7AIVyJEiWcLospIynJ0G3mkCFDEB4ejoEDB1qcT6fTYeTIkQCAvLw81KhRQ9q2QkJCMG7cOISE\nhCAzMxN6vR4bNmzA4sWLUa5cuUeF5OUB+cneOM/o0aMVf49qo9frceLECaSlpUGn02HmzJm4cuWK\n2TzGHga/NH65TezatQsAMGfOnELfMbPv8P37wPXrAAyf3/r16/HLL78AMIzax4qHkzZzypIlwNix\nhsevvvoq2rRpI722ZcsW7Nq1C9euXSu0XOPGjZGVlSU9j4yMhE6nQ2RkpDQtPDwcDRs2BAD89ttv\nDsdW5A6FqcKECcDnn3+O7777Dnfu3MH8+fMBAHUtXH9u3LgxAODixYsAIG1b9+7dw6xZswAYhqz0\n8fHB1atX8e233+LIkSPmhbz4IgBI85TywGEmfX190ahRI/j5+YGI8OGHH6JmzZpm8xi/d8b/ALB9\n+3YAQNOmTaVBdky/Y5999lmh7xy2bJHma9iwIUaMGAEA+PXXX+V9U56IiNT4x1Sse/fC09q1a0dE\nROnp6WbTMzMziyzr6tWrZv8L2rt3L02YMMGJKA0ePnzo9LLM9Rz5vHJycixOv3LlCi1YsKDwCzDf\ntVy/ft2h2LRsyBCi7du3m02ztv4KunPnDhERZWVlWXz98OHDVK1atcIvTJxo9vTQoUN21efhbOZH\nbj3O7JaXZxjso8DZbej1evj4+IgJijFmk5Ktx7Ozs+Hn52f5RQ8efc1J3HqcyeOjjwzteQombACc\nsJlDLl8WWLmxIwEmG6sJGzAkbJPLYaz4OGkzm/z8gJ9+Eh0Fcxf57crEGD5cnrFhmf0iIkRH4FY4\nabMivfN/uPhAAAAgAElEQVQO7+OYfHQ6YM0aoHp1gUFs2CCwcg/EvS7JipM2s0ivN/wtWSI6Egfp\ndI/+mCplZwPx8QID6NEDGDZMYAAeaPVq0RG4DU7arBAvL0OfFJq+VJ2XJzoCZsGiRYC3t+goYBhi\nTK8XHYVnGT5cdARugZM2M+PtrfF8Z2wUw0faQk2fPh09e/ZEeHg4atasiddeew2TJ09Gv35ATo7o\n6PKFhoqOwLNMmgSULSs6Cs3jpM0kH34I5OaKjqKYsrKAKVNER+FRjh49ikqVKiHB5Naejz/+GKtW\nrcKlS5dw+fJlrFmzBsPyT0nrdEBaWhrCw8Oxc+dOUWEDAQH8487VHj40DOBiocc1Zh9O2gwDBxqO\nrmfOFB2J47744gt4eXlhyJAhOHv2rGHiJ58gJycH//vf/1C5cmV07twZ6enpYgN1Q8HBwQCAJk2a\n4Pbt2yhfvrzdy5YqVQqXLl3CSy+9BAAoWbKkIjEWqXVrw3+T3viMBg8eDJ1Oh88++wyXTe5RS0tL\nw7x58xAYGIjatWtzb3uO8vYGzp8Hvv/e4ss5OTkICwtDq1atEBMTg0yTQQ0OHDiAqKgo6HQ6rFy5\n0lURqw53ruLhfH1VdLrSTjqdDleuXEF1J5ogV61aFRcvXiz63lJWJC8vL+QpdA2lRIkSZt3cKu6v\nv4DmzQEihISE4MqVKyhdurTDxQQEBGDXrl1o1qyZAkEWn0uG5rTXyy8DMTGGsVlh6PZ4w4YNmOnE\nUcO9e/fQrFmzQn2oaxh3rsKs0+m0lbAHDBiAnJwcEJFTCRsArl27Bj8/P/zrX/+SNzgP0aFDB8US\nNgBkZWXh3XffVaz8Qp59FrFBQQAMCcCZhA0A6enpaNasGf8YtMcffwB16iAjIwNlypRBx44dnUrY\nAKQfWhs3bsTRo0dlDlSdOGl7oOxsYNQo6Yeu6n3++ec4d+4cFi5cCF9fX1nKvHXrFgAgMDBQlvLc\n3csvvwwA2Lp1q+J1LViwAADQoEEDReu5cOEC0tLS8NzDh7KVmZ3fqYHSsWtdpaQklCxZEsnJybKU\n16VLFzRp0gR37tzBvXv3ZClTrfj0uIfp3RtYvlx0FParXbs2Lly4oGgdW7ZsQe3atREeHq5oPVrl\n4+MDvaDbo5Q6FR8REWFx6Fg56fV65OXlqeLoWy2nxxMTE6W2EEoqXbo0UlNTFa9HAXx6nD3i76+t\nhB0bG6t4wgYM43+Hh4ejd+/eitelNd7e3sISNmAYQ9tb5hu7Hzx4oHjCBgw/dvz8/LB06VLF69IC\nLy8vlyRsAEhNTcXjjz/ukrpcjZO2h9DpAJOGmKrXoEEDPPfccy6tc/ny5WatVT3dzZs3kauCewBz\nc3MLj4/tpPfffx/lypWTpSx7vfXWWy6tT40GDx6saFsIS86ePYv27du7tE5X4KTt5tLTDdeu1XkV\nxLK//voLp0+fFlK3v6VhzDzQgQMHEKqizkeaNm1a7CPWpk2bYs6cOTJF5Bgiwrlz54TULdqOHTsw\nd+5cIXVv27ZNVduxHDhpq8yMGTOg0+kwYsQIxMXF4cSJE1iwYAECAgIwYcIEh8rq0kV7/UfUq1cP\nzz77rNAY3Oj2Eac1b95cdAiFFOeIdejQobIdrTtDp9Ohbt26wuoXqW3btkLrv3nzJqpWrSo0BlkR\nkRr/PE5gYKDd844fP54SExOLnKdUqeJG5HqRkZGiQ5AcPXpUdAjCBAUFiQ7Bqnr16jm8TPXq1RWI\nxDlff/21kHqHDBFSLS1fvlxMxdplMz9y63HBlixZgooVK6Jjx44OLztmzBiMHj1aej5gALBuHRAV\npc1eAuPj452+/1oJmzZtQufOnUWHwdxMSkqKy281FNF6/MKFC6hdu7ZrKy1C9erVES90eDm72Dwv\nyklbID8/P+m+TmfpdDqpK0XjafDjx4HGjYsbnWu9/vrrquuaMCkpCUH5HW94CtPtSa0c6TWtbNmy\neCjjfdhyWLVqFXr27Omy+kwvj7nyoz1z5gzq16/vugrt8MQTT+DUqVOiwygKJ221kvPeV29vb3h5\n5UKv11aDM1NEBJ0KL75PmzYNUVFRosNgbubq1auoVq2aS+rq3RtYsQL47jvXnYHTwo8/leL7tNVo\n7Nixst77mpubi+DgippN2DqdTpUJGwDCwsJEh+Ay3377regQ7PbGG2/YnGfQoEEuiMQ5w104trSx\nbwZXXjJTc8KeNm2a6BCKhY+0Bbh06RL3vmXCVb0ksaJVqVIFN27cEB2GXdq1a4ft27cXOU+nTp2w\nefNmF0Wkbq4cGCg5ORllypRxTWVOqFatGq5evSo6DGv4SFttnn76acUStlY7cVB7wp4xY4boEFxC\nKwkbgM2EnZeXp/qEXdTZJbnPPMmdsIvq31vt7UBUnLDtwknbxQ4fPqxY2VrsLlHNpzCNxowZIzoE\nZkG7du2svubI2N6i/PHHH6JDUIRxMB6mDE7aLvTSSy8pXkelSpUUr0NOyzXQGbonDPm3ceNG0SE4\n7MyZM1Zf6969uwsjcU6bNm1Eh6AILeyDPv74Y9EhOI2Ttgtdu3ZN8TqaNWumeB1y+uyzz0SHYJNb\n9aZkhahuJotjyJAhVl9zZUMvpj1avuTFSduFijsWsemvQ2sDW6xdu7ZYdbiaq37xTp8+vdA0R0Z6\nun79upzhqM769etdXufixYsLTXPkMxk8eLDV11zVZej8+fMBAN9//7007eDBg9JjOX4M5RRxQXrs\n2LFmzwt2wTt79mwAQFZWFmJjY6XpGzZsAPBoP2KMPzo6WvoMjMsap6tJdHQ01q1bh0OHDkmxmca+\nbNmyIpevXLmy4jEqxp5u0wT8eZRz584REVFKSgqdPHmSiIjGjh0rvX7hwoVCy9jqxtRdTJgwgYiI\n0tLSSK/X0/z584mIaNq0aURENG/ePJo6dSoRkbTu+vTpIy3v4+NDderUkZ7PmDGDiIhGjRpFixYt\nIiKikSNH2oxj2bJlxXwn6mbYFVi3adMmunHjBqWnp9PPP/9Mt2/fpqSkJCIi2rJlC61bt46IDOtp\n2rRp9Msvv1BmZibNmTOHiAzbsPFzIiLq16+f9DgrK4uIDNt07969ici+z6Q4jNtVbm4uERH9+eef\nREQ0evRoIiI6depUoe2qTJky0vI+Pj6Ul5cnPU9ISKBGjRoREdHGjRuJiKRttSimZZiy9nncv3+f\n0tPT6caNG7Rr1y5p/ZqqX78+ERHdu3dPeh4fHy+9bnzPRIZ1/tFHH5nVOWvWLPLy8pKWBVAozrt3\n71qMz7iurDHu686cOUNTpkyRYklOTqZr167RunXrKDk5mYiIduzYQWPGjKGYmBhp+8rJySlUpun7\nadeuHZUtW7bIGIhI2s5UyGZ+FJ2cOWkT0YMHD6h27doUEBBApfI7DR+S31nwc889J81X8Mvm7nJz\nc2n69OlERHT27FkCQEuXLqWnnnqKiAw7mSpVqlCTJk3Mllu4cCEdO3aMtm7dSkRE4eHhtGrVqkLl\np6WlEZF9CWLevHnFei9q5+PjY3MeALRgwQKz582aNTObp0WLFgSA+vbtK03T6XSFyvruu++IiEiv\n10tJ25SSSdt0uyIi6tu3L4WHh9PatWuJiOiTTz6h+fPnS4nCqGrVqmbbFRHRiy++SEREc+fOpW++\n+YaIDD8ajcva+jFk/OFTkKXlunbtSnXr1qUzZ86YvZ6eni79ERF16NCBiIhiYmKIiKhEiRKFyjJ+\n3omJidSqVSuzOlu1aiU9Ni5bpUoVs+WtJe3Y2FiL042M+7qwsDBp2rvvvkubNm2iJUuWmM07cOBA\nunjxovTc2vgMoaGhZs99fX3N9pWWDBw4sMjXBbKZH/n0uAr06NEDCQkJaNSokdTyslevXgCA/fv3\no1u3bsjIyICX16OPyxPGff73v/+NixcvAoDUbWV2drZ0CmzhwoW4ceMGEhISpGWICLGxsXjyySfR\nvn17dOzYERcvXsTIkSPNyo6KipJOJT548MBmLK7qvUoUW539bNq0CYCh32xTcXFx0jjJixYtwuXL\nlwE8WqdEhLy8PHTr1s1suNWpU6cCMPTmZ5x38uTJUqcc9nwmzjJuV8buTRMTE5GdnY2uXbsCACZM\nmIDFixfjt99+k5YhIty6dctsu9qxYwdef/11AMB7772HyZMno3PnzmjRogVeffVVxMbGol69ekXG\n4sj9zOvWrUPv3r0LdQ1asmRJ6e+3335DQEAAgEcNXzMzM/HEE09I+wwiki5NZGZmYvfu3QAgjYK2\ne/duqY/uzMxMHDt2zO7Tyba+J8Z9XVpamjQtMTERvXv3RmpqKgAgLS0NDx48QHZ2trQdXLp0CcnJ\nyfjxxx8BwGycbOP32DitT58+Nsfu1kAf5FZx5youNGXKFHz66adWX587dy4CAwPRp08fp+vYsmWL\nU4OPiBIbG4vnnntOdBgeLygoCElJSaLDcMj+/fvx/PPPiw5DEWrvBvTevXsICQkRHYbTVLx+ue9x\nNfH19S2yUYkcVLwxWtSyZUvs27dPdBgez9YPSjVq3749tm3bZvG1n376CR999JGLI5KP2r/HWk/a\nL774onSGQWU4aavJ+fPnUadOHUXrWLNmDV577TVF65CT2ndOADBx4kSMGDFCdBisgKK2nVKlSpmd\ngtUatX8vtJ60Dxw4gObNm4sOwxLuxlRNlE7YADSVsAHg77//Fh2CTdwjmjoZr7NbooUfWZ06dRId\ngiKGDRsmOgSbVJqw7eIjOgBP0717d/z++++KlF2/fv0ie4lSo8cff1x0CDZlZGSIDsElNm7ciC5d\nuogOwy5XrlwpMul99dVXSElJQWBgoAujckzDhg2LfF3NYwkYG4RZovYf4seOHcNTTz0lOgyn8elx\nAby8vGy2bnTUvn370LJlS1nLdJU2bdogJiZGdBgW5ebmwtvbW3QYLlG3bl2cO3dOdBh2KVmypM0f\nUwEBAUhPT3dRRI7Zs2cPWrVqJToMj1SnTh2cP39edBjW8OlxNcrLy5P9epWPj3ZPmqj5x0aJEiVE\nh+AyWknYAOxq6a7WhA0AnTt3Fh2Conbs2CE6BKtUnLDtwklbEDmH3svNzdX0NZpvv/1Wtbfu2Lp/\n2d2sWbNGdAg2nTt3Dn5+fnbNO23aNIWjcY7xnmR35YpxFpzx3nvviQ6h2DhpC2Ic66A4tx0Yj4zc\n4fTt/v37RYdQSOnSpUWH4HJaeM+O9OHfv39/BSNxjq+vr+gQFDdgwADZxwSXw5w5c0SHUGyctAUY\nOBCYNcvw+MUXX8TKlSsd/uUdFBTkskERXEVtX3J3PxqypG3btphl3DhVaPfu3Q7dfx0UFITHHntM\nwYgco9frFe+rQS3U1l5q5syZZr1KapX234HG9O8P5A8MJHn99ddRunRpu5KWTqfD/fv3Ndd7lT3U\n9CVXc6tjpVWoUEF0CFY502Dx7t27CkTinLZt24oOwaVq1KghOgQAhvuyP/zwQ9FhyIJbj7tQ377A\nr7/aP/+RI0eQl5eHJk2auMUpcHuooRvWMmXKIDk5WWgMolWpUgU3btwQHYaZFi1a4M8//3Rq2UGD\nBmHevHkyR+SY2rVr48KFC0JjEOHll1/GH3/8IToMreAe0dSid29g+XLRUWhDo0aNcOLECSF1BwcH\nIzExUUjdalOxYkXcuXNHdBgAgCeffBLHjx8vVhm7du1C69at5QnIQUrc5qklGzZswCuvvCKk7osX\nL6JWrVpC6nYC3/KlBj16cMJ2xIkTJ4Qc5ZUvX54Ttok7d+5g9OjRosNAdHR0sRM2ALRu3Vqxjo2K\nUk6n8+iEDQCvvPIKVq5c6dI6U1JScOTIES0lbLvwkbbCunUD1q4VHYU29e/fH4sWLXJJXR9//DGm\nT5/ukrqYY4hI1kaK9erVc1mvXU2bNjUMeVmnDqDx+4Pl0LVrV6xbt84ldT18+BBly5Z1SV0y4iNt\nkTp35oRdHIsWLUJOTo7ZeNlKmDVrFidsG5Qc39oaYytrue8q+Pvvv3H16lXF7+GuW7euNEY1zp8H\nwsIUrU8L1q1bp3hr/qFDhwIAytaurWg9onDSVkj79kAR4xkwO/n6+qJ8+fKYNGmS7K3Ly5UrBwAY\nYrxpnlk0caJhXZUsWdJldXp5eSl6P3O1atUQFRWFN954Q/ayjbcVFeph7vp1QHtHfrIztuYfPny4\nrOXGxMQgKioKU6dONUy4d89w5ORm+PS4Atq0AVTalbbmTJoEmH63Bw8ejI8++ggNGjRwukxubGa/\nmjWBy5cfPU9LS0N8fHyx1n9Rbt68CW9vb1SsWFGR8q1p3bo1evTo4fQY3BkZGQgICLDvh6W/P5CZ\n6VQ97sjHx6dYPQ+OGTMG7du3R4sWLSzPoNMB6sxzltg+rUREavzTrFatREfgPo4csf7a8uXLqXHj\nxnaXtX79eoooVUqGqDzHDz9Yfy00NFT2+urVqyd7mY6aPHkydejQwe7558+fT3Xr1nW8olGjHF/G\njeXl5REAWrZsmV3zZ2dnk7e3N/3nP/+xr4L09GJE51I28yMfacvo+ecBFfbG6TH27duHAwcOICcn\nB6GhoejZs6d5t5y3bwOVKokLUENq1ACuXLE9X6tWrTBv3jyne+e7efMmOnTogJMnTzq1vCvExMTg\n8OHDiI8vixdeCESvXr3kGaBHW0eALnf69BUcO7Yfly5dQrly5fDyyy8rdoZHRfhI21WaNRMdgXsJ\nCREdgecaO9a55X799VcCQIsXL7Y6T3R0NAGw/whJRXbuVKBQX18FCnUPCxbIXGBMjMwFKoKPtF2h\naVPA2EiUFZ+iByD16wNnzihUuPb98w9QubJChR84AGh4NLpduwBF+mYJDARSUhQoWNs6dAAcGBvG\nPg8fqr0xIN/ypbSnnuKELTdFf0c+95yChWvfvXsKFi77HthNpKQAp0+LjkJ1du1SoNCyZQGNDxqi\n7egFa9gQOHZMdBTu5fPPFa6g4GgtTNK/P/DkkwpWsGSJgoVrXIMGQHi46ChUJTtboYLz8oCICIUK\nVx4nbSfVrw+ouO2MJj3/PPD99y6oaM0aF1SiLWXLAop3PmdPyzZPdukS8PTToqNQje7dFSw8OtrQ\nMFWDOGk7oW5dviyqBJe1vFdBf9pq8v77hkt9ivOwYSmdcvgw0KmT6ChUoXdvhSuoVAlYvVrhSuQn\nw30LnqVWLeDiRdFRuB+XtsWJi3NRRerXogXg5GiXjuvQwUUVadzmzYZTuBq/9lpcL73kLpXIy7O3\nCgfVqMEJWwleXgIazzZr5uIK1WfFChcmbAB45x0XVqZxXl4uaOChbgp3UW5QrhzwwQcuqEg+nLTt\nVLWqfJfk7t+/L09BbuDhQ8NBhaPS0tKKV/GnnxZveY1LSwP+/W9nlivGeg8JcX5ZT/T998BPP4mO\nwv3Nng3I0VmOi3DStkOVKsC1a4Ber8eJEycQHh6OgQMHYubMmbhSIJOvXLkSI0eONJvWunVrREdH\nY1f+PQwhISHINOl7ePHixdLgFQUVHOHorbfekuEdqYeXF1C9enUAQGpqqtX5pEEAAOTl5eHtt9+W\n1n9IfjIw9qql1+uh0+lQsmRJdO3a1XKB+RfM9Ho9NmzYgKCgIJz3kKETly4FSpUC1q9fbza9Y8eO\nheY1Xe/+/v54++23pf61C/bx0KZNG+h0OowaNQpHjx61WLfpdp+UlCT7CF6u1LFjR+k7XVBGRgZK\nlChRZE9vOp0OTZs2RXJyMoDC6xMA8NFHQP441Mb5f/nlFwDuty8oqGnTpmbPDxw4IO1LC1q9ejX8\n/f3NplnaPtevXy+tbzN6vXQ7qE6nw/r165Gd33y9RIkSxXkb8rOnBxYBf6rxr389emxYXUSlTPqw\nNk4zOnv2LEVHRxcqZ+LEiZScnEwJCQmFlqlRowZdunTJYv06nc7secFltaxLF8P/n3/+md588036\n5JNPpNe2bdtmNu8Ck+6Rzp07RxEREUT0aH14e3vTuXPnpHkA0JEjR+iVV16xHsD27URENHPmTAoM\nDCzWe9GSrKxHj0+dOkVJSUnS86LWOwBpvXt7e5v9JyIaOHAgAaCcnBz64IMPbMah1+vpu+++c+o9\niLBzJ9Hu3bvN1pfxO23J448/Lm2TderUMftPZFif77zzDhGRxX2GJC/PbP5/5e+U3GlfYM3u3bsL\nTZs4cWKhaRcvXjRbH8b1PGTIEGnawIED6cqVK0RUxPpes4Z0Oh1duXKF5s6dS0REvq7ttc5mfhSd\nnFWdtB97zPz59vydvFFOTo5D5WWZ7i1NXLlyxWznaM3Zs2cdqk+Lrl69SkREd+7cISKi999/3+q8\nxvVv7XMYNmwYbdy40XplNWuaPc3MzHQkVE3assX8+YMHD4iIKD1/QAXjel++fLnTdWzbto22bt1K\nEyZMKHK+vPxkpBU7dxKNHDmSiAw/dogefadtra+HDx+a/TeVmZlJMTExVK1aNesFZGYSHT8uPT10\n6JADkWuXcX3Hx8ebTb97926RyxnXs3F/Ysq4vrOzs23Wb+nzUhh3Y+qskBCFe4fycOPHA6NGiY7C\ns7RvD2zb9ui5Xq+XZ+ALR+j1mrp+aEqxbkwdce2aoYGNmztyBHjiiSwxp6ZzcgAFx3K3gbsxdUb5\n8pywlfTiiypK2AkJoiNwiRIlzBM2ANcnbADIvx7LnFS1KvDggegoFLd8ucBryb6+gElbDrXhpF1A\nuXIesx8XIiMD2L1bdBQmqlQRHYHiAgKArCzRUeTj/seL7+5d0REoTpF+xx0xdCjwwguCg7CMk3a+\n9u2BoCCV/YjV6Qx/LunbU3m+vkBSkugoTMyeDWRmGtaxG/LyMnQFmZ4uOpJ83t6Gm8M1uL51OkM/\nHMOHi44Ehi4ZY2OB3FxNrktbdDrg0CHD5iLU3r1AmTKqW8d8TRuGS2y5uYbrVTt3urJmG4wbizo/\nI4dMnAgY74RT1dtxo3VckOreWk4O4OcHnD8P1K4tOhqHqG5dAoZfZYbWxKIjkdW2bYbO8/R6wYl7\nxYpHfanm5bkqedusxOOTtl7/qM3B2bOGH7GqUaMGcPOmis5tOk+VOz0AOHECaNxYhYEVT4kShlGS\nunYF1q4VHY0JRQdLV0758kBioopCJ3rUzWm1akB8vNBw5KaazeS//wX69jV8oUz6GFAQJ21bdDrD\nNezgYFfV6Jm8vJzr+cwl/PwUHAdQDNXs9Aoq2IRdQ7p2BdatEx2FBar9sJ3XsiWwb5/oKEx4extO\nxypPu63HX3JRR+5E8ibs6dOny1eYzHbs2CGs7uIk7MjISPkCsUShhK143EVwZh9+z8otE7J+F12Y\nsOXeh7gqYc+aNcuxBVyQsO3tuU6uHu6UTNhOxahwwnZkX6HapM0YY4wxc5y0GWOMMY3gpM0YY4xp\nhCaSdk5OjtXXhhe4cXL27NlmzxcvXgwAZqNxRUdHIz3/5tWDBw8CANbmN7HduHGjNIrMlClTpGWM\n8znijz/+sPpa37597Yp727ZtyMu/IGyM8dq1a1i6dCkAWBxlyBh/dHR0oVHI7DF06FCL05OTk5GY\nmGgxTqO7+R0/GN9Pamqq2ee3YcMGAMCKFSsAAN+b3IMen98C1tl2AdbinjVrVpFxX7lyRYrbOP3h\nw4fIzb+OZdwOYmJipGWMcZt+FqbbiyMGDhxocfrevXsLTYuNjZUeZ2VlSduGkWmMI0aMkKb9888/\nAMzf94QJExyOtajv4vvvv2/2vOC2Z9wmTKd//fXXAAzbqul2CwApKSnS99S4bk3XtyOK+i46sw+J\ni4vDjRs38Ndff0nTjDEuXLiw0LRvvvnG4ZgBw/u1puA+ZFuB9gLGbdQ0buN3a+HChdKoeuPGjQMA\ns/dibfQyRxS1rdi7/zMVHR2NNWvWWPweqiXGYcOGFZpmzDem38NitX2yp4NyAX/UunXrQj2pGwfV\nMI6c07lzZ/rnn3+k1+/du0dERPXr1yciIi8vL7Pn9evXp1GjRlGNGjWIiGj16tU0e/ZsQy/t+SPE\nAJAGMsjNzS00ny3Tpk2THp8+fZqIHg00smnTJiIyjGq1du1am3EbrVixolCMRh999BERPepY3zT+\nggoOeGKNsfwxY8YQEdGnn35KREQ+Pj5m8xWMs1+/fmbT69evTwAoJCSEiMzXJwAp9hUrVlDp0qVp\n4sSJFBYWRkREZcuWlco1HanHnrh37NghxZ2SkkJz5syxGGdBBd+PcYQwY9xERPv27ZPqWbFihdln\nYTqfI3FXqVKFiAzbhTFuIqLXXnvNbD7TuogMgygYtw3j52+M8f79+2bzvvDCC2bvLzs7mypWrGg2\nj7VBGOz9LpquZ3u+iwXNmjWLEhMTqXfv3tK0Z555htLS0oiIKCMjw6GRrUzjNn4XjXEvWbJEitue\nfYjp9GHDhknzN2vWjBo1akSlS5eWpi1dupT+97//EZH5NmFv7MbtwNTYsWOJ6NH61ul0ZvsQ43ep\n4D7CuH3Ur19f+m4ZLVmyxOwzN74X0/Icjd3SfKb7vxs3bji8/yOiQttFwe+hI4qKMT093aEYLW3H\nBRnzDZHhe1jwcyAy21fYzI+aONI2OnLkCH799Vc89thjqFOnDurWrYvQ0FBkZGQgIyNDGrs2LCwM\nAKSjkEuXLknTx40bh8uXLwMAunfvXqgOIkLV/A75jb9yLc3niN9//x3NmjVDp06d8L///Q99+/ZF\n165dC8VtjNMYt/FIz/h+TGO01gLSNH4qRqvSefPm4c6dOwCAAQMGSHEZY87IyCgUZ3h4uFn8ly5d\nwoQJExAQEAAA8PLygq+vr7Q+jx8/DgA4d+4c4uLiMGLECBw6dAirV6/GU0895VTcoaGh0hjQAwYM\nQOnSpTF48GCzuI1x5hZoEVrw/WzevBmAYTvYuHEjAKBly5bwzu/x4dy5c2afRVFHRbZUrlwZp0+f\nluIGgCZNmpjF7V2gp4lq1aoV2jaMMZYvX97s/aWkpJi9v+K28j1y5Ahq1apl9l00Xc/2fBcB4N13\n31gcpC8AAAQ1SURBVJXKXLZsGcqVK4dly5ZJ0+bOnYuAgACEhobC39+/yG3f3rh//fVXvP3220Xu\nQwp+F43balhYGCZPngwA+PLLL+Hj44MePXogLi4OoaGhAICdO3fi+eefB2C+TRTn+3jz5k00a9ZM\nWt9EZLYPMcZnZLqNGuM2frcAoF+/fpgxYwZCQkKkM3jG9yI30/1flSpVHN7/AZC2i99//x1A4e+h\nXDHGx8cXGaO17dmUcV9hZMw3gOF7aPo5OMWezC7gz+Kv+++//56Sk5PppZdeouDgYPr3v/9t8ZdN\n9+7d6eTJk2bDua1YsYK6d+9OH3/8sdm8BY+0iYieffZZ2rx5s8X5bDE90o6NjSUioubNm9PMmTNp\n48aNNGnSJDpz5ozV5T/77DOzuPPy8qSxYY0xpqWl0cqVK4mI6L333iOiR0NYHj16lJ599lkiIgoK\nCjIr25Ej7czMTDpz5gydOHGC9uzZQykpKRaP9Ixat25NwcHB9J///MdseqVKleibb76R3suvv/5q\n9l6M/P39KTU1lYjMxxwmcuxIOywsjJYvXy7FnZycXOgIMjg4WHq8f/9+WrFiRaG4T548SS+88ILZ\ndpCWlkYzZsyQjoSN01auXFloe3Ek7nLlytGsWbPo6NGjZuu74PKffvopZWRkEJFhfTdo0EBaV8bP\n3xgjEdGMGTPop59+orS0NJowYUKh4Q2Lc6T9/fffU69evYr8Lhq/c9a+i0amR6M//PCDdKbo2Wef\npWnTpklD0q5cudJs27fFNG7jd9G4D0lKSipyH0JEVuM2DsuZk5NDs2bNotKlS5O/v7/Z+2nUqJHZ\nNjFjxgzpzJUtlo60d+7cSTNnzpTWd8HvD5Hh+3XkyBE6efKk2TZqur6N20tqairduXOHsrOzacaM\nGZSdnS29F6JH+xUjS/VZYmk+0/0fgCL3f0SFh+EkMt8uLH0PHVFUjDD0EWI1RuN6tBSjpX2A8cyI\nMW7jcLVF7ONs5kfRydmhpK0FpklbbexN2mpjb/JTG63F7UjS1gKtxm0paYtWnKStNmqM0W1PjzPG\nGGOejJM2Y4wxphGctBljjDGNUO2AIXL1YasUa+uN41YGx628hIQEBFvoiF+NsZrS0jo2pbW47ckV\naopdo+uXR/lijDHGNMJm0vZxRRROUOdPIcYYY0wgvqbNGGOMaQQnbcYYY0wjOGkzxhhjGsFJmzHG\nGNMITtqMMcaYRnDSZowxxjSCkzZjjDGmEZy0GWOMMY3gpM0YY4xpBCdtxhhjTCM4aTPGGGMawUmb\nMcYY0whO2owxxphGcNJmjDHGNIKTNmOMMaYRnLQZY4wxjeCkzRhjjGkEJ23GGGNMIzhpM8YYYxrB\nSZsxxhjTCE7ajDHGmEZw0maMMcY0gpM2Y4wxphGctBljjDGN4KTNGGOMaQQnbcYYY0wjOGkzxhhj\nGvH/DaSDseFYqwcAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x11b491630>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "xgb.plot_tree(model_xgboost, num_trees=39)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Prediction using xgboost"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "xgb_predict = model_xgboost.predict(xgb.DMatrix(X_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1546,)"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xgb_predict.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Run the model for 200 trees  \n",
    "2. Using that model, predict and observe the AUC on test\n",
    "3. Now, predict only using the first 120 trees from that model. *Hint* : Use `ntree_limit` option  \n",
    "4. Use `xgboost's` cross-validation method. Sample code:\n",
    "\n",
    "```xgb.cv(parameters, train_matrix, num_round, nfold,\n",
    "       metrics={'error'}, seed = 0,\n",
    "       callbacks=[xgb.callback.print_evaluation(show_stdv=True)])```\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Early Stopping**\n",
    "\n",
    "Source: [xgboost docs](http://xgboost.readthedocs.io/en/latest/python/python_intro.html)  \n",
    "\n",
    "If you have a validation set, you can use early stopping to find the optimal number of boosting rounds.\n",
    "\n",
    "`train(..., evals=evals, early_stopping_rounds=10)`\n",
    "\n",
    "The model will train until the validation score stops improving. Validation error needs to decrease at least every early_stopping_rounds to continue training.\n",
    "\n",
    "If early stopping occurs, the model will have three additional fields: `bst.best_score, bst.best_iteration and bst.best_ntree_limit`. Note that train() will return a model from the last iteration, not the best one.\n",
    "\n",
    "This works with both metrics to minimize (RMSE, log loss, etc.) and to maximize (MAP, NDCG, AUC). Note that if you specify more than one evaluation metric the last one in param['eval_metric'] is used for early stopping."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### References\n",
    "\n",
    "- https://www.analyticsvidhya.com/blog/2015/09/complete-guide-boosting-methods/\n",
    "- https://www.analyticsvidhya.com/blog/2015/05/boosting-algorithms-simplified/\n",
    "- https://xgboost.readthedocs.io/\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [conda root]",
   "language": "python",
   "name": "conda-root-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}