train-model.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "25f705fb-dd83-4242-afd5-f791fe198e1c",
   "metadata": {},
   "source": [
    "# Titanic: Survival Model\n",
    "\n",
    "Build and train a model to predict survival on the Titanic using a [cleaned and split dataset](https://huggingface.co/datasets/jamieoliver/titanic-2410), and upload the model to Hugging Face.\n",
    "\n",
    "Based on https://github.com/fastai/course22/blob/master/clean/05-linear-model-and-neural-net-from-scratch.ipynb using the dataset from https://www.openml.org/search?type=data&sort=runs&id=40945&status=active.\n",
    "\n",
    "Plan\n",
    "- [x] Download [cleaned and split dataset](https://huggingface.co/datasets/jamieoliver/titanic-2410) from Hugging Face\n",
    "- [x] Prepare data for model\n",
    "    - [x] Load dataset splits as PyTorch tensors\n",
    "    - [x] Normalise dataset splits\n",
    "- [x] Train linear model\n",
    "    - [x] Set up coefficients\n",
    "    - [x] Set up gradient descent step\n",
    "    - [x] Run training loop\n",
    "- [x] Train neural network\n",
    "    - [x] Set up coefficients\n",
    "    - [x] Run training loop\n",
    "- [x] Train deep neural network\n",
    "    - [x] Set up coefficients\n",
    "    - [x] Run training loop\n",
    "- [x] Recreate as PyTorch module\n",
    "- [x] Tune model hyperparameters\n",
    "- [x] Test model\n",
    "- [x] Upload model to Hugging Face"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "813bd16e-9fbd-4d9d-a400-fabc8ff7f2e6",
   "metadata": {},
   "source": [
    "##  Download Dataset from Hugging Face"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "928b3381-2e66-47ea-b937-b848141f0cf4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DatasetDict({\n",
       "    train: Dataset({\n",
       "        features: ['survived', 'name', 'age', 'sibsp', 'parch', 'ticket', 'fare', 'cabin', 'log_fare', 'pclass_1', 'pclass_2', 'pclass_3', 'sex_female', 'sex_male', 'embarked_C', 'embarked_Q', 'embarked_S'],\n",
       "        num_rows: 1047\n",
       "    })\n",
       "    validation: Dataset({\n",
       "        features: ['survived', 'name', 'age', 'sibsp', 'parch', 'ticket', 'fare', 'cabin', 'log_fare', 'pclass_1', 'pclass_2', 'pclass_3', 'sex_female', 'sex_male', 'embarked_C', 'embarked_Q', 'embarked_S'],\n",
       "        num_rows: 131\n",
       "    })\n",
       "    test: Dataset({\n",
       "        features: ['survived', 'name', 'age', 'sibsp', 'parch', 'ticket', 'fare', 'cabin', 'log_fare', 'pclass_1', 'pclass_2', 'pclass_3', 'sex_female', 'sex_male', 'embarked_C', 'embarked_Q', 'embarked_S'],\n",
       "        num_rows: 131\n",
       "    })\n",
       "})"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from datasets import *\n",
    "\n",
    "datasetDict = load_dataset('jamieoliver/titanic-2410')\n",
    "datasetDict"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "50258aa3-5330-4970-a655-2e9eacd3c502",
   "metadata": {},
   "source": [
    "## Prepare Data for Model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "04c91b29-6e0c-4952-ab70-59eb24254d5e",
   "metadata": {},
   "source": [
    "### Load Dataset Splits as PyTorch Tensors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "7d762c5e-33d5-4f62-a8a8-5b6327c26e43",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "from torch import tensor\n",
    "\n",
    "torch.set_default_device('cuda')\n",
    "torch.set_printoptions(linewidth=120, edgeitems=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ee93576a-78cf-4ce4-840b-c967435777e7",
   "metadata": {},
   "source": [
    "The dependent variable is the variable we are predicting i.e. `survived`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "ca8bd00a-69b5-4923-bb8e-39612871d383",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'train': tensor([1., 0., 0., 0., 0., 0., 0., 1., 0., 0.,  ..., 1., 0., 0., 0., 0., 1., 0., 1., 0., 0.], device='cuda:0'),\n",
       " 'validation': tensor([1., 0., 0., 1., 0., 1., 0., 0., 1., 0., 1., 1., 0., 1., 0., 1., 0., 1., 0., 1., 0., 1., 0., 0., 1., 0., 0., 1.,\n",
       "         0., 0., 1., 0., 1., 1., 0., 0., 1., 0., 0., 1., 1., 1., 0., 1., 1., 0., 1., 1., 0., 1., 0., 0., 0., 0., 0., 1.,\n",
       "         0., 0., 0., 1., 0., 1., 0., 0., 0., 1., 0., 0., 0., 1., 0., 0., 1., 0., 1., 0., 1., 0., 1., 1., 0., 0., 1., 0.,\n",
       "         0., 1., 0., 0., 0., 1., 0., 0., 0., 1., 0., 0., 1., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "         0., 0., 0., 0., 1., 0., 0., 0., 1., 1., 0., 1., 0., 0., 0., 0., 0., 0., 1.], device='cuda:0'),\n",
       " 'test': tensor([1., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 1., 0., 1., 0., 1., 1., 0.,\n",
       "         0., 1., 1., 1., 0., 0., 0., 0., 1., 0., 0., 0., 1., 0., 1., 1., 0., 0., 0., 0., 0., 1., 0., 1., 0., 1., 1., 0.,\n",
       "         1., 1., 1., 0., 0., 0., 1., 0., 1., 1., 0., 0., 0., 1., 0., 1., 0., 0., 0., 1., 0., 0., 0., 1., 0., 1., 1., 1.,\n",
       "         1., 0., 1., 1., 0., 0., 1., 1., 0., 0., 0., 1., 0., 0., 0., 0., 0., 1., 0., 0., 0., 1., 0., 1., 1., 0., 0., 0.,\n",
       "         0., 0., 0., 0., 0., 0., 1., 1., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 1.], device='cuda:0')}"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dependent_var = {split: tensor(dataset.to_pandas()['survived'].values, dtype=torch.float) for split, dataset in datasetDict.items()}\n",
    "dependent_var"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "ef193d95-c2f7-47cb-9798-7c4a49a1fd8d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'train': torch.Size([1047]),\n",
       " 'validation': torch.Size([131]),\n",
       " 'test': torch.Size([131])}"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "{split: var.shape for split, var in dependent_var.items()}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "403fedd0",
   "metadata": {},
   "source": [
    "Transpose the dependent variable into a column vector."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "0e0530e4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1.],\n",
       "        [0.],\n",
       "        [0.],\n",
       "        [0.],\n",
       "        [0.],\n",
       "        [0.],\n",
       "        [0.],\n",
       "        [1.],\n",
       "        [0.],\n",
       "        [0.]], device='cuda:0')"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dependent_var = {split: var[:,None] for split, var in dependent_var.items()}\n",
    "dependent_var['train'][:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6faac882-da41-45a9-a4b2-9cc35fbcfb9a",
   "metadata": {},
   "source": [
    "The independent variables are the variables we will use to make the prediction. Note that we use a trick in mutiplying the Pandas DataFrame by 1 to convert booleans to integers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "18bdd66b-2dd7-4d0e-9434-d4b185d39558",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'train': tensor([[ 4.0000,  1.0000,  1.0000,  3.1781,  0.0000,  1.0000,  0.0000,  1.0000,  0.0000,  0.0000,  0.0000,  1.0000],\n",
       "         [20.0000,  0.0000,  0.0000,  2.1889,  0.0000,  0.0000,  1.0000,  0.0000,  1.0000,  0.0000,  0.0000,  1.0000],\n",
       "         [32.5000,  0.0000,  0.0000,  5.3589,  1.0000,  0.0000,  0.0000,  0.0000,  1.0000,  1.0000,  0.0000,  0.0000],\n",
       "         [23.0000,  0.0000,  0.0000,  2.7754,  0.0000,  1.0000,  0.0000,  0.0000,  1.0000,  1.0000,  0.0000,  0.0000],\n",
       "         [47.0000,  0.0000,  0.0000,  3.9703,  1.0000,  0.0000,  0.0000,  0.0000,  1.0000,  0.0000,  0.0000,  1.0000],\n",
       "         [30.0000,  1.0000,  0.0000,  3.0910,  0.0000,  1.0000,  0.0000,  0.0000,  1.0000,  0.0000,  0.0000,  1.0000],\n",
       "         [40.0000,  1.0000,  0.0000,  3.2958,  0.0000,  1.0000,  0.0000,  0.0000,  1.0000,  0.0000,  0.0000,  1.0000],\n",
       "         [18.0000,  0.0000,  2.0000,  4.3901,  1.0000,  0.0000,  0.0000,  1.0000,  0.0000,  0.0000,  0.0000,  1.0000],\n",
       "         [32.0000,  0.0000,  0.0000,  2.1856,  0.0000,  0.0000,  1.0000,  0.0000,  1.0000,  0.0000,  0.0000,  1.0000],\n",
       "         [16.0000,  2.0000,  0.0000,  2.9444,  0.0000,  0.0000,  1.0000,  0.0000,  1.0000,  0.0000,  0.0000,  1.0000],\n",
       "         ...,\n",
       "         [40.0000,  1.0000,  1.0000,  4.9090,  1.0000,  0.0000,  0.0000,  1.0000,  0.0000,  1.0000,  0.0000,  0.0000],\n",
       "         [21.0000,  0.0000,  0.0000,  2.1719,  0.0000,  0.0000,  1.0000,  0.0000,  1.0000,  0.0000,  0.0000,  1.0000],\n",
       "         [24.0000,  0.0000,  0.0000,  2.1077,  0.0000,  0.0000,  1.0000,  0.0000,  1.0000,  1.0000,  0.0000,  0.0000],\n",
       "         [28.0000,  1.0000,  1.0000,  2.7344,  0.0000,  0.0000,  1.0000,  1.0000,  0.0000,  0.0000,  0.0000,  1.0000],\n",
       "         [21.0000,  1.0000,  0.0000,  2.3819,  0.0000,  0.0000,  1.0000,  1.0000,  0.0000,  0.0000,  0.0000,  1.0000],\n",
       "         [24.0000,  1.0000,  2.0000,  4.1897,  0.0000,  1.0000,  0.0000,  1.0000,  0.0000,  0.0000,  0.0000,  1.0000],\n",
       "         [24.0000,  0.0000,  0.0000,  2.7754,  0.0000,  1.0000,  0.0000,  0.0000,  1.0000,  1.0000,  0.0000,  0.0000],\n",
       "         [45.0000,  0.0000,  1.0000,  4.1645,  1.0000,  0.0000,  0.0000,  1.0000,  0.0000,  1.0000,  0.0000,  0.0000],\n",
       "         [24.0000,  1.0000,  0.0000,  2.7381,  0.0000,  0.0000,  1.0000,  1.0000,  0.0000,  1.0000,  0.0000,  0.0000],\n",
       "         [24.0000,  0.0000,  0.0000,  2.1459,  0.0000,  0.0000,  1.0000,  0.0000,  1.0000,  0.0000,  0.0000,  1.0000]],\n",
       "        device='cuda:0'),\n",
       " 'validation': tensor([[23.0000,  0.0000,  1.0000,  4.1645,  1.0000,  0.0000,  0.0000,  0.0000,  1.0000,  1.0000,  0.0000,  0.0000],\n",
       "         [24.0000,  0.0000,  0.0000,  3.9318,  1.0000,  0.0000,  0.0000,  0.0000,  1.0000,  0.0000,  0.0000,  1.0000],\n",
       "         [43.0000,  0.0000,  0.0000,  2.1856,  0.0000,  0.0000,  1.0000,  0.0000,  1.0000,  0.0000,  0.0000,  1.0000],\n",
       "         [21.0000,  0.0000,  0.0000,  3.3160,  1.0000,  0.0000,  0.0000,  1.0000,  0.0000,  0.0000,  0.0000,  1.0000],\n",
       "         [25.0000,  1.0000,  0.0000,  3.2958,  0.0000,  1.0000,  0.0000,  0.0000,  1.0000,  0.0000,  0.0000,  1.0000],\n",
       "         [24.0000,  0.0000,  0.0000,  2.1691,  0.0000,  0.0000,  1.0000,  1.0000,  0.0000,  0.0000,  1.0000,  0.0000],\n",
       "         [40.0000,  1.0000,  4.0000,  3.3638,  0.0000,  0.0000,  1.0000,  0.0000,  1.0000,  0.0000,  0.0000,  1.0000],\n",
       "         [36.0000,  0.0000,  0.0000,  2.6391,  0.0000,  1.0000,  0.0000,  0.0000,  1.0000,  0.0000,  0.0000,  1.0000],\n",
       "         [36.0000,  0.0000,  0.0000,  4.9173,  1.0000,  0.0000,  0.0000,  1.0000,  0.0000,  1.0000,  0.0000,  0.0000],\n",
       "         [37.0000,  0.0000,  0.0000,  2.3597,  0.0000,  0.0000,  1.0000,  1.0000,  0.0000,  0.0000,  0.0000,  1.0000],\n",
       "         ...,\n",
       "         [48.0000,  0.0000,  2.0000,  3.6310,  0.0000,  1.0000,  0.0000,  1.0000,  0.0000,  0.0000,  0.0000,  1.0000],\n",
       "         [24.0000,  0.0000,  0.0000,  2.1691,  0.0000,  0.0000,  1.0000,  1.0000,  0.0000,  0.0000,  1.0000,  0.0000],\n",
       "         [60.0000,  1.0000,  0.0000,  4.3340,  1.0000,  0.0000,  0.0000,  1.0000,  0.0000,  1.0000,  0.0000,  0.0000],\n",
       "         [34.0000,  0.0000,  0.0000,  2.2028,  0.0000,  0.0000,  1.0000,  0.0000,  1.0000,  0.0000,  0.0000,  1.0000],\n",
       "         [21.0000,  0.0000,  0.0000,  2.2442,  0.0000,  0.0000,  1.0000,  0.0000,  1.0000,  0.0000,  0.0000,  1.0000],\n",
       "         [30.5000,  0.0000,  0.0000,  2.1691,  0.0000,  0.0000,  1.0000,  1.0000,  0.0000,  0.0000,  1.0000,  0.0000],\n",
       "         [33.0000,  0.0000,  0.0000,  2.5859,  0.0000,  1.0000,  0.0000,  0.0000,  1.0000,  0.0000,  0.0000,  1.0000],\n",
       "         [28.0000,  0.0000,  0.0000,  2.1856,  0.0000,  0.0000,  1.0000,  0.0000,  1.0000,  0.0000,  0.0000,  1.0000],\n",
       "         [46.0000,  0.0000,  0.0000,  4.3845,  1.0000,  0.0000,  0.0000,  0.0000,  1.0000,  1.0000,  0.0000,  0.0000],\n",
       "         [18.0000,  0.0000,  2.0000,  2.6391,  0.0000,  1.0000,  0.0000,  1.0000,  0.0000,  0.0000,  0.0000,  1.0000]],\n",
       "        device='cuda:0'),\n",
       " 'test': tensor([[20.0000,  0.0000,  0.0000,  2.6988,  0.0000,  1.0000,  0.0000,  0.0000,  1.0000,  1.0000,  0.0000,  0.0000],\n",
       "         [44.0000,  0.0000,  0.0000,  2.6391,  0.0000,  1.0000,  0.0000,  0.0000,  1.0000,  0.0000,  0.0000,  1.0000],\n",
       "         [26.0000,  2.0000,  0.0000,  2.2683,  0.0000,  0.0000,  1.0000,  0.0000,  1.0000,  0.0000,  0.0000,  1.0000],\n",
       "         [33.0000,  0.0000,  0.0000,  3.3160,  1.0000,  0.0000,  0.0000,  0.0000,  1.0000,  0.0000,  0.0000,  1.0000],\n",
       "         [24.0000,  0.0000,  0.0000,  2.2734,  0.0000,  0.0000,  1.0000,  0.0000,  1.0000,  1.0000,  0.0000,  0.0000],\n",
       "         [24.0000,  1.0000,  1.0000,  2.7408,  0.0000,  0.0000,  1.0000,  0.0000,  1.0000,  0.0000,  0.0000,  1.0000],\n",
       "         [19.0000,  1.0000,  1.0000,  3.6310,  0.0000,  1.0000,  0.0000,  0.0000,  1.0000,  0.0000,  0.0000,  1.0000],\n",
       "         [37.0000,  1.0000,  0.0000,  3.2958,  0.0000,  1.0000,  0.0000,  0.0000,  1.0000,  0.0000,  0.0000,  1.0000],\n",
       "         [15.0000,  0.0000,  0.0000,  2.1072,  0.0000,  0.0000,  1.0000,  1.0000,  0.0000,  1.0000,  0.0000,  0.0000],\n",
       "         [16.0000,  0.0000,  0.0000,  2.4423,  0.0000,  1.0000,  0.0000,  0.0000,  1.0000,  0.0000,  0.0000,  1.0000],\n",
       "         ...,\n",
       "         [26.0000,  0.0000,  0.0000,  2.1846,  0.0000,  0.0000,  1.0000,  0.0000,  1.0000,  0.0000,  0.0000,  1.0000],\n",
       "         [10.0000,  0.0000,  2.0000,  3.2249,  0.0000,  0.0000,  1.0000,  1.0000,  0.0000,  0.0000,  0.0000,  1.0000],\n",
       "         [31.0000,  1.0000,  1.0000,  3.6377,  0.0000,  1.0000,  0.0000,  0.0000,  1.0000,  1.0000,  0.0000,  0.0000],\n",
       "         [17.0000,  0.0000,  0.0000,  2.2683,  0.0000,  0.0000,  1.0000,  0.0000,  1.0000,  0.0000,  0.0000,  1.0000],\n",
       "         [28.0000,  0.0000,  0.0000,  2.1856,  0.0000,  0.0000,  1.0000,  0.0000,  1.0000,  0.0000,  0.0000,  1.0000],\n",
       "         [63.0000,  1.0000,  0.0000,  3.2958,  0.0000,  1.0000,  0.0000,  0.0000,  1.0000,  0.0000,  0.0000,  1.0000],\n",
       "         [28.0000,  0.0000,  0.0000,  2.2028,  0.0000,  0.0000,  1.0000,  0.0000,  1.0000,  0.0000,  0.0000,  1.0000],\n",
       "         [22.0000,  0.0000,  1.0000,  4.1428,  1.0000,  0.0000,  0.0000,  1.0000,  0.0000,  1.0000,  0.0000,  0.0000],\n",
       "         [24.0000,  0.0000,  0.0000,  2.1856,  0.0000,  0.0000,  1.0000,  0.0000,  1.0000,  0.0000,  0.0000,  1.0000],\n",
       "         [24.0000,  0.0000,  0.0000,  2.1691,  0.0000,  0.0000,  1.0000,  0.0000,  1.0000,  0.0000,  1.0000,  0.0000]],\n",
       "        device='cuda:0')}"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "independent_cols = ['age', 'sibsp', 'parch', 'log_fare', 'pclass_1', 'pclass_2', 'pclass_3', 'sex_female', 'sex_male',\n",
    "                    'embarked_C', 'embarked_Q', 'embarked_S']\n",
    "\n",
    "independent_vars = {split: tensor((dataset.to_pandas()*1)[independent_cols].values, dtype=torch.float) for split, dataset in datasetDict.items()}\n",
    "independent_vars"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "56b4dfcd-b5b4-4ea6-8a80-d581396f41b1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'train': torch.Size([1047, 12]),\n",
       " 'validation': torch.Size([131, 12]),\n",
       " 'test': torch.Size([131, 12])}"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "{split: vars.shape for split, vars in independent_vars.items()}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5c3fcf76-c29a-4cd5-8d01-84c4e5196247",
   "metadata": {},
   "source": [
    "### Normalise Dataset Splits"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "ba95ae83-5b4a-4198-a8f6-d527714d8c13",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'train': tensor([[0.0526, 0.1250, 0.1111, 0.5092, 0.0000, 1.0000, 0.0000, 1.0000, 0.0000, 0.0000, 0.0000, 1.0000],\n",
       "         [0.2632, 0.0000, 0.0000, 0.3507, 0.0000, 0.0000, 1.0000, 0.0000, 1.0000, 0.0000, 0.0000, 1.0000],\n",
       "         [0.4276, 0.0000, 0.0000, 0.8587, 1.0000, 0.0000, 0.0000, 0.0000, 1.0000, 1.0000, 0.0000, 0.0000],\n",
       "         [0.3026, 0.0000, 0.0000, 0.4447, 0.0000, 1.0000, 0.0000, 0.0000, 1.0000, 1.0000, 0.0000, 0.0000],\n",
       "         [0.6184, 0.0000, 0.0000, 0.6362, 1.0000, 0.0000, 0.0000, 0.0000, 1.0000, 0.0000, 0.0000, 1.0000],\n",
       "         [0.3947, 0.1250, 0.0000, 0.4953, 0.0000, 1.0000, 0.0000, 0.0000, 1.0000, 0.0000, 0.0000, 1.0000],\n",
       "         [0.5263, 0.1250, 0.0000, 0.5281, 0.0000, 1.0000, 0.0000, 0.0000, 1.0000, 0.0000, 0.0000, 1.0000],\n",
       "         [0.2368, 0.0000, 0.2222, 0.7034, 1.0000, 0.0000, 0.0000, 1.0000, 0.0000, 0.0000, 0.0000, 1.0000],\n",
       "         [0.4211, 0.0000, 0.0000, 0.3502, 0.0000, 0.0000, 1.0000, 0.0000, 1.0000, 0.0000, 0.0000, 1.0000],\n",
       "         [0.2105, 0.2500, 0.0000, 0.4718, 0.0000, 0.0000, 1.0000, 0.0000, 1.0000, 0.0000, 0.0000, 1.0000],\n",
       "         ...,\n",
       "         [0.5263, 0.1250, 0.1111, 0.7866, 1.0000, 0.0000, 0.0000, 1.0000, 0.0000, 1.0000, 0.0000, 0.0000],\n",
       "         [0.2763, 0.0000, 0.0000, 0.3480, 0.0000, 0.0000, 1.0000, 0.0000, 1.0000, 0.0000, 0.0000, 1.0000],\n",
       "         [0.3158, 0.0000, 0.0000, 0.3377, 0.0000, 0.0000, 1.0000, 0.0000, 1.0000, 1.0000, 0.0000, 0.0000],\n",
       "         [0.3684, 0.1250, 0.1111, 0.4381, 0.0000, 0.0000, 1.0000, 1.0000, 0.0000, 0.0000, 0.0000, 1.0000],\n",
       "         [0.2763, 0.1250, 0.0000, 0.3817, 0.0000, 0.0000, 1.0000, 1.0000, 0.0000, 0.0000, 0.0000, 1.0000],\n",
       "         [0.3158, 0.1250, 0.2222, 0.6713, 0.0000, 1.0000, 0.0000, 1.0000, 0.0000, 0.0000, 0.0000, 1.0000],\n",
       "         [0.3158, 0.0000, 0.0000, 0.4447, 0.0000, 1.0000, 0.0000, 0.0000, 1.0000, 1.0000, 0.0000, 0.0000],\n",
       "         [0.5921, 0.0000, 0.1111, 0.6673, 1.0000, 0.0000, 0.0000, 1.0000, 0.0000, 1.0000, 0.0000, 0.0000],\n",
       "         [0.3158, 0.1250, 0.0000, 0.4387, 0.0000, 0.0000, 1.0000, 1.0000, 0.0000, 1.0000, 0.0000, 0.0000],\n",
       "         [0.3158, 0.0000, 0.0000, 0.3438, 0.0000, 0.0000, 1.0000, 0.0000, 1.0000, 0.0000, 0.0000, 1.0000]],\n",
       "        device='cuda:0'),\n",
       " 'validation': tensor([[0.3594, 0.0000, 0.2000, 0.7667, 1.0000, 0.0000, 0.0000, 0.0000, 1.0000, 1.0000, 0.0000, 0.0000],\n",
       "         [0.3750, 0.0000, 0.0000, 0.7239, 1.0000, 0.0000, 0.0000, 0.0000, 1.0000, 0.0000, 0.0000, 1.0000],\n",
       "         [0.6719, 0.0000, 0.0000, 0.4024, 0.0000, 0.0000, 1.0000, 0.0000, 1.0000, 0.0000, 0.0000, 1.0000],\n",
       "         [0.3281, 0.0000, 0.0000, 0.6105, 1.0000, 0.0000, 0.0000, 1.0000, 0.0000, 0.0000, 0.0000, 1.0000],\n",
       "         [0.3906, 0.1250, 0.0000, 0.6068, 0.0000, 1.0000, 0.0000, 0.0000, 1.0000, 0.0000, 0.0000, 1.0000],\n",
       "         [0.3750, 0.0000, 0.0000, 0.3993, 0.0000, 0.0000, 1.0000, 1.0000, 0.0000, 0.0000, 1.0000, 0.0000],\n",
       "         [0.6250, 0.1250, 0.8000, 0.6193, 0.0000, 0.0000, 1.0000, 0.0000, 1.0000, 0.0000, 0.0000, 1.0000],\n",
       "         [0.5625, 0.0000, 0.0000, 0.4859, 0.0000, 1.0000, 0.0000, 0.0000, 1.0000, 0.0000, 0.0000, 1.0000],\n",
       "         [0.5625, 0.0000, 0.0000, 0.9053, 1.0000, 0.0000, 0.0000, 1.0000, 0.0000, 1.0000, 0.0000, 0.0000],\n",
       "         [0.5781, 0.0000, 0.0000, 0.4344, 0.0000, 0.0000, 1.0000, 1.0000, 0.0000, 0.0000, 0.0000, 1.0000],\n",
       "         ...,\n",
       "         [0.7500, 0.0000, 0.4000, 0.6685, 0.0000, 1.0000, 0.0000, 1.0000, 0.0000, 0.0000, 0.0000, 1.0000],\n",
       "         [0.3750, 0.0000, 0.0000, 0.3993, 0.0000, 0.0000, 1.0000, 1.0000, 0.0000, 0.0000, 1.0000, 0.0000],\n",
       "         [0.9375, 0.1250, 0.0000, 0.7979, 1.0000, 0.0000, 0.0000, 1.0000, 0.0000, 1.0000, 0.0000, 0.0000],\n",
       "         [0.5312, 0.0000, 0.0000, 0.4055, 0.0000, 0.0000, 1.0000, 0.0000, 1.0000, 0.0000, 0.0000, 1.0000],\n",
       "         [0.3281, 0.0000, 0.0000, 0.4132, 0.0000, 0.0000, 1.0000, 0.0000, 1.0000, 0.0000, 0.0000, 1.0000],\n",
       "         [0.4766, 0.0000, 0.0000, 0.3993, 0.0000, 0.0000, 1.0000, 1.0000, 0.0000, 0.0000, 1.0000, 0.0000],\n",
       "         [0.5156, 0.0000, 0.0000, 0.4761, 0.0000, 1.0000, 0.0000, 0.0000, 1.0000, 0.0000, 0.0000, 1.0000],\n",
       "         [0.4375, 0.0000, 0.0000, 0.4024, 0.0000, 0.0000, 1.0000, 0.0000, 1.0000, 0.0000, 0.0000, 1.0000],\n",
       "         [0.7188, 0.0000, 0.0000, 0.8072, 1.0000, 0.0000, 0.0000, 0.0000, 1.0000, 1.0000, 0.0000, 0.0000],\n",
       "         [0.2812, 0.0000, 0.4000, 0.4859, 0.0000, 1.0000, 0.0000, 1.0000, 0.0000, 0.0000, 0.0000, 1.0000]],\n",
       "        device='cuda:0'),\n",
       " 'test': tensor([[0.2500, 0.0000, 0.0000, 0.4324, 0.0000, 1.0000, 0.0000, 0.0000, 1.0000, 1.0000, 0.0000, 0.0000],\n",
       "         [0.5500, 0.0000, 0.0000, 0.4229, 0.0000, 1.0000, 0.0000, 0.0000, 1.0000, 0.0000, 0.0000, 1.0000],\n",
       "         [0.3250, 0.5000, 0.0000, 0.3634, 0.0000, 0.0000, 1.0000, 0.0000, 1.0000, 0.0000, 0.0000, 1.0000],\n",
       "         [0.4125, 0.0000, 0.0000, 0.5313, 1.0000, 0.0000, 0.0000, 0.0000, 1.0000, 0.0000, 0.0000, 1.0000],\n",
       "         [0.3000, 0.0000, 0.0000, 0.3643, 0.0000, 0.0000, 1.0000, 0.0000, 1.0000, 1.0000, 0.0000, 0.0000],\n",
       "         [0.3000, 0.2500, 0.2000, 0.4392, 0.0000, 0.0000, 1.0000, 0.0000, 1.0000, 0.0000, 0.0000, 1.0000],\n",
       "         [0.2375, 0.2500, 0.2000, 0.5818, 0.0000, 1.0000, 0.0000, 0.0000, 1.0000, 0.0000, 0.0000, 1.0000],\n",
       "         [0.4625, 0.2500, 0.0000, 0.5281, 0.0000, 1.0000, 0.0000, 0.0000, 1.0000, 0.0000, 0.0000, 1.0000],\n",
       "         [0.1875, 0.0000, 0.0000, 0.3376, 0.0000, 0.0000, 1.0000, 1.0000, 0.0000, 1.0000, 0.0000, 0.0000],\n",
       "         [0.2000, 0.0000, 0.0000, 0.3913, 0.0000, 1.0000, 0.0000, 0.0000, 1.0000, 0.0000, 0.0000, 1.0000],\n",
       "         ...,\n",
       "         [0.3250, 0.0000, 0.0000, 0.3501, 0.0000, 0.0000, 1.0000, 0.0000, 1.0000, 0.0000, 0.0000, 1.0000],\n",
       "         [0.1250, 0.0000, 0.4000, 0.5167, 0.0000, 0.0000, 1.0000, 1.0000, 0.0000, 0.0000, 0.0000, 1.0000],\n",
       "         [0.3875, 0.2500, 0.2000, 0.5829, 0.0000, 1.0000, 0.0000, 0.0000, 1.0000, 1.0000, 0.0000, 0.0000],\n",
       "         [0.2125, 0.0000, 0.0000, 0.3634, 0.0000, 0.0000, 1.0000, 0.0000, 1.0000, 0.0000, 0.0000, 1.0000],\n",
       "         [0.3500, 0.0000, 0.0000, 0.3502, 0.0000, 0.0000, 1.0000, 0.0000, 1.0000, 0.0000, 0.0000, 1.0000],\n",
       "         [0.7875, 0.2500, 0.0000, 0.5281, 0.0000, 1.0000, 0.0000, 0.0000, 1.0000, 0.0000, 0.0000, 1.0000],\n",
       "         [0.3500, 0.0000, 0.0000, 0.3530, 0.0000, 0.0000, 1.0000, 0.0000, 1.0000, 0.0000, 0.0000, 1.0000],\n",
       "         [0.2750, 0.0000, 0.2000, 0.6638, 1.0000, 0.0000, 0.0000, 1.0000, 0.0000, 1.0000, 0.0000, 0.0000],\n",
       "         [0.3000, 0.0000, 0.0000, 0.3502, 0.0000, 0.0000, 1.0000, 0.0000, 1.0000, 0.0000, 0.0000, 1.0000],\n",
       "         [0.3000, 0.0000, 0.0000, 0.3476, 0.0000, 0.0000, 1.0000, 0.0000, 1.0000, 0.0000, 1.0000, 0.0000]],\n",
       "        device='cuda:0')}"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def normalise(vars):\n",
    "  max_vals, indices = vars.max(dim=0)  \n",
    "  return vars / max_vals\n",
    "\n",
    "independent_vars = {split: normalise(vars) for split, vars in independent_vars.items()}\n",
    "independent_vars"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0c19fa10-2f6d-43e8-8100-def837cde605",
   "metadata": {},
   "source": [
    "## Train Linear Model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "82078812-7143-40fe-9e74-f32670dac63f",
   "metadata": {},
   "source": [
    "### Set Up Coefficients"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5859de46-3042-4423-b4ca-56a194c867d9",
   "metadata": {},
   "source": [
    "Initialise random coefficients as a column vector."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "1927f9ef-0db6-4418-b3cb-28c7129fe2c4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.1130],\n",
       "        [-0.4899],\n",
       "        [-0.1016],\n",
       "        [-0.4597],\n",
       "        [-0.3437],\n",
       "        [-0.0175],\n",
       "        [ 0.2362],\n",
       "        [-0.0940],\n",
       "        [ 0.0189],\n",
       "        [-0.2133],\n",
       "        [-0.2584],\n",
       "        [ 0.4228]], device='cuda:0')"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "num_coeffs = independent_vars['train'].shape[1]\n",
    "torch.manual_seed(42)\n",
    "coeffs = torch.rand(num_coeffs, 1) - 0.5\n",
    "coeffs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "9b588ef8-8ac8-496a-89a3-29fb778016db",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.0106],\n",
       "        [ 0.5465],\n",
       "        [-0.8845],\n",
       "        [-0.3822],\n",
       "        [-0.1246],\n",
       "        [ 0.1799],\n",
       "        [ 0.1797],\n",
       "        [-0.3341],\n",
       "        [ 0.5646],\n",
       "        [ 0.3624]], device='cuda:0')"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predictions = independent_vars['train']@coeffs\n",
    "predictions[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "9b044342-370b-4fd2-a0e0-c6e6a6c76c1f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(0.6817, device='cuda:0')"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "loss = torch.abs(predictions - dependent_var['train']).mean()\n",
    "loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "d17c8a05-65df-476a-a3e6-639094683c82",
   "metadata": {},
   "outputs": [],
   "source": [
    "def calc_predictions(coeffs, independent_vars):\n",
    "    return torch.sigmoid(independent_vars@coeffs).sum(axis=1)\n",
    "\n",
    "def calc_loss(coeffs, independent_vars, dependent_var):\n",
    "    return torch.abs(calc_predictions(coeffs, independent_vars) - dependent_var).mean()\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b2a6fd49-6af5-4d39-82d1-edf765b6434c",
   "metadata": {},
   "source": [
    "### Set Up Gradient Descent Step"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "ae0997df-6ae7-4081-a255-3bb99e360ddb",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.1130],\n",
       "        [-0.4899],\n",
       "        [-0.1016],\n",
       "        [-0.4597],\n",
       "        [-0.3437],\n",
       "        [-0.0175],\n",
       "        [ 0.2362],\n",
       "        [-0.0940],\n",
       "        [ 0.0189],\n",
       "        [-0.2133],\n",
       "        [-0.2584],\n",
       "        [ 0.4228]], device='cuda:0', requires_grad=True)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "coeffs.requires_grad_()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "8f0a6a13-e5dd-4ecd-9da0-114a57149df9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(0.5017, device='cuda:0', grad_fn=<MeanBackward0>)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "loss = calc_loss(coeffs, independent_vars['train'], dependent_var['train'])\n",
    "loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "764db880-8b78-4e2b-bd5a-7feed49d3b3b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0.0195],\n",
       "        [0.0033],\n",
       "        [0.0022],\n",
       "        [0.0248],\n",
       "        [0.0122],\n",
       "        [0.0117],\n",
       "        [0.0282],\n",
       "        [0.0188],\n",
       "        [0.0333],\n",
       "        [0.0104],\n",
       "        [0.0051],\n",
       "        [0.0365]], device='cuda:0')"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "loss.backward()\n",
    "coeffs.grad"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "b8d334b3-d06e-41e0-b642-416916b0d158",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor(0.5007, device='cuda:0')\n"
     ]
    }
   ],
   "source": [
    "loss = calc_loss(coeffs, independent_vars['train'], dependent_var['train'])\n",
    "loss.backward()\n",
    "with torch.no_grad():\n",
    "    coeffs.sub_(coeffs.grad * 0.1)\n",
    "    coeffs.grad.zero_()\n",
    "    print(calc_loss(coeffs, independent_vars['train'], dependent_var['train']))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f6ceee0f-04ab-4e66-be71-0d8d7f5aea3f",
   "metadata": {},
   "source": [
    "### Run Training Loop"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "0fe3a0b4-da48-46b4-9034-3966e4dbeff0",
   "metadata": {},
   "outputs": [],
   "source": [
    "def update_coeffs(coeffs, learning_rate):\n",
    "    coeffs.sub_(coeffs.grad * learning_rate)\n",
    "    coeffs.grad.zero_()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "c6684e67-08d9-42cb-9a60-4d79ba748aa9",
   "metadata": {},
   "outputs": [],
   "source": [
    "def one_epoch(coeffs, learning_rate):\n",
    "    loss = calc_loss(coeffs, independent_vars['train'], dependent_var['train'])\n",
    "    loss.backward()\n",
    "    with torch.no_grad(): update_coeffs(coeffs, learning_rate)\n",
    "    print(f'{loss:.3f}', end='; ')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "2d58c531-4b82-4936-b2f1-1eb11554f5a2",
   "metadata": {},
   "outputs": [],
   "source": [
    "def init_coeffs():\n",
    "    return (torch.rand(num_coeffs, 1) - 0.5).requires_grad_()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "9af7154d-6074-45ba-9100-d9a1babccec7",
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_model(epochs=30, learning_rate=0.01):\n",
    "    torch.manual_seed(442)\n",
    "    coeffs = init_coeffs()\n",
    "    for i in range (epochs):\n",
    "        one_epoch(coeffs, learning_rate)\n",
    "        \n",
    "    return coeffs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "7c80e4c8-0369-47b5-b58e-0f00c46aaeda",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.514; 0.391; 0.391; 0.391; 0.391; 0.391; 0.391; 0.391; 0.391; 0.391; 0.391; 0.391; 0.391; 0.391; 0.391; 0.391; 0.391; 0.391; 0.391; 0.391; "
     ]
    }
   ],
   "source": [
    "coeffs = train_model(epochs=20, learning_rate=100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "7706e362-bbad-4a8f-8264-6be55227156b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'age': tensor([-1.6475], device='cuda:0'),\n",
       " 'sibsp': tensor([-0.8166], device='cuda:0'),\n",
       " 'parch': tensor([-0.0429], device='cuda:0'),\n",
       " 'log_fare': tensor([-2.3991], device='cuda:0'),\n",
       " 'pclass_1': tensor([-1.1872], device='cuda:0'),\n",
       " 'pclass_2': tensor([-1.5524], device='cuda:0'),\n",
       " 'pclass_3': tensor([-3.3380], device='cuda:0'),\n",
       " 'sex_female': tensor([-1.9546], device='cuda:0'),\n",
       " 'sex_male': tensor([-3.1221], device='cuda:0'),\n",
       " 'embarked_C': tensor([-0.8663], device='cuda:0'),\n",
       " 'embarked_Q': tensor([-0.3180], device='cuda:0'),\n",
       " 'embarked_S': tensor([-3.6144], device='cuda:0')}"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def show_coeffs():\n",
    "    return dict(zip(independent_cols, coeffs.requires_grad_(False)))\n",
    "\n",
    "show_coeffs()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "49a8b6cc",
   "metadata": {},
   "source": [
    "## Train Neural Network"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "93bc5c1c",
   "metadata": {},
   "source": [
    "### Set up Coefficients\n",
    "\n",
    "Initialisation of the coefficients for the neural network is similar to the linear model. However, we need to initialise the coefficients for each layer in the network. We do this by creating a list of tensors, one for each layer. The number of columns in the first layer is equal to the number of independent variables. The number of columns in each subsequent layer is equal to the number of hidden coefficients, which we set to a constant value. The number of columns in the last layer is equal to the number of dependent variables i.e. 1 in our case.\n",
    "\n",
    "We divide the number of hidden coefficients by the number of columns in the first layer to ensure that the weights are distributed evenly across each layer. This ensures that the gradients for each layer will be similar and have a similar impact on the model's performance.\n",
    "\n",
    "We add a constant to the last layer of coefficients so that the model can learn a bias term."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "f7b50490",
   "metadata": {},
   "outputs": [],
   "source": [
    "def init_coeffs(num_hidden_coeffs=20):\n",
    "    layer_1 = (torch.rand(num_coeffs, num_hidden_coeffs) - 0.5) / num_hidden_coeffs\n",
    "    layer_2 = torch.rand(num_hidden_coeffs, 1) - 0.3\n",
    "    const = torch.rand(1)[0]\n",
    "    return layer_1.requires_grad_(), layer_2.requires_grad_(), const.requires_grad_()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aefb84c6",
   "metadata": {},
   "source": [
    "Calculation of predictions is also similar to the linear model except that we take the outputs from the first layer and pass them to the second layer. Note the use of `relu` to rectify the outputs of the first layer to ensure that they are positive."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "d916c1fd",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch.nn.functional as F\n",
    "\n",
    "def calc_predictions(coeffs, independent_vars):\n",
    "  layer_1, layer_2, const = coeffs\n",
    "  result = F.relu(independent_vars@layer_1)\n",
    "  result = result@layer_2 + const\n",
    "  return torch.sigmoid(result)\n",
    "     "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "cb6561a3",
   "metadata": {},
   "outputs": [],
   "source": [
    "def update_coeffs(coeffs, learning_rate):\n",
    "  for layer_coeffs in coeffs:\n",
    "    layer_coeffs.sub_(layer_coeffs.grad * learning_rate)\n",
    "    layer_coeffs.grad.zero_()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa4b6a40",
   "metadata": {},
   "source": [
    "### Run Training Loop"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "e1dc7956",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.536; 0.433; 0.273; 0.371; 0.268; 0.219; 0.219; 0.218; 0.217; 0.213; 0.210; 0.209; 0.207; 0.207; 0.206; 0.206; 0.206; 0.206; 0.205; 0.205; 0.205; 0.205; 0.205; 0.205; 0.205; 0.205; 0.205; 0.204; 0.204; 0.204; 0.204; 0.204; 0.204; 0.204; 0.204; 0.204; 0.204; 0.204; 0.204; 0.204; 0.204; 0.204; 0.204; 0.204; 0.204; 0.203; 0.203; 0.203; 0.203; 0.203; 0.203; 0.203; 0.203; 0.203; 0.203; 0.203; 0.203; 0.203; 0.203; 0.203; 0.203; 0.203; 0.203; 0.203; 0.203; 0.203; 0.203; 0.203; 0.203; 0.203; 0.203; 0.202; 0.202; 0.202; 0.202; 0.202; 0.202; 0.202; 0.202; 0.202; 0.202; 0.202; 0.202; 0.202; 0.202; 0.202; 0.202; 0.202; 0.201; 0.201; 0.201; 0.201; 0.201; 0.201; 0.201; 0.201; 0.201; 0.201; 0.201; 0.201; "
     ]
    }
   ],
   "source": [
    "coeffs = train_model(epochs=100, learning_rate=20)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3a287fe5",
   "metadata": {},
   "source": [
    "## Train Deep Neural Network"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12dfa42a",
   "metadata": {},
   "source": [
    "### Set up Coefficients\n",
    "\n",
    "We now genericise the number of hidden layers and their sizes. We also add a constant term to each layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "eff6350f",
   "metadata": {},
   "outputs": [],
   "source": [
    "def init_coeffs(num_hidden_coeffs=[10, 10]):\n",
    "  layer_sizes = [num_coeffs] + num_hidden_coeffs + [1]\n",
    "  num_layers = len(layer_sizes)\n",
    "\n",
    "  layers = [(torch.rand(layer_sizes[i], layer_sizes[i+1]) - 0.5) / layer_sizes[i + 1] * 4 for i in range(num_layers - 1)]\n",
    "  consts = [(torch.rand(1)[0] - 0.5) * 0.1 for i in range(num_layers - 1)]\n",
    "\n",
    "  for layer in layers + consts:\n",
    "    layer.requires_grad_()\n",
    "\n",
    "  return layers, consts"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "16f1c0de",
   "metadata": {},
   "source": [
    "Calculation of predictions proceeds largely as before but instead we loop over each layer rather than explicitly writing them out."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "b95bf7be",
   "metadata": {},
   "outputs": [],
   "source": [
    "def calc_predictions(coeffs, independent_vars):\n",
    "  layers, consts = coeffs\n",
    "  num_layers = len(layers)\n",
    "  result = independent_vars\n",
    "  \n",
    "  for i, layer in enumerate(layers):\n",
    "    result = result@layer + consts[i]\n",
    "    if i != num_layers - 1:\n",
    "      result = F.relu(result)\n",
    "\n",
    "  return torch.sigmoid(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "46ee050c",
   "metadata": {},
   "source": [
    "A minor change to to the training loop is required as we now have a list of coefficients rather than a single set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "8fc352aa",
   "metadata": {},
   "outputs": [],
   "source": [
    "def update_coeffs(coeffs, learning_rate):\n",
    "  layers, consts = coeffs\n",
    "  for layer in layers + consts:\n",
    "    layer.sub_(layer.grad * learning_rate)\n",
    "    layer.grad.zero_()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ce540c16",
   "metadata": {},
   "source": [
    "### Run Training Loop"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "276d16e3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.505; 0.488; 0.454; 0.380; 0.358; 0.343; 0.337; 0.325; 0.408; 0.307; 0.289; 0.266; 0.239; 0.221; 0.215; 0.212; 0.210; 0.210; 0.209; 0.209; 0.207; 0.207; 0.206; 0.205; 0.205; 0.205; 0.204; 0.204; 0.204; 0.204; 0.204; 0.204; 0.203; 0.203; 0.203; 0.203; 0.203; 0.203; 0.203; 0.203; 0.203; 0.203; 0.203; 0.203; 0.203; 0.203; 0.203; 0.202; 0.202; 0.202; 0.202; 0.202; 0.202; 0.202; 0.202; 0.202; 0.202; 0.202; 0.202; 0.202; 0.202; 0.202; 0.202; 0.201; 0.201; 0.201; 0.201; 0.201; 0.201; 0.201; 0.201; 0.201; 0.201; 0.201; 0.201; 0.201; 0.201; 0.201; 0.201; 0.200; 0.200; 0.200; 0.200; 0.200; 0.200; 0.200; 0.200; 0.200; 0.200; 0.200; 0.200; 0.200; 0.200; 0.200; 0.200; 0.200; 0.200; 0.200; 0.200; 0.200; "
     ]
    }
   ],
   "source": [
    "coeffs = train_model(epochs=100, learning_rate=4)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "675cbb91",
   "metadata": {},
   "source": [
    "## Recreate as PyTorch Module"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "97c03100",
   "metadata": {},
   "source": [
    "We subclass the PyTorch Sequential class to create a model that we can train. The layers from the previous example are recreated as members of the class, and are initialised with random weights as before."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "c238d09d",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch.nn as nn\n",
    "\n",
    "class TitanicModel(nn.Sequential):\n",
    "  def __init__(self):\n",
    "    super(TitanicModel, self).__init__(nn.Linear(num_coeffs, 20),\n",
    "                                       nn.ReLU(),\n",
    "                                       nn.Linear(20, 20),\n",
    "                                       nn.ReLU(),\n",
    "                                       nn.Linear(20, 1),\n",
    "                                       nn.Sigmoid())\n",
    "    \n",
    "    self.apply(self._init_weights)\n",
    "    \n",
    "  def _init_weights(self, module):\n",
    "    if isinstance(module, nn.Linear):\n",
    "      module.weight.data.normal_(mean=0.0, std=0.5)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "456d4aca",
   "metadata": {},
   "source": [
    "The training loop is similar to before with the exception that we use the PyTorch optimisation machinery rather than creating our own.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "d9ab947f",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch.optim as optim\n",
    "\n",
    "def train_model(epochs=30, learning_rate=10):\n",
    "  torch.manual_seed(442)\n",
    "  model = TitanicModel()\n",
    "  calc_loss = nn.L1Loss()\n",
    "  optimizer = optim.SGD(model.parameters(), lr=learning_rate)\n",
    "\n",
    "  for epoch in range(epochs):\n",
    "    optimizer.zero_grad()\n",
    "\n",
    "    output = model(independent_vars['train'])\n",
    "    loss = calc_loss(output, dependent_var['train'])\n",
    "\n",
    "    loss.backward()\n",
    "    optimizer.step()\n",
    "    print(f'{loss:.3f}', end='; ')\n",
    "\n",
    "  return model\n",
    "\n",
    "def calc_predictions(model, independent_vars):\n",
    "  return model(independent_vars)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eabe3a01",
   "metadata": {},
   "source": [
    "Training progresses similar to before:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "51a207ee",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.479; 0.390; 0.390; 0.389; 0.378; 0.232; 0.389; 0.358; 0.214; 0.208; 0.263; 0.232; 0.219; 0.218; 0.206; 0.206; 0.203; 0.203; 0.202; 0.202; 0.202; 0.202; 0.202; 0.202; 0.202; "
     ]
    }
   ],
   "source": [
    "model = train_model(epochs=25, learning_rate=15)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d34d5be1",
   "metadata": {},
   "source": [
    "## Tune Model Hyperparameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "ffbf2cb4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[False],\n",
       "        [ True],\n",
       "        [ True],\n",
       "        [ True],\n",
       "        [ True],\n",
       "        [ True],\n",
       "        [ True],\n",
       "        [ True],\n",
       "        [ True],\n",
       "        [ True],\n",
       "        [ True],\n",
       "        [False],\n",
       "        [ True],\n",
       "        [False],\n",
       "        [ True],\n",
       "        [False],\n",
       "        [ True],\n",
       "        [False],\n",
       "        [ True],\n",
       "        [ True]], device='cuda:0')"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predictions = calc_predictions(model, independent_vars['validation'])\n",
    "results = dependent_var['validation'].bool() == (predictions > 0.5)\n",
    "results[:20]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "e795488e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(0.8092, device='cuda:0')"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results.float().mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "457f4a15",
   "metadata": {},
   "outputs": [],
   "source": [
    "def calc_accuracy(model, independent_vars, dependent_var):\n",
    "  return (dependent_var.bool() == (calc_predictions(model, independent_vars) > 0.5)).float().mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "c33bd86b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(0.8092, device='cuda:0')"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "calc_accuracy(model, independent_vars['validation'], dependent_var['validation'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "134cd12c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.479; 0.390; 0.390; 0.389; 0.378; 0.232; 0.389; 0.358; 0.214; 0.208; "
     ]
    },
    {
     "data": {
      "text/plain": [
       "tensor(0.7557, device='cuda:0')"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = train_model(epochs=10, learning_rate=15)\n",
    "calc_accuracy(model, independent_vars['validation'], dependent_var['validation'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "fd0faca2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.479; 0.390; 0.390; 0.389; 0.378; 0.232; 0.389; 0.358; 0.214; 0.208; 0.263; 0.232; 0.219; 0.218; 0.206; "
     ]
    },
    {
     "data": {
      "text/plain": [
       "tensor(0.8092, device='cuda:0')"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = train_model(epochs=15, learning_rate=15)\n",
    "calc_accuracy(model, independent_vars['validation'], dependent_var['validation'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "d8ea20df",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.479; 0.357; 0.264; 0.231; 0.230; 0.215; 0.211; 0.211; 0.210; 0.211; 0.208; 0.206; 0.205; 0.204; 0.204; "
     ]
    },
    {
     "data": {
      "text/plain": [
       "tensor(0.8092, device='cuda:0')"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = train_model(epochs=15, learning_rate=5)\n",
    "calc_accuracy(model, independent_vars['validation'], dependent_var['validation'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "0366e2eb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.479; 0.357; 0.264; 0.231; 0.230; 0.215; 0.211; 0.211; 0.210; 0.211; "
     ]
    },
    {
     "data": {
      "text/plain": [
       "tensor(0.8092, device='cuda:0')"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = train_model(epochs=10, learning_rate=5)\n",
    "calc_accuracy(model, independent_vars['validation'], dependent_var['validation'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "77e15874",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.479; 0.355; 0.273; 0.232; 0.222; 0.216; 0.212; 0.210; 0.209; 0.208; "
     ]
    },
    {
     "data": {
      "text/plain": [
       "tensor(0.8092, device='cuda:0')"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = train_model(epochs=10, learning_rate=4)\n",
    "calc_accuracy(model, independent_vars['validation'], dependent_var['validation'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "723bdb32",
   "metadata": {},
   "source": [
    "## Test Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "70236199",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(0.8168, device='cuda:0')"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "calc_accuracy(model, independent_vars['test'], dependent_var['test'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e12a7819",
   "metadata": {},
   "source": [
    "## Upload Model to Hugging Face"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e2845f7e",
   "metadata": {},
   "source": [
    "We redefine `TitanicModel` to inherit from `PyTorchModelHubMixin` and supply additional metadata for the model card:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "375e2b1f",
   "metadata": {},
   "outputs": [],
   "source": [
    "from huggingface_hub import *\n",
    "\n",
    "class TitanicModel(nn.Sequential,\n",
    "                   PyTorchModelHubMixin,\n",
    "                   library_name='titanic-2410',\n",
    "                   license='mit',\n",
    "                   repo_url='https://github.com/jamieoliver/titanic-model-2410',\n",
    "                   docs_url='https://github.com/jamieoliver/titanic-model-2410'):\n",
    "  def __init__(self):\n",
    "    super(TitanicModel, self).__init__(nn.Linear(num_coeffs, 20),\n",
    "                                       nn.ReLU(),\n",
    "                                       nn.Linear(20, 20),\n",
    "                                       nn.ReLU(),\n",
    "                                       nn.Linear(20, 1),\n",
    "                                       nn.Sigmoid())\n",
    "    \n",
    "    self.apply(self._init_weights)\n",
    "    \n",
    "  def _init_weights(self, module):\n",
    "    if isinstance(module, nn.Linear):\n",
    "      module.weight.data.normal_(mean=0.0, std=0.5)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "04327cf6",
   "metadata": {},
   "source": [
    "Then we retrain the model and check the results are the same as before:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "eb9ee519",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.479; 0.355; 0.273; 0.232; 0.222; 0.216; 0.212; 0.210; 0.209; 0.208; "
     ]
    },
    {
     "data": {
      "text/plain": [
       "tensor(0.8168, device='cuda:0')"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = train_model(epochs=10, learning_rate=4)\n",
    "calc_accuracy(model, independent_vars['test'], dependent_var['test'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aad6c07f",
   "metadata": {},
   "source": [
    "Finally we create a branch in the model repository and upload the model to it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "746384f3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'2410222130'"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from datetime import *\n",
    "\n",
    "timestamp = datetime.now().strftime(\"%y%m%d%H%M\")\n",
    "timestamp"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "ed8ead28",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "CommitInfo(commit_url='https://huggingface.co/jamieoliver/titanic-2410/commit/05ac0350f227015ede26d88204f41a11ccfd576f', commit_message='Model timestamp 2410222130', commit_description='', oid='05ac0350f227015ede26d88204f41a11ccfd576f', pr_url=None, repo_url=RepoUrl('https://huggingface.co/jamieoliver/titanic-2410', endpoint='https://huggingface.co', repo_type='model', repo_id='jamieoliver/titanic-2410'), pr_revision=None, pr_num=None)"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "repo = 'jamieoliver/titanic-2410'\n",
    "branch = f'timestamp-{timestamp}'\n",
    "\n",
    "create_branch(repo,\n",
    "              branch=branch,\n",
    "              repo_type='model')\n",
    "\n",
    "model.push_to_hub(repo,\n",
    "                  branch=branch,\n",
    "                  commit_message=f'Model timestamp {timestamp}')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}