homeworks/homework5.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#  Homework 5 - Berkeley STAT 157\n",
    "\n",
    "**Your name: XX, SID YY** (Please add your name, and SID to ease Ryan and Rachel to grade.)\n",
    "\n",
    "**Please submit your homework through [gradescope](http://gradescope.com/) instead of Github, so you will get the score distribution for each question. Please enroll in the [class](https://www.gradescope.com/courses/42432) by the Entry code: MXG5G5** \n",
    "\n",
    "Handout 2/19/2019, due 2/26/2019 by 4pm in Git by committing to your repository.\n",
    "\n",
    "In this homework, we will model covariate shift and attempt to fix it using logistic regression. This is a fairly realistic scenario for data scientists. To keep things well under control and understandable we will use [Fashion-MNIST](http://d2l.ai/chapter_linear-networks/fashion-mnist.html) as the data to experiment on. \n",
    "\n",
    "Follow the instructions from the Fashion MNIST notebook to get the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-02-12T19:52:59.411749Z",
     "start_time": "2019-02-12T19:52:28.081528Z"
    },
    "attributes": {
     "classes": [],
     "id": "",
     "n": "3"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "from mxnet import autograd, gluon, init, nd\n",
    "from mxnet.gluon import data as gdata, loss as gloss, nn, utils\n",
    "import numpy as np\n",
    "\n",
    "mnist_train = gdata.vision.FashionMNIST(train=True)\n",
    "mnist_test = gdata.vision.FashionMNIST(train=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Logistic Regression\n",
    "\n",
    "1. Implement the logistic loss function $l(y,f) = -\\log(1 + \\exp(-y f))$ in Gluon.\n",
    "2. Plot its values and its derivative for $y = 1$ and $f \\in [-5, 5]$, using automatic differentiation in Gluon.\n",
    "3. Generate training and test datasets for a binary classification problem using Fashion-MNIST with class $1$ being a combination of `sneaker` and `pullover` and class $-1$ being the combination of `sandal` and `shirt` categories. \n",
    "4. Train a binary classifier of your choice (it can be linear or a simple MLP such as from a previous lecture) using half the data (i.e. $12,000$ observations mixed as abvove) and one using the full dataset (i.e. $24,000$ observations as arising from the 4 categories) and report its accuracy. \n",
    "\n",
    "Hint - you should encapsulate the training and reporting code in a callable function since you'll need it quite a bit in the following. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Covariate Shift\n",
    "\n",
    "Your goal is to introduce covariate shit in the data and observe the accuracy. For this, compose a dataset of $12,000$ observations, given by a mixture of `sneaker` and `pullover` and of `sandal` and `shirt` respectively, where you use a fraction $\\lambda \\in \\{0.05, 0.1, 0.2, \\ldots 0.8, 0.9, 0.95\\}$ of one and a fraction of $1-\\lambda$ of  the other datasets respectively. For instance, you might pick for $\\lambda = 0.1$ a total of $600$ `sneaker` and $5,400$ `pullover` images and likewise $600$ `sandal` and $5,400$ `shirt` photos, yielding a total of $12,000$ images for training. Note that the test set remains unbiased, composed of $2,000$ photos for the `sneaker` + `pullover` category and of the `sandal` + `shirt` category each.\n",
    "\n",
    "1. Generate training sets that are appropriately biased. You should have 11 datasets.\n",
    "2. Train a binary classifier using this and report the test set accuracy on the unbiased test set. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Covariate Shift Correction\n",
    "\n",
    "Having observed that covariate shift can be harmful, let's try fixing it. For this we first need to compute the appropriate propensity scores $\\frac{dp(x)}{dq(x)}$. For this purpose pick a biased dataset, let's say with $\\lambda = 0.1$ and try to fix the covariate shift.\n",
    "\n",
    "1. When training a logistic regression binary classifier to fix covariate shift, we assumed so far that both sets are of equal size. Show that re-weighting data in training and test set appropriately can help address the issue when both datasets have different size. What is the weighting?\n",
    "2. Train a binary classifier (using logistic regression) distinguishing between the biased training set and the unbiased test set. Note - you need to weigh the data. \n",
    "3. Use the scores to compute weights on the training set. Do they match the weight arising from the biasing distribution $\\lambda$? \n",
    "4. Train a binary classifier of the covariate shifted problem using the weights obtained previously and report the accuracy. Note - you will need to modify the training loop slightly such that you can compute the gradient of a weighted sum of losses. \n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}