Polynomial Regression.ipynb

{
 "metadata": {
  "name": ""
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Work in progress!\n",
      "\n",
      "A polynomial of order N is a function that can be expressed as $p(x)=\\sum_{n=0}^{N}a_n x^n$\n",
      "\n",
      "We first generate some data, then fit it with polynomials of variable orders and compare error on training set and the generalization error on the test set.\n",
      "\n",
      "Let's first define some x-coordinates of our training and test dataset."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "n_points = 22\n",
      "x = linspace(0, 1, n_points)\n",
      "x_train = x[1::2] # x[0, 2, 4, ...]\n",
      "x_test = x[::2] # x[1, 3, 5, ...]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Now we define a toy funcion $\\tt{our\\_function}$ taking a parameter called $\\tt a$ as $f(a) = 0.2a + 0.01\\sin(10\\pi a)$."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "def our_function(a):\n",
      "    return 0.2 * a + 0.01 * sin(10 * pi * a)\n",
      "x_fine = linspace(0, 1, 100)\n",
      "plot(x_fine, our_function(x_fine))"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Now we can create our training and test target variables."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "y_train = our_function(x_train)\n",
      "y_test = our_function(x_test)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Now let's do some fitting. $\\tt polyfit$ takes the x-coordinates, the y-coordinates and the order as arguments and returns an array with the coefficients of the polynomial $a_0$, $a_1$, ..."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from numpy import polyfit\n",
      "popt = polyfit(x_train, y_train, 1) "
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "We can use the convenience functon poly1d to create the polynomial with the parameters we just found. And then use it to make predictions."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from numpy import poly1d\n",
      "p = poly1d(popt)\n",
      "y_train_pred = p(x_train)\n",
      "y_test_pred = p(x_test)\n",
      "plot(x_train, y_train, 'ob', label='training truth')\n",
      "plot(x_train, y_train_pred, 'sb', label='training prediction')\n",
      "plot(x_test, y_test, 'or', label='test truth')\n",
      "plot(x_test, y_test_pred, 'sr', label='test prediction')\n",
      "legend()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "def error(x, y):\n",
      "    return sum(pow(x - y, 2))\n",
      "ns = [1, 2, 5, n_points/2]\n",
      "x2 = linspace(0, 1, 101)\n",
      "for n in ns:\n",
      "    popt, residuals, _, _, _ = polyfit(x_train, y_train, n, full=True)\n",
      "    p = poly1d(popt)\n",
      "    prediction  = p(x_test)\n",
      "    generalization_error = error(y_test, p(x_test))\n",
      "    plot(x2, p(x2), label='polynomial of order {}'.format(n))\n",
      "plot(x_train, y_train, 'ob', label='Training')\n",
      "plot(x_test, y_test, 'sr', label='Test')\n",
      "legend()\n",
      "residuals(y_test, p(xtest)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    }
   ],
   "metadata": {}
  }
 ]
}