Motivation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "![](https://web-assets.domo.com/blog/wp-content/uploads/2016/06/16_domo_data-never-sleeps-4.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The amount of information that is being generated daily (minutely!) is pretty astounding.  So, whether we like it or not, we have entered the realm of \"Big Data\".  \n",
    "\n",
    "The question is \"What kinds of things can we learn from this data and how do we do it?\"  \n",
    "\n",
    "*What* we can learn depends on what you might hope to learn.  But *how* we can interact with Big Data is what **Data Mining** and **Machine Learning** are all about.  Data mining is exactly what it sounds like: sifting through piles of data in order to find something useful---like digging rock from the ground and extracting metal ores from it.  Machine learning is about how to do this using computers to leverage our ability to extract useful (hopefully!) information from the data.  \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Who does data mining and uses machine learning?  Seems like everyone these days\n",
    "\n",
    "    - Amazon and Target to predict things that you might buy or ads that may be of interest\n",
    "    - Google for everything, but most interesting these days (I think) is self-driving cars: http://dataconomy.com/how-data-science-is-driving-the-driverless-car/\n",
    "    - Insurance companies to predict how much of a risk it is to insure you\n",
    "    - Financial institutions to predict the future prices of their investments\n",
    "    - Goverments for all sorts of things\n",
    "    - Election prognosticators, e.g., http://fivethirtyeight.com/, http:election.princeton.edu\n",
    "    - Netflix to predict what movies you are likely to want to watch: https://en.wikipedia.org/wiki/Netflix_Prize\n",
    "    - Sports teams, e.g., https://en.wikipedia.org/wiki/Moneyball\n",
    "    - Face recognition tools\n",
    "    - and, of course, physicists!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "So there's definitely some interesting things to be learned and some money to be made if you have knowledge of the basics of data mining and machine learning.  \n",
    "\n",
    "But don't get too cocky as things can go [wrong](https://www.wired.com/2015/10/can-learn-epic-failure-google-flu-trends/), the world is made of [people](http://www.slate.com/blogs/how_not_to_be_wrong/2014/06/09/big_data_what_s_even_creepier_than_target_guessing_that_you_re_pregnant.html) and not data, and machine learning by itself [doesn't *intrinsically* make our lives better](https://www.amazon.com/Weapons-Math-Destruction-Increases-Inequality/dp/0553418815/ref=sr_1_1?ie=UTF8&qid=1473514058&sr=8-1&keywords=Weapons+of+Math\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**My** interest in the topic stems from the new generation of astronomical sky surveys like the [Large Synoptic Survey Telescope (LSST)](https://www.lsst.org/), of which Drexel is a member institution.  While the Big Data examples above are impressive, in many ways, projects like LSST *define* Big Data.  \n",
    "\n",
    "What do I mean by that?  Well, LSST is a project that is going to generate about 200 PB of data by the end of its 10 year mission.  During that time, it will have measured a hundred or more properties for some 40 billion objects---*every 3 nights*.  \n",
    "\n",
    "To put that into perspective, there are 7.4 billion people on the Earth today.  So, LSST will be equivalent to collecting dozens of pieces of information *per day* for *every* person on the Earth.  Google would go crazy with that!\n",
    "\n",
    "Even that is small potatoes for particle physicists.  In astronomy (at least up to now), we at least keep all of the data.  In particle physics, they throw away *most* of the data because there is so much.  Instead they have the notion of a *trigger*, which is basically as series of \"if-then\" statements that decide whether or not an \"event\" is worth saving (for future analsyis) or not."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### What the class is NOT\n",
    "\n",
    "* A Statistics Class\n",
    "* A Math Methods Class\n",
    "* A Computer Science Class\n",
    "* A Programming Class\n",
    "\n",
    "There are other classes for those things and I am **NOT** a Stats or CS professor.  There will definitely be some things that we are learning together rather than me transferring my knowledge to you!  Moreover, we only have 10 weeks, so I can't cover *everything*."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### What the class IS\n",
    "\n",
    "* An introduction to Big Data and (python-based) Machine Learning algorithms needed to handle it\n",
    "* Whatever stats we need to get there\n",
    "* My field of expertise and the book will bias things towards astronomy, but I'll try to make it as general as possible.\n",
    "\n",
    "\n",
    "In just 10 weeks we won't be able to go into as much detail as you might really need to do \"real\" data mining/machine learning.  My assumption is that if you need this, you'll follow through and make sure to understand the details later (otherwise, I'm just creating monsters!)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "This figure from drewconway.com nicely illustrates where we are trying to get: somewhere close to the middle!\n",
    "\n",
    "![http://static1.squarespace.com/static/5150aec6e4b0e340ec52710a/t/51525c33e4b0b3e0d10f77ab/1364352052403/Data_Science_VD.png?format=750w](http://static1.squarespace.com/static/5150aec6e4b0e340ec52710a/t/51525c33e4b0b3e0d10f77ab/1364352052403/Data_Science_VD.png?format=750w)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "At some level, you don't need this class at all.  You *could* just go through the book on your own (but how often does that happen without someone making you?) But I found that it isn't entirely self-contained and that you really need other resources too.  So, what I am trying to do with this class is to bring those resources together in one place.\n",
    "\n",
    "A group project is intended to help you delve into details for one algorithm/method (and learn from your classmates about others).  See the [syllabus](http://www.physics.drexel.edu/~gtr/teaching/physT480/) for more information on it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Almost everything that we will do can be categorized into one of two different pairs of things.\n",
    "\n",
    "- **Supervised learning** vs. **unsupervised learning**\n",
    "  - (Where it is the learning process, not *you* that is supervised or not.)\n",
    "  - Supervised learning means that we know the \"truth\" for (some) of the data that we are analyzing.  It is this \"truth\" that is \"supervising\" the learning process\n",
    "  - Unsupervised learning means that we do NOT know the \"right\" answer.  We are trying to find it.  Or, generally speaking, not so much as the right answer as *any* answer that increases our knowledge.\n",
    "  \n",
    "- **Classification** vs. **Regression**\n",
    "  - Classification means that we are trying to put our data into different discrete categories\n",
    "  - Regression is the limit where the classification \"bins\" become continuous.\n",
    "  \n",
    "These can be combined together:\n",
    " * supervised classification\n",
    " * unsupervised classification (aka clustering)\n",
    " * supervised regression\n",
    " * unsupervised regression (aka dimensional reduction)\n",
    "\n",
    "We'll talk about this in more detail when I introduce the scikit-learn package."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Graphically the course can be represented as a tour of the following [flowchart](http://scikit-learn.org/stable/tutorial/machine_learning_map/index.html) (where the link has links to each of these algorithms):\n",
    "\n",
    "![](http://scikit-learn.org/stable/_static/ml_map.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Lastly, here are some reading resources that you might find useful:\n",
    "\n",
    "[Ball & Brunner 2010](http://www.worldscientific.com/doi/pdf/10.1142/S0218271810017160): astro-specific, but serves as a good general introduction too\n",
    "\n",
    "[Bloom & Richards 2011](http://adsabs.harvard.edu/abs/2012amld.book...89B), where that Richards is not yours truly (in this case).\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}