AI-progress-metrics.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Measuring the Progress of AI Research\n",
    "\n",
    "This pilot project collects problems and metrics/datasets from the AI research literature, and tracks progress on them. \n",
    "\n",
    "You can use this [Notebook](https://jupyter.org/#about-notebook) to see how things are progressing in specific subfields or AI/ML as a whole, as a place to report new results you've obtained, as a place to look for problems that might benefit from having new datasets/metrics designed for them, or as a source to build on for data science projects.\n",
    "\n",
    "At EFF, we're ultimately most interested in how this data can influence our understanding of the likely implications of AI. To begin with, we're [focused on gathering it](https://www.eff.org/deeplinks/2017/06/help-eff-track-progress-ai-and-machine-learning).\n",
    "\n",
    "<a name=\"sources\"></a>\n",
    "_Original authors: [Peter Eckersley](https://www.eff.org/about/staff/peter-eckersley) and [Yomna Nasser](https://ynasser.github.io/) at EFF_. Contact: [ai-metrics@eff.org](mailto:ai-metrics@eff.org).<br>\n",
    "\n",
    "With contributions from: [Yann Bayle](http://yannbayle.fr/french/index.php), [Owain Evans](https://www.fhi.ox.ac.uk/team/owain-evans/),  [Gennie Gebhart](https://www.eff.org/about/staff/gennie-gebhart) and [Dustin Schwenk](https://github.com/drschwenk).<br>\n",
    "\n",
    "Inspired by and merging data from:\n",
    "\n",
    "* Rodrigo Benenson's [\"Who is the Best at X / Are we there yet?\"](https://rodrigob.github.io/are_we_there_yet/build/#about) collating machine vision datasets & progress\n",
    "* Jack Clark and Miles Brundage's [collection of AI progress measurements](https://raw.githubusercontent.com/AI-metrics/master_text/master/archive/AI-metrics-data.txt)\n",
    "* Sarah Constantin's [Performance Trends in AI](https://srconstantin.wordpress.com/2017/01/28/performance-trends-in-ai/)\n",
    "* Katja Grace's [Algorithmic Progress in Six Domains](https://intelligence.org/files/AlgorithmicProgress.pdf)\n",
    "* The Swedish Computer Chess Association's [History of Computer Chess performance](https://en.wikipedia.org/wiki/Swedish_Chess_Computer_Association#Rating_list_year-end_leaders)\n",
    "* Gabriel Synnaeve's [WER are We](https://github.com/syhw/wer_are_we) collation of speech recognition performance data\n",
    "* Qi Wu _et al._'s [Visual Question Answering: A survey of Methods and Datasets](https://arxiv.org/abs/1607.05910)\n",
    "* Eric Yuan's [Comparison of Machine Reading Comprehension Datasets](http://eric-yuan.me/compare-popular-mrc-datasets/)\n",
    "\n",
    "Thanks to many others for valuable conversations, suggestions and corrections, including: Dario Amodei, James Bradbury, Miles Brundage, Mark Burdett, Breandan Considine, Owen Cotton-Barrett, Marc Bellemare, Will Dabny, Eric Drexler, Otavio Good, Katja Grace, Hado van Hasselt, Anselm Levskaya, Clare Lyle, Toby Ord, Michael Page, Maithra Raghu, Anders Sandberg, Laura Schatzkin, Daisy Stanton, Gabriel Synnaeve, Stacey Svetlichnaya, Helen Toner, and Jason Weston. EFF's work on this project has been supported by the [Open Philanthropy Project](http://www.openphilanthropy.org/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Table of Contents\n",
    "\n",
    "1. [Taxonomy](#Taxonomy)\n",
    "2. [Source code](#Source-Code) for defining and importing data\n",
    "3. [Problems, Metrics and Datasets](#Problems,-Metrics,-and-Datasets)\n",
    "    4. [Game Playing](#Game-Playing)\n",
    "        5. [Abstract Strategy Games](#Abstract-Strategy-Games)\n",
    "        6. [Real-time Video Games](#Real-time-video-games)\n",
    "    7. [Vision and image modelling](#Vision)\n",
    "        9. [Image recognition](#Vision)\n",
    "        8. [Visual Question Answering](#Visual-Question-Answering)\n",
    "        10. [Video recognition](#Vision)\n",
    "        11. [Generating images](#Image-Generation)\n",
    "    7. Written Language\n",
    "        8. [Reading Comprehension](#Reading-Comprehension)\n",
    "        9. [Language Modelling](#Language-Modelling-and-Comprehension)\n",
    "        10. [Conversation](#Conversation:-Chatbots-&-Conversational-Agents)\n",
    "        11. [Translation](#Translation)\n",
    "    8. Spoken Language\n",
    "        12. [Speech recognition](#Speech-recognition)\n",
    "    8. [Music Information Retrieval](#Music-Information-Retrieval)\n",
    "        12. [Instrumentals tracks recognition](#Instrumentals-recognition)\n",
    "    9. [Scientific and Technical Capabilities](#Scientific-and-Technical-capabilities)\n",
    "        10. [Solving constrained, well-specified technical problems](#Scientific-and-Technical-capabilities)\n",
    "        10. [Reading technical papers](#Scientific-and-Technical-capabilities)\n",
    "        10. [Solving real-world technical problems](#Scientific-and-Technical-capabilities)\n",
    "        10. [Generating computer programs from specifications](#Generating-computer-programs-from-specifications)\n",
    "        10. [Answering Science Exam Questions](#Answering-Science-Exam-Questions)\n",
    "\n",
    "    8. Learning to Learn Better\n",
    "        7. [Generalization](#Generalisation-and-Transfer-Learning)\n",
    "        7. [Transfer Learning](#Generalisation-and-Transfer-Learning)\n",
    "        7. [One-shot Learning](#Generalisation-and-Transfer-Learning)\n",
    "    8. [Safety and Security](#Safety-and-Security-Problems)\n",
    "        9. [\"Adversarial Examples\" and Manipulation of Classifiers](#\"Adversarial-Examples\"-and-manipulation-of-ML-classifiers)\n",
    "        9. [Safety for Reinforcement Learning Agents](#Safety-of-Reinforcement-Learning-Agents-and-similar-systems)\n",
    "        9. [Automated Hacking Systems](#Automated-Hacking-Systems)\n",
    "        9. [Pedestrian Detection for self-driving vehicles](#Pedestrian-Detection)\n",
    "    8. [Transparency, Explainability & Interpretability](#Explainability-and-Interpretability)\n",
    "    8. [Fairness and Debiasing](#Fairness-and-Debiasing)\n",
    "    8. [Privacy Problems](#Privacy)\n",
    "12. [Taxonomy and recorded progress to date](#Taxonomy-and-recorded-progress-to-date)\n",
    "    13. [Breakdown of Problems and Metrics by Type/Category](#Problems-and-Metrics-by-category)\n",
    "    \n",
    "14. [How to contribute to this project](#How-to-contribute-to-this-notebook)\n",
    "    15. [Notes on importing data](#Notes-on-importing-data)\n",
    "    15. [Exporting / building on this data](#Building-on-this-data)\n",
    "    15. [License](#License)\n",
    "    15. [How to cite this document](#How-to-cite-this-document)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "### Taxonomy\n",
    "\n",
    "It collates data with the following structure:\n",
    "\n",
    "```\n",
    "problem \n",
    "    \\   \\\n",
    "     \\   metrics  -  measures \n",
    "      \\\n",
    "       - subproblems\n",
    "            \\\n",
    "          metrics\n",
    "             \\\n",
    "            measure[ment]s\n",
    "```\n",
    "\n",
    "Problems describe the ability to learn an important category of task.\n",
    "\n",
    "Metrics should ideally be formulated in the form \"software is able to learn to do X given training data of type Y\". In some cases X is the interesting part, but sometimes also Y.\n",
    "\n",
    "Measurements are the score that a specific instance of a specific algorithm was able to get on a Metric.\n",
    "\n",
    "problems are tagged with attributes:\n",
    "eg, vision, abstract-games, language, world-modelling, safety\n",
    "\n",
    "Some of these are about performance relative to humans (which is of course a very arbitrary standard, but one we're familiar with)\n",
    "* agi -- most capable humans can do this, so AGIs can do this (note it's conceivable that an agent might pass the Turing test before all of these are won)\n",
    "* super -- the very best humans can do this, or human organisations can do this\n",
    "* verysuper -- neither humans nor human orgs can presently do this\n",
    "\n",
    "problems can have \"subproblems\", including simpler cases and preconditions for solving the problem in general\n",
    "\n",
    "a \"metric\" is one way of measuring progress on a problem, commonly associated with a test dataset. There will often be several metrics\n",
    "for a given problem, but in some cases we'll start out with zero metrics and will need to start proposing some...\n",
    "\n",
    "a measure[ment] is a score on a given metric, by a particular codebase/team/project, at a particular time\n",
    "\n",
    "The present state of the actual taxonomy is [at the bottom of this notebook](#Taxonomy-and-recorded-progress-to-date)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "# Source Code\n",
    "\n",
    "* Code implementing the taxonomy of Problems and subproblems, Metrics and Measurements is defined in a free-standing Python file, [taxonomy.py](https://github.com/AI-metrics/AI-metrics/blob/master/taxonomy.py). [scales.py](https://github.com/AI-metrics/AI-metrics/blob/master/scales.py) contains definitions of various unit systems used by `Metric`s.\n",
    "* Most source data is now defined in a series of separate files by topic:\n",
    "    * [data/vision.py](https://github.com/AI-metrics/AI-metrics/blob/master/data/vision.py) for hand-entered computer vision data\n",
    "    * [data/language.py](https://github.com/AI-metrics/AI-metrics/blob/master/data/language.py) for hand-entered and merged language data\n",
    "    * [data/strategy_games.py](https://github.com/AI-metrics/AI-metrics/blob/master/data/strategy_games.py) for data on abstract strategy games\n",
    "    * [data/video_games.py](https://github.com/AI-metrics/AI-metrics/blob/master/data/video_games.py) a combination of hand-entered and scraped Atari data (other video game data can also go here)\n",
    "    * [data/stem.py](https://github.com/AI-metrics/AI-metrics/blob/master/data/stem.py) for data on scientific & technical problems\n",
    "\n",
    "    * data imported from specific scrapers (and then subsequently edited):\n",
    "        * [Are We There Yet? image data](https://github.com/AI-metrics/AI-metrics/blob/master/data/awty.py), generated by `scrapers/awty.py` but then edited by hand\n",
    "    * For now, some of the `Problem`s and `Metric`s are still defined in this Notebook, especially in areas that do not have many active results yet.\n",
    "*  Scrapers for specific data sources:\n",
    "  * [scrapers/awty.py](https://github.com/AI-metrics/AI-metrics/blob/master/scrapers/awty.py) for importing data from Rodriguo Benenson's Are We There Yey? site\n",
    "  * [scrapers/es.py](https://github.com/AI-metrics/AI-metrics/blob/master/scrapers/es.py) for processing a pasted table of data from the Evolutionary Strategies Atari paper (is probably a useful model for other Atari papers)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "from IPython.display import HTML\n",
    "HTML('''\n",
    "<script>\n",
    "    if (typeof code_show == \"undefined\") {\n",
    "        code_show=false;\n",
    "    } else {\n",
    "        code_show = !code_show; // FIXME hack, because we toggle on load :/\n",
    "    }\n",
    "    function toggle_one(mouse_event) {\n",
    "        console.log(\"Unhiding \"+button + document.getElementById(button.region));\n",
    "        parent = button.parentNode;\n",
    "        console.log(\"Parent\" + parent)\n",
    "        input = parent.querySelector(\".input\");\n",
    "        console.log(\"Input\" + input + \" \" + input.classList + \" \" + input.style.display)\n",
    "        input.style.display = \"block\";\n",
    "        //$(input).show();\n",
    "    }\n",
    "    function code_toggle() {\n",
    "        if (!code_show) {\n",
    "            inputs = $('div.input');\n",
    "            for (n = 0; n < inputs.length; n++) {\n",
    "                if (inputs[n].innerHTML.match('# hidd' + 'encode'))\n",
    "                    inputs[n].style.display = \"none\";\n",
    "                    button = document.createElement(\"button\");\n",
    "                    button.innerHTML=\"unhide code\";\n",
    "                    button.style.width = \"100px\";\n",
    "                    button.style.marginLeft = \"90px\";\n",
    "                    button.addEventListener(\"click\", toggle_one);\n",
    "                    button.classList.add(\"cell-specific-unhide\")\n",
    "                    // inputs[n].parentNode.appendChild(button);\n",
    "            }\n",
    "        } else { \n",
    "            $('div.input').show();\n",
    "            $('button.cell-specific-unhide').remove()\n",
    "        } \n",
    "        code_show = !code_show;\n",
    "    } \n",
    "    \n",
    "    $( document ).ready(code_toggle);\n",
    "    \n",
    "</script>\n",
    "<form action=\"javascript:code_toggle()\">\n",
    "    <input type=\"submit\" value=\"Click here to show/hide source code cells.\"> <br><br>(you can mark a cell as code with <tt># hiddencode</tt>)\n",
    "</form>\n",
    "''')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "hidden": true,
    "scrolled": true,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "# hiddencode\n",
    "from __future__ import print_function\n",
    "\n",
    "%matplotlib inline  \n",
    "import matplotlib as mpl\n",
    "try:\n",
    "    from lxml.cssselect import CSSSelector\n",
    "except ImportError:\n",
    "    # terrifying magic for Azure Notebooks\n",
    "    import os\n",
    "    if os.getcwd() == \"/home/nbuser\":\n",
    "        !pip install cssselect\n",
    "        from lxml.cssselect import CSSSelector\n",
    "    else:\n",
    "        raise\n",
    "\n",
    "import datetime\n",
    "import json\n",
    "import re\n",
    "\n",
    "from matplotlib import pyplot as plt\n",
    "\n",
    "date = datetime.date\n",
    "\n",
    "import taxonomy\n",
    "#reload(taxonomy)\n",
    "from taxonomy import Problem, Metric, problems, metrics, measurements, all_attributes, offline, render_tables\n",
    "from scales import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    " # Problems, Metrics, and Datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "## Vision\n",
    "\n",
    "<div style=\"float: right\">\n",
    "<br>\n",
    "<img src=\"images/imagenet.jpg\" style=\"width:80%; height: 80%\">\n",
    "<div style=\"text-align:center\">(Imagenet example data)</div>\n",
    "</div>\n",
    "\n",
    "The simplest vision subproblem is probably image classification, which determines what objects are present in a picture. From 2010-2017, Imagenet has been a closely watched contest for progress in this domain.\n",
    "\n",
    "Image classification includes not only recognising single things within an image, but localising them and essentially specifying which pixels are which object. MSRC-21 is a metric that is specifically for that task:\n",
    "\n",
    "<br>\n",
    "<div style=\"float:left; margin-left: 5%\">\n",
    "<img src=\"images/msrc_21.png\" style=\"margin-bottom: 10px\">\n",
    "<div style=\"text-align:center\">(MSRC 21 example data)</div>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "from data.vision import *\n",
    "imagenet.graph()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "hidden": true
   },
   "outputs": [],
   "source": [
    "from data.vision import *\n",
    "from data.awty import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "for m in sorted(image_classification.metrics, key=lambda m:m.name): \n",
    "    if m != imagenet: m.graph()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "AWTY, not yet imported:\n",
    "\n",
    "```\n",
    "Handling 'Pascal VOC 2011 comp3' detection_datasets_results.html#50617363616c20564f43203230313120636f6d7033\n",
    "Skipping 40.6 mAP Fisher and VLAD with FLAIR CVPR 2014\n",
    "Handling 'Leeds Sport Poses' pose_estimation_datasets_results.html#4c656564732053706f727420506f736573\n",
    "69.2 %                  Strong Appearance and Expressive Spatial Models for Human Pose Estimation  ICCV 2013\n",
    "64.3 %                                    Appearance sharing for collective human pose estimation  ACCV 2012\n",
    "63.3 %                                                   Poselet conditioned pictorial structures  CVPR 2013\n",
    "60.8 %                                Articulated pose estimation with flexible mixtures-of-parts  CVPR 2011\n",
    " 55.6%           Pictorial structures revisited: People detection and articulated pose estimation  CVPR 2009\n",
    "Handling 'Pascal VOC 2007 comp3' detection_datasets_results.html#50617363616c20564f43203230303720636f6d7033\n",
    "Skipping 22.7 mAP Ensemble of Exemplar-SVMs for Object Detection and Beyond ICCV 2011\n",
    "Skipping 27.4 mAP Measuring the objectness of image windows PAMI 2012\n",
    "Skipping 28.7 mAP Automatic discovery of meaningful object parts with latent CRFs CVPR 2010\n",
    "Skipping 29.0 mAP Object Detection with Discriminatively Trained Part Based Models PAMI 2010\n",
    "Skipping 29.6 mAP Latent Hierarchical Structural Learning for Object Detection CVPR 2010\n",
    "Skipping 32.4 mAP Deformable Part Models with Individual Part Scaling BMVC 2013\n",
    "Skipping 34.3 mAP Histograms of Sparse Codes for Object Detection CVPR 2013\n",
    "Skipping 34.3 mAP Boosted local structured HOG-LBP for object localization CVPR 2011\n",
    "Skipping 34.7 mAP Discriminatively Trained And-Or Tree Models for Object Detection CVPR 2013\n",
    "Skipping 34.7 mAP Incorporating Structural Alternatives and Sharing into Hierarchy for Multiclass Object Recognition and Detection CVPR 2013\n",
    "Skipping 34.8 mAP Color Attributes for Object Detection CVPR 2012\n",
    "Skipping 35.4 mAP Object Detection with Discriminatively Trained Part Based Models PAMI 2010\n",
    "Skipping 36.0 mAP Machine Learning Methods for Visual Object Detection archives-ouvertes 2011\n",
    "Skipping 38.7 mAP Detection Evolution with Multi-Order Contextual Co-occurrence CVPR 2013\n",
    "Skipping 40.5 mAP Segmentation Driven Object Detection with Fisher Vectors ICCV 2013\n",
    "Skipping 41.7 mAP Regionlets for Generic Object Detection ICCV 2013\n",
    "Skipping 43.7 mAP Beyond Bounding-Boxes: Learning Object Shape by Model-Driven Grouping ECCV 2012\n",
    "Handling 'Pascal VOC 2007 comp4' detection_datasets_results.html#50617363616c20564f43203230303720636f6d7034\n",
    "Skipping 59.2 mAP Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition ECCV 2014\n",
    "Skipping 58.5 mAP Rich feature hierarchies for accurate object detection and semantic segmentation CVPR 2014\n",
    "Skipping 29.0 mAP Multi-Component Models for Object Detection ECCV 2012\n",
    "Handling 'Pascal VOC 2010 comp3' detection_datasets_results.html#50617363616c20564f43203230313020636f6d7033\n",
    "Skipping 24.98 mAP Learning Collections of Part Models for Object Recognition CVPR 2013\n",
    "Skipping 29.4 mAP Discriminatively Trained And-Or Tree Models for Object Detection CVPR 2013\n",
    "Skipping 33.4 mAP Object Detection with Discriminatively Trained Part Based Models PAMI 2010\n",
    "Skipping 34.1 mAP Segmentation as selective search for object recognition ICCV 2011\n",
    "Skipping 35.1 mAP Selective Search for Object Recognition IJCV 2013\n",
    "Skipping 36.0 mAP Latent Hierarchical Structural Learning for Object  Detection CVPR 2010\n",
    "Skipping 36.8 mAP Object Detection by Context and Boosted HOG-LBP ECCV 2010\n",
    "Skipping 38.4 mAP Segmentation Driven Object Detection with Fisher Vectors ICCV 2013\n",
    "Skipping 39.7 mAP Regionlets for Generic Object Detection ICCV 2013\n",
    "Skipping 40.4 mAP Fisher and VLAD with FLAIR CVPR 2014\n",
    "Handling 'Pascal VOC 2010 comp4' detection_datasets_results.html#50617363616c20564f43203230313020636f6d7034\n",
    "Skipping 53.7 mAP Rich feature hierarchies for accurate object detection and semantic segmentation CVPR 2014\n",
    "Skipping 40.4 mAP Bottom-up Segmentation for Top-down Detection CVPR 2013\n",
    "Skipping 33.1 mAP Multi-Component Models for Object Detection ECCV 2012\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "from IPython.display import HTML\n",
    "HTML(image_classification.tables())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "<img src=\"images/vqa.jpg\" style=\"width: 50%; height: 50%; float: right\" float=\"right\">\n",
    "\n",
    "### Visual Question Answering\n",
    "\n",
    "\n",
    "Comprehending an image involves more than recognising what objects or entities are within it, but recognising events, relationships, and context from the image. This problem requires both sophisticated image recognition, language, world-modelling, and \"image comprehension\". There are several datasets in use. The illustration is from VQA, which was generated by asking Amazon Mechanical Turk workers to propose questions about photos from Microsoft's COCO image collection."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "plot = vqa_real_oe.graph(keep=True, title=\"COCO Visual Question Answersing (VQA) real open ended\", llabel=\"VQA 1.0\")\n",
    "vqa2_real_oe.graph(reuse=plot, llabel=\"VQA 2.0\", fcol=\"#00a0a0\", pcol=\"#a000a0\")\n",
    "for m in image_comprehension.metrics:\n",
    "    m.graph() if not m.graphed else None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "HTML(image_comprehension.tables())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "scrolled": true
   },
   "source": [
    "# Game Playing\n",
    "\n",
    "In principle, games are a sufficiently open-ended framework that all of intelligence could be captured within them. We can imagine a \"ladder of games\" which grow in sophistication and complexity, from simple strategy and arcade games to others which require very sophisticated language, world-modelling, vision and reasoning ability. At present, published reinforcement agents are climbing the first few rungs of this ladder.\n",
    "\n",
    "\n",
    "## Abstract Strategy Games\n",
    "\n",
    "As an easier case, abstract games like chess, go, checkers etc can be played with no knowldege of the human world or physics. Although this domain has largely been solved to super-human performance levels, there are a few ends that need to be tied up, especially in terms of having agents learn rules for arbitrary abstract games effectively given various plausible starting points (eg, textual descriptions of the rules or examples of correct play)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "from data.strategy_games import *\n",
    "computer_chess.graph()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "HTML(computer_chess.table())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "## Real-time video games\n",
    "\n",
    "Computer and video games are a very open-ended domain. It is possible that some existing or future games could be so elaborate that they are \"AI complete\". In the mean time, a lot of interesting progress is likely in exploring the \"ladder of games\" of increasing complexity on various fronts.\n",
    "\n",
    "#### Atari 2600\n",
    "\n",
    "Atari 2600 games have been a popular target for reinforcement learning, especially at DeepMind and OpenAI.  RL agents now play most but not all of these games better than humans.\n",
    "\n",
    "In the Atari 2600 data, the label \"noop\" indicates the game was played with a random number, up to 30, of \"no-op\" moves at the beginning, while the \"hs\" label indicates that the starting condition was a state sampled from 100 games played by expert human players. These forms of randomisation give RL systems a diversity of game states to learn from."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "from data.video_games import *\n",
    "from scrapers.atari import *\n",
    "simple_games.graphs()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "HTML(simple_games.tables())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "## Speech recognition\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "from data.acoustics import *\n",
    "from data.wer import *\n",
    "speech_recognition.graphs()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "HTML(speech_recognition.tables())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "## Music Information Retrieval\n",
    "\n",
    "### Instrumentals recognition\n",
    "\n",
    "Instrumentals recognition in a representative musical dataset for Instrumentals playlist generation.\n",
    "- Experiments tested on _SATIN_ database from [Bayle et al. (2017)](https://www.researchgate.net/publication/317824533_SATIN_A_Persistent_Musical_Database_for_Music_Information_Retrieval).\n",
    "- The ratio Instrumentals/Songs (11%/89%) of _SATIN_ is representative of real uneven musical databases.\n",
    "- The human performance is at 99% of correct instrumentals detection because there are [known examples of possible confusion](https://youtu.be/OcioBFokUnQ?t=56s)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "from data.acoustics import *\n",
    "instrumentals_recognition.graphs()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "HTML(instrumentals_recognition.tables())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "# Image Generation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "from data.generative import *\n",
    "image_generation_metric.graph()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "HTML(image_generation_metric.table())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "# Language Modelling and Comprehension"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Text compression is one way to see how well machine learning systems are able to model human language. Shannon's [classic 1951 paper]() obtained an expiermental measure of human text compression performance at 0.6 - 1.3 bits per character: humans know, better than classic algorithms, what word is likely to come next in a piece of writing. More recent work ([Moradi 1998](https://pdfs.semanticscholar.org/48bc/ce35ceb72068723d5f360f388a073aadadca.pdf), Cover 1978) provides estimates that are text-relative and in the 1.3 bits per character (and for some texts, much higher) range."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "from data.language import *\n",
    "ptperplexity.graph()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "HTML(ptperplexity.table())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "hp_compression.graph()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "HTML(hp_compression.table())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "LAMBADA is a challenging language modelling dataset in which the model has to predict a next word in a discourse, when that exact word has not occurred in the test. For instance, given a context like this:\n",
    "\n",
    "> He shook his head, took a step back and held his hands up as he tried to smile without losing a cigarette. “Yes you can,” Julia said in a reassuring voice. “I’ve already focused on my friend. You just have to click the shutter, on top, here.”\n",
    "\n",
    "And a target sentence: \n",
    "\n",
    "> He nodded sheepishly, through his cigarette away and took the \\_\\_\\_\\_\\_\\_\\_\\_\\_.\n",
    "\n",
    "The task is to guess the target word \"**camera**\".\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "lambada.graph()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Also consider adding the Microsoft Sentence Completion Challenge; see eg http://www.fit.vutbr.cz/~imikolov/rnnlm/thesis.pdf table 7.4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "## Translation\n",
    "\n",
    "Translation is a tricky problem to score, since ultimately it is human comprehension or judgement that determines whether a translation is accurate. Google for instance uses human evaluation to determine when their algorithms have improved. But that kind of measurement is expensive and difficult to replicate accurately, so automated scoring metrics are also widely used in the field. Perhaps the most common of these are [BLEU scores](https://en.wikipedia.org/wiki/BLEU) for corpora that have extensive professional human translations, which forms the basis for the measurements included here:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "from data.language import *\n",
    "en_fr_bleu.graph()\n",
    "plot = en_de_bleu.graph(keep=True, title=\"En-De Traslation BLEU scores\", llabel=\"news-test-2014\")\n",
    "en_de_bleu15.graph(reuse=plot, llabel=\"news-test-2015\", fcol=\"#00a0a0\", pcol=\"#a000a0\", suppress_target=True)\n",
    "en_ro_bleu.graph()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "HTML(translation.tables())\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "# Conversation: Chatbots & Conversational Agents\n",
    "\n",
    "Conversation is the classic AI progress measure! There is the Turing test, which involves a human judge trying to tell the difference between a humand and computer that they are chatting to online, and also easier variants of the Turing test in which the judge limits themselves to more casual, less probing conversation in various ways.\n",
    "\n",
    "The Loebner Prize is an annual event that runs a somewhat easier version of the test. Since 2014, the event has also been giving standard-form tests to their entrants, and scoring the results (each question gets a plausible/semi-plausible/implausible rating). This metric is not stable, because the test questions have to change every year, they are somewhat indicative of progress. Ideally the event might apply each year's test questions to the most successful entrants from prior years. Here is an example from 2016:\n",
    "\n",
    "<img src=\"images/loebner.png\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "loebner.graph()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "HTML(loebner.table())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "## Reading Comprehension"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "The Facebook BABI 20 QA dataset is an example of a basic reading comprehension task. It has been solved with large training datasets (10,000 examples per task) but not with a smaller training dataset of 1,000 examples for each of the 20 categories of tasks. It involves learning to answer simple reasoning questions like these:\n",
    "\n",
    "<img src=\"images/babi20qa.png\" style=\"width: 66%; height: 66%\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "There are numerous other reading comprehension metrics that are in various ways harder than bAbi 20 QA. They are generally not solved, though progress is fairly promising."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "for m in reading_comprehension.metrics: m.graphed = False\n",
    "plot = bAbi1k.graph(keep=True, title=\"bAbi 20 QA reading comprehension\", llabel=\"1k training examples\")\n",
    "bAbi10k.graph(reuse=plot, llabel=\"10k training examples\", fcol=\"#00a0a0\", pcol=\"#a000a0\", suppress_target=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another reading comprehension dataset that has received significant recent attention is the [Stanford Question Answering Dataset (SQuAD)](https://rajpurkar.github.io/SQuAD-explorer/). The literature reports both [F1 scores](https://en.wikipedia.org/wiki/F1_score) and exact match scores, though these are closely correlated:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot = squad_f1.graph(keep=True, title=\"Stanford Question Answering Dataset (SQuAD)\", tcol=\"g\", llabel=\"F1 score\")\n",
    "squad_em.graph(reuse=plot, llabel=\"Exact Match (EM)\", fcol=\"#00a0a0\", pcol=\"#a000a0\", tcol=\"#00a0a0\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "for m in reading_comprehension.metrics:\n",
    "    if not m.graphed: m.graph()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "HTML(reading_comprehension.tables())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Scientific and Technical capabilities\n",
    "\n",
    "Arguably reading and understanding scientific, technical, engineering and medical documents would be taxonomically related to general reading comprehension, but these technical tasks are probably much more difficult, and will certainly be solved with separate efforts. So we classify them separately for now. We also classify some of these problems as superintelligent, because only a tiny fraction of humans can read STEM papers, and only a miniscule fraction of humans are capable of reasonably comprehending STEM papers across a large range of fields.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from data.stem import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"float:left; position:relative\">\n",
    "  <div style=\"float:right; text-align:center\">\n",
    "      <img src=\"images/cards.jpg\" style=\"width: 33%; height: 33%\">\n",
    "      Example Magic The Gathering (MTG) and Hearthstone (HS) cards\n",
    "      <img src=\"images/mtg-dragon-code.png\" style=\"width: 33%; height: 33%\">\n",
    "      Corresponding MTG card implementation in Java\n",
    "  </div>\n",
    "  <div style=\"text-align:left\">\n",
    "    <h2>Generating computer programs from specifications</h2>\n",
    "\n",
    "        A particularly interesting technical problem, which may be slightly harder than problems with very clear constraints like circuit design, is generating computer programs from natural language specifications (which will often contain ambiguities of various sorts). This is presently a very unsolved problem, though there is now at least one good metric / dataset for it, which is [Deepmind's \"card2code\" dataset](https://github.com/deepmind/card2code) of Magic the Gathering and Hearthstone cards, along with Java and Python implementations (respectively) of the logic on the cards. Shown below is a figure from [_Ling, et al. 2016_](https://arxiv.org/abs/1603.06744v1) with their Latent Predictor Networks generating part of the code output for a Hearthstone card:\n",
    "    <img src=\"images/hs-lpn.jpg\" style=\"width: 66%; height: 66%\">\n",
    "  </div>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "card2code_hs_acc.graph()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "HTML(card2code_hs_acc.table())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Answering Science Exam Questions\n",
    "\n",
    "Science exam question answering is a multifaceted task that pushes the limits of artificial intelligence. As indicated by the example questions pictured, successful science exam QA requires natural language understanding, reasoning, situational modeling, and commonsense knowledge; a challenge problem for which information-retrieval methods alone are not sufficient to earn a \"passing\" grade. \n",
    "\n",
    "The [AI2 Science Questions](http://data.allenai.org/ai2-science-questions/) dataset provided by the <a href=\"http://allenai.org/\">Allen Institute for Artificial Intelligence</a> (AI2) is a freely available collection of 5,059 real science exam questions derived from a variety of regional and state science exams. <a href=\"http://allenai.org/aristo/\">Project Aristo</a> at AI2 is focused on the task of science question answering – the Aristo system is composed of a suite of various knowledge extraction methods, diagram processing tools, and solvers. As a reference point, the system currently achieves the following scores on these sets of non-diagram multiple choice (NDMC) and diagram multiple choice (DMC) science questions at two different grade levels. Allen Institute staff [claim these states of the art](https://github.com/AI-metrics/AI-metrics/pull/60) for Aristo [<i>Scores are listed as \"Subset (Train/Dev/Test)\"</i>]:\n",
    "\n",
    "<ul>\n",
    "<li>Elementary NDMC (63.2/60.2/61.3)\n",
    "<li>Elementary DMC (41.8/41.3/36.3)\n",
    "<li>Middle School NDMC (55.5/57.6/57.9) \n",
    "<li>Middle School DMC (38.4/35.3/34.3)\n",
    "</ul>\n",
    "\n",
    "<img src=\"images/science_qa.jpg\" style=\"width: 66%; height: 66%\">\n",
    "\n",
    "Another science question answering dataset that has been studied in the literature is based specifically on New York Regents 4th grade science exam tests:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ny_4_science.graph()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "HTML(science_question_answering.tables())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Learning to Learn\n",
    "\n",
    "## Generalisation and Transfer Learning\n",
    "\n",
    "ML systems are making strong progress at solving specific problems with sufficient training data. But we know that humans are\n",
    "capable of _transfer learning_ -- applying things they've learned from one context, with appropriate variation, to another context.\n",
    "Humans are also very general; rather than just being taught to perform specific tasks, a single agent is able to do a very\n",
    "wide range of tasks, learning new things or not as required by the situation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "generalisation = Problem(\"Building systems that solve a wide range of diverse problems, rather than just specific ones\")\n",
    "generalisation.metric(\"Solve all other solved problems in this document, with a single system\", solved=False)\n",
    "\n",
    "transfer_learning = Problem(\"Transfer learning: apply relevant knowledge from a prior setting to a new slightly different one\")\n",
    "arcade_transfer = Problem(\"Transfer of learning within simple arcade game paradigms\")\n",
    "\n",
    "generalisation.add_subproblem(transfer_learning)\n",
    "transfer_learning.add_subproblem(arcade_transfer)\n",
    "\n",
    "# These will need to be specified a bit more clearly to be proper metrics, eg \"play galaga well having trained on Xenon 2\" or whatever\n",
    "# the literature has settled on\n",
    "# arcade_transfer.metric(\"Transfer learning of platform games\")\n",
    "# arcade_transfer.metric(\"Transfer learning of vertical shooter games\")\n",
    "# arcade_transfer.metric(\"Transfer from a few arcade games to all of them\")\n",
    "\n",
    "one_shot_learning = Problem(\"One shot learning: ingest important truths from a single example\", [\"agi\", \"world-modelling\"])\n",
    "\n",
    "uncertain_prediction = Problem(\"Correctly identify when an answer to a classification problem is uncertain\")\n",
    "uncertain_prediction.notes = \"Humans can usually tell when they don't know something. Present ML classifiers do not have this ability.\"\n",
    "\n",
    "interleaved_learning = Problem(\"Learn a several tasks without undermining performance on a first task, avoiding catastrophic forgetting\", url=\"https://arxiv.org/abs/1612.00796\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "# Safety and Security Problems\n",
    "\n",
    "The notion of \"safety\" for AI and ML systems can encompass many things. In some cases it's about ensuring that the system meets various sorts of constraints, either in general or for specifically safety-critical purposes, such as [correct detection of pedestrians](#pedestrian detection) for self driving cars."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "### \"Adversarial Examples\" and manipulation of ML classifiers\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "hidden": true
   },
   "outputs": [],
   "source": [
    "adversarial_examples = Problem(\"Resistance to adversarial examples\", [\"safety\", \"agi\", \"security\"], url=\"https://arxiv.org/abs/1312.6199\")\n",
    "\n",
    "adversarial_examples.notes = \"\"\"\n",
    "We know that humans have significant resistance to adversarial examples.  Although methods like camouflage sometimes\n",
    "work to fool us into thinking one thing is another, those\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### Safety of Reinforcement Learning Agents and similar systems"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# This section is essentially on teaching ML systems ethics and morality. Amodei et al call this \"scaleable supervision\".\n",
    "scalable_supervision = Problem(\"Scalable supervision of a learning system\", [\"safety\", \"agi\"], url=\"https://arxiv.org/abs/1606.06565\")\n",
    "cirl = Problem(\"Cooperative inverse reinforcement learning of objective functions\", [\"safety\", \"agi\"], url=\"https://arxiv.org/abs/1606.03137\")\n",
    "cirl.notes = \"This is tagged agi because most humans are able to learn ethics from their surrounding community\"\n",
    "# Co-operative inverse reinforcement learning might be equivalent to solving scalable supervision, or there might other subproblems here\n",
    "scalable_supervision.add_subproblem(cirl)\n",
    "\n",
    "safe_exploration = Problem(\"Safe exploration\", [\"safety\", \"agi\", \"world-modelling\"], url=\"https://arxiv.org/abs/1606.06565\")\n",
    "safe_exploration.notes = \"\"\"\n",
    "Sometimes, even doing something once is catastrophic. In such situations, how can an RL agent or some other AI system\n",
    "learn about the catastrophic consequences without even taking the action once? This is an ability that most humans acquire\n",
    "at some point between childhood and adolescence.\n",
    "\"\"\"\n",
    "# safe exploration may be related to one shot learning, though it's probably too early to mark that so clearly.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The work by <a href=\"https://arxiv.org/pdf/1707.05173.pdf\">Saunders <em>et al.</em> (2017)</a> is an example of attempting to deal with the safe exploration problem by human-in-the-loop supervision. Without this oversight, a reinforcement learning system may engage in \"reward hacking\" in some Atari games. For instance in the Atari 2600 Road Runner game, an RL agent may deliberately kill itself to stay on level 1, because it can get more points on that level than it can on level 2 (particularly when it has not yet learned to master level 2). Human oversight overcomes this problem:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# hiddencode\n",
    "HTML(\"\"\"\n",
    "<video id=\"video\" width=\"80%\" height=\"%45\" controls poster=\"images/road-runner-poster.jpg\" onclick=\"this.paused?this.play():this.pause();\">\n",
    "   <source src=\"video/saunders-roadrunner.mp4\" type=\"video/mp4\">\n",
    "</video>\n",
    "\n",
    "<div style=\"text-align:right; margin-right:20%\">\n",
    "  Futher videos from that project are on <a href=\"https://www.youtube.com/playlist?list=PLjs9WCnnR7PCn_Kzs2-1afCsnsBENWqor\">on YouTube</a>.\n",
    "</div>\n",
    "\"\"\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "avoiding_reward_hacking = Problem(\"Avoiding reward hacking\", [\"safety\"], url=\"https://arxiv.org/abs/1606.06565\")\n",
    "avoiding_reward_hacking.notes = \"\"\"\n",
    "Humans have only partial resistance to reward hacking.\n",
    "Addiction seems to be one failure to exhibit this resistance.\n",
    "Avoiding learning something because it might make us feel bad, or even building elaborate systems of self-deception, are also sometimes\n",
    "seen in humans. So this problem is not tagged \"agi\".\n",
    "\"\"\"\n",
    "\n",
    "avoiding_side_effects = Problem(\"Avoiding undesirable side effects\", [\"safety\"], url=\"https://arxiv.org/abs/1606.06565\")\n",
    "avoiding_side_effects.nodes = \"\"\"\n",
    "Many important constraints on good behaviour will not be explicitly\n",
    "encoded in goal specification, either because they are too hard to capture\n",
    "or simply because there are so many of them and they are hard to enumerate\n",
    "\"\"\"\n",
    "\n",
    "robustness_to_distributional_change = Problem(\"Function correctly in novel environments (robustness to distributional change)\", [\"safety\", \"agi\"], url=\"https://arxiv.org/abs/1606.06565\")\n",
    "\n",
    "copy_bounding = Problem(\"Know how to prevent an autonomous AI agent from reproducing itself an unbounded number of times\", [\"safety\"])\n",
    "\n",
    "safety = Problem(\"Know how to build general AI agents that will behave as expected\")\n",
    "safety.add_subproblem(adversarial_examples)\n",
    "safety.add_subproblem(scalable_supervision)\n",
    "safety.add_subproblem(safe_exploration)\n",
    "safety.add_subproblem(avoiding_reward_hacking)\n",
    "safety.add_subproblem(avoiding_side_effects)\n",
    "safety.add_subproblem(robustness_to_distributional_change)\n",
    "safety.add_subproblem(copy_bounding)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "### Automated Hacking Systems\n",
    "\n",
    "Automated tools are becoming increasingly effective both for offensive and defensive computer security purposes.\n",
    "\n",
    "On the defensive side, fuzzers and static analysis tools have been used for some time by well-resourced software development teams to reduce the number of vulnerabilities in the code they ship.\n",
    "\n",
    "Assisting both offense and defense, DARPA has recently started running the [Cyber Grand Challenge](https://www.cybergrandchallenge.com/) contest to measure and improve the ability of agents to either break into systems or defend those same systems against vulnerabilities. It [isn't necessarily clear](https://www.eff.org/deeplinks/2016/08/darpa-cgc-safety-protocol) how such initiatives would change the security of various systems.\n",
    "\n",
    "This section includes some clear AI problems (like learning to find exploitable vulnerabilities in code) and some less pure AI problems, such as ensuring that defensive versions of this technology (whether in the form of fuzzers, IPSes, or other things) are deployed on all critical systems."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# It isn't totally clear whether having automated systems be good at finding bugs in and of itself will make the deployment\n",
    "# of AI technologies safer or less safe, so we tag this both with \"safety\" and as a potentialy \"unsafe\" development\n",
    "bug_finding = Problem(\"Detect security-related bugs in codebases\", [\"safety\", \"security\", \"unsafe\"])\n",
    "\n",
    "# However what\n",
    "defensive_deployment = Problem(\"Deploy automated defensive security tools to protect valuable systems\")\n",
    "defensive_deployment.notes = \"\"\"\n",
    "It is clearly important is ensuring that the state of the art in defensive technology is deployed everywhere\n",
    "that matters, including systems that perform important functions or have sensitive data on them (smartphones, for instance), and \n",
    "systems that have signifcant computational resources. This \"Problem\" isn't \n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "### Pedestrian Detection\n",
    "\n",
    "Detecting pedestrians from images or video is a specific image classification problem that has received a lot of attention because of\n",
    "its importance for self-driving vehicles. Many metrics in this space are based on the [Caltech pedestrians toolkit](http://www.vision.caltech.edu/Image_Datasets/CaltechPedestrians/), thought the [KITTI Vision Benchmark](http://www.cvlibs.net/datasets/kitti/eval_object.php) goes beyond that to include cars and cyclists in addition to pedestrians. We may want to write scrapers for Caltech's published results and KITTI's live results table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "pedestrian_detection = Problem(\"Pedestrian, bicycle & obstacle detection\", [\"safety\", \"vision\"])\n",
    "image_classification.add_subproblem(pedestrian_detection)\n",
    "\n",
    "# TODO: import data from these pedestrian datasets/metrics.\n",
    "# performance on them is a frontier of miss rate / false positive tradeoffs, \n",
    "# so we'll need to chose how to handle that as a scale\n",
    "\n",
    "# http://www.vision.caltech.edu/Image_Datasets/CaltechPedestrians/rocs/UsaTestRocReasonable.pdf\n",
    "pedestrian_detection.metric(\"Caltech Pedestrians USA\", url=\"http://www.vision.caltech.edu/Image_Datasets/CaltechPedestrians/\")\n",
    "# http://www.vision.caltech.edu/Image_Datasets/CaltechPedestrians/rocs/InriaTestRocReasonable.pdf\n",
    "pedestrian_detection.metric(\"INRIA persons\", url=\"http://pascal.inrialpes.fr/data/human/\")\n",
    "# http://www.vision.caltech.edu/Image_Datasets/CaltechPedestrians/rocs/ETHRocReasonable.pdf\n",
    "pedestrian_detection.metric(\"ETH Pedestrian\", url=\"http://www.vision.ee.ethz.ch/~aess/dataset/\")\n",
    "# http://www.vision.caltech.edu/Image_Datasets/CaltechPedestrians/rocs/TudBrusselsRocReasonable.pdf\n",
    "pedestrian_detection.metric(\"TUD-Brussels Pedestrian\", url=\"http://www.d2.mpi-inf.mpg.de/tud-brussels\")\n",
    "# http://www.vision.caltech.edu/Image_Datasets/CaltechPedestrians/rocs/DaimlerRocReasonable.pdf\n",
    "pedestrian_detection.metric(\"Damiler Pedestrian\", url=\"http://www.gavrila.net/Datasets/Daimler_Pedestrian_Benchmark_D/Daimler_Mono_Ped__Detection_Be/daimler_mono_ped__detection_be.html\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Explainability and Interpretability\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "explainability = Problem(\"Modify arbitrary ML systems in order to be able to provide comprehensible human explanations of their decisions\")\n",
    "\n",
    "statistical_explainability = Problem(\"Provide mathematical or technical explanations of decisions from classifiers\")\n",
    "statistical_explainability.notes = \"\"\"\n",
    "Providing explanations with techniques such as monte carlo analysis may in general\n",
    "be easier than providing robust ones in natural language (since those may or may not\n",
    "exist in all cases)\n",
    "\"\"\"\n",
    "\n",
    "explainability.add_subproblem(statistical_explainability)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Fairness and Debiasing\n",
    "\n",
    "Biased decision making is a problem exhibited both by very simple machine learning classifiers as well as much more complicated ones. Large drivers of this problem include [omitted-variable bias](https://en.wikipedia.org/wiki/Omitted-variable_bias), reliance on inherently biased data sources for training data, attempts to make predictions from insufficient quantities of data, and deploying systems that create real-world incentives that change the behaviour they were measuring (see [Goodhart's Law](https://en.wikipedia.org/wiki/Goodhart%27s_law)).\n",
    "\n",
    "These problems are severe and widespread in the deployment of scoring and machine learning systems in contexts that include [criminal justice](https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing), [education policy](https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2388436), [insurance](https://www.propublica.org/article/minority-neighborhoods-higher-car-insurance-premiums-white-areas-same-risk) and [lending](https://arxiv.org/abs/1610.02413)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "avoiding_bias = Problem(\"Build systems which can recognise and avoid biases decision making\", [\"safety\"])\n",
    "\n",
    "avoiding_bias.notes = \"\"\"\n",
    "Legally institutionalised protected categories represent only the most extreme and socially recognised\n",
    "forms of biased decisionmaking. Attentive human decision makers are sometime capable of recognising\n",
    "and avoiding many more subtle biases. This problem tracks AI systems' ability to do likewise.\n",
    "\"\"\"\n",
    "\n",
    "avoid_classification_biases = Problem(\"Train ML classifiers in a manner that corrects for the impact of omitted-variable bias on certain groups\", solved=True)\n",
    "avoid_classification_biases.notes = '''\n",
    "Several standards are available for avoiding classification biases.\n",
    "\n",
    "They include holding false-positive / false adverse prediction rates constant across protected categories (which roughly maps \n",
    "to \"equal opportunity\"), holding both false-positive and false-negative rates equal (\"demographic parity\"), and ensuring\n",
    "that the fraction of each protected group that receives a given prediction is constant across all groups \n",
    "(roughly equivalent to \"affirmative action\").'''\n",
    "\n",
    "avoid_classification_biases.metric(\"Adjust prediction models to have constant false-positive rates\", url=\"https://arxiv.org/abs/1610.02413\", solved=True)\n",
    "avoid_classification_biases.metric(\"Adjust prediction models tos have constant false-positive and -negative rates\", url=\"http://www.jmlr.org/proceedings/papers/v28/zemel13.pdf\", solved=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Privacy\n",
    "\n",
    "Many of the interesting privacy problems that will arise from AI and machine learning will come from choices about the applications of\n",
    "the technology, rather than a lack of algorithmic progress within the field. But there are some exceptions, which we will track here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "private_training = Problem(\"Train machine learning systems on private user data, without transferring sensitive facts into the model\")\n",
    "private_training.metric(\"Federated Learning (distributed training with thresholded updates to models)\", solved=True, url=\"https://arxiv.org/abs/1602.05629\")\n",
    "\n",
    "avoid_privacy_bias = Problem(\"Fairness in machine learning towards people with a preference for privacy\")\n",
    "avoid_privacy_bias.notes = \"\"\"\n",
    "People who care strongly about their own privacy take many measures to obfuscate their tracks through\n",
    "technological society, including using fictitious names, email addresses, etc in their routine dealings with\n",
    "corporations, installing software to block or send inacurate data to online trackers. Like many other groups,\n",
    "these people may be subject to unfairly adverse algorithmic decisionmaking. Treating them as a protected\n",
    "group will be more difficult, because they are in many respects harder to identify.\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# hiddencode\n",
    "def counts():\n",
    "    print (\"Included thus far:\")\n",
    "    print (\"=================================\")\n",
    "    print (len(problems), \"problems\")\n",
    "    print (len(metrics), \"metrics\", len([m for m in metrics.values() if m.solved]), \"solved\")\n",
    "    print (len(measurements), \"measurements\")\n",
    "    print (len([p for p in problems.values() if not p.metrics]), \"problems which do not yet have any metrics (either not in this notebook, or none in the open literature)\")\n",
    "    print (\"=================================\\n\")\n",
    "    print (\"Problems by Type:\")\n",
    "    print (\"=================================\")\n",
    "\n",
    "    by_attr = {}\n",
    "    solved_by_attr = {}\n",
    "    for a in all_attributes:\n",
    "        print (a, len([p for p in problems.values() if a in p.attributes]), )\n",
    "        print (\"solved:\", len([p for p in problems.values() if p.solved and a in p.attributes]))\n",
    "\n",
    "    print (\"\\nMetrics by Type:\")\n",
    "    print (\"=================================\")\n",
    "\n",
    "    by_attr = {}\n",
    "    solved_by_attr = {}\n",
    "    for a in all_attributes:\n",
    "        print (a, sum([len(p.metrics) for p in problems.values() if a in p.attributes]), )\n",
    "        print (\"solved:\", sum([len([m for m in p.metrics if m.solved]) for p in problems.values() if a in p.attributes]))\n",
    "    print (\"=================================\\n\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# hiddencode\n",
    "def list_problems():\n",
    "    for p in sorted(problems.values(), key=lambda x: x.attributes):\n",
    "        if not p.superproblems:\n",
    "            p.print_structure()\n",
    "            print(\"\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# hiddencode\n",
    "def venn_report():\n",
    "    print(\"Sample of problems characterized thus far:\")\n",
    "    lang = set(p for p in problems.values() if \"language\" in p.attributes)\n",
    "    world = set(p for p in problems.values() if \"world-modelling\" in p.attributes)\n",
    "    vision = set(p for p in problems.values() if \"vision\" in p.attributes)\n",
    "\n",
    "    from matplotlib_venn import venn3\n",
    "    venn3((lang, world, vision), ('Language Problems', 'World-Modelling Problems', 'Vision Problems'))\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# hiddencode\n",
    "def graphs():\n",
    "    print(\"Graphs of progress:\")\n",
    "    for name, metric in metrics.items():\n",
    "        if len(metric.measures) > 2 and not metric.graphed:\n",
    "            print(name, \"({0} measurements)\".format(len(metric.measures)))\n",
    "            metric.graph()\n",
    "    plt.show()\n",
    "                \n",
    "graphs()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Taxonomy and recorded progress to date"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "list_problems()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Problems and Metrics by category"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "counts()\n",
    "venn_report()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# How to contribute to this notebook\n",
    "\n",
    "This notebook is an open source, community effort. It lives on Github at https://github.com/AI-metrics/AI-metrics. You can help by adding new metrics, data and problems to it! If you're feeling ambitious you can also improve its semantics or build new analyses into it. Here are some high level tips on how to do that.\n",
    "\n",
    "### 0. The easiest way -- just hit the edit button\n",
    "\n",
    "Next to every table of results (not yet next to the graphs) you'll find an \"Add/edit data on Github\" link. You can just click it, and you should get a link to the Github's online editor that should make it easy to add new results, or fix existing ones, and send us a pull request. For best results, make sure you're logged in to Github \n",
    "\n",
    "### 1. If you're comfortable with git and Jupyter Notebooks, or are happy to learn\n",
    "\n",
    "If you're interested in making more extensive changes to the Notebook, and you've already worked a lot with `git` and IPython/Jupyter Notebooks, you can run and edit copy locally.  This is a fairly involved process (Jupyter Notebook and git interact in a somewhat complicated way) but here's a quick list of things that should mostly work:\n",
    "\n",
    "1. Install [Jupyter Notebook](https://jupyter.readthedocs.io/en/latest/install.html) and [git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git).\n",
    "    * On an Ubuntu or Debian system, you can do: <br><pre>sudo apt-get install git\n",
    "      sudo apt-get install ipython-notebook || sudo apt-get install jupyter-notebook || sudo apt-get install python-notebook</pre>\n",
    "    * Make sure you have IPython Notebook version 3 or higher. If your OS\n",
    "      doesn't provide it, you might need to enable backports, or use `pip` to\n",
    "      install it.\n",
    "2. Install this notebook's Python dependencies:<br>\n",
    "    * On Ubuntu or Debian, do: <br><pre>    sudo apt-get install python-{cssselect,lxml,matplotlib{,-venn},numpy,requests,seaborn}</pre>\n",
    "    * On other systems, use your native OS packages, or use `pip`: <br><pre>    pip install cssselect lxml matplotlib{,-venn} numpy requests seaborn</pre>\n",
    "3. Fork our repo on\n",
    "   github: https://github.com/AI-metrics/AI-metrics#fork-destination-box\n",
    "4. [Clone](https://help.github.com/articles/cloning-a-repository/) the repo on your machine, and `cd` into the directory it's using\n",
    "4. Configure your copy of git to use [IPython Notebook merge filters](http://pascalbugnion.net/blog/ipython-notebooks-and-git.html) to prevent conflicts when multiple people edit the Notebook simultaneously. You can do that with these two commands in the cloned repo:\n",
    "    <pre>git config --file .gitconfig filter.clean_ipynb.clean $PWD/ipynb_drop_output</pre>\n",
    "    <pre>git config --file .gitconfig filter.clean_ipynb.smudge cat</pre>\n",
    "5. Run Jupyter Notebok in the project directory (the command may be `ipython notebook`, `jupyter notebook`, `jupyter-notebook`, or `python notebook` depending on your system), then go to [localhost:8888](http://localhost:8888) and edit the Notebook to your heart's content\n",
    "\n",
    "6. Save and commit your work (`git commit -a -m \"DESCRIPTION OF WHAT YOU CHANGED\"`)\n",
    "7. Push it to your remote repo\n",
    "8. Send us a pull request!\n",
    "\n",
    "\n",
    "## Notes on importing data\n",
    "\n",
    "* Each `.measure()` call is a data point of a specific algorithm on a specific metric/dataset. Thus one paper will often produce multiple measurements on multiple metrics. It's most important to enter results that were at or near the frontier of best performance on the date they were published. This isn't a strict requirement, though; it's nice to have a sense of the performance of the field, or of algorithms that are otherwise notable even if they aren't the frontier for a sepcific problem.\n",
    "* When multiple revisions of a paper (typically on arXiv) have the same results on some metric, use the date of the first version (the CBTest results in [this paper](https://arxiv.org/abs/1606.02245v4) are an example)\n",
    "* When subsequent revisions of a paper improve on the original results ([example](https://arxiv.org/abs/1606.01549v3)), use the date and scores of the first results, or if each revision is interesting / on the frontier of best performance, include each paper\n",
    "  * We didn't check this carefully for our first ~100 measurement data points :(. In order to denote when we've checked which revision of an arXiv preprint first published a result, cite the specific version (https://arxiv.org/abs/1606.01549v3 rather than https://arxiv.org/abs/1606.01549). That way, we can see which previous entries should be double-checked for this form of inaccuracy.\n",
    "* Where possible, use a clear short name or acronym for each algorithm. The full paper name can go in the `papername` field (and is auto-populated for some papers). When matplotlib 2.1 ships we may be able to get nice [rollovers](https://github.com/matplotlib/matplotlib/pull/5754) with metadata like this. Or perhaps we can switch to D3 to get that type of interactivity.\n",
    "\n",
    "\n",
    "\n",
    "## What to work on\n",
    "\n",
    "* If you know of ML datasets/metrics that aren't included yet, add them\n",
    "* If there are papers with interesting results for metrics that aren't included, add them\n",
    "* If you know of important problems that humans can solve, and machine learning systems may or may not yet be able to, and they're missing from our taxonomy, you can propose them\n",
    "* Look at our [Github issue list](https://github.com/AI-metrics/master_text), perhaps starting with those tagged as [good volunteer tasks](https://github.com/AI-metrics/master_text/issues?q=is%3Aissue+is%3Aopen+label%3A%22Good+volunteer+task%22).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building on this data\n",
    "\n",
    "If you want to use this data for some purpose that is beyond the scope of this Notebook, all of the raw data exported as a JSON blob. This is not yet a stable API, but you can get the data at:\n",
    "\n",
    "\n",
    "https://raw.githubusercontent.com/AI-metrics/AI-metrics/master/export-api/v01/progress.json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## License\n",
    "\n",
    "<a rel=\"license\" href=\"http://creativecommons.org/licenses/by-sa/4.0/\"><img alt=\"Creative Commons License\" style=\"border-width:0; float:left\" src=\"images/cc-by-sa.png\" /></a><br><br>Much of this Notebook is uncopyrightable data. The copyrightable portions of this Notebook that are written by EFF and other Github contributors are licensed under the <a rel=\"license\" href=\"http://creativecommons.org/licenses/by-sa/4.0/\">Creative Commons Attribution-ShareAlike 4.0 International License</a>. Illustrations from datasets and text written by other parties remain copyrighted by their respective owners, if any, and may be subject to different licenses.\n",
    "\n",
    "The source code is also dual-licensed under the [GNU General Public License, version 2 or greater](https://www.gnu.org/licenses/old-licenses/gpl-2.0.en.html).\n",
    "\n",
    "### How to cite this document\n",
    "\n",
    "In academic contexts, you can cite this document as: <tt>Peter Eckersley, Yomna Nasser _et al._, EFF AI Progress Measurement Project, (2017-) https://eff.org/ai/metrics, accessed on 2017-09-09</tt>, or the equivalent in the format you are working in.  \n",
    "\n",
    "If you would like to deep-link an exact version of the text of the Notebook for archival or historical purposes, you can do that using the [Internet Archive](https://web.archive.org/web/*/www.eff.org/ai/metrics) or [Github](https://github.com/AI-metrics/AI-metrics/commits/master). In addition to keeping a record of changes, Github will render a specific version of the Notebook using URLs like this one: https://github.com/AI-metrics/AI-metrics/blob/008993c84188094ba804882f65815c7e1cfc4d0e/AI-progress-metrics.ipynb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# hiddencode\n",
    "\n",
    "def export_json(indent=False, default_name=\"export-api/v01/progress.json\"):\n",
    "    \"\"\"Export all the data in here to a JSON file! Default name: progress.json.\"\"\"\n",
    "    output = {'problems':[]}\n",
    "    for problem in problems:\n",
    "        problem = problems[problem]\n",
    "        problem_data = {}        \n",
    "        for problem_attr in problem.__dict__:\n",
    "            if problem_attr in ['subproblems', 'superproblems']:\n",
    "                problem_data[problem_attr] = list(map(lambda x: x.name, getattr(problem, problem_attr)))\n",
    "            elif problem_attr != 'metrics':\n",
    "                problem_data[problem_attr] = getattr(problem, problem_attr)\n",
    "            elif problem_attr == 'metrics':\n",
    "                problem_data['metrics'] = []\n",
    "                for metric in problem.metrics:\n",
    "                    metric_data = {}\n",
    "                    metric_data['measures'] = []\n",
    "                    for metric_attr in metric.__dict__: \n",
    "                        if metric_attr == 'scale':\n",
    "                            metric_data[metric_attr] = getattr(metric, metric_attr).axis_label\n",
    "                        elif metric_attr != 'measures':\n",
    "                            metric_data[metric_attr] = getattr(metric, metric_attr)\n",
    "                        elif metric_attr == 'measures':\n",
    "                            for measure in getattr(metric, 'measures'):\n",
    "                                measure_data = {}\n",
    "                                for measure_attr in measure.__dict__:\n",
    "                                    measure_data[measure_attr] = getattr(measure, measure_attr)\n",
    "                                metric_data['measures'].append(measure_data)\n",
    "                    problem_data['metrics'].append(metric_data)\n",
    "        output['problems'].append(problem_data)\n",
    "    if indent:\n",
    "        with open(default_name, 'w') as f:\n",
    "            f.write(json.dumps(output, default=str, indent=4))\n",
    "    else:\n",
    "        with open(default_name, 'w') as f:\n",
    "            f.write(json.dumps(output, default=str))\n",
    "        \n",
    "export_json(indent=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%javascript\n",
    "// # hiddencode\n",
    "if (document.location.hostname == \"localhost\" && document.location.port != null) {\n",
    "    console.log($(\".local-edit\").show(500));\n",
    "}"
   ]
  }
 ],
 "metadata": {
  "git": {
   "suppress_outputs": true
  },
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}