1_SHARE_API_Basics_py3.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Calling the SHARE API\n",
    "----\n",
    "Here are some working examples of how to query the current SHARE database for individual results, metrics, and statistics.\n",
    "\n",
    "These particular queries are just examples, and the data is open for anyone to use, so feel free to make your own and experiment!\n",
    "\n",
    "Soon, we'll need a URL to access the SHARE Search API\n",
    "\n",
    "If you want to learn more about Python, the language that we are using to access the API and play with the data, these are a few great guides:\n",
    "+ [The Hitchhiker's Guide to Python](http://docs.python-guide.org/en/latest/)\n",
    "+ [Code Academy Python](https://www.codecademy.com/learn/python)\n",
    "+ [The Python Tutorial](https://docs.python.org/3/tutorial/index.html)\n",
    "+ [SoloLearn Python -- App for Android](https://play.google.com/store/apps/details?id=com.sololearn.python&hl=en)\n",
    "+ [SoloLearn Python -- App for iOS](https://itunes.apple.com/us/app/learn-python-pro/id953972812?mt=8)\n",
    "\n",
    "And a quick introduction to Jupyter Notebooks:\n",
    "+ [Running the Jupyter Notebook](https://jupyter-notebook-beginner-guide.readthedocs.io/en/latest/execute.html)\n",
    "+ [Basics of Jupyter Notebook](https://www.packtpub.com/books/content/basics-jupyter-notebook-and-python)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "SHARE_API = 'https://staging-share.osf.io/api/search/abstractcreativework/_search'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The SHARE Search Schema\n",
    "\n",
    "The SHARE search API is built on a tool called elasticsearch. It lets you search a subset of SHARE's normalized metadata in a simple format.\n",
    "\n",
    "Here are the fields available in SHARE's elasticsearch endpoint:\n",
    "\n",
    "    - 'title'\n",
    "    - 'language'\n",
    "    - 'subject'\n",
    "    - 'description'\n",
    "    - 'date'\n",
    "    - 'date_created'\n",
    "    - 'date_modified\n",
    "    - 'date_updated'\n",
    "    - 'date_published'\n",
    "    - 'tags'\n",
    "    - 'links'\n",
    "    - 'awards'\n",
    "    - 'venues'\n",
    "    - 'sources'\n",
    "    - 'contributors'\n",
    "\n",
    "You can see a formatted version of the base results from the API by visiting the [SHARE Search API URL](https://staging-share.osf.io/api/search/abstractcreativework/_search)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Service Names for Reference\n",
    "----\n",
    "Each provider harvested from has a specific . Let's make an API call to generate a table to get all of those \"internal\" names, along with the official name of the repository that they represent.\n",
    "\n",
    "The SHARE API has different endpoints. One of those endpoints returns a list of all of the providers that SHARE is harvesting from, along with their internal names, official names, links to their homepages, and a simple version of an icon representing their service, in a parsable format called json.\n",
    "\n",
    "Let's make a call to that API endpoint using the requests libarary, get the json data, and print out all of the shortnames and longnames."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Here are the first 10 Providers:\n",
      "Research Online @ University of Wollongong\n",
      "http://ro.uow.edu.au\n",
      "au.uow\n",
      "\n",
      "Ghent University Academic Bibliography\n",
      "https://biblio.ugent.be/\n",
      "be.ghent\n",
      "\n",
      "Pontifical Catholic University of Rio de Janeiro\n",
      "http://www.maxwell.vrac.puc-rio.br\n",
      "br.pcurio\n",
      "\n",
      "Lake Winnipeg Basin Information Network\n",
      "http://130.179.67.140\n",
      "ca.lwbin\n",
      "\n",
      "PAPYRUS - Dépôt institutionnel de l'Université de Montréal\n",
      "http://papyrus.bib.umontreal.ca\n",
      "ca.umontreal\n",
      "\n",
      "Western University\n",
      "http://ir.lib.uwo.ca\n",
      "ca.uwo\n",
      "\n",
      "BioMed Central\n",
      "http://www.springer.com/us/\n",
      "com.biomedcentral\n",
      "\n",
      "Social Science Research Network\n",
      "http://papers.ssrn.com/\n",
      "com.dailyssrn\n",
      "\n",
      "figshare\n",
      "https://figshare.com/\n",
      "com.figshare\n",
      "\n",
      "Nature Publishing Group\n",
      "http://www.nature.com/\n",
      "com.nature\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Requests library allows you to send organic, grass-fed HTTP/1.1 requests, no need to manually add query strings \n",
    "    # to your URLs, or to form-encode your POST data. Docs: http://docs.python-requests.org/en/master/\n",
    "import requests\n",
    "\n",
    "# Json library parses JSON from strings or files. The library parses JSON into a Python dictionary or list. \n",
    "    # It can also convert Python dictionaries or lists into JSON strings. \n",
    "    # https://docs.python.org/2.7/library/json.html\n",
    "import json\n",
    "\n",
    "# This takes the URL and puts it into a variable (so we only need to ever reference this variable, \n",
    "    # and so we don't have to repeat adding this URL when we want to work with the data)\n",
    "SHARE_PROVIDERS = 'https://staging-share.osf.io/api/providers/'\n",
    "\n",
    "# this requests the data from the SHARE_PROVIDERS and uses the json library to parse it into this list variable\n",
    "data = requests.get(SHARE_PROVIDERS).json()\n",
    "\n",
    "# literally prints out the sentence in the quotes on the screen\n",
    "print('Here are the first 10 Providers:')\n",
    "\n",
    "# this is a for loop (https://wiki.python.org/moin/ForLoop) to repeat the same tasks for each of the items in the \n",
    "    # list of our SHARE providers (what we put into the variable \"data\").\n",
    "# for every item (called 'source' below) in the list, we print out the title, website, and provider name, \n",
    "    # formatted so each is on a new line (\\n)\n",
    "for source in data['results']:\n",
    "    print(\n",
    "        '{}\\n{}\\n{}\\n'.format(\n",
    "            source['long_title'],\n",
    "            source['home_page'],\n",
    "            source['provider_name']\n",
    "        )\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### SHARE Schema\n",
    "\n",
    "You can make queries against any of the fields defined in the [SHARE Schema](https://github.com/CenterForOpenScience/SHARE-Schema/blob/master/share.yaml). If we were able to harvest the information from the original source, it should appear in SHARE. However, not all fields are required for every document. \n",
    "\n",
    "Required fields include:\n",
    "- title\n",
    "- contributors\n",
    "- uris\n",
    "- providerUpdatedDateTime\n",
    "\n",
    "We add some information after each document is harvested inside the field shareProperties, including:\n",
    "- source (where the document was originally harvested)\n",
    "- docID  (a unique identifier for that object from that source)\n",
    "\n",
    "These two fields can be combined to make a unique document identifier."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Simple Queries"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's get the first 3 results from the most basic query - the first page of the most recently updated research release events in SHARE.\n",
    "\n",
    "We'll use the URL parsing library furl to keep track of all of our arguments to the URL, because we'll be modifying them as we go along. We'll print the URL as we go to take a look at it, so we know what we're requesting.\n",
    "\n",
    "We'll print out the result's title and sources where it appears."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The request URL is https://staging-share.osf.io/api/search/abstractcreativework/_search?size=3\n",
      "----------\n",
      "LEDAPS corrected Landsat Enhanced Thematic Mapper image data for Shortgrass Steppe collected on 2011-06-17 -- from ['providers.org.datacite']\n",
      "Test entry from ezid service for identifier: doi:10.6085//TEST/20152611351448160.0758426051207266 -- from ['providers.org.datacite']\n",
      "LEDAPS corrected Landsat Enhanced Thematic Mapper image data for Shortgrass Steppe collected on 1990-04-20 -- from ['providers.org.datacite']\n"
     ]
    }
   ],
   "source": [
    "# furl is a Python library that allows you to easily manipulate URLs. https://github.com/gruns/furl\n",
    "import furl\n",
    "\n",
    "# In cell 1, we put the URL for the SHARE API into the variable SHARE_API. We can use it even down here!\n",
    "    # We are parsing it using furl and putting it into a new variable called search_url\n",
    "search_url = furl.furl(SHARE_API)\n",
    "\n",
    "# We are limiting the arguments that we can attach to the URL to 3 -- so we grab the first 3 entries from the API\n",
    "search_url.args['size'] = 3\n",
    "\n",
    "# We are requsting the information from the search_url and parsing the JSON that we requested and got back \n",
    "    # (requests.get!) like we did in cell 2. We put it into this list variable \n",
    "recent_results = requests.get(search_url.url).json()\n",
    "\n",
    "# This is called a 2 dimensional array -- a list that looks kind of like a matrix. A list can store other lists. \n",
    "    # This is a way to create two-dimensional (2D) lists in Python -- 2Dlist=[hi,my,name,is,erin][but,my,name,is,vicky]. We can print out 2Dlist[1][4] and we would see \"my Vicky\" because we start counting at 0 instead of 1\n",
    "recent_results = recent_results['hits']['hits']\n",
    "\n",
    "# we need to do this to initialize/define the list -- look at this for more information: \n",
    "    # http://stackoverflow.com/questions/6667201/how-to-define-two-dimensional-array-in-python\n",
    "recent_results\n",
    "\n",
    "#We are printing out on the screen the variable search_url\n",
    "print('The request URL is {}'.format(search_url.url))\n",
    "\n",
    "#This is just so we have a nice visual queue between the url that we searched and the actual data we are grabbing\n",
    "print('----------')\n",
    "\n",
    "#Another for loop! For all of the items in the results we just grabbed, we are printing out the variables \n",
    "    # called source and title\n",
    "for result in recent_results:\n",
    "    print(\n",
    "        '{} -- from {}'.format(\n",
    "            result['_source']['title'],\n",
    "            result['_source']['sources']\n",
    "        )\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's limit that query to only documents mentioning \"giraffes\" somewhere in the title, description, or in any of the metadata. We'd do that by adding a query search term."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The request URL is https://staging-share.osf.io/api/search/abstractcreativework/_search?size=3&q=giraffes\n",
      "---------\n",
      "Genome reveals why giraffes have long necks -- from ['providers.org.crossref']\n",
      "Odd creature was ancient ancestor of today’s giraffes -- from ['providers.org.crossref']\n",
      "Genome reveals why giraffes have long necks -- from ['providers.com.nature']\n"
     ]
    }
   ],
   "source": [
    "# we are reusing that variable search_url from the cell above! We are querying (hence the args['q']) the API to try \n",
    "    # and get items that have the word 'giraffes' in them\n",
    "search_url.args['q'] = 'giraffes'\n",
    "\n",
    "# We are requsting the information from the search_url and parsing the JSON that we requested and got back \n",
    "    # (requests.get!) like we did in cell 2. We put it into this list variable \n",
    "recent_results = requests.get(search_url.url).json()\n",
    "\n",
    "# This is called a 2 dimensional array -- a list that looks kind of like a matrix. A list can store other lists. \n",
    "    # This is a way to create two-dimensional (2D) lists in Python -- \n",
    "    # Exmaple: 2Dlist=[hi,my,name,is,erin][but,my,name,is,vicky]. We can print out 2Dlist[1][4] and we would see \n",
    "    # \"my Vicky\" because we start counting lists at 0 instead of 1 (computer science quirk)\n",
    "recent_results = recent_results['hits']['hits']\n",
    "\n",
    "# We are printing out on the screen the variable search_url\n",
    "print('The request URL is {}'.format(search_url.url))\n",
    "\n",
    "# This is just so we have a nice visual queue between the url that we searched and the actual data we are grabbing\n",
    "print('---------')\n",
    "\n",
    "# Another for loop! For all of the items in the results we just grabbed, we are printing out the entries that have \n",
    "    # the keyword 'giraffes'\n",
    "for result in recent_results:\n",
    "    print(\n",
    "        '{} -- from {}'.format(\n",
    "            result['_source']['title'],\n",
    "            result['_source']['sources']\n",
    "        )\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's search for documents from the source CrossRef"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The request URL is https://staging-share.osf.io/api/search/abstractcreativework/_search?size=3&q=sources:providers.org.crossref\n",
      "---------\n",
      "Communicating Accessibility Resources Benefits Everyone -- from ['providers.org.crossref']\n",
      "The Devil Is in the Details -- from ['providers.org.crossref']\n",
      "Progression of coronary artery calcification by cardiac computed tomography -- from ['providers.org.crossref']\n"
     ]
    }
   ],
   "source": [
    "# we are reusing that variable search_url from the cell above! We are querying (see that arg again) for everything \n",
    "    # from the source provide CrossRef\n",
    "search_url.args['q'] = 'sources:providers.org.crossref'\n",
    "\n",
    "# We are requsting the information from the search_url and parsing the JSON that we requested and got back \n",
    "    # (requests.get!) like we did in cell 2. We put it into this list variable \n",
    "recent_results = requests.get(search_url.url).json()\n",
    "\n",
    "# This is called a 2 dimensional array -- a list that looks kind of like a matrix. A list can store other lists. \n",
    "    # This is a way to create two-dimensional (2D) lists in Python -- \n",
    "    # Example: 2Dlist=[hi,my,name,is,erin][but,my,name,is,vicky]. We can print out 2Dlist[1][4] \n",
    "    # and we would see \"my Vicky\" because we start counting at 0 instead of 1 (computer science quirk)\n",
    "recent_results = recent_results['hits']['hits']\n",
    "\n",
    "# We are printing out on the screen the variable search_url\n",
    "print('The request URL is {}'.format(search_url.url))\n",
    "\n",
    "# This is just so we have a nice visual queue between the url that we searched and the actual data we are grabbing\n",
    "print('---------')\n",
    "\n",
    "# Another for loop! For all of the items in the results we just grabbed, we are printing out the entries that are \n",
    "    # from CrossRef\n",
    "for result in recent_results:\n",
    "    print(\n",
    "        '{} -- from {}'.format(\n",
    "            result['_source']['title'],\n",
    "            result['_source']['sources']\n",
    "        )\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's combine the two and find documents from CrossRef that mention giraffes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The request URL is https://staging-share.osf.io/api/search/abstractcreativework/_search?size=3&q=sources:providers.org.crossref+AND+giraffes\n",
      "---------\n",
      "Genome reveals why giraffes have long necks -- from ['providers.org.crossref']\n",
      "Odd creature was ancient ancestor of today’s giraffes -- from ['providers.org.crossref']\n",
      "Of Caucasians, Asians, and Giraffes: The Influence of Categorization and Target Valence on Social Projection -- from ['providers.org.crossref']\n"
     ]
    }
   ],
   "source": [
    "# we are reusing that variable search_url from the cell above! We are querying (see that arg again) for entries \n",
    "    # that use the keyword \"giraffes\" from the source provider CrossRef\n",
    "search_url.args['q'] = 'sources:providers.org.crossref AND giraffes'\n",
    "\n",
    "# We are requsting the information from the search_url and parsing the JSON that we requested and got back \n",
    "    # (requests.get!) like we did in cell 2. We put it into this list variable \n",
    "recent_results = requests.get(search_url.url).json()\n",
    "\n",
    "# This is called a 2 dimensional array -- a list that looks kind of like a matrix. A list can store other lists. \n",
    "    # This is a way to create two-dimensional (2D) lists in Python -- 2Dlist=[hi,my,name,is,erin][but,my,name,is,vicky]. We can print out 2Dlist[1][4] and we would see \"my Vicky\" because we start counting at 0 instead of 1\n",
    "recent_results = recent_results['hits']['hits']\n",
    "\n",
    "# We are printing out on the screen the variable search_url\n",
    "print('The request URL is {}'.format(search_url.url))\n",
    "\n",
    "# This is just so we have a nice visual queue between the url that we searched and the actual data we are grabbing\n",
    "print('---------')\n",
    "\n",
    "# Another for loop! For all of the items in the results we just grabbed, we are printing out the entries that are \n",
    "    # from CrossRef that are about giraffes\n",
    "for result in recent_results:\n",
    "    print(\n",
    "        '{} -- from {}'.format(\n",
    "            result['_source']['title'],\n",
    "            result['_source']['sources']\n",
    "        )\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Complex Queries\n",
    "The SHARE Search API runs on elasticsearch - meaning that it can accept complicated queries that give you a wide variety of information.\n",
    "\n",
    "Here are some examples of how to make more complex queries using the raw elasticsearch results. You can read a [lot more about elasticsearch queries here](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'https://staging-share.osf.io/api/search/abstractcreativework/_search'"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# reset the args so that we remove our old query arguments.\n",
    "search_url.args = None  \n",
    "\n",
    "# Show the URL that we'll be requesting to make sure the args were cleared\n",
    "search_url.url "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Query Setup\n",
    "\n",
    "We can define a few functions that we can reuse to make querying simpler. Elasticsearch queries are passed through as json blobs specifying how to return the information you want."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#just like the json library from cell 2\n",
    "import json\n",
    "\n",
    "#this is called a function -- we DEFINE it (def) and name it something useful for us. It makes it easy for us to \n",
    "    #reuse this later on by just calling the function by typng it's name, and adding the appropriate values in the \n",
    "    #parentheses (called arguments). To learn more about functions: \n",
    "        #http://www.tutorialspoint.com/python/python_functions.htm\n",
    "        #http://docs.python-guide.org/en/latest/writing/style/?highlight=function\n",
    "        \n",
    "# This is a helper function that will use the requests library, pass along the correct headers, and make the query\n",
    "    # we want\n",
    "def query_share(url, query):\n",
    "    headers = {'Content-Type': 'application/json'}\n",
    "    data = json.dumps(query)\n",
    "    return requests.post(url, headers=headers, data=data).json()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Some Queries\n",
    "The SHARE schema has many spots for information, and many of the original sources do not provide this information. We can do a query to find out if a certain field exists or not within certain records. The SHARE API is set up to show an empty list if the field is empty.\n",
    "\n",
    "Let's query for the counts of documents that have a content in their tags field."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# we are making another list of all the items with tags\n",
    "tags_query = {\n",
    "    \"query\": {\n",
    "        \"exists\": {\n",
    "            \"field\": \"tags\"\n",
    "        }\n",
    "    }\n",
    "}\n",
    "\n",
    "\n",
    "# we are making another list of all the items without tags\n",
    "missing_tags_query = {\n",
    "    \"query\": {\n",
    "        \"bool\": {\n",
    "            \"must_not\": {\n",
    "                \"exists\": {\n",
    "                    \"field\": \"tags\"\n",
    "                }\n",
    "            }\n",
    "        }      \n",
    "    }\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2443294 results out of 4914457, or 49.72%, have tags.\n",
      "2471163 results out of 4914457, or 50.28%, do NOT have tags.\n",
      "------------\n",
      "As a little sanity check....\n",
      "49.71645901062925 + 50.28354098937074 = 100.00%\n"
     ]
    }
   ],
   "source": [
    "# we are making a list of the results from searching items with tags\n",
    "with_tags = query_share(search_url.url, tags_query)\n",
    "\n",
    "# we are making a list of the results from searching items without tags\n",
    "missing_tags = query_share(search_url.url, missing_tags_query)\n",
    "\n",
    "#Gets the total number of hits from each search\n",
    "total_results = requests.get(search_url.url).json()['hits']['total']\n",
    "\n",
    "# getting the percentages of hits with and without tags, respectively, using the built-in float function, which\n",
    "    # returns a float number (written with a decimal point dividing the integer and fractional parts.)\n",
    "    # read more about float function: https://docs.python.org/3/library/functions.html#float\n",
    "with_tags_percent = (float(with_tags['hits']['total'])/total_results)*100\n",
    "missing_tags_percent = (float(missing_tags['hits']['total'])/total_results)*100\n",
    "\n",
    "\n",
    "# this simply prints out the list of results that have tags by iterating over the list we already made \"with_tags\"\n",
    "    # it then prints out the percentage of total \n",
    "print(\n",
    "    '{} results out of {}, or {}%, have tags.'.format(\n",
    "        with_tags['hits']['total'],\n",
    "        total_results,\n",
    "        format(with_tags_percent, '.2f')\n",
    "    )\n",
    ")\n",
    "\n",
    "# this simply prints out the list of results without tags by iterating over the list we already made \"missing_tags\"\n",
    "    # it then prints out the percentage of tota\n",
    "print(\n",
    "    '{} results out of {}, or {}%, do NOT have tags.'.format(\n",
    "        missing_tags['hits']['total'],\n",
    "        total_results,\n",
    "        format(missing_tags_percent, '.2f')\n",
    "    )\n",
    ")\n",
    "\n",
    "# Visual cue, printing the percentage of results with tags + percent with no tags (we make sure it equals 100)\n",
    "print('------------')\n",
    "print('As a little sanity check....')\n",
    "print('{} + {} = {}%'.format(with_tags_percent, missing_tags_percent, format(with_tags_percent + missing_tags_percent, '.2f')))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using SHAREPA for SHARE Parsing and Analysis\n",
    "\n",
    "While you can always pass raw elasticsearch queries to the SHARE API, there is also a pip-installable python library that you can use that makes elasticsearch aggregations a little simpler. This library is called [sharepa - short for SHARE Parsing and Analysis](https://github.com/CenterForOpenScience/sharepa#sharepa)\n",
    "\n",
    "### Basic Actions\n",
    "\n",
    "A basic search will provide access to all documents in SHARE in 10 document slices.\n",
    "\n",
    "#### Count\n",
    "You can use sharepa and the basic search to get the total number of documents in SHARE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4914457"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Sharepa is a python client for  browsing and analyzing SHARE data specifically using elasticsearch querying.\n",
    "    # We can use this to aggregate, graph, and analyze the data. \n",
    "    # Helpful Links:\n",
    "        # https://github.com/CenterForOpenScience/sharepa\n",
    "        # https://pypi.python.org/pypi/sharepa\n",
    "    # here, we import the specific function from Sharepa called basic_search\n",
    "from sharepa import basic_search\n",
    "\n",
    "# this performs a basic search over the SHARE dataset and returns a count of the items in it\n",
    "basic_search.count()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Iterating Through Results\n",
    "Executing the basic search will send the actual basic query to the SHARE API and then let you iterate through results, 10 at a time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LEDAPS corrected Landsat Enhanced Thematic Mapper image data for Shortgrass Steppe collected on 2011-06-17\n",
      "Test entry from ezid service for identifier: doi:10.6085//TEST/20152611351448160.0758426051207266\n",
      "LEDAPS corrected Landsat Enhanced Thematic Mapper image data for Shortgrass Steppe collected on 1990-04-20\n",
      "Chemical composition of essential oils of three Pistacia cultivars in Khorasan Razavi, Iran\n",
      "Test entry from ezid service for identifier: doi:10.6085//TEST/20152611351448190.5563383046761334\n",
      "Test entry from ezid service for identifier: doi:10.6085//TEST/20152611351448200.9788127569002799\n",
      "Test entry from ezid service for identifier: doi:10.6085//TEST/20152611351448150.3484667860365027\n",
      "Compiled Tree-ring Dates from the Southwestern United States (Unrestricted)\n",
      "None\n",
      "Area-based Amino Acid Composition for three types of interactions in the BNCP-CS dataset\n"
     ]
    }
   ],
   "source": [
    "# this captures 10 items in the SHARE dataset -- that's what the basic search function is for\n",
    "results = basic_search.execute()\n",
    "\n",
    "# this prints out the 10 items we added to the list variable, results (above)\n",
    "for hit in results:\n",
    "    print(hit.title)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we don't want 10 results, or we want to offset the results, we can use slices"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['providers.org.datacite']\n",
      "['providers.org.datacite']\n",
      "['providers.org.datacite']\n",
      "['providers.org.datacite']\n",
      "['providers.org.datacite']\n"
     ]
    }
   ],
   "source": [
    "# this performs a basic search for items number 20-25 in the SHARE dataset (useful if you know where your things are\n",
    "    # in SHARE dataset) \n",
    "results = basic_search[20:25].execute()\n",
    "\n",
    "# prints out the provider for the items capture in the list, results (above)\n",
    "for hit in results:\n",
    "    print(hit.sources)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Advanced Search with sharepa\n",
    "\n",
    "You can make your own search object, which allows you to pass in custom queries for certain terms or SHARE fields. Queries are formed using [lucene query syntax](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html#query-string-syntax), just like we used in the above examples.\n",
    "\n",
    "This type of query accepts an exists field. Other options include a query_string, a match query, a multi-match query, a bool query, and any other query structure available in the elasticsearch API.\n",
    "\n",
    "We can see what that query that we're about to send to elasticsearch by using the pretty print helper function. You'll see that it looks very similar to the queries we defined by hand earlier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "    \"query\": {\n",
      "        \"exists\": {\n",
      "            \"field\": \"tags\"\n",
      "        }\n",
      "    }\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "# Sharepa is a python client for  browsing and analyzing SHARE data specifically using elasticsearch querying.\n",
    "    # We can use this to aggregate, graph, and analyze the data. \n",
    "    # Helpful Links:\n",
    "        # https://github.com/CenterForOpenScience/sharepa\n",
    "        # https://pypi.python.org/pypi/sharepa\n",
    "    # here, we import the specific function from Sharepa called ShareSearch and pretty_print\n",
    "from sharepa import ShareSearch\n",
    "from sharepa.helpers import pretty_print\n",
    "\n",
    "#we just created a local name for ShareSearch function for us to use\n",
    "my_search = ShareSearch()\n",
    "\n",
    "# Lucene supports fielded data. When performing a search you can either specify a field, or use the default field. \n",
    "my_search = my_search.query(\n",
    "    'exists', # Type of query, will accept a lucene query string\n",
    "    field='tags', # This lucene query string will find all documents that don't have tags\n",
    ")\n",
    "\n",
    "# this prints out (prettily!) our search, transformed into a dictionary data type\n",
    "    # read more about dictionaries here: http://learnpythonthehardway.org/book/ex39.html\n",
    "pretty_print(my_search.to_dict())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When you execute that query, you can then iterate through the results the same way that you could with the simple search query."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['CDL.LTERNET', 'CDL', 'dataPackage', 'Dataset']\n",
      "['CDL.PISCO', 'CDL']\n",
      "['CDL.LTERNET', 'CDL', 'dataPackage', 'Dataset']\n",
      "['CDL.DIGSCI', 'CDL', 'Paper', 'Dataset']\n",
      "['CDL.PISCO', 'CDL']\n",
      "['CDL.PISCO', 'CDL']\n",
      "['CDL.PISCO', 'CDL']\n",
      "['CDL.DIGANT', 'CDL', 'Dataset']\n",
      "['TIB.R-GATE', 'TIB']\n",
      "['CDL.DIGSCI', 'CDL', 'Image']\n"
     ]
    }
   ],
   "source": [
    "# we are taking the my_search variable from the cell above and executing the search, placing the results into a \n",
    "    # new list called new_results\n",
    "new_results = my_search.execute()\n",
    "\n",
    "# this for loop prints out the tags for each item in the results we gathered \n",
    "for hit in new_results:\n",
    "    print(hit.tags)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Debugging and Problem Solving\n",
    "\n",
    "Not everything always goes as planned when querying an unfamillar API. Here are some debugging and problem solving strategies when you're querying the SHARE API.\n",
    "\n",
    "### Schema issues\n",
    "The SHARE schema has a lot of parts, and much of the information is nested within sections. Making a query isn't always as straight forward as you might think, if you're not looking in the right part of the schema.\n",
    "\n",
    "Let's say you were trying to query for all SHARE documents that specify the language as not being in English.\n",
    "\n",
    "We'll guess as to what that query might be, and try to make it using sharepa."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# this creates a new search for us to use!\n",
    "language_search = ShareSearch()\n",
    "\n",
    "# this sets the search query we are using for our new search! all the items that aren't in english\n",
    "language_search = language_search.query(\n",
    "    'query_string', # Type of query, will accept a lucene query string\n",
    "    query='NOT languages=english', # This lucene query string will find all documents that don't have tags\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "ename": "AttributeError",
     "evalue": "'Result' object has no attribute 'languages'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyError\u001b[0m                                  Traceback (most recent call last)",
      "\u001b[0;32m/Users/erin/miniconda3/envs/share_tutorials/lib/python3.5/site-packages/elasticsearch_dsl/utils.py\u001b[0m in \u001b[0;36m__getattr__\u001b[0;34m(self, attr_name)\u001b[0m\n\u001b[1;32m    119\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 120\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0m_wrap\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_d_\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mattr_name\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    121\u001b[0m         \u001b[0;32mexcept\u001b[0m \u001b[0mKeyError\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mKeyError\u001b[0m: 'languages'",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[0;31mAttributeError\u001b[0m                            Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-17-f7e8b99a9f64>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      2\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mhit\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mresults\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 4\u001b[0;31m     \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mhit\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlanguages\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m/Users/erin/miniconda3/envs/share_tutorials/lib/python3.5/site-packages/elasticsearch_dsl/utils.py\u001b[0m in \u001b[0;36m__getattr__\u001b[0;34m(self, attr_name)\u001b[0m\n\u001b[1;32m    121\u001b[0m         \u001b[0;32mexcept\u001b[0m \u001b[0mKeyError\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    122\u001b[0m             raise AttributeError(\n\u001b[0;32m--> 123\u001b[0;31m                 '%r object has no attribute %r' % (self.__class__.__name__, attr_name))\n\u001b[0m\u001b[1;32m    124\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    125\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m__delattr__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mattr_name\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mAttributeError\u001b[0m: 'Result' object has no attribute 'languages'"
     ]
    }
   ],
   "source": [
    "# this allows us to search through 10 results and find the languages of each\n",
    "results = language_search.execute()\n",
    "\n",
    "# for each item in results, print out the language it is in\n",
    "for hit in results:\n",
    "    print(hit.languages)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So the result does not have an attribute called languages! Let's try to figure out what went wrong here.\n",
    "\n",
    "Step one could be that we are trying to find something that does NOT match a given parameter. Since languages is not required, this is returning results that do not include the languages result at all!\n",
    "\n",
    "So let's fix this up a bit to make sure that we're querying for items that specify language in the first place."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# let's try that again! creating a new search from the ShareSearch() function\n",
    "language_search = ShareSearch()\n",
    "\n",
    "# this sets up our new query: if the field called 'language' exists, grab those results\n",
    "language_search = language_search.filter(\n",
    "    'exists',\n",
    "    field=\"language\"\n",
    ")\n",
    "\n",
    "# count the number of entries that have a language field\n",
    "language_search.count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# grab the results of the search for a language field!\n",
    "results = language_search.execute()\n",
    "\n",
    "# Let's see how many documents have language results.\n",
    "print('There are {} documents with languages specified'.format(language_search.count()))\n",
    "\n",
    "print('Here are the languages for the first 10 results:')\n",
    "\n",
    "# for each item in results, print out the language it is in\n",
    "for hit in results:\n",
    "    print(hit.language)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So now we're better equipped to add on to this filter, and then narrow down to results that are not in English.\n",
    "\n",
    "When we printed out the first few results, we might have noticed a second problem with our query -- going back to the [SHARE Schema](https://github.com/CenterForOpenScience/SHARE-Schema/blob/master/share.yaml), we might notice that there is a restriction on how languages are captured - as a three letter lowercase representation. Instead of \"english\" let's look for the three letter abbreviation - \"eng\"\n",
    "\n",
    "We can modify our new and improved language query by adding on another query to our started language_search. We'll use the elasticsearch query object Q, and invert it with a ~ symbol, and search for the term \"eng.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Elasticsearch DSL is a high-level library whose aim is to help with writing and running queries against Elasticsearch.\n",
    "    # Read more about elasticsearch here:\n",
    "        # http://elasticsearch-dsl.readthedocs.io/en/latest/search_dsl.html\n",
    "        # https://pypi.python.org/pypi/elasticsearch-dsl\n",
    "        # https://github.com/elastic/elasticsearch-dsl-py\n",
    "    # this imports the function Q from the library \n",
    "from elasticsearch_dsl import Q\n",
    "\n",
    "# sets up a new search -- for results that have english ('eng') in their language field\n",
    "language_search = language_search.query(~Q(\"term\", language=\"eng\"))\n",
    "\n",
    "# execute our search and throw it into a new list\n",
    "results = language_search.execute()\n",
    "\n",
    "# Let's see how many documents have language results that aren't eng\n",
    "print('There are {} documents that do not have \"eng\" listed.'.format(language_search.count()))\n",
    "\n",
    "print('Here are the languages for the first 10 results:')\n",
    "\n",
    "# Check out the first few results, make sure \"eng\" isn't in there\n",
    "for hit in results:\n",
    "    print(hit.language)\n",
    "    print(hit.title)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}