Upload-Trove-newspapers-to-Omeka.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Upload Trove newspaper articles to Omeka-S\n",
    "\n",
    "I was keen to play around with [Omeka-S](https://omeka.org/s/) – the new, Linked Open Data powered version of Omeka. In particular I wanted to understand its [API](https://omeka.org/s/docs/developer/key_concepts/api/), its use of [JSON-LD](https://json-ld.org/), and how I could create and use relationships between different types of entities. I decided that one way of exploring these things was to build a pipeline that would make it easy to upload Trove newspaper articles to an Omeka site. I recently figured out how to generate nice thumbnails from newspaper articles, so I could add images as well. But what about the 'Linked' part of LOD? I thought I'd not only create records for the newspaper articles, I'd create records for each cited newspaper and link them to the articles. Things not strings!\n",
    "\n",
    "Of course, this is not the only way to get Trove records into Omeka. You can save them to [Zotero](https://www.zotero.org/) and then use the [Zotero Import module](https://omeka.org/s/docs/user-manual/modules/zoteroimport/) to upload to Omeka. My [first Omeka 'how to'](http://discontents.com.au/some-exhibition-magic-with-zotero-and-omeka/) (written 8 years ago!) talked about using the Zotero Import plugin to upload records from the National Archives of Australia. While the import module works well, I wanted the flexibility to generate new images on the fly, modify the metadata, and unflatten the Zotero records to build newspaper-to-article relationships. I also wanted to be able to get newspaper articles from other sources, such as Trove lists.\n",
    "\n",
    "As I started playing around I realised I could rejig my thumbnail generator to [get an image of the whole article](https://nbviewer.jupyter.org/github/GLAM-Workbench/trove-newspapers/blob/master/Save-Trove-newspaper-article-as-image.ipynb). This is useful because Trove's article download options tend to slice up images in unpleasant ways. So I decided to upload the complete image (or images if the article is published across multiple pages) and let Omeka create the derivative versions. I also decided to upload the PDF version of the article generated by Trove as an alternative.\n",
    "\n",
    "In summary, you can use this notebook to:\n",
    "\n",
    "* Get a Trove newspaper article using the Trove API\n",
    "* Generate an image for the article\n",
    "* Search the Omeka site using the API to see if the newspaper that published the article already has a record, if not create one.\n",
    "* Upload the article details to Omeka using the API, including a link to the newspaper record, as well as the article image(s) and PDF files.\n",
    "\n",
    "But what newspaper articles are we uploading? I've created four different ways of selecting articles for upload. There are further details and configuration options for each of these  methods below:\n",
    "\n",
    "* [Option 1: Upload a Trove newspaper search](#Option-1%3A-Upload-a-Trove-newspaper-search)\n",
    "* [Option 2: Upload newspaper articles from a Trove list](#Option-2%3A-Upload-newspaper-articles-from-a-Trove-list)\n",
    "* [Option 3: Upload Trove newspaper articles saved in Zotero](#Option-3%3A-Upload-Trove-newspaper-articles-saved-in-Zotero)\n",
    "* [Option 4: Upload a list of article ids](#Option-4%3A-Upload-a-list-of-article-ids)\n",
    "\n",
    "There's a fair bit of configuration involved in getting this all working, so make sure you follow the instructions under [Basic configuration](#Basic-configuration) and [Preparing Omeka-S](#Preparing-Omeka-S) before proceeding.\n",
    "\n",
    "## An example\n",
    "\n",
    "Here's an example of a [newspaper article](https://trove.nla.gov.au/newspaper/article/162833980) in Trove:\n",
    "\n",
    "<img src=\"images/162833980.png\" style=\"width: 600px;\">\n",
    "\n",
    "Here's the same article after it's been imported into Omeka. Note that the article is linked to the newspaper that published it using the `isPartOf` property. You can also see the images and PDF of the article that were automatically uploaded along with the metadata.\n",
    "\n",
    "<img src=\"images/article_omeka.png\" style=\"width: 600px;\">\n",
    "\n",
    "And here's the record for the linked newspaper:\n",
    "\n",
    "<img src=\"images/newspaper_omeka.png\" style=\"width: 600px;\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Basic configuration\n",
    "\n",
    "First of all you need an Omeka-S site to upload to! If you don't have a site, a handy server, or a preferred web host, I'd suggest you set up an account with [Reclaim Hosting](https://reclaimhosting.com/) – they're affordable, have excellent support, and provide a one-click installer for Omeka-S. You'll be up and running very quickly.\n",
    "\n",
    "Once you have your Omeka-S site, you need to add some configuration values in the cell below:\n",
    "\n",
    "* `API_URL` – the url (or endpoint) of your Omeka site's API. This is basically just the url of your site with `/api` on the end.\n",
    "* `KEY_IDENTITY` and `KEY_CREDENTIAL` these are the authentication keys you need to upload new records to Omeka-S.\n",
    "* `TROVE_API_KEY` – authentication key to access the Trove API\n",
    "\n",
    "See below for instructions on generating your keys.\n",
    "\n",
    "### Generating your Omeka keys\n",
    "\n",
    "1. To generate your keys, log in to the admin interface of your Omeka-S site and click on 'Users' in the side menu. \n",
    "2. Find your user account in the list and then click on the pencil icon to edit your details. \n",
    "3. Click on the 'API keys' tab.\n",
    "4. In the 'New key label' box, enter a name for your key – something like 'Trove upload' would be fine.\n",
    "5. Click on the 'Save' button.\n",
    "6. A message will appear with your `key_identity` and `key_credential` values – the `key_credential` is only ever displayed once, so copy them both now!\n",
    "\n",
    "### Trove API key\n",
    "\n",
    "To get your Trove API key, just [follow these instructions](http://help.nla.gov.au/trove/building-with-trove/api).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# CONFIGURATION\n",
    "# Things you need to change!\n",
    "# Paste your values in below!\n",
    "\n",
    "# The url of your Omeka site's api (basically your Omeka site site url with '/api' on the end)\n",
    "API_URL = 'http://your.omeka.site/api'\n",
    "\n",
    "# The keys to your Omeka site\n",
    "KEY_IDENTITY = 'YOUR OMEKA KEY IDENTITY'\n",
    "KEY_CREDENTIAL = 'YOUR OMEKA KEY CREDENTIAL'\n",
    "\n",
    "# Your Trove API key\n",
    "TROVE_API_KEY = 'YOUR TROVE API KEY'\n",
    "\n",
    "# Resize images so this is the max dimension -- the Trove page images are very big, so you might want to resize before uploading to Omeka\n",
    "# Set this to None if you want them as big as possible (this might be useful if you're using the Omeka IIIF server & Universal viewer modules)\n",
    "MAX_IMAGE_SIZE = 3000"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preparing Omeka-S\n",
    "\n",
    "Before we start uploading newspaper articles we need to set up the vocabularies and templates we need in Omeka to describe them.\n",
    "\n",
    "### Installing the schema.org vocabulary\n",
    "\n",
    "Omeka uses defined vocabularies to describe items. A number of vocabularies are built-in, but we're going to add the widely-used [Schema.org](https://schema.org/) vocabularly. Amongst many other things, Schema.org includes classes for both `Newspaper` and `NewsArticle`. You can download the [Schema.org definition files here](https://schema.org/docs/developers.html). In the instructions below, I suggest installing the 'all layers' version of the definition file, as this includes the [Bib extension](https://bib.schema.org/) as well as [Pending changes](https://pending.schema.org/) such as `ArchiveComponent` and `ArchiveOrganization`.\n",
    "\n",
    "To install Schema.org in Omeka:\n",
    "\n",
    "1. Log in to the admin interface of your Omeka-S site and click on 'Vocabularies' in the side menu.\n",
    "2. Click on the 'Import new vocabulary button.\n",
    "3. In the `Vocabularly url` field enter http://schema.org/version/latest/all-layers.rdf\n",
    "4. In the `File format` field select 'RDF/XML'\n",
    "5. In the `Prefix` field enter 'schema'\n",
    "6. In the `Namespace URI` field enter http://schema.org/\n",
    "7. In the `Label` field enter 'Schema.org'\n",
    "8. Click 'Import' to complete the installation.\n",
    "\n",
    "This is what the import form should look like:\n",
    "\n",
    "<img src=\"images/omeka-vocab-import.png\" width=\"600\">\n",
    "\n",
    "See the Omeka-S documentation for [more information on managing vocabularies](https://omeka.org/s/docs/user-manual/content/vocabularies/).\n",
    "\n",
    "### Installing the Numeric Data Type module\n",
    "\n",
    "The [Numeric Data Type module](https://omeka.org/s/docs/user-manual/modules/numericdatatypes/) gives you more options in defining the data type of a field. In particular, you can identify certain values as ISO formatted dates so they can then be properly formatted, sorted, and searched. The template I've created for newspaper articles (see below) uses the 'timestamp' data type to import the dates of the newspaper articles, so you'll need to install this module before importing the templates.\n",
    "\n",
    "1. Download the [Numeric Data Type module](https://omeka.org/s/modules/NumericDataTypes/) to your computer.\n",
    "2. Upload the zipped module to the modules folder of your Omeka site.\n",
    "3. Unzip the module.\n",
    "4. Log in to the admin interface of your Omeka-S site and click on 'Modules' in the side menu.\n",
    "5. Click on the 'Install' button.\n",
    "\n",
    "See the Omeka-S documentation for [more information on installing modules](https://omeka.org/s/docs/user-manual/modules/#installing-modules).\n",
    "\n",
    "### Importing the resource templates \n",
    "\n",
    "One of the powerful features of Omeka is the ability to define the types of items you're working with using 'resource templates'. These resource templates associate the items with specific vocabulary classes and list the expected properties. I've created resource templates for 'Newspaper' and 'Newspaper article' and exported them as JSON. The Trove upload code looks for these templates, so they need to be imported into your Omeka site.\n",
    "\n",
    "1. Download [Newspaper.json](templates/Newspaper.json) and [Newspaper_article.json](templates/Newspaper_article.json) to your own computer.\n",
    "2. Log in to the admin interface of your Omeka-S site and click on 'Resource templates' in the side menu.\n",
    "3. Click on the 'Import' button\n",
    "4. Click on 'Browse' and select the `Newspaper.json` file you downloaded.\n",
    "5. Click on the 'Review import' button.\n",
    "6. Click on the 'Complete import' button.\n",
    "7. Click the 'Edit Resource Template' button.\n",
    "8. Find the `name` property and click on the pencil icon to edit it.\n",
    "9. Check the box next to 'Use for resource title', then click on the 'Set changes' button.\n",
    "10. Find the `description` property and click on the pencil icon to edit it.\n",
    "11. Check the box next to 'Use for resource description', then click on the 'Set changes' button.\n",
    "12. Click on the 'Save' button.\n",
    "13. Repeat the same procedure to import `Newspaper_article.json`, but check at the 'Review import' stage to make sure that the selected data type for the `datePublished` property is `Timestamp`.\n",
    "\n",
    "Note that I've deliberately kept the templates fairly simple. You might want to add additional properties. However, if you change or remove any of the existing properties, you'll probably also have to edit the `add_article()` function below.\n",
    "\n",
    "See the Omeka-S documentation for [more information on managing resource templates](https://omeka.org/s/docs/user-manual/content/resource-template/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Define all the functions that we need\n",
    "\n",
    "Hit **Shift+Enter** to run each of the cells below and set up all the basic functions we need."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "from IPython.display import display, HTML, JSON\n",
    "import json\n",
    "from bs4 import BeautifulSoup\n",
    "from PIL import Image\n",
    "from io import BytesIO\n",
    "import re\n",
    "import arrow\n",
    "import os\n",
    "from tqdm.auto import tqdm\n",
    "from pyzotero import zotero\n",
    "from requests.adapters import HTTPAdapter\n",
    "from requests.packages.urllib3.util.retry import Retry\n",
    "\n",
    "s = requests.Session()\n",
    "retries = Retry(total=10, backoff_factor=1, status_forcelist=[ 502, 503, 504, 524 ])\n",
    "s.mount('http://', HTTPAdapter(max_retries=retries))\n",
    "s.mount('https://', HTTPAdapter(max_retries=retries))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# SOME GENERAL FUNCTIONS FOR WORKING WITH OMEKA\n",
    "\n",
    "def get_resource_class(resource_template_id):\n",
    "    '''\n",
    "    Gets the resource class associated with a given resource template.\n",
    "    '''\n",
    "    response = s.get('{}/resource_templates/{}'.format(API_URL, resource_template_id))\n",
    "    data = response.json()\n",
    "    return data['o:resource_class']\n",
    "    \n",
    "def get_resource_properties(resource_template_id):\n",
    "    '''\n",
    "    Gets basic info about the properties in a resource template including:\n",
    "        - property term\n",
    "        - property id\n",
    "        - data type\n",
    "    '''\n",
    "    properties = []\n",
    "    response = s.get('{}/resource_templates/{}'.format(API_URL, resource_template_id))\n",
    "    data = response.json()\n",
    "    for prop in data['o:resource_template_property']:\n",
    "        prop_url = prop['o:property']['@id']\n",
    "        # The resource template doesn't include property terms, so we have to go to the property data\n",
    "        prop_response = s.get(prop_url)\n",
    "        prop_data = prop_response.json()\n",
    "        # Give a default data type of literal if there's not one defined in the resource template\n",
    "        data_type = 'literal' if prop['o:data_type'] is None else prop['o:data_type']\n",
    "        properties.append({'term': prop_data['o:term'], 'property_id': prop_data['o:id'], 'type': data_type})\n",
    "    return properties     \n",
    "\n",
    "def prepare_payload(resource_template_id):\n",
    "    '''\n",
    "    Create a skeleton payload for the creation of a new item based on the given resource template.\n",
    "    '''\n",
    "    resource_class = get_resource_class(resource_template_id)\n",
    "    properties = get_resource_properties(resource_template_id)\n",
    "    payload = {\n",
    "        'o:resource_class': {\n",
    "            '@id': resource_class['@id'],\n",
    "            'o:id': resource_class['o:id']\n",
    "        },\n",
    "        'o:resource_template': {\n",
    "            '@id': '{}/resource_templates/{}'.format(API_URL, resource_template_id),\n",
    "            'o:id': resource_template_id\n",
    "        },\n",
    "        \"o:thumbnail\": None,\n",
    "        \"o:media\": [ ],\n",
    "        \"o:item_set\": [ ]\n",
    "    }\n",
    "    for prop in properties:\n",
    "        payload[prop['term']] = [{\n",
    "            'type': prop['type'],\n",
    "            'property_id': prop['property_id']\n",
    "        }]\n",
    "    return payload\n",
    "\n",
    "def get_template_id(label):\n",
    "    params = {\n",
    "        'label': label\n",
    "    }\n",
    "    response = s.get('{}/resource_templates/'.format(API_URL), params=params)\n",
    "    return response.json()[0]['o:id']\n",
    "\n",
    "# SPECIFIC FUNCTIONS FOR UPLOADING TROVE NEWSPAPERS\n",
    "\n",
    "def get_article(article_id):\n",
    "    '''Retrieve an individual newspaper article from the Trove API.'''\n",
    "    url = 'http://api.trove.nla.gov.au/v2/newspaper/{}'.format(article_id)\n",
    "    params = {\n",
    "        'include': 'articleText',\n",
    "        'encoding': 'json',\n",
    "        'key': TROVE_API_KEY\n",
    "    }\n",
    "    response = s.get(url, params=params)\n",
    "    return response.json()['article']\n",
    "\n",
    "def get_total_results(params):\n",
    "    '''\n",
    "    Get the total number of results for a Trove search.\n",
    "    '''\n",
    "    these_params = params.copy()\n",
    "    these_params['n'] = 0\n",
    "    response = s.get('https://api.trove.nla.gov.au/v2/result', params=these_params)\n",
    "    data = response.json()\n",
    "    return int(data['response']['zone'][0]['records']['total'])\n",
    "\n",
    "def get_newspaper_id_property(template_resource_id):\n",
    "    properties = get_resource_properties(template_resource_id)\n",
    "    for prop in properties:\n",
    "        if prop['term'] == 'schema:identifier':\n",
    "            return prop['property_id']\n",
    "\n",
    "def find_newspaper(newspaper_id, template_resource_id):\n",
    "    id_prop = get_newspaper_id_property(template_resource_id)\n",
    "    newspaper_class_id = get_resource_class(template_resource_id)\n",
    "    params = {\n",
    "        'property[0][joiner]': 'and', # and / or joins multiple property searches\n",
    "        'property[0][property]': id_prop, # property id\n",
    "        'property[0][type]': 'eq', # See above for options\n",
    "        'property[0][text]': newspaper_id,\n",
    "        'resource_class_id': [newspaper_class_id],\n",
    "        'item_set_id': []  \n",
    "    }\n",
    "    response = s.get('{}/items'.format(API_URL), params=params)\n",
    "    data = response.json()\n",
    "    try:\n",
    "        newspaper_id = data[0]['o:id']\n",
    "    except (IndexError, KeyError):\n",
    "        newspaper_id = None\n",
    "    return newspaper_id\n",
    "\n",
    "def get_newspaper(newspaper):\n",
    "    '''\n",
    "    Checks to see if there's already a record for the given newspaper in Omeka.\n",
    "    If not, it creates one.\n",
    "    Returns the Omeka id for the newspaper.\n",
    "    '''\n",
    "    template_id = get_template_id('Newspaper')\n",
    "    newspaper_id = find_newspaper(newspaper['id'], template_id)\n",
    "    if not newspaper_id:\n",
    "        payload = prepare_payload(template_id)\n",
    "        payload['schema:identifier'][0]['@value'] = newspaper['id']\n",
    "        payload['schema:name'][0]['@value'] = newspaper['value']\n",
    "        payload['schema:url'][0]['@id'] = 'http://nla.gov.au/nla.news-title{}'.format(newspaper['id'])\n",
    "        # print(payload)\n",
    "        params = {\n",
    "            'key_identity': KEY_IDENTITY,\n",
    "            'key_credential': KEY_CREDENTIAL\n",
    "        }\n",
    "        response = s.post('{}/items/'.format(API_URL), json=payload, params=params)\n",
    "        data = response.json()\n",
    "        newspaper_id = data['o:id']\n",
    "    return newspaper_id\n",
    "\n",
    "def get_newspaper_pdf(article_id):\n",
    "    '''\n",
    "    Use my proxy app to get the url to the PDF copy of an article.\n",
    "    '''\n",
    "    response = s.get('https://trove-proxy.herokuapp.com/pdf/{}'.format(article_id))\n",
    "    response = s.get(response.text)\n",
    "    filename = '{}.pdf'.format(article_id)\n",
    "    with open(filename, 'wb') as pdf_file:\n",
    "        pdf_file.write(response.content)\n",
    "    return filename\n",
    "\n",
    "def find_article(article_id, template_resource_id):\n",
    "    id_prop = get_newspaper_id_property(template_resource_id)\n",
    "    article_class_id = get_resource_class(template_resource_id)\n",
    "    params = {\n",
    "        'property[0][joiner]': 'and', # and / or joins multiple property searches\n",
    "        'property[0][property]': id_prop, # property id\n",
    "        'property[0][type]': 'eq', # See above for options\n",
    "        'property[0][text]': article_id,\n",
    "        'resource_class_id': [article_class_id],\n",
    "        'item_set_id': []  \n",
    "    }\n",
    "    response = s.get('{}/items'.format(API_URL), params=params)\n",
    "    data = response.json()\n",
    "    try:\n",
    "        article_id = data[0]['o:id']\n",
    "    except (IndexError, KeyError):\n",
    "        article_id = None\n",
    "    return article_id\n",
    "\n",
    "def add_article(article):\n",
    "    '''\n",
    "    Add a newspaper aticle (and image) to Omeka.\n",
    "    '''\n",
    "    template_id = get_template_id('Newspaper article')\n",
    "    article_id = find_article(article['id'], template_id)\n",
    "    if article_id is None:\n",
    "        payload = prepare_payload(template_id)\n",
    "        payload['schema:identifier'][0]['@value'] = article['id']\n",
    "        payload['schema:name'][0]['@value'] = article['heading']\n",
    "        # A basic citation\n",
    "        payload['schema:description'][0]['@value'] = '{}, {}, page {}'.format(arrow.get(article['date']).format('D MMM YYYY'), article['title']['value'], article['page'])\n",
    "        payload['schema:datePublished'][0]['@value'] = article['date']\n",
    "        # Fails silently if you don't turn the page number into a string\n",
    "        payload['schema:pagination'][0]['@value'] = str(article['page'])\n",
    "        # Get the id of the newspaper item record\n",
    "        newspaper_id = get_newspaper(article['title'])\n",
    "        # Connect to the newspaper record\n",
    "        payload['schema:isPartOf'][0]['type'] = 'resource:item'\n",
    "        payload['schema:isPartOf'][0]['@id'] = '{}/items/{}'.format(API_URL, newspaper_id)\n",
    "        payload['schema:isPartOf'][0]['value_resource_id'] = newspaper_id\n",
    "        payload['schema:isPartOf'][0]['value_resource_name'] = 'items'\n",
    "        # Link to Trove\n",
    "        payload['schema:url'][0]['@id'] = 'http://nla.gov.au/nla.news-article{}'.format(article['id'])\n",
    "        # Remove html tags from article text\n",
    "        try:\n",
    "            soup = BeautifulSoup(article['articleText'])\n",
    "            article_text = soup.get_text()\n",
    "            # Add full text\n",
    "            payload['schema:text'][0]['@value'] = article_text\n",
    "        except KeyError:\n",
    "            pass\n",
    "        # Data for image file upload\n",
    "        payload['o:media'] = []\n",
    "        files = {}\n",
    "        images = get_page_images(article['id'])\n",
    "        for index, image in enumerate(images):\n",
    "            page_id = image[:-4]\n",
    "            payload['o:media'].append({'o:ingester': 'upload', 'file_index': str(index), 'o:item': {}, 'dcterms:title': [{'property_id': 1, '@value': 'Page: {}'.format(page_id), 'type': 'literal'}]})\n",
    "            with open(image, 'rb') as f:\n",
    "                files['file[{}]'.format(index)] = f.read()\n",
    "        pdf_file = get_newspaper_pdf(article['id'])\n",
    "        payload['o:media'].append({'o:ingester': 'upload', 'file_index': str(index+1), 'o:item': {}, 'dcterms:title': [{'property_id': 1, '@value': 'PDF: {}'.format(article['id']), 'type': 'literal'}]})\n",
    "        with open(pdf_file, 'rb') as f:\n",
    "            files['file[{}]'.format(str(index+1))] = f.read()\n",
    "        files['data'] = (None, json.dumps(payload), 'application/json')\n",
    "        params = {\n",
    "            'key_identity': KEY_IDENTITY,\n",
    "            'key_credential': KEY_CREDENTIAL\n",
    "        }\n",
    "        # Post the files!\n",
    "        response = s.post('{}/items/'.format(API_URL), files=files, params=params)\n",
    "        # Clean up\n",
    "        for image in images:\n",
    "            os.remove(image)\n",
    "        os.remove(pdf_file)\n",
    "        # display(JSON(response.json()))\n",
    "    \n",
    "def get_box(zones):\n",
    "     # Start with the first element, but...\n",
    "    left = 10000\n",
    "    right = 0\n",
    "    top = 10000\n",
    "    bottom = 0\n",
    "    page_id = zones[0]['data-page-id']\n",
    "    for zone in zones:\n",
    "        if int(zone['data-y']) < top:\n",
    "            top = int(zone['data-y'])\n",
    "        if int(zone['data-x']) < left:\n",
    "            left = int(zone['data-x'])\n",
    "        if (int(zone['data-x']) + int(zone['data-w'])) > right:\n",
    "            right = int(zone['data-x']) + int(zone['data-w'])\n",
    "        if (int(zone['data-y']) + int(zone['data-h'])) > bottom:\n",
    "            bottom = int(zone['data-y']) + int(zone['data-h'])\n",
    "    return {'page_id': page_id, 'left': left, 'top': top, 'right': right, 'bottom': bottom}\n",
    "    \n",
    "def get_article_boxes(article_url):\n",
    "    '''\n",
    "    Positional information about the article is attached to each line of the OCR output in data attributes.\n",
    "    This function loads the HTML version of the article and scrapes the x, y, and width values for each line of text \n",
    "    to determine the coordinates of a box around the article.\n",
    "    '''\n",
    "    boxes = []\n",
    "    response = s.get(article_url)\n",
    "    soup = BeautifulSoup(response.text, 'lxml')\n",
    "    # Lines of OCR are in divs with the class 'zone'\n",
    "    # 'onPage' limits to those on the current page\n",
    "    zones = soup.select('div.zone.onPage')\n",
    "    boxes.append(get_box(zones))\n",
    "    off_page_zones = soup.select('div.zone.offPage')\n",
    "    if off_page_zones:\n",
    "        current_page = off_page_zones[0]['data-page-id']\n",
    "        zones = []\n",
    "        for zone in off_page_zones:\n",
    "            if zone['data-page-id'] == current_page:\n",
    "                zones.append(zone)\n",
    "            else:\n",
    "                boxes.append(get_box(zones))\n",
    "                zones = [zone]\n",
    "                current_page = zone['data-page-id']\n",
    "        boxes.append(get_box(zones))    \n",
    "    return boxes\n",
    "\n",
    "def get_page_images(article_id):\n",
    "    '''\n",
    "    Extract an image of the article from the page image(s), save it, and return the filename(s).\n",
    "    '''\n",
    "    images = []\n",
    "    # Get position of article on the page(s)\n",
    "    boxes = get_article_boxes('http://nla.gov.au/nla.news-article{}'.format(article_id))\n",
    "    for box in boxes:\n",
    "        # print(box)\n",
    "        # Construct the url we need to download the page image\n",
    "        page_url = 'https://trove.nla.gov.au/ndp/imageservice/nla.news-page{}/level{}'.format(box['page_id'], 7)\n",
    "        # Download the page image\n",
    "        response = s.get(page_url)\n",
    "        # Open download as an image for editing\n",
    "        img = Image.open(BytesIO(response.content))\n",
    "        # Use coordinates of top line to create a square box to crop thumbnail\n",
    "        points = (box['left'], box['top'], box['right'], box['bottom'])\n",
    "        # Crop image to article box\n",
    "        cropped = img.crop(points)\n",
    "        # Resize if necessary\n",
    "        if MAX_IMAGE_SIZE:\n",
    "            cropped.thumbnail((MAX_IMAGE_SIZE, MAX_IMAGE_SIZE), Image.ANTIALIAS)\n",
    "        # Save and display thumbnail\n",
    "        cropped_file = '{}.jpg'.format(box['page_id'])\n",
    "        cropped.save(cropped_file)\n",
    "        images.append(cropped_file)\n",
    "    return images"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Select your upload method\n",
    "\n",
    "Select the method you want to use to select and process your articles and follow the associated instructions.\n",
    "\n",
    "----\n",
    "\n",
    "### Option 1: Upload a Trove newspaper search\n",
    "\n",
    "This will attempt to upload **all** the newspaper articles returned by a search to your Omeka site. Obviously you want to **restrict your search** to make sure you're only getting the articles you want – use things like facets and date ranges to keep the result set to a manageable size. **You have been warned...**\n",
    "\n",
    "Once you've constructed your search, enter the various parameters below. You might want to add additional facets such as `l-decade` or `l-year`. Check out [Trove's API documentation](https://help.nla.gov.au/trove/building-with-trove/api-version-2-technical-guide) to see all the options.\n",
    "\n",
    "Once you've edited the search details, hit **Shift+Enter** to run the cells below and start your upload."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def upload_trove_search(params):\n",
    "    start = '*'\n",
    "    total = get_total_results(params)\n",
    "    with tqdm(total=total) as pbar:\n",
    "        while start:\n",
    "            params['s'] = start\n",
    "            response = s.get('https://api.trove.nla.gov.au/v2/result', params=params)\n",
    "            data = response.json()\n",
    "            # The nextStart parameter is used to get the next page of results.\n",
    "            # If there's no nextStart then it means we're on the last page of results.\n",
    "            try:\n",
    "                start = data['response']['zone'][0]['records']['nextStart']\n",
    "            except KeyError:\n",
    "                start = None\n",
    "            for article in data['response']['zone'][0]['records']['article']:\n",
    "                add_article(article)\n",
    "                pbar.update(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Edit/add search values and parameters as required. These are an example only!\n",
    "trove_params = {\n",
    "        'q': '\"inigo jones\"', # required -- change to anything you might enter in the Trove search box (including double quotes for phrases and boolean operators like AND)\n",
    "        'zone': 'newspaper', # don't change this\n",
    "        'l-illustrated': 'true', # edit or remove -- limits to illustrated articles\n",
    "        'l-illtype': 'Photo', # edit or remove -- limits to illustrations with photos\n",
    "        'l-word': '1000+ Words', # edit or remove -- limits to article with more than 1000 words\n",
    "        'include': 'articleText', # don't change this\n",
    "        'encoding': 'json', # don't change this\n",
    "        'key': TROVE_API_KEY # don't change this\n",
    "    }\n",
    "\n",
    "upload_trove_search(trove_params)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----\n",
    "\n",
    "### Option 2: Upload newspaper articles from a Trove list\n",
    "\n",
    "You can upload any newspaper articles stored in a Trove list to your Omeka site. \n",
    "\n",
    "To find the `list_id`, just go to the list's web page. The `list_id` is the string of numbers that appears after 'id=' in the url. Once you have your `list_id`, paste it where indicated in the cell below.\n",
    "\n",
    "Once you've edited the list details, hit **Shift+Enter** to run the cells below and start your upload."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def upload_trove_list(list_id):\n",
    "    '''\n",
    "    Upload any newspaper articles in the given Trove list to Omeka.\n",
    "    '''\n",
    "    url = 'http://api.trove.nla.gov.au/v2/list/{}'.format(list_id)\n",
    "    params = {\n",
    "        'include': 'listItems',\n",
    "        'encoding': 'json',\n",
    "        'key': TROVE_API_KEY\n",
    "    }\n",
    "    response = s.get(url, params=params)\n",
    "    data = response.json()\n",
    "    for item in tqdm(data['list'][0]['listItem']):\n",
    "        for zone, record in item.items():\n",
    "            if zone == 'article':\n",
    "                article = get_article(record['id'])\n",
    "                add_article(article)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Paste the identifier of your list between the quotes\n",
    "list_id = '[Your list ID]'\n",
    "upload_trove_list(list_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----\n",
    "\n",
    "### Option 3: Upload Trove newspaper articles saved in Zotero\n",
    "\n",
    "You can upload any Trove newspaper articles stored in a Zotero collection to your Omeka site.\n",
    "\n",
    "To access Zotero you need four pieces of information:\n",
    "\n",
    "* `ZOTERO_KEY` – generate an API for this application by [going here](https://www.zotero.org/settings/keys/new) once you've logged into the Zotero web site. Your key only need read access.\n",
    "* `LIBRARY_ID` – a string of numbers that identifies your library. For personal libraries, go to [this page](https://www.zotero.org/settings/keys/) once you've logged into the Zotero web site and look for the line that says 'Your userID for use in API calls is...'. For group libraries, just open the groups web page – the `LIBRARY_ID` will be the string of numbers after '/groups/' in the url.\n",
    "* `LIBRARY_TYPE` – either 'user' or 'group', depending on whether it's a personal library of a group library.\n",
    "* `collection_id` – the id of the collection that contains your Trove newspaper articles. Just open the collection on the Zotero website – the `collection_id` is the string of letters and numbers that comes after '/collectionKey/' in the url.\n",
    "\n",
    "For additional information see the [Pyzotero](https://github.com/urschrei/pyzotero) documention.\n",
    "\n",
    "When you have all the values you need, simply paste them where indicated in the cells below.\n",
    "\n",
    "Once you've added your details, hit **Shift+Enter** to run the cells below and start your upload."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ENTER YOUR VALUES BETWEEN THE QUOTES WHERE INDICATED\n",
    "ZOTERO_KEY = 'YOUR ZOTERO KEY' # The Zotero API key you generated\n",
    "LIBRARY_TYPE = 'user' # user or group\n",
    "LIBRARY_ID = 'YOUR ZOTERO ID' # Either a personal user id or a group id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def upload_zotero_collection(coll_id):\n",
    "    '''\n",
    "    Upload any Trove newspaper articles in the given collection to Omeka.\n",
    "    '''\n",
    "    zot = zotero.Zotero(LIBRARY_ID, LIBRARY_TYPE, ZOTERO_KEY)\n",
    "    items = zot.everything(zot.collection_items(coll_id))\n",
    "    articles = []\n",
    "    for item in items:\n",
    "        # Filter out things that aren't newspaper articles\n",
    "        try:\n",
    "            url = item['data']['url']\n",
    "            if item['data']['itemType'] == 'newspaperArticle' and 'nla.news-article' in url:\n",
    "                article_id = re.search('(\\d+)$', url).group(1)\n",
    "                articles.append(article_id)\n",
    "        except KeyError:\n",
    "            pass\n",
    "    for article_id in tqdm(articles):       \n",
    "        article = get_article(article_id)\n",
    "        add_article(article)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Paste your collection ID between the quotes below.\n",
    "collection_id = 'YOUR COLLECTION ID'\n",
    "upload_zotero_collection(collection_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---- \n",
    "\n",
    "### Option 4: Upload a list of article ids\n",
    "\n",
    "This is useful for testing – just get some Trove newspaper article identifiers (the number in the url) and upload them to your Omeka site.\n",
    "\n",
    "Once you've edited the list of article, hit **Shift+Enter** to run the cells below and start your upload."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Edit the list of articles as you see fit...\n",
    "article_ids = [\n",
    "    135348864,\n",
    "    70068753,\n",
    "    193580452\n",
    "]\n",
    "\n",
    "for article_id in tqdm(article_ids):\n",
    "    article = get_article(article_id)\n",
    "    add_article(article)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Future developments\n",
    "\n",
    "* I'd originally hoped to use the Omeka-S API to do most of the configuration, including importing vocabularies and resource templates. But I [just couldn't get it to work properly](https://forum.omeka.org/t/upload-resource-template-via-api/8855/5). If I figure it out I'll add the details.\n",
    "\n",
    "* It would be good to be able to upload other types of things from Trove such as images, maps and journal articles. This should be possible, however, things are not as standardised as they are with the newspaper articles, so it'll take a bit of work. Let me know if you're interested and it'll give me a bit more motivation.\n",
    "\n",
    "* Also, going back to my Omeka how-to from 8 years back, I'll be creating a RecordSearch to Omeka-S pipeline as well."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----\n",
    "\n",
    "Created by [Tim Sherratt](https://timsherratt.org/) for the [GLAM Workbench](https://glam-workbench.github.io/)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}