Create caching screenshot service for OA material

[anjackson]

Add an endpoint that uses https://github.com/ukwa/webrender-api to render screenshots of OA items via pywb in proxy mode. It should cache them essentially permanently, and keep the URL lists to retain the transclusions for whitelisting.

This has been made much easier by using an off-the-shelf caching IIIF server (Cantaloupe) as an intermediary. This now works reasonably well, see: ukwa/ukwa-services#24 (comment)

Some work remains:

• Remove banner by having separate pywb config for rendering.
• Chase up T:73492 (all o' Twitter)
• Review exposed API and tidy up.
• Actually do thumbs/full/card sizes properly
• Update pywb and add in IIIF API hooks for social cards.
• Note switch to https://flask-restx.readthedocs.io/en/latest/
• Extend Flask-RESTx to add logo etc as per this
• Switch to general OG markup rather than proprietary Twitter tags, as per
• Report pywb returning that 304. (???)
• Implement test suite to cover new functionality.

[anjackson]

So, started working.

[anjackson]

As per ukwa/ukwa-services#24 (comment) added an IIIF server which makes a lot of this much easier. Still lots of those details to tie off.

[anjackson]

Also need to access rendered versions from crawl time. Simplest is to use the screenshot: prefix, but PyWB doesn't support that at this time.

• Lookup works fine: https://beta.webarchive.org.uk/api/query/lookup?sort=reverse&url=screenshot:http://bbc.co.uk/news
• But WARC retrieval fails when checking access rights using PyWB: https://beta.webarchive.org.uk/api/query/warc/20201219140038/screenshot:http://www.bbc.co.uk/news
• PyWB doesn't like it: https://beta.webarchive.org.uk/wayback/archive/20201219140038/screenshot:http://www.bbc.co.uk/news >>> Invalid Url: http://screenshot:http://www.bbc.co.uk/news
• IIIF gives 400 reported as 500: https://beta.webarchive.org.uk/api/iiif/2/urn:pwid:webarchive.org.uk:2020-12-19T14:14:38Z:page:screenshot%253Ahttp%253A%252F%252Fwww.bbc.co.uk%252Fnews/0%2C0%2C1024%2C1024/600%2C/0/default.png

[anjackson]

I monkeypatched ukwa-pywb to allow the screenshot:http URL scheme through, but now we get a 451 because the access rights are not matched up. See ukwa/ukwa-pywb#62

[anjackson]

This mess of ideas needs splitting into separate tickets. Deferring for now.