views: support for elasticsearch scroll to retrieve all records

[lnielsen]

In order to be able to use the records REST API to retrieve all records we need to add support for scroll over the elasticsearch results.

Related to #28

[lnielsen]

@nharraud Is this something you would have time to look into?

[jma]

It would be great to integrate this feature as we it is an alternative to the legacy OAI-PMH harvesting.

[slint]

The logic wouldn't change a lot compared to what invenio-oaiserver already does. I guess introducing a scroll_id/cursor querystring parameter and doing a similar call is all that's needed... Good extras would be having the ability to enable/disable/rate-limit the feature

[lnielsen]

Agreed, it would probably need separate rate-limiting. The alternative to this feature is something like the exporter we have in Zenodo, that can produce a full dump of the records on a regular basis. On Zenodo e.g. a full-dump of 400k records is something like 250MB BZip2 compressed JSON.