- Incremental harvesting during a given time window
- Option to resume from a given token
- Support for Python < 3.6
- Adopt codestyle from black
- Refactor out fetching records
- Changes file follows recognised convention
- Support for Python 2.6
- Log complete traceback on errors
- Decode binary metadata string returned by pyoai in Python 3
- Output files as UTF-8
- Support for building a development virtualenv and testing builds using Tox
- Continuous Integration testing with Travis CI
- Working in Python 3 (thanks to @ulikoehler)
- Allow providers with https transport scheme (thanks to @ulikoehler)
- Fix build by using
ez_setup.py
instead ofdistribute_setup.py
for bootstrapping setuptools.
- Report an appropriate error when a given URL does not appear to resolve to an OAI-PMH server.
- Option to specify a string that, when it occurs in identifiers, is treated as a separator to store harvested records in sub-directories.
- File name escaping on Windows. On Windows do not protect
:
characters from being escaped.
- Option for creating sub-directories based on slashes in identifiers.
- Automatically create download directory if missing, thanks to @mhoffman
- Removed rogue debugging
raise
statement - Update registry with
--until
option when given - Only update registry if harvest completed (i.e. not truncated with
--limit=N
)
- Handle deletions
- Selective harvesting by set
- Limit harvesting to an arbitrary number of records (e.g. to avoid throttling / blacklisting by provider server)
- Don't ERROR when encountering deleted records
- Harvest from a given URL
- Selective harvesting by modification date
- Facilitate regular harvesting from 1 or more providers
- Register parameters - e.g. URL, Schema (
metadataPrefix
), target directory - for regular harvesting from a named provider - Specify provider using registered name when harvesting, or harvest from all registered providers using the reserved name
all
- Harvest only records modified since last harvest (unless overridden)
- Register parameters - e.g. URL, Schema (