This small aplication allows you to query the CKAN implementation at iatiregistry.org to find all 'end point' urls of data recorded on the registry, and to then download that data.
The registry holds records about where data can be found on the interent.
The application is basically 2 scripts that you run one after the other.
grab_urls.php
queries the registry and creates a text file for each group on the registry of all url end points of IATI data files
fetch_data.sh
uses wget to pull all the data from those url text files and deposit them in their own directory.
IATI Registry Refresher requires PHP version 5.2.0 or later.
It also requires curl. on Ubuntu
sudo apt-get install curl
sudo apt-get install php5-curl php5-cli
Place all files in the same directory.
Create an empty directories called urls, data and ckan
mkdir urls data ckan
From a terminal, use php-cli (command line interface) to run:
php grab_urls.php
(if you want to set up your own paths, copy this file to e.g. grab_my_urls.php and edit the paths.)
This gives the data endpoints for all the files in the IATI registry (see
http://iatiregistry.org/)
Run fetch_data.sh to get all the data.
(In a terminal type ./fetch_data.sh)
(if you want to set up your own paths, copy this file to e.g. fetch_my_urls.sh and edit the paths.)
The code in git.sh can be used to update a git repository (in the data directory) with a new commit each time it is run. The IATI Tech Team maintains a git repository with nightly snapshot commits, but it is not public, see https://github.com/IATI/IATI-Stats#getting-some-data-to-run-stats-on
If your copy of wget is compiled against an old version of gnutls, then some https downloads will fail. Please make sure your system has the latest version of gnutls installed.
If you find any bugs, note any issues or have any feature requests, please report them at https://github.com/caprenter/IATI-Registry-Refresher
Copyright 2012 caprenter <[email protected]>
This file is part of IATI Registry Refresher.
IATI Registry Refresher is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
IATI Registry Refresher is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with IATI Registry Refresher. If not, see <http://www.gnu.org/licenses/>.