================= The yum-repo-server is a server that allows you to host and manage YUM repositories using a RESTful API.
- RESTful api for repository management (including creation, metadata generation, RPM upload, RPM propagation,...)
- Configurable scheduling system for periodic metadata generation on repositories with high activity
- Repository cleanup routines
- Graphical web interface to browse repositories and their contents
- Link system to create virtual repositories that can dynamically point to other repositories
- Easily extensible due to good test coverage
- Propagation of RPMs from one staging repository to the next
- Command line wrapper for more comfort, see below
The aim of this project is to provide a simple, straightforward and extensible implementation of a server that is able to manage YUM repositories. While creating a standalone YUM repository is easy, there was no easy way to manage many such repositories at the time of writing.
Our company is migrating towards a CLD-friendly deployment solution. Our solution involves release repositories that need to be dynamically referenceable in order to update hosts or entire host groups without changing the host's repositories. This is done like so :
- The yum-repo-server enables you to access repository management operations from other routines or automations, such as build servers or delivery chains.
- For instance you can dynamically create a repository when needed (e.G. after compiling sources), upload RPMs into it, generate metadata and then use it right away!
- The virtual repository system provides an additional layer of abstraction over repositories and allows you to create "fake" (virtual) repositories that forward any requests they obtain to a real repository.
- Since consumers cannot differentiate between virtual and regular repositories, it is possible to change the repositories used by hosts dynamically in one simple operation (instead of fiddling on the file system level in
/etc/yum/repos.d/for instance).- As a consequence, the virtual repository system enables you to use one (virtual) repository for a group of hosts, and change the link as needed, e.G. when updating packages.
- Since consumers cannot differentiate between virtual and regular repositories, it is possible to change the repositories used by hosts dynamically in one simple operation (instead of fiddling on the file system level in
- The yum-repo-server comes with built-in cleanup and metadata generation routines, meaning you do not need to use other tools (like CRON jobs) to manage repositories
- Install the packages that are not part of PyPi :
sudo apt-get install python-rpm
python-lxml
python2.7-dev
python-pycurl
PyYAML- Install the pip packages in a virtualenv (remember to use the system-site-packages switch to make the modules that are not on PyPi available to the virtualenv)
virtualenv --system-site-packages ve
. ve/bin/activate
pip install django==1.3 django-piston nose python-daemon lockfile stdeb mockito-
Build it :
python setup.py build -
Run the tests :
python setup.py test -
Install it :
python setup.py install -
Try it out! Running
python manage.py runserverwill start up a django developement server (not for production!!) featuring the yum-repo-server.- This development server is fully fledged and you can use it to determine if the yum-repo-server is what you want very quickly.
- Use the included yum-repo-client for more comfortable tryouts. (cd into client/ and install it with
python setup.py install)
For production usage we recommend an apache webserver (httpd) with mod_wsgi. If you build a RPM by python setup.py bdist_rpm an apache configuration file is automatically included. But the wsgi.py should work as well with other WSGI-compatible servers like CherryPy, twisted.web, Gunicorn, etc.
The yum-repo-server relies on django and piston to handle browser and API requests made through HTTP. You need to put a HTTP web server in front of the yum-repo-server, apache with mod_wsgi will do fine. Most of the scheduling routines use the APScheduler (Advanced Python Scheduler). In order to generate repository metadata, the createrepo package is used. Have a look at updaterepo [https://github.com/is24-herold/updaterepo] if you need a more performant createrepo.
At the core of the yum-repo-server is a directory that contains repository information and contents.
This directory's location is stored in a settings.py file, like so :
REPO_CONFIG = {'REPO_DIR' : '/var/yum-repos', [shortened] }
This makes it easy to backup or replicate your repository.
In a nutshell, when yum checks for updates it sends HTTP GET requests to repositories it is aware of (usually through repository files in /etc/yum/repos.d/) and queries repository metadata.
If it decides a package has to be updated (or installed) it will then directly download the RPM package through a HTTP request.
This is handled by django and quite straightforward.
A virtual repository does look exactly like a regular repository for consumers, but it is actually an empty repository that contains a YAML file named repo.yaml. The file contains an entry with a relative path to a regular repository, and requests to the virtual repository are rerouted to the regular one.
The metadata generation is located in a YAML file called metadata-generation.yaml that lives in the repository it describes.
The file looks like this :
generation_type : scheduled
This will schedule a periodic createrepo that will be executed every 40 seconds.
generation_interval : 40
rpm_max_keep : 3
rpm_max_keep means there will also be a cleanup routine before the createrepo that will delete older
RPMs when there are more than three RPMs with the same canonical name.
You may omit rpm_max_keep to disable the cleanup routine and set generation_type to manual or remove the file
if you do not wish to have a periodic createrepo scheduled.
API requests are handled by piston and use a REST like format. For maximal comfort, use the yum-repo-client. The examples below should give you a good understanding of how the requests look like.
Creating a new repository involves sending a POST request with the name of the repository in the body to $host/$repo_base.
This will create a new resource (the new repository) underneath the repository base, which means you can access your new repository at $host/$repo_base/$new_repo_name
A static repository can be deleted when sending a DELETE request to the repository (/repo/repo-to-delete). It can be protected from deletion when its name is listed within the /etc/yum-repo-server/non-deletable-repositories file.
Virtual repositories that were linked to the deleted static repository, will not be deleted or changed. The virtual repositories will deliver HTTP 404 sites as long as the static repository does not exist again or the link is changed manually.
As a consequence, uploading a RPM to an existing repository involves sending a POST request containing the RPM file in a form element called rpmFile. The request is send to $host/$repo_base/$repo_name
It creates a new resource underneath $repo_name.
The RPM can then be retrieved with a GET request sent to
$host/$repo_base/$repo_name/$rpm_architecture/$rpm_filename.
Generating metadata involves a POST request to $host/$repo_base/$repo_name/repodata since it creates a new resource (the actual metadata files) underneath repodata/.
You can propagate a RPM from a source repository to a destination repository on the same host by sending a POST request to $host/propagation/ with parameter source and destination.
source must be $source-repo-name/$architecture/artifact-name.rpm. destination is just name of the target repository.
Propagation does not work with virtual repositories.
For example:
curl -F "source=test-repo/noarch/ test-artifact&destination=test-repo2" http://myyum-repo-server/propagation/
will search for the latest test-artifact-XX-X.noarch.rpm and propagate the rpm from test-repo repository to test-repo2.
You can retrieve a list of static or virtual repositories for static repos via
http://myyum-repo-server/repo.txt
for virtual repos:
http://myyum-repo-server/repo/virtual.txt
Optionally you can get the destination for virtual repositories with the showDestination parameter. If set to true the list will contain entries with the following pattern: repo_name:destination. The destination is the path to the static repository or it could also be a url to an external repository.
To filter the list you have several url parameters:
- Filter by name regex:
http://myyum-repo-server/repo.txt?name=any_regex.* - Filter by tags:
http://myyum-repo-server/repo.txt?tag=tag1,tag2This will retrieve all repositories marked with tag1 or tag2. - Filter by tags exclusive:
http://myyum-repo-server/repo.txt?notag=tag1,tag2This will retrieve all repositories marked not marked with tag1 or tag2. - Filter by newer then X days:
http://myyum-repo-server/repo.txt?newer=10This will retrieve all repositories newer then 10 days. - Filter by older then X days:
http://myyum-repo-server/repo.txt?older=10This will retrieve all repositories older then 10 days.
All filters are concatable and are combined via and, so http://myyum-repo-server/repo.txt?older=10&newer=30 will retrieve all repositories older then 10 days and newer then 30 days.
===============
yum-repo-client is a command line interface for interacting with the yum-repo-server The aim is to provide a command line wrapper for every functionality the yum-repo-server provides so you don't have to fiddle with REST requests. This is especially good for automations that can run command line tools because you can modify the yum-repo-server as you wish without breaking your automation (provided you include the modifications in the yum-repo-client).
The yum-repo-client currently supports
- Remote repository creation and deletion
- RPM upload and remote deletion
- Virtual repository creation, linking, redirection and deletion
- Optional authentication and yum-repo-server host parametrization
- RPM propagation
- Smart bash autocompletion
The yum-repo-client comes bundled with the yum-repo-server. Everything you need is located in the project subfolder 'client'.
First of all it is recommended to run the tests:
python setup.py test
These should always be successfull.
Optionally after that you can run:
python setup.py bdist_rpm
to get an rpm of the yum-repo-client.
If you have build a rpm file in the step above, then you can install it as usual.
Without the rpm file you can install the yum-repo-client with:
python setup.py install
Simply call
[you@yourhost ~]$ repoclient
to display the help text that includes call syntax and operation description
To set the default host and port used by the yum-repo-client, you need to edit (or create) the file /etc/yum-repo-client.yaml.
This file should contain the following entries :
DEFAULT_HOST : localhost
DEFAULT_PORT : 8000
The configuration above is for usage with the django development server (not for production use!).