This repository holds an example implementation of an Jenkins HA infrastructure setup solely based on Open Source components, whose reason of existence and development process is described in a blog post.
Prerequisites:
- Ruby
- Vagrant
- VirtualBox
- This setup may also serve as a simple Jenkins playground on your local machine (provides Linux and Windows agents)!
- It neither implements nor configurates any security measures other then
- private network for all component located behind the load balancer
- enabled firewall and some simple rules
- STONITH is not (yet) implemented
- cluster only implements cold-standby mode
- components: HAProxy, GlusterFS, Jenkins, Jenkins Swarm Plugin, Pacemaker, Corosync
- adjust
/conf.envaccording to your needs (and available host resources) vagrant up- go to
http[s]:${EXTERNAL_LOAD_BALANCER_IP}:${EXTERNAL_LOAD_BALANCER_PORT}to visit Jenkins UI
NOTE: must be executed as privileged user
$ pcs status
$ pcs cluster status
$ pcs status corosyncTo move resources form one node to another (simulate failure), you could stop one node by going into
the node and do pcs cluster stop $NODE_NAME (node name defaults to local), or maybe change the
configuration for the preferred resource location (e.g. pcs constraint location jenkins-master--rsc prefers jenkins-master-2=INFINITY).
Another way would be to just send the active node into standby (pcs cluster standby jenkins-master-1)
- the HAProxy instance could play a role in fencing implementation, e.g. by preventing the dead node from getting traffic.
- closing down firewall on all gluster nodes to prevent unwanted access by a dead jenkins master node
- depending on availability, using cloud provider's API to shutdown a jenkins master node in question
- ...?
Both approaches require to write a fence agent from scratch, see
- https://github.com/ClusterLabs/fence-agents
- https://docs.pagure.org/ClusterLabs.fence-agents/FenceAgentAPI.md
/usr/share/fence/
The entire provisioning and system setup is implemented with scripts (bash, batch). Today of course, this is done in a more declarative way, that would also ensure idempotency. Since this examples is based on centos7 it would only make sense to use Ansible here. But, any technology supported by vagrant would suffice.