Skip to content

Commit

Permalink
Updates for linchpin 1.0.4 compatibility
Browse files Browse the repository at this point in the history 
* cinch/bin - The 'cinchpin' command has been fully removed and
deprecated.  cinch and linchpin are tracking different versions of
Ansible and cannot live in the same Python virtualenv, hence the need to
remove this convenience wrapper.  The new 'teardown' command is now
available to assist with removing Jenkins slaves from Jenkins masters.
This functionality was previously provided by 'cinchpin'.  Thanks to
greg-hellings for providing the code for this.

* cinch/playbooks/install-rhel7.yml - we now create two separate
virtualenvs for linchpin and cinch

* cinch/playbooks/install-rhel7.yml - the Python virtualenv creation
process has been stramlined by using native Ansible to work around
issues with the selinux Python module and outdated versions of pip and
setuptools

* cinch/playbooks/install-rhel7.yml - the Beaker Python package is no
longer installed in the virtualenv since with system-site-packages we
can use the Beaker version that's installed via RPM

* cinch/playbooks/install-rhel7.yml - the 'latest tip' install option
has been removed.  This is a developer tool that's no longer necessary
at this time

* Sphinx docs have been updated to reflect the changes in this commit

* All references to 'linch-pin' have been changed to 'linchpin' as the
project has been renamed

* jjb/ci-jslave-project-sample.yaml - This example template now uses the
two separate Python virtualenvs for linchpin and cinch

* jjb/ci-jslave-project-sample.yaml - The Jenkins jobs have been renamed
for less visual clutter

* jjb/ci-jslave-project-sample.yaml - A bug with teardown has been
fixed, where teardown would fail if the Jenkins slave never connected to
the Jenkins master

* jjb/install-rhel7.yml - Removed options for 'latest tip' and 'beaker'
to corresponded with the related Ansible playbook changes described
previously

* setup.py - Bumped version to 0.9.0

* setup.py - Replaced 'cinchpin' command entry point with new 'teardown'
command

* setup.py - Removed linchpin as a dependency
  • Loading branch information
David Roble committed Nov 20, 2017
1 parent c05173a commit 9252c96
Show file tree
Hide file tree
Showing 8 changed files with 144 additions and 341 deletions.
12 changes: 12 additions & 0 deletions CHANGELOG
View file
Original file line number Diff line number Diff line change
@@ -1,3 +1,15 @@
v 0.9.0 (16 Nov 2017)
- IMPORTANT - The 'cinchpin' command has been REMOVED for linchpin 1.0.4
compatibility. The new 'teardown' command replaces Jenkins slave
disconnection functionality previously handled by the 'cinchpin' command.
- The RHEL7 installer now creates two virtualenvs, one for linchpin and one for
cinch
- Removed 'latest tip' and Beaker python package installation options from
RHEL7 installer as they are no longer necessary
- Fixed a bug where Jenkins slaves would not be removed from the master during
a provisioning failure in our JJB example workflow
(ci-jslave-project-sample.yaml)

v 0.8.5 (10 Oct 2017)
- Remove management of the executor setting on masters (GH #182)
- Remove stale, unused repo key that began failing (GH #184)
Expand Down
60 changes: 23 additions & 37 deletions cinch/bin/entry_point.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,23 +2,12 @@
from __future__ import print_function
from argparse import ArgumentParser, REMAINDER
from os import getcwd, path
from wrappers import call_ansible, call_linchpin
from wrappers import call_ansible

import sys


def cinch():
"""
Entry point for the "cinch" CLI that merely wraps the ansible-playbook
command and pre-fills its path to the site.yml file for Cinch. The cinch
tool requires a single argument - the Ansible inventory file - and accepts
an arbitrary number of extra arguments that are passed through to the
ansible-playbook executable.
:return: Exit code 0 if the execution is completed successfully, or 255
if an unknown error occurs. If ansible-playbook exits with an error code,
this executable will exit with the same code.
"""
def cinch_generic(playbook):
# Parse the command line arguments
parser = ArgumentParser(description='A wrapper around Cinch for the most '
'common use case')
Expand All @@ -35,38 +24,35 @@ def cinch():
inventory = path.join(getcwd(), args.inventory)
else:
raise Exception('Inventory path needs to be non-empty')
exit_code = call_ansible(inventory, 'site.yml', args.args)
exit_code = call_ansible(inventory, playbook, args.args)
sys.exit(exit_code)


def cinchpin():
def cinch():
"""
Entry point for the "cinchpin" CLI that wraps the linchpin command and
loads the linch-pin PinFile to provision resources and then uses the
generated inventory file to pass to cinch. The cinchpin tool requires a
single argument - a valid linchpin subcommand - and accepts an arbitrary
number of extra arguments that are passed through to the linchpin
executable. If a linch-pin PinFile is not found in the current working
directory, a path to a linch-pin working directory may be optionally
provided.
Entry point for the "cinch" CLI that merely wraps the ansible-playbook
command and pre-fills its path to the site.yml file for Cinch. The cinch
tool requires a single argument - the Ansible inventory file - and accepts
an arbitrary number of extra arguments that are passed through to the
ansible-playbook executable.
:return: Exit code 0 if the execution is completed successfully, or 255
if an unknown error occurs. If linchpin exits with an error code,
if an unknown error occurs. If ansible-playbook exits with an error code,
this executable will exit with the same code.
"""
# Parse the command line arguments
parser = ArgumentParser(description='A wrapper around linchpin for the '
'most common use case')
# The linch-pin working directory containing a PinFile that the user
# provides which will get passed along to linchpin for its consumption
parser.add_argument('-w', '--workdir', default=getcwd(),
help='''path to linch-pin working directory containing a
PinFile''')
# All remaining arguments are passed through, untouched, to linchpin
parser.add_argument('arg', help='argument to pass to the linchpin command')
args = parser.parse_args()
exit_code = call_linchpin(args.workdir, args.arg)
sys.exit(exit_code)
cinch_generic('site.yml')


def teardown():
"""
Entry point for the "teardown" CLI that wraps ansible-playbook commands and
pre-fills its path to the teardown.yml file.
:return: Exit code 0 if the execution is completed successfully, or 255 if
an unknown error occurs. If ansible-playbook exits with an error code, this
executable will exit with the same code.
"""
cinch_generic('teardown.yml')


if __name__ == '__main__':
Expand Down
139 changes: 2 additions & 137 deletions cinch/bin/wrappers.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,6 @@

import os
import sys
import yaml


BASE = os.path.abspath(os.path.join(os.path.dirname(__file__), '..'))
Expand Down Expand Up @@ -36,149 +35,15 @@ def call_ansible(inventory, playbook, *args):
os.path.join(BASE, playbook),
'-i', inventory,
'-v',
'--ssh-common-args=-o StrictHostKeyChecking=no'
'--ssh-common-args=-o StrictHostKeyChecking=no ' +
'-o UserKnownHostsFile=/dev/null'
]
ansible_args.extend(args)
ansible = local['ansible-playbook']
exit_code = command_handler(ansible, ansible_args)
return exit_code


def call_linchpin(work_dir, arg):
"""
Wraps a call out to the linchpin executable, and then kicks off a cinch
Ansible playbook if necessary.
:param work_dir: The linch-pin working directory that contains a PinFile
and associated configuration files
:param arg: A single argument to pass to the linchpin command
:return: The exit code returned from linchpin, or 255 if errors come from
elsewhere
"""
# cinch will only support a subset of linchpin subcommands
supported_cmds = ['up', 'destroy', 'init']
if arg not in supported_cmds:
sys.exit('linchpin command "{0}" not '
'supported by cinch'.format(arg))

# If we are to ask linch-pin to interact with infrastructure we will check
# for some required configuration items and set up them for later use
if arg != 'init':
inventory_file = get_inventory(work_dir)
inventory_path = os.path.join(work_dir, 'inventories', inventory_file)

# For destroy/teardown, we must run our teardown playbook(s) *before*
# linchpin terminates the instance(s)
if arg == 'destroy':
exit_code = call_ansible(inventory_path, 'teardown.yml')

# Construct the arguments to pass to linch-pin by munging the arguments
# provided to this method
linchpin_args = [
'-v',
'-w', work_dir,
'--creds-path', os.path.join(work_dir, 'credentials')
]
linchpin_args.append(arg)
# Execute the 'linchpin' command
linchpin = local['linchpin']
exit_code = command_handler(linchpin, linchpin_args)

# Set up a linch-pin+cinch configuration skeleton for later use if the
# 'init' subcommand was executed previously
if arg == 'init':
cinchpin_init(work_dir)

# If linchpin is asked to provision resources, we will then run our
# cinch provisioning playbook
if arg == 'up' and exit_code == 0:
exit_code = call_ansible(inventory_path, 'site.yml')
return exit_code


def cinchpin_init(work_dir):
"""
Set up a linch-pin+cinch configuration skeleton
:param work_dir: The linch-pin working directory that contains a PinFile
and associated configuration files
"""
# Consistent filename to use for various linch-pin YAML configurations
# for 'cinchpin'
config_file = 'cinch.yml'
# Cinch layout and topology paths to be added to linch-pin PinFile
config_setup = {
'cinch': {
'topology': config_file,
'layout': config_file
}
}

# Ansible group_vars directory that will be created for later use
group_vars = os.path.join('inventories', 'group_vars')

# Dictionary of workspace directories and target filenames where we will
# put our skeletons
local_paths = {
'layouts': config_file,
'topologies': config_file,
'credentials': config_file,
group_vars: 'all'
}

# Overwrite the PinFile that linch-pin created with our configuration
pin_file = os.path.join(work_dir, 'PinFile')
with open(pin_file, 'w') as f:
yaml.dump(config_setup, f, default_flow_style=False)

# Create Ansible group_vars directory since linch-pin doesn't provide this
os.mkdir(os.path.join(work_dir, group_vars))

# Write out the skeletons and inform the user that they exist
for directory, filename in local_paths.items():
path = os.path.join(work_dir, directory, filename)
with open(path, 'w') as f:
f.write(SKEL_TEXT.format(directory, DOCS))
print('Please configure this file to use cinch: ' + path)
print('Example configurations: ' + DOCS)


def get_inventory(work_dir):
"""
Basic checks for cinch compatibility in the linch-pin working directory,
and if successful, we produce a topology file for cinch to use.
:param work_dir: The linch-pin working directory as created by 'linchpin
init' or 'cinchpin init'
:return: The topology file to pass to the 'cinch' command
"""
# Attempt to open the linch-pin PinFile
try:
with open(os.path.join(work_dir, 'PinFile'), 'r') as f:
pin_file_yaml = yaml.safe_load(f)
except IOError:
sys.exit('linch-pin PinFile not found in ' + work_dir)
# We must find a topology section named 'cinch' to determine where our
# inventory file will live
try:
cinch_topology = 'cinch'
topology = pin_file_yaml[cinch_topology]['topology']
except KeyError:
sys.exit('linch-pin PinFile must contain a topology '
'section named "{0}"'.format(cinch_topology))
# The inventory file generated by linchpin that will be used by cinch for
# configuration
try:
topology_path = os.path.join(work_dir, 'topologies', topology)
with open(topology_path) as topology_file:
topology_yaml = yaml.safe_load(topology_file)
inventory_file = topology_yaml['topology_name'] + '.inventory'
except (IOError, TypeError):
sys.exit('linch-pin topology file not found or malformed: ' +
topology_path)
return inventory_file


def command_handler(command, args):
"""
Generic function to run external programs.
Expand Down
91 changes: 30 additions & 61 deletions cinch/playbooks/install-rhel7.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,18 +4,16 @@
# on RHEL7 to work with newer Python libraries such as those required by
# 'cinch'.
#
# This playbook was tested with Ansible 1.8.4.
# This playbook was tested with Ansible 2.4.1.0

- name: install cinch into a virtualenv on RHEL7
hosts: localhost
vars:
jenkins_home: /var/lib/jenkins
venv_dir: "{{ jenkins_home }}/opt/cinch"
temp_dir: "{{ venv_dir }}/tmp"
python: "{{ venv_dir }}/bin/python"
venvs:
cinch: "{{ jenkins_home }}/opt/cinch"
linchpin: "{{ jenkins_home }}/opt/linchpin"
delete_venv: false
latest_tip: false
beaker_kerberos: true

tasks:
- name: fail if we are not running this playbook on RHEL7
Expand All @@ -38,77 +36,48 @@
msg: "directory {{ jenkins_home }} must exist for this playbook to run"
when: not jenkins_home_stat_result.stat.exists

- name: check for /var/lib/jenkins/opt/cinch directory
- name: check for existing virtualenvs
stat:
path: "{{ venv_dir }}"
path: "{{ item.value }}"
with_dict: "{{ venvs }}"
register: venv_stat_result

- name: >-
fail if pre-existing cinch installation at
/var/lib/jenkins/opt/cinch is found and it cannot be deleted
fail if pre-existing virtualenvs are found and cannot be deleted because
delete_venv is set to 'false'
fail:
msg: "directory {{ venv_dir }} exists, but 'delete_venv' setting is False"
when: venv_stat_result.stat.exists and not (delete_venv|bool)
msg: "directory {{ item.item.value }} exists, but 'delete_venv' setting is False"
with_items: "{{ venv_stat_result.results }}"
when: item.stat.exists == true and not (delete_venv|bool)
- name: >-
delete existing virtualenv directory (disabled by default, override
with DELETE_VENV Jenkins job parameter or delete_venv playbook variable)
file:
path: "{{ venv_dir }}"
path: "{{ item.value }}"
state: absent
with_dict: "{{ venvs }}"
when: (delete_venv|bool)
- name: create virtualenv
command: virtualenv --no-setuptools "{{ venv_dir }}"
args:
creates: "{{ venv_dir }}"

- name: create temp dir in root of virtualenv
file:
path: "{{ temp_dir }}"
state: directory

- name: download latest version of pip (version included with RHEL7 is too old)
get_url:
url: https://bootstrap.pypa.io/get-pip.py
dest: "{{ temp_dir }}"

- name: install pip manually by running get-pip.py script
command: "{{ python }} {{ temp_dir }}/get-pip.py"
args:
creates: "{{ venv_dir }}/lib/python2.7/site-packages/setuptools"

- name: install released versions of cinch+linch-pin using pip
- name: >-
create virtualenvs with --system-site-packages to allow for selinux
module compatibility, then upgrade setuptools and pip
pip:
name: cinch
virtualenv: "{{ venv_dir }}"
name: setuptools pip
virtualenv: "{{ item.value }}"
extra_args: -U
when: not (latest_tip|bool)
virtualenv_site_packages: true
with_dict: "{{ venvs }}"
# This pip install should be non-editable, but the pip module in Ansible
# 1.8.4. does not support that flag
- name: install latest tip of cinch+linch-pin instead of latest release from pypi
command: >-
"{{ venv_dir }}/bin/pip" install -U
https://github.com/CentOS-PaaS-SIG/linchpin/archive/develop.tar.gz
https://github.com/RedHatQE/cinch/archive/master.tar.gz
when: (latest_tip|bool)

- name: install beaker-client and python-krbV with pip to use kerberos with Beaker
- name: install version 1.0.4 of linchpin using pip
pip:
name: "{{ item }}"
virtualenv: "{{ venv_dir }}"
name: linchpin
virtualenv: "{{ venvs.linchpin }}"
extra_args: -U
with_items:
- beaker-client
- python-krbV
when: (beaker_kerberos|bool)
version: 1.0.4

## https://dmsimard.com/2016/01/08/selinux-python-virtualenv-chroot-and-ansible-dont-play-nice/
- name: >-
set up symlink in virtualenv for selinux module in system site-packages
since it's not pip installable
file:
src: /usr/lib64/python2.7/site-packages/selinux
dest: "{{ venv_dir }}/lib/python2.7/site-packages/selinux"
state: link
- name: install released version of cinch using pip
pip:
name: cinch
virtualenv: "{{ venvs.cinch }}"
extra_args: -U
Loading

0 comments on commit 9252c96

Please sign in to comment.