Merge branch 'develop'

README.md

@@ -34,6 +34,34 @@ Currently tested on two remote hosts (not vagrant).
 
 View the comments in the playbook for more info.
 
+### ansible-pull-setup.yml
+
+A playbook for setting up an [ansible-pull](http://docs.ansible.com/ansible/playbooks_intro.html#ansible-pull) scenario. 
+
+`ansible-pull` is a command that inverts the usual ansible architecture from a 'push' mode to a 'pull' mode. Instead of having a control machine that SSH's into your remote server and runs the tasks, the remote server itself will pull an ansible project from a git repo, and run it in place on the server.
+
+This can be particularly useful for cron jobs. While ansible has a [cron module](http://docs.ansible.com/ansible/cron_module.html) for scheduling cron jobs on a server, it is less obvious as to how one should write the script that performs the actual job. You might think that writing a .sh script is the best solution, but having already invested time into learning ansible, wouldn't it be great if you could define your cron jobs as ansible playbooks? `ansible-pull` allows you to do just that.
+
+`ansible-pull` can pull from any public repo you specify (and probably a private repo too, with some extra config on the server). However, it makes sense to keep all the code for a project together, so this project assumes we'll be pulling the very same repo that stores all the rest of our tasks - i.e. the one you're looking at now.
+
+#### Step 1: prepare the server and schedule the job
+
+This is a two-part process. Firstly, you need a playbook that is going to prepare your server to run the `ansible-pull` command, and schedule a cron job to run that command. This involves installing git and ansible, and using the cron module to schedule the job. This is demonstrated in playbooks/ansible-pull-setup.yml.
+
+The tricky part is choosing the correct options for the `ansible-pull` command. Here's the formula used in this project:
+
+`ansible-pull --directory {{ workdir }} --url {{ repo_url }} --checkout {{ repo_branch }} --full --force -i '127.0.0.1,' playbooks/ansible-pull-action.yml`
+
+The important flags here are:
+* `--full` - checks out the full repo. This seems to only be necessary if checking out a branch other than master.
+* `-i '127.0.0.1,'` - the inventory to use. This can either be a directory, file, or a comma-separated list of hostnames. Ideally we would not supply this flag at all, and instead default to the inventory specified in ansible.cfg. Unfortunately, doing so results in an error about a missing file, due to a symlink with a missing destination (see hosts/README.md for more info on why we have symlink here). Thus, we opt for the simplest option of specifying the host directly. Cron jobs usually take care of some maintenance on the host they run on, so the host we specify is '127.0.0.1,'. We could still delegate some jobs to other hosts with `delegate_to` if we wanted to. The comma is important here, as otherwise ansible-pull looks for a file of this name, rather than treating it as a list of hostnames.
+* --force - forces the playbook to run even if the repo couldn't be pulled this time (using the last sucessfully pulled version).
+
+#### Step 2: write the job
+
+The second part is to define the job itself as an ansible playbook. Remember that this playbook will be running from the remote host itself, and thus any tasks that should operate on the host itself should not use the external IP, but the internal IP. Hence, we set `hosts: 127.0.0.1`. In all other respects, writing the tasks is exactly the same as when running ansible from the control machine. This is demonstrated in playbooks/ansible-pull-action.yml.
+
 ## TODO
 
-* Add a POC for ansible-pull
+## DONE
+* ~~Add a POC for ansible-pull~~

Vagrantfile

@@ -61,25 +61,16 @@ Vagrant.configure(2) do |config|
   #   push.app = "YOUR_ATLAS_USERNAME/YOUR_APPLICATION_NAME"
   # end
 
-  config.vm.define "primary" do |primary|
+  config.vm.define "primary"
+  config.vm.define "secondary"
 
-    primary.vm.provision "ansible" do |ansible|
-      ansible.playbook = "playbooks/vagrant.yml"
-    end
-
-  end
-
-  config.vm.define "secondary" do |secondary|
-
-    secondary.vm.provision "ansible" do |ansible|
-      ansible.playbook = "playbooks/vagrant.yml"
-      # ansible.groups must be defined in the last vm, as it will overwrite
-      # the settings listed in all previous vms.
-      ansible.groups = {
-        'development' => ['primary', 'secondary']
-      }
-    end
+  config.vm.provision "ansible" do |ansible|
 
+    ansible.playbook = "playbooks/vagrant.yml"
+    ansible.groups = {
+      "development" => ['primary', 'secondary']
+    }
+
   end
 
 end

host_vars/127.0.0.1

@@ -0,0 +1,4 @@
+---
+# this vars file will be used by ansible-pull (where the host will always be 127.0.0.1)
+
+my_remote_user: admin

hosts/README.md

@@ -18,17 +18,19 @@ ansible.groups = {
 
 We can then reference the 'development' group in our playbooks.
 
+For this reason, we have defined '\_development\_vars' as a separate file - although we could possibly add these vars via the vagrant config too. (TODO!)
+
 ## Why are some inventory file names prefixed with underscores?
 
 This is a workaround for a problem. The contents of the hosts directory are processed in alphabetical order. This means that if you had inventory files 'a' and 'b', and in 'a' you tried to reference a group name that is defined in 'b', you would get an error like: `"Section [mysection:children] includes undefined group: mygroup"`
 
 To work around this, we make sure all groups are defined first (hence adding the underscore to make them first alphabetically).
 
-## What is the insecure_ssh inventory file?
+There is currently no need for this workaround in this project, as some extra host groups have been removed since development began - however, this workaround is left in place as a reminder.
 
-(The insecure_ssh file is based on a solution found here: http://stackoverflow.com/a/35564773/3293805)
+## What is '-o StrictHostKeyChecking=no' for?
 
-This file adds a nested inventory group that allows us to disable strict host key checking for a group of servers. Strict host key checking is how SSH determines whether the machine you're connecting to is the machine you really think it is, by checking its 'fingerprint'.
+This setting allows us to disable strict host key checking for a group of servers. Strict host key checking is how SSH determines whether the machine you're connecting to is the machine you really think it is, by checking its 'fingerprint'.
 
 But why would we want to disable this?
 
@@ -38,11 +40,11 @@ Disabling string host key checking is particularly useful for the development gr
 
 In fact, nothing nasty is happening, we've just rebuilt our machine. By disabling strict host key checking, we allow ansible to continue to connect to the machine.
 
-Disabling string host key checking can also be useful if you test your playbooks against a remote host, and you occasionally rebuild that host (e.g. on AWS or Digital Ocean). ~~For this reason, this project also disables strict host key checking for the 'staging' group.~~ (no longer - see notes below)
+Disabling string host key checking can also be useful if you test your playbooks against a remote host, and you occasionally rebuild that host (e.g. on AWS or Digital Ocean). However, this should not be done on production, or indeed any server you plan to deploy sensitive information to (which might include staging). Use at your own risk.
 
-To disable strict host key checking for a group, add the group name under `[insecure_ssh:children]` in the 'insecure_ssh' file. This should not be done on production, or indeed any server you plan to deploy sensitive information to (which might include staging). Use at your own risk.
+To disable strict host key checking for a group, add `ansible_ssh_extra_args='-o StrictHostKeyChecking=no'` to the group's vars (see hosts/\_development\_vars for an example.
 
-Note:
-* Running `vagrant provision` after rebuilding your vagrant machine does not throw up the above warning, so disabling strict host key checking is only necessary if you're trying to run ansible directly, e.g. via `ansible-playbook`. Likewise, `vagrant ssh` does not throw up the warning.
-* Disabling strict host key checking in ansible does not have an effect when SSH'ing into the machine using the ssh command - you will see the above warning. A workaround in this case is to remove the old 'fingerprint' from your ~/.ssh/known_hosts file. Look for the IP address of the machine, and remove that line from the file.
-* There is a limitation to using group vars in inventory files, e.g. `[insecure_ssh:vars]` - if you set a variable on this group, and you've set the same variable on another group, and a particular host is a member of both groups - the variable will be overwritten. As such, staging has been removed from the insecure_ssh group, so that the variable can be set with a different value via another group.
+### Notes
+* Running `vagrant provision` after rebuilding your vagrant machine does not throw up the identity warning, so disabling strict host key checking is only necessary if you're trying to run ansible directly, e.g. via `ansible-playbook`. Likewise, `vagrant ssh` does not throw up the warning.
+* Disabling strict host key checking in ansible does not have an effect when SSH'ing into the machine using the ssh command - you will see the above warning. A workaround in this case is to remove the old 'fingerprint' from your ~/.ssh/known_hosts file. Look for the IP address of the machine, and remove that line from the file. This can also be achieved with `ssh-keygen -R <ip address>`.
+* There is a limitation to using group vars in inventory files, e.g. `[development:vars]` - if you set a variable on this group, and you've set the same variable on another group, and a particular host is a member of both groups - the variable will be overwritten by the most recent setting.

playbooks/ansible-pull-action.yml

@@ -0,0 +1,11 @@
+---
+
+- name: run simple action (to be run by ansible pull on remote host)
+  hosts: 127.0.0.1
+  remote_user: "{{ my_remote_user }}"
+  become: no
+  become_method: sudo
+  become_user: root
+
+  roles:
+    - ansible_pull_action

playbooks/ansible-pull-setup.yml

@@ -0,0 +1,12 @@
+---
+
+- name: set up simple ansible pull cron job
+  hosts: primary
+  remote_user: "{{ my_remote_user }}"
+  become: no
+  become_method: sudo
+  become_user: root
+
+  roles:
+    - { role: admin_user, when: "'development' not in group_names" }
+    - ansible_pull_setup

roles/ansible_pull_action/tasks/main.yml

@@ -0,0 +1,4 @@
+---
+
+- name: print working directory
+  command: pwd

roles/ansible_pull_setup/tasks/main.yml

@@ -0,0 +1,48 @@
+---
+
+- name: add ansible ppa
+  apt_repository:
+    repo: ppa:ansible/ansible
+  become: yes
+
+- name: install git and ansible
+  apt:
+    name: "{{ item }}"
+    update_cache: yes
+  become: yes
+  with_items:
+    - git
+    - ansible
+
+- name: create local dir to work from
+  file:
+    path: "{{ workdir }}"
+    state: directory
+    owner: "{{ my_remote_user }}"
+    group: "{{ my_remote_user }}"
+    mode: 0751
+  become: yes
+
+- name: create initial empty log file
+  file:
+    state: touch
+    path: "{{ logfile }}"
+    owner: "{{ my_remote_user }}"
+    group: "{{ my_remote_user }}"
+    mode: 0644
+  become: yes
+
+- name: Create crontab entry to ansible-pull
+  cron:
+    name: ansible-pull action
+    user: "{{ my_remote_user }}"
+    job: "ansible-pull --directory {{ workdir }} --url {{ repo_url }} --checkout {{ repo_branch }} --full --force -i '127.0.0.1,' playbooks/ansible-pull-action.yml >> {{ logfile }} 2>&1"
+
+- name: Create logrotate entry for ansible-pull.log
+  template:
+    src: etc_logrotate.d_ansible-pull.j2
+    dest: /etc/logrotate.d/ansible-pull
+    owner: "{{ my_remote_user }}"
+    group: "{{ my_remote_user }}"
+    mode: 0644
+  become: yes

roles/ansible_pull_setup/templates/etc_logrotate.d_ansible-pull.j2

@@ -0,0 +1,7 @@
+{{ logfile }} {
+  rotate 7
+  daily
+  compress
+  missingok
+  notifempty
+}

roles/ansible_pull_setup/vars/main.yml

@@ -0,0 +1,9 @@
+# File that ansible will use for logs
+logfile: /var/log/ansible-pull.log
+
+# Directory to where repository will be cloned
+workdir: /var/lib/ansible/local
+
+repo_url: git://github.com/discopatrick/ansible-pocs.git
+
+repo_branch: feature/ansible_pull