Inspec Tests

Overview

• Inspec tests are used to verify whether a node convergence meets the desired configuration.
• Inspec tests are ran over ssh using Inspec CLI. So it does require ssh_user, host name, and the private key.

Installing Inspec

Inspec isn't currently installed as part of mu. Installation instructions here

Inspec Vocabs

• Inspec Profile : A Inspec profile is a Test Suite. An Inspec profile can have multiple tests(controls). It is very much like a cookbook
• Controls : A control is responsible for verifying a recipe. It contains tests to test each individual resource of a recipe.

Pre-requistes of writing inspec tests for your BOK

1. Each BOK should have its own Inspec profile to run
2. Each BOK appname must be unique. This helps locating the correct deploy_id in the deployments directory.
3. Each servers run_list MUST have recipe[demo::store_attr] as the LAST recipe to run.
   • Why? It's a limitation from Inspec. Inspec cannot access node attributes directly from cookbooks. So the store_attr.rb is a recipe that dumps node attributes on the node in a temporary file /tmp/chef_node.json. Now to access node attributes from the node itself, add this in your test node =json('/tmp/chef_node.json').params.
4. Each servers recipe name from run_list should become a corresponding inspec control in that inspec profile. You will see a step-by-step walkthrough in the next section.
5. Make sure you have performed mu-deploy after adding the store_attr recipe to BOK. the

Creating Inspec tests ( Assuming you have a working BOK)

1. Ensure you have performed a mu-deploy after adding following recipe in run_list at the end recipe[demo::store_attr] of each servers run_list.
2. Navigate to test directory /opt/mu/lib/test
3. Generate Inspec Profile inspec init profile <your_profile_name>
4. Create a new control.rb file in /opt/mu/lib/test/<your_profile_name>/controls/. Name of the file can be anything.
   • Preferred way;
     • Create 1 control.rb file and add all controls in that one file.
     • So if your BOK has 2 servers with run_lists as recipe[demo::flask] for server_1 and recipe[demo::apache2] for server_2
     • Then, in your inspec profile, create a control called 'all-in-one.rb'
     • Finally add control blocks that match the recipe names.
     • Sample control.rb file in a inspec_profile;

     node =json('/tmp/chef_node.json').params
     control 'flask' do
         title 'This will test Flask Recipe from Demo Cookbook'
         # inspec-resources to test flask recipe
     end
     
     control 'apache2' do
         title 'This will test Apache2 Recipe from Demo Cookbook'
         # inspec-resources to test flask recipe
     end
     

     • Notice how control names match the recipe names. This is a MUST and it is case-sensitive
     • Inspec Resource Docs : https://www.inspec.io/docs/reference/resources/
5. Add Mu-Base Tests in ALL your inspec profiles. ( MUST ).
6. Steps to include base tests in your inspec_profile;
   1. Navigate in your Inspec Profile.
   2. Edit the inspec.yml file and add the following (YAML is indent space sensitive!). Note that the path to mu-tools-test is relative to the test you are running, not the directory you are running the test from

      depends:
      - name: mu-tools-test
        path: ../mu-tools-test
      

   3. Path to the inspec root /opt/mu/lib/test and re-build inspec.lock file, insesrting your test directory appropriately, e.g.;: inspec vendor <your_profile_name> --overwrite
   4. Edit your control.rb file and add the following to the top and outside an existing control block. include_controls 'mu-tools-test'
7. Your inspec control.rb file should look something similar to this;

       node =json('/tmp/chef_node.json').params
       
       include_controls 'mu-tools-test'
       
       control 'flask' do
           title 'This will test Flask Recipe from Demo Cookbook'
           # inspec-resources to test flask recipe
       end
   
       control 'apache2' do
           title 'This will test Apache2 Recipe from Demo Cookbook'
           # inspec-resources to test flask recipe
       end
   

8. To run this profile, simply run python exec_inspec -p <your_profile> -b <bok_name.yaml> -l

Running an Inspec profile (Make sure to be inside the test directory)

• Testing locally after a mu-deploy, but only possible if clean up was NOT performed; python exec_inspec.py -p <inspec_profile_name> -b <bok_file_name.yaml> -l
  • -p => Inspec Profile Name
  • -b => BOK file name, including the extension
  • -l => local mode
• You should be using the -l flag all the time, the only time when its not used is when inspec is running on a CI Server Job.

Do we have any base tests?

• YES! When we perform a mu-deploy or when we create a new mu-master we do run few base cookbooks.
• We have 2 inspec_profiles handling this;
  • New mu-master tests are inside /opt/mu/lib/test/mu-master-test
  • mu-deploy base tests are inside /opt/mu/lib/test/mu-tools-test ** This inspec_profile is automatically included when running the exec_inspec.py script to test mu-deploy

How Inspec Works with a MU Deployment

• When a Mu deploy is done, a deployment directory is created inside /opt/mu/var/deployments/<deploy_id>. The deploy_id directory contains metadata which includes; ssh_key file name, each server's information, and more. We have utilized each deploy_id's deployment.json and basket_of_kittens.json to parse out values we need for inspec to run tests. Inside /opt/mu/lib/test/ we have a exec_inspec.py script that handles parsing out deployment metadata + running inspec tests as well. The exec_inspec.py does take 2 positional parameters. One is the inspec_profile and other one is the corresponding bok.yaml file (Just the file name). Then inspec runs tests for each server by mapping each servers run_list recipe name to its corresponding control name inside the inspec_profile. So if you have A BOK with multiple recipes, then there should be A inspec_profile with multiple controls. For example, you might have 1 BOK with 2 servers whose run_list for server_a is recipe[demo::flask] and for server_b recipe[demo::rails]. Then there should be A inspec_profile named 'demo-test-profile' or 'flask-rails-test-profile' or any other name. However what really matters is that there MUST be a control name 'flask' and a control name 'rails'( it is case sensitive) in that inspec_profile. Now why this design? Because a BOK can contain multiple servers with same or different run_lists. In-order to make sure Inspec runs the right tests on the right server, we must follow this convention. But how is Inspec making sure controls are running on right server??? Well this is where deployment metadata comes in handy and it has the run_list of each individual server, so the exec_inspec.py is parsing out the recipe_names from each servers run_list and using that name as a control name to run.