Two-Point UwU

Chef (Client) 14

Chef 12 reached EOL 4/20/2018. We have been working hard to bring a jump to Chef 14, the latest and greatest (for now). At the same time we want to make sure we develop processes to allow us to keep up with Chefs new yearly release cycle. So Mu 2 contains the following in terms of Chef Client upgrades.

• Muby has been upgraded to 2.5 (See below).
• Chef 14 has been rolled out for both the Mu master, and any nodes that the master manages.
• All of Mu's internal cookbooks have been updated to support Chef 14+.
• We have resolved all of the FoodCritic warnings, and have setup the CI/CD pipeline to fail if new issues are introduced. This will allow us to continually keep our cookbooks in line with Chef best practices, and to identify deprecations.
• The Mu Demo Platform contains a sample GitLab CI/CD pipeline that will allow you to run the same tests against your Platform Repos.

These new updates and processes will allow future updates of the Chef Client to be a lot smoother and faster.

Berkshelf enhancements

Remember the days when you had to do something like this to get mu to work with your platform repos?

# --- AUTO-INCLUDE STANDARD MU COOKBOOKS --- #
# --- DO NOT DELETE THESE COMMENTS UNLESS YOU HAVE A PLAN --- #
addtl_cookbooks = {}
if ENV.include? "MU_COOKBOOK_ROOT"
  cookbookPath = "#{ENV['MU_COOKBOOK_ROOT']}/cookbooks"
  siteCookbookPath = "#{ENV['MU_COOKBOOK_ROOT']}/site_cookbooks"
  if ENV.include? "MU_DEPRESOLVE"
    ["cookbooks", "site_cookbooks"].each { |dir|
      next if !Dir.exists?(ENV['MU_COOKBOOK_ROOT']+"/"+dir)
      Dir.foreach(ENV['MU_COOKBOOK_ROOT']+"/"+dir).each { |cb|
        next if cb == "." or cb == ".."
        path = ENV['MU_COOKBOOK_ROOT']+"/"+dir+"/"+cb
        next if !File.exists?(path+"/metadata.rb") and !File.exists?(path+"/metadata.json")
        addtl_cookbooks[cb] = { "path" => path }
      }
    }
    # now to smoke some rocks
    if File.exists?("#{ENV['MU_COOKBOOK_ROOT']}/Berksfile.lock")
      in_deps = false
      File.open("#{ENV['MU_COOKBOOK_ROOT']}/Berksfile.lock").each { |line|
        if in_deps
          if line.match(/^  ([^\s]+)\s+\((.*?)\)/)
            cb = Regexp.last_match[1]
            vers = Regexp.last_match[2]
            if vers
              addtl_cookbooks[cb] ||= {}
              addtl_cookbooks[cb]["version"] = vers
            end
          elsif line.match(/^\s*$/)
            break
          end
        end
        in_deps = true if line.match(/^DEPENDENCIES$/)
      }
    end
  end
end

Thanks to Berkshelf, and Chef updates, those days are behind us. You can now replace all of that rigmarole with something along the lines of this:

source "https://supermarket.chef.io"
source :chef_server if File.file?('/etc/chef/client.pem')
source chef_repo: "cookbooks/"

Full Berksfile Example

This will allow Berkshelf to check the chef server to see if the mu cookbooks already exist without us having to tie the platform repos and the mu chef repo Berksfiles together.

This allows you to manage your supermarket dependencies completely separate from the mu dependencies. Enabling multiple versions of a single cookbook without quickly falling into dependency hell.

Along with these enhancements, we have cleaned all of the Berksfile dependencies, and greatly simplified mu's root Berksfile. We have moved the dependency declarations into individual cookbooks this allows us to simply declare our internal cookbooks in our root berksfile.

One final note on this, since we now have proper dependency versioning in Berksfile and metadata.rb, we no longer need the Berskfile.lock, so those have been killed with fire.

CI/CD Updates and Automated Smoke Tests

We have continued to improve our CI/CD pipeline, and have enabled more automated tests. Most notabily we are now able to build a fresh version of cloud-mu.gem and to use that to run automated BoK parser tests that we have written. We are working to increase coverage of these tests, but we have a pretty good set thusfar. Currently we have a simple test and a complicated test. The simple test tests that mu is able to generate sensiable defaults on its own, and the complicated test ensures that we are able to validate and support all of the functionality that we say we can. This will enable us to get faster feedback on changes to the parser, and hopefully reduce breaking changes on platform repos.

As we continue down the CI/CD path we will be looking to useing cloud-mu.gem more to enable rich CI/CD pipelines.

IAM artifacts now first-class resources

https://cloudamatic.gitlab.io/mu/MU/Config/BasketofKittens/users.html
https://cloudamatic.gitlab.io/mu/MU/Config/BasketofKittens/groups.html
https://cloudamatic.gitlab.io/mu/MU/Config/BasketofKittens/roles.html

Role, User, and Group resources can now be declared and referenced like anything else. This is implemented in both AWS and Google. Mu features which previously created one-off roles for their own purposes now properly inject a Role object as a dependency where applicable and address it like any other sibling resource.

In AWS, the Role functionality also includes language for declaring inline policies, and even for defining free-floating policies without a containing role, or attached to a User or Group directly when supported. See: https://cloudamatic.gitlab.io/mu/MU/Config/BasketofKittens/roles/policies.html

Google Cloud does not truly have native user management, but rather relies on external directories- typically GMail accounts or an enterprise GSuite directory. We have implemented the ability to bind these Google groups and users, but creation and destruction are not currently possible outside of corporate GSuite environments. Additionally, the custom role functionality in GCP is alpha/beta quality, and so we've skipped fully supporting this functionality until its interface stabilizes.

The following BoK exercises these three resource types in an AWS deploy, and demonstrates role-based policies as well as inline policies:

---
appname: iamtests
roles:
- name: somerole
  can_assume:
  - entity_id: ec2.amazonaws.com
    entity_type: service
  import:
  - AmazonLexReadOnly
  - arn:aws:iam::aws:policy/AmazonRDSFullAccess
  policies:
  - name: a_basic_policy
    permissions:
    - ec2:CreateSnapshot
    targets:
    - identifier: thing1
      type: user
  iam_policies:
  - CloudWatch_Logs:
      Version: '2012-10-17'
      Statement:
      - Sid: Stmt1406256819000
        Effect: Allow
        Action:
        - logs:CreateLogGroup
        - logs:CreateLogStream
        - logs:DeleteRetentionPolicy
        - logs:DescribeLogGroups
        - logs:DescribeLogStreams
        - logs:DescribeMetricFilters
        - logs:GetLogEvents
        - logs:PutLogEvents
        - logs:PutMetricFilter
        - logs:PutRetentionPolicy
        - logs:TestMetricFilter
        Resource:
        - "*"
  - Snapshots_and_Tags:
      Version: '2012-10-17'
      Statement:
      - Sid: Stmt1385828567000
        Effect: Allow
        Action:
        - ec2:CreateSnapshot
        - ec2:DeleteSnapshot
        - ec2:DescribeSnapshotAttribute
        - ec2:DescribeSnapshots
        - ec2:DescribeTags
        - ec2:DescribeInstanceAttribute
        - ec2:DescribeInstanceStatus
        - ec2:DescribeInstances
        - ec2:CreateTags
        - ec2:DescribeVolumes
        - ec2:DescribeVolumeAttribute
        - ec2:DescribeVolumeStatus
        - ec2:ModifySnapshotAttribute
        Resource: "*"
- name: somepolicies
  bare_policies: true
  iam_policies:
  - AllowCertListing:
      Version: '2012-10-17'
      Statement:
      - Effect: Allow
        Action: acm:ListCertificates
        Resource: "*"
users:
- name: thing1
  cloud: AWS
  tags:
  - key: thisisatag
    value: thisisatagvalue
  groups:
  - developers
  - impliedgroup
  - declaredawsgroup
  create_console_password: true
  create_api_key: true
  iam_policies:
  - Thing1CertListing:
      Version: '2012-10-17'
      Statement:
      - Effect: Allow
        Action: acm:ListCertificates
        Resource: "*"
groups:
- name: admin
  cloud: AWS
  members:
  - thing1
- name: declaredgroup
  cloud: AWS
  purge_extra_members: true
  members:
  - [email protected]
  iam_policies:
  - S3_List_Get_Objects:
      Version: '2012-10-17'
      Statement:
      - Effect: Allow
        Action:
        - s3:GetObject
        - s3:ListBucket
        - s3:ListAllMyBuckets
        Resource:
        - "*"

Cross-account resource management

A single Mu Master is now able to manage resources in multiple accounts, given appropriate credentials in /opt/mu/etc/mu.yaml. In the following example, three AWS accounts have been configured.

aws:
  egtdev:
    account_number: '616552976502'
    region: us-east-1
    log_bucket_name: stange-mu-dev
    default: true
    name: egtdev
  egtprod:
    access_key: [redacted]
    access_secret: [redacted]
    log_bucket_name: stange-mu-prod
    name: egtprod
  egtmaster:
    credentials_file: "/root/root_aws_egt_labs"
    log_bucket_name: stange-mu-dev-in-master
    name: egtmaster

This example covers several methods of specifying credentials. In egtdev, none are specified- the account is leveraging the IAM Instance Profile of the Mu Master itself, which happens to grant privileges over the account in which the master resides. The egtprod example specifies an access_key and access_secret inline directly in the config file, while egtmaster references an awscli-style credentials file.

The mu-configure menu system has been updated to facilitate the addition of credentials.

Targeting alternate accounts is done through the credentials parameter in applicable resource types. The following example creates two VPCs in different accounts, and then, just for flourish, peers them with one another as well as with the Mu Master's host VPC.

---
appname: hotcrossbuns
vpcs:
- name: "vpc"
  credentials: egtdev
  ip_block: 10.120.0.0/16
  create_nat_gateway: true
  peers:
  - vpc:
      vpc_name: vpc2
  - vpc:
      vpc_id: <%= MU.myVPC %>
- name: "vpc2"
  credentials: egtprod
  ip_block: 10.130.0.0/16
  create_nat_gateway: true
  peers:
  - vpc:
      vpc_id: <%= MU.myVPC %>
  - vpc:
      vpc_name: vpc

Function and Endpoint

https://cloudamatic.gitlab.io/mu/MU/Config/BasketofKittens/functions.html
https://cloudamatic.gitlab.io/mu/MU/Config/BasketofKittens/endpoints.html
https://cloudamatic.gitlab.io/mu/MU/Config/BasketofKittens/endpoints/methods/integrate_with.html

A continuation of existing work on AWS Lambda, this is now even more feature complete. This release includes a number of bugfixes and integration with API Gateway, which we expose as a resource called Endpoint. The following example deploys a Lambda function from a bundled zip file, grants it permission to interact with DynamoDB, and integrates it with an API Gateway called test-api.

---
appname: myapp
functions:
- name: mylambda
  credentials: egtdev
  runtime: python3.6
  memory: 256
  handler: lambda_function.handler
  timeout: 15
  code:
    zip_file: "/home/me/myapp.zip"
  vpc:
    vpc_name: vpc
    subnet_pref: all_private
    deploy_id: HOTCROSSBUNS-DEV-2019012112-LH
  triggers:
  - name: test-api
    service: apigateway
endpoints:
- name: test-api
  methods:
  - path: do_the_thing
    type: GET
    cors: true
    integrate_with:
      name: mylambda
      type: function
  vpc:
    vpc_name: vpc
    deploy_id: HOTCROSSBUNS-DEV-2019012112-LH

Habitat and Folder

https://cloudamatic.gitlab.io/mu/MU/Config/BasketofKittens/folders.html
https://cloudamatic.gitlab.io/mu/MU/Config/BasketofKittens/habitats.html

Google Cloud folder and project resources are now declarable as Mu resources. This functionality is only available for accounts that are members of organizations- see https://cloud.google.com/resource-manager/docs/creating-managing-organization for details.

The following BoK will create a folder named folder1 at the root of the organization of which our service credentials are a member, then a sub-folder named folder2 underneath folder1, and finally a project at the bottom of the tree.

---
appname: habitattest
folders:
- name: folder1
  cloud: Google
- name: folder2
  cloud: Google
  parent:
    name: folder1
habitats:
- name: project
  cloud: Google
  parent:
    name: folder2

This functionality is not currently recommended for use in AWS. We have partially implemented the Habitat resource, however automated deletion is not possible at this time.

Notifier

https://cloudamatic.gitlab.io/mu/MU/Config/BasketofKittens/notifiers.html

AWS SNS support has existed in Mu for some time, but was not fleshed out as a proper first-class resource- merely a few class methods used to support things like the Alarm resource. It has now been promoted.

Example usage:

---
appname: foo
notifiers:
- name: notifier
  subscriptions:
  - endpoint: [email protected]
    type: email-json
  - endpoint: 703-555-5555

Bucket

https://cloudamatic.gitlab.io/mu/MU/Config/BasketofKittens/buckets.html

Amazon S3 and Google Cloud Storage buckets can now be created as first-class resources.

---
appname: backups
buckets:
- name: buckit
  cloud: AWS
  acl: public-read
  versioning: true
  storage_class: REDUCED_REDUNDANCY
  web: true
  web_error_object: doh.html
  web_index_object: hi.html
- name: buckthistoo
  cloud: Google
  versioning: false
  storage_class: STANDARD
  policies:
  - name: CloudFrontAllow
    permissions:
    - "s3:GetObject"
    targets:
    - type: buckthistoo
      identifier: buckthistoo
      path: "/*"
    conditions:
    - comparison: IpAddress
      variable: "aws:SourceIp"
      values:
      - "158.71.132.242"
- name: gcpbucket
  cloud: Google
  policies:
  - name: testpermissions
    grant_to:
    - identifier: [email protected]
    targets:
    - type: gcpbucket
      identifier: gcpbucket

NoSQLDB

https://cloudamatic.gitlab.io/mu/MU/Config/NoSQLDB.html

Basic DynamoDB table and schema creation support. Note that our parser validation does not attempt to inspect the minutiae of secondary indexes, which are nuanced enough that we rely on the cloud provider to accept or reject incorrect schemas at creation-time, so mu-deploy -d will not necessarily catch a wrong-for-Dynamo schema in all cases.

---
appname: dynamo
nosqldbs:
- name: humm
  stream: NEW_IMAGE
  read_capacity: 50
  write_capacity: 50
  attributes:
  - name: name
    type: S
    primary_partition: true
  - name: date
    type: S
    primary_sort: true
  - name: things
    type: N
  - name: blob
    type: B
  secondary_indexes:
  - index_name: whee
    type: global
    read_capacity: 51
    key_schema:
    - attribute: things
      type: HASH
    - attribute: name
      type: RANGE
    projection:
      type: INCLUDE
      non_key_attributes:
      - blob
      - date
  - index_name: hurp
    type: local
    key_schema:
    - attribute: name
      type: HASH
    - attribute: blob
      type: RANGE

cloud-mu gem

We can now bundle a large portion of a Mu Master's ecosystem as a Ruby gem. This is primarily of use in leveraging our cloud APIs as a lightweight alternative to other tools, which we're actively doing for certain clients.

This gem is published to rubygems.org as cloud-mu, meaning you can just list it in your Gemfile or do a gem install cloud-mu, given a minimum version of Ruby 2.4.

Our internal Gemfile imports the gemspec file for this gem to import many of its dependencies, and our GitLab integration uses this gem to perform tests and build documentation inside of Docker containers.

Miscellaneous

• RDS implementation now supports cluster_mode: serverless to expose the Aurora mode of the same name. See also: https://aws.amazon.com/rds/aurora/serverless/
• Improvements for cloud auto-detection. In both full installs and in gem deployments, the Mu library should be quicker at identifying when it is hosted in a supported cloud provider and self-configure accordingly.
• Lots of minor quality-of-life improvements to our YARD documentation generation for Baskets of Kittens
• Mu Master's local web server slightly smarter about proxying on https, so that you get things you want if you're specific enough, or the Chef Server if not
• We now have proper spec and build script for Mu's bundled Ruby RPM (muby). See /opt/mu/lib/extras/ruby_rpm/ if you're curious.
• Cloud layer implementations can now add to the list of required methods for their child resource implementations. In AWS, for example, we now require each resource to implement the #arn object method.
• This release includes (empty) stubs for Azure support. Real work is ongoing in the Azure_you_want_azure branch.
• The groom flag now exists, and can be set to false to skip grooming of Server and ServerPool resources entirely. This is of use for using Mu as a pure provisioning tool a la Terraform, or in scenarios where we use the cloud-mu gem, which does not have a bundled Chef server.
• Bugfix for a long-standing, rare bug in MU::Config that would cause validation to be skipped on on-the-fly generated resources.
• The Mu Master motd and index.html links to Mu documentation should now link to the public documentation at https://cloudamatic.gitlab.io/mu/ when on the master branch or detached to a tagged release.
• Fresh CentOS 6 and CentOS 7 base AMIs have been generated for AWS.
• Berksfile.lock is now no more
• Documentation is automatically built and uploaded to https://cloudamatic.gitlab.io/mu/