Skip to content

Commit

Permalink
Merge of Chef Stack (https://github.com/ncerny/chef_stack) resource a…
Browse files Browse the repository at this point in the history 
…nd helpers into Chef Ingredient (https://github.com/chef-cookbooks/chef-ingredient)

Specific files were selected in their entirety versus merging the chef_stack repo to keep the commits clean and avoid conflicts.

Background

Chef Stack is a series of primitives for installing, configuring, and managing over time all of Chef's products.  Its intention is to be very prescriptive about how Chef's software stack is configured and maintained, while being very non-prescriptive about how its deployed, or where its configuration comes from.
It was born out of a Customer need to focus on the complexities of their environment, without having to focus on Chef's core software.  Moving the management of Chef's software to easy-to-consume primitives allows the Customer to focus on the things that are important to them -- their environment and their workflow.

Contributions

Special Thanks to Edmund DeSmet <[email protected]> who heavily influenced the design of Chef Stack.

andy-dufour <[email protected]> (10):
      Fixing push jobs client config
      Fix Issue #5: Run delivery token before delivery api so output is only JSON
      Version bump
      Adding non-standard platform support
      Adding platform options to client resource
      Adding readme
      Fixing formating
      Fixing spacing
      Adding descriptions to custom resources
      Fixing some formatting
Brandon Raabe <[email protected]> (3):
      Fix tables in the readme
      Remove default value of `nil`
      Fix regex to determine if source is a URI
Jeremy J. Miller <[email protected]> (5):
      adding package_source to automate resource
      changing property_is_set call
      adding second guard for resource
      fixing 'source is required' error
      Merge pull request #10 from itmustbejj/wip
Josh Hudson <[email protected]> (20):
      Merge pull request #6 from andy-dufour/ad/v2-runner-token-fix
      Move write_vault to helpers
      Update write_vault to reflect renamed cookbook
      Add gather_secrets action for chef_backend resource.
      Use node['chef_stack']['admin'] defaulted to workflow instead of hardcoded string in write_vault function
      Implement read_vault helper function
      Change the location of the chef vault encryption key
      Use an attribute to determine the client key
      Default chef_backend_secrets to nil
      Don't have non-bootstrap try to join-cluster if they already have.
      Allow :chef_config_file property for wf_builder resource instead of hardcoding /etc/chef/client.rb
      Don't explicitly pass in node name when tagging runner
      Use Chef::Config['node_name'] instead of node['fqdn'] in wf_builder resource
      Reverting a bunch of changes we didn't need to make
      Don't write chef-backend-secrets.json if it's not set (bootstrap)
      Revert partial implementation of package_source property for resources for another feature branch.
      Fix chef_backend_secrets property for use case of bootstrap backend generating it's own secrets.
      Lint and syntax fixes
      Merge pull request #11 from ncerny/fix_source
      Merge pull request #15 from andy-dufour/ad/multi_platform
Nathan Cerny <[email protected]> (55):
      Add generated delivery configuration
      Add generated cookbook content
      Add generated delivery build cookbook
      Merge branch 'add-delivery-configuration'
      Initial build out of cookbook for managing Chef-Client, Chef-Server, and Chef-Automate.
      Update v1 build node search string to delivery-build-node to better match legacy workflows.
      Supermarket recipe and tweaks for performance and workflow.
      Move client config to shared recipe, and update all components to latest.
      cleanup and switch all components to latest
      Fix issues with spinning up fresh cluster.
      Create directory and place config before reconfigure of automate so nginx picks up config.
      Bug fixes.
      Add initial chef-backend resource and recipe
      Fix bug with key on chef_user resource.
      Same bug in chef_org
      Be consistent in declaration of new_resource in chef_user
      Pull client.rb template from this cookbook.
      Rename 'key' to 'key_path' to be more explicit.
      Lay down validation pem in install, if it's set.
      property error - test if new_resource works in this context.
      Lazy-load contents of delivery-cmd for v2 builders.
      Ensure authorized_keys file exists for v2 job runner.
      Enterprise should be configurable in workflow_builder.
      Merge pull request #1 from andy-dufour/ad/push_jobs_config
      Rename Cookbook to chef_stack
      Merge pull request #2 from ncerny/nc/rename-cookbook
      Move ensurekv to helpers to avoid duplication.
      Testing should be done with chef-services
      Merge pull request #3 from ncerny/nc/cleanup
      Merge pull request #4 from ncerny/lauck/fix_cookbook_name
      Merge pull request #16 from andy-dufour/ad/add_readme
      Merge pull request #18 from brandocorp-cookbooks/chef-backend-secrets-deprecation
      Merge pull request #17 from brandocorp-cookbooks/readme-fixes
      Version Bump for PRs - fix deprecation on chef_backend, and fix README tables.
      Switch chef_backend to use peers instead of bootstrap.
      Change chef-backend-secrets to default to empty string.
      suppress error messages in chef-backend guard
      Fix lint failures.
      Remove old delivery configuration.
      Merge pull request #23 from ncerny/nc/remove-delivery
      Remove old .chef directory.
      Merge pull request #24 from ncerny/nc/remove-chef
      Merge branch 'master' into nc/fix-lints
      Default chef_file source to chef_file if source property is not set.
      Fix incorrect log_location in wf_builder.
      Version bump - 0.8.0 - Clear up ambiguous versions
      Merge pull request #26 from ncerny/nc/fix-source-req-error
      Merge branch 'master' into nc/wf-log-bug
      Merge pull request #25 from ncerny/nc/wf-log-bug
      Merge branch 'master' into nc/fix-lints
      Merge pull request #21 from ncerny/nc/fix-lints
      Merge branch 'master' into nc/allow-backend-peers
      Merge pull request #19 from ncerny/nc/allow-backend-peers
      Merge pull request #31 from brandocorp-cookbooks/chef-file-regex
      Release 0.8.1
Stephen Lauck <[email protected]> (1):
      Rename cookbook chef to chef_stack in client resource

Signed-off-by: Patrick Wright <[email protected]>

Signed-off-by: Patrick Wright <[email protected]>
  • Loading branch information
Patrick Wright committed Apr 27, 2017
1 parent 5543cf5 commit 928ad14
Show file tree
Hide file tree
Showing 16 changed files with 1,265 additions and 0 deletions.
6 changes: 6 additions & 0 deletions .kitchen.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -78,3 +78,9 @@ suites:
<% end %>
- recipe[test]
- recipe[test::inspec]

- name: chef_server
includes: [ 'ubuntu-16.04' ]
run_list:
- recipe[test]
- recipe[test::chef_server]
142 changes: 142 additions & 0 deletions CHEFSTACK.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,142 @@
# chef_stack

Chef stack is a library cookbook that provides custom resources to build and manage your Chef infrastructure.

An accompanying project, [Chef-Services](https//github.com/stephenlauck/chef-services) exists as an example implementation of Chef Stack.

## Custom Resources

Below are the custom resources provided by this cookbook.

### General Properties

These properties exist for all resources

| Name | Type | Default Value | Description |
|---|---|---|---|
| name | String | N/A | A name for the resource |
| channel | Symbol | stable | The channel from our package repository to install. Most of the time you want stable. |
| version | [String, Symbol] | latest | The version of Automate you want to install |
| config | String | N/A | The configuration that will be written to the appropriate configuration file for the product. |
| accept_license | [TrueClass, FalseClass] | false | Do you accept Chef's license agreements. |
| platform | String | Auto-detected | Use only if you need to over-ride the default platform. |
| platform_version | String | Auto-detected | Use only if you need to over-ride the default platform. |


### chef_automate

Installs Chef Automate.

#### Properties
| Name | Type | Default Value | Description |
|---|---|---|---|
| enterprise | [String, Array] | chef | The Enterprise to create in Automate|
| license | String | N/A | Your license file | we recommend using the chef_file resource |
| chef_user | String | workflow | The user you will connect to the Chef server as |
| chef_user_pem | String | N/A | The private key of the above Chef user |
| validation_pem | String | N/A | The validator key of the Chef org we're connecting to |
| builder_pem | String | N/A | The private key of the build nodes |

### chef_backend

#### Properties

| Name | Type | Default Value | Description |
|---|---|---|---|
| bootstrap_node | String | N/A | The node we'll bootstrap secrets with. |
| publish_address | String | node['ipaddress'] | The address you want Chef-Backend to listen on. |
| chef_backend_secrets | String | nil | A location where your secrets are | we recommend using the chef_file resource. |

### chef_org

#### Properties


| Name | Type | Default Value | Description |
|---|---|---|---|
| org | String | N/A | The short name of the org. |
| org_full_name | String | node['ipaddress'] | The full name of the org you want to create. |
| admins | Array | N/A | An array of admins for the org. |
| users | Array | [] | An array of users for the org. |
| remove_users | Array | [] | An array of users to remove from the org. |
| key_path | String | N/A | Where to store the validator key that is created with the org. |

### chef_user

| Name | Type | Default Value | Description |
|---|---|---|---|
| username | String | N/A | The username of the user. |
| first_name | String | N/A | The first name of the user. |
| last_name | Array | N/A | The last name of the user. |
| email | Array | [] | N/A | The users e-mail. |
| password | Array | [] | The users password. |
| key_path | String | N/A | Where to store the users private key that is created with the user. |
| serveradmin | [TrueClass, FalseClass] | F | Is the user a serveradmin? |

### chef_client
| Name | Type | Default Value | Description |
|---|---|---|---|
| node_name | String | true | The name of the node. |
| version | [String, Symbol] | latest | The version of chef-client to install. |
| chefdk | [TrueClass, FalseClass] | false | Do you want to install chefdk? |
| chef_server_url | [String, Symbol] | local | What is hte Chef server URL to connect to. |
| ssl_verify | [TrueClass, FalseClass] | true | Validate ssl certificates? |
| log_location | String | 'STDOUT' | Where to log. |
| log_level | Symbol | auto | Log level. |
| config | String | | Any configuration for client.rb. |
| run_list | Array | | The clients runlist. |
| environment | String | | Which Chef Environment the client belongs to. |
| validation_pem | String | | The validation pem to validate with. |
| validation_client_name | String | | The validation client name. |
| tags | [String, Array] | '' | Any tags for the node. |
| interval | Integer | 1800 | The interval to run chef-client on. |
| splay | Integer | 1800 | The randomization to add to the interval. |
| data_collector_token | String | '93a49a4f2482c64126f7b6015e6b0f30284287ee4054ff8807fb63d9cbd1c506' | The data collector token to talk to Visibility. |
| data_collector_url | String | | The Visibility URL to send data. |

### chef_file


| Name | Type | Default Value | Description |
|---|---|---|---|
| filename | String | | The name of the resource. |
| source | String | | The source of the file. |
| user | String | default 'root' | The owner of the file. |
| group | String | default 'root' | The group owner of the file. |
| mode | String | default '0600' | The mode for the file. |

### chef_server

| Name | Type | Default Value | Description |
|---|---|---|---|
| addons | Hash | | A set of addons to install with the Chef Server. |
| data_collector_token | String | default '93a49a4f2482c64126f7b6015e6b0f30284287ee4054ff8807fb63d9cbd1c506' | The data collector token to authenticate with Chef Visiblity. |
| data_collector_url | String | | The URL to connect to Visibility. |

### chef_supermarket

| Name | Type | Default Value | Description |
|---|---|---|---|
| chef_server_url | String | Chef::Config['chef_server_url'] | The Chef server's URL. |
| chef_oauth2_app_id | String | | The oauth2 app id from the Chef server. |
| chef_oauth2_secret | String | | The oauth2 secret from the Chef server. |
| chef_oauth2_verify_ssl | [TrueClass, FalseClass] | true | Whether to validate SSL certificates. |

### workflow_builder

| Name | Type | Default Value | Description |
|---|---|---|---|
| pj_version | [String, Symbol] | :latest | The version of Push-Jobs to install. |
| chef_user | String | 'workflow' | The Chef user to authenticate with the Chef Server. |
| chef_user_pem | String | | The private key of the Chef user to authenticate with the Chef Server. |
| builder_pem | String | | The builder users private key to communicate with Chef Automate. |
| chef_fqdn | String | URI.parse(Chef::Config['chef_server_url']).host | The FQDN of the Chef server. |
| automate_fqdn | String | | | The FQDN of the automate server. |
| supermarket_fqdn | String | | The FQDN of the Supermarket server. |
| job_dispatch_version | String | 'v2' | Which job dispatch version to use. V1 is push-jobs, V2 is SSH runners. |
| automate_user | String | 'admin' | What is the Automate user we're connecting to Automate as. |
| automate_password | String | | The password for the user above. |
| automate_enterprise | String | 'chef' | The Enterprise to connect to. |
| chef_config_path | String | '/etc/chef/client.rb' | The config path for chef-client. |

## Contributers
24 changes: 24 additions & 0 deletions libraries/helpers.rb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -300,3 +300,27 @@ def installer
end
end
end

#
# Chef Stack Helpers
#
def prefix
(platform_family?('windows') ? 'C:/Chef/' : '/etc/chef/')
end

def ensurekv(config, hash)
hash.each do |k, v|
if v.is_a?(Symbol)
v = v.to_s
str = v
else
str = "'#{v}'"
end
if config =~ /^ *#{v}.*$/
config.sub(/^ *#{v}.*$/, "#{k} #{str}")
else
config << "\n#{k} #{str}"
end
end
config
end
119 changes: 119 additions & 0 deletions resources/automate.rb
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,119 @@
#
# Author:: Nathan Cerny <[email protected]>
#
# Cookbook Name:: chef_stack
# Resource:: automate
#
# Copyright 2017 Chef Software Inc
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

resource_name 'chef_automate'
default_action :create

property :name, String, name_property: true
property :channel, Symbol, default: :stable
property :version, [String, Symbol], default: :latest
property :config, String, required: true
property :accept_license, [TrueClass, FalseClass], default: false
property :enterprise, [String, Array], default: 'chef'
property :license, String
property :chef_user, String, default: 'workflow'
property :chef_user_pem, String, required: true
property :validation_pem, String, required: true
property :builder_pem, String, required: true
property :platform, String
property :platform_version, String

load_current_value do
# node.run_state['chef-users'] ||= Mixlib::ShellOut.new('chef-server-ctl user-list').run_command.stdout
# node.run_state['chef-orgs'] ||= Mixlib::ShellOut.new('chef-server-ctl org-list').run_command.stdout
# current_value_does_not_exist! unless node.run_state['chef-orgs'].index(/^#{org}$/)
end

action :create do
# https://github.com/chef/delivery/issues/469
new_resource.config << "\ndelivery['chef_server_proxy'] = false" unless new_resource.config.include?('chef_server_proxy')

# Always make sure user and key provided to resource is at the bottom of the config, overriding duplicates.
new_resource.config << "\ndelivery['chef_username'] = '#{new_resource.chef_user}'"
new_resource.config << "\ndelivery['chef_private_key'] = '/etc/delivery/#{new_resource.chef_user}.pem'"

# Hardcode v1 runner search to automate-build-node
new_resource.config << "\ndelivery['default_search'] = 'tags:delivery-build-node'"

chef_ingredient 'automate' do
action :upgrade
channel new_resource.channel
version new_resource.version
config new_resource.config
accept_license new_resource.accept_license
platform new_resource.platform if new_resource.platform
platform_version new_resource.platform_version if new_resource.platform_version
end

directory '/etc/delivery'
directory '/etc/chef'

directory '/var/opt/delivery/license/' do
recursive true
end

{
'/var/opt/delivery/license/delivery.license' => new_resource.license,
"/etc/delivery/#{new_resource.chef_user}.pem" => new_resource.chef_user_pem,
'/etc/chef/validation.pem' => new_resource.validation_pem,
'/etc/delivery/builder_key' => new_resource.builder_pem,
}.each do |file, src|
chef_file file do
source src
user 'root'
group 'root'
mode '0600'
end
end

file '/etc/delivery/builder_key.pub' do
content lazy { "ssh-rsa #{[OpenSSL::PKey::RSA.new(::File.read('/etc/delivery/builder_key')).to_blob].pack('m0')}" }
user 'root'
group 'root'
mode '0644'
end

directory '/var/opt/delivery/nginx/etc/addon.d/' do
recursive true
end

file '/var/opt/delivery/nginx/etc/addon.d/99-installer_internal.conf' do
content <<-EOF
location /installer {
alias /opt/delivery/embedded/service/omnibus-ctl/installer;
}
EOF
end

ingredient_config 'automate' do
notifies :reconfigure, 'chef_ingredient[automate]', :immediately
end

if new_resource.enterprise.is_a?(String)
new_resource.enterprise = [new_resource.enterprise]
new_resource.enterprise.each do |ent|
execute "create enterprise #{ent}" do
command "delivery-ctl create-enterprise #{ent} --ssh-pub-key-file=/etc/delivery/builder_key.pub > /etc/delivery/#{ent}.creds"
not_if "delivery-ctl list-enterprises --ssh-pub-key-file=/etc/delivery/builder_key.pub | grep -w #{ent}"
only_if 'delivery-ctl status'
end
end
end
end
93 changes: 93 additions & 0 deletions resources/backend.rb
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
#
# Author:: Nathan Cerny <[email protected]>
#
# Cookbook Name:: chef_stack
# Resource:: backend
#
# Copyright 2017 Chef Software Inc
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

resource_name 'chef_backend'
default_action :create

property :name, String, name_property: true
property :channel, Symbol, default: :stable
property :version, [String, Symbol], default: :latest
property :config, String, default: ''
property :accept_license, [TrueClass, FalseClass], default: false
property :peers, [String, Array], required: true
property :publish_address, String, default: node['ipaddress']
property :chef_backend_secrets, String, default: ''
property :platform, String
property :platform_version, String

alias :bootstrap_node :peers

load_current_value do
# node.run_state['chef-users'] ||= Mixlib::ShellOut.new('chef-server-ctl user-list').run_command.stdout
# current_value_does_not_exist! unless node.run_state['chef-users'].index(/^#{username}$/)
end

action :create do
raise 'Must accept the Chef License agreement before continuing.' unless new_resource.accept_license

new_resource.config = ensurekv(new_resource.config, publish_address: new_resource.publish_address)
chef_ingredient 'chef-backend' do
action :upgrade
channel new_resource.channel
version new_resource.version
config new_resource.config # TODO: Figure out why this isn't working in chef-ingredient
accept_license new_resource.accept_license
platform new_resource.platform if new_resource.platform
platform_version new_resource.platform_version if new_resource.platform_version
end

file '/etc/chef-backend/chef-backend.rb' do
content new_resource.config
end

chef_file '/etc/chef-backend/chef-backend-secrets.json' do
source new_resource.chef_backend_secrets
user 'root'
group 'root'
mode '0600'
not_if { new_resource.chef_backend_secrets.empty? }
end

http_retry_count = Chef::Config['http_retry_count']
Chef::Config['http_retry_count'] = 0
existing_peer = false
peers = (new_resource.peers.is_a?(Array) ? new_resource.peers : [new_resource.peers])

peers.each do |peer|
begin
Chef::HTTP.new("http://#{peer}:2379").get('/version')
existing_peer = peer
break
rescue
next
end
end
Chef::Config['http_retry_count'] = http_retry_count

execute 'chef-backend-ctl create-cluster --accept-license --yes' do
not_if 'chef-backend-ctl cluster-status &> /dev/null'
not_if { existing_peer }
end

execute "chef-backend-ctl join-cluster #{existing_peer} --accept-license --yes" do
not_if 'chef-backend-ctl cluster-status &> /dev/null'
only_if { existing_peer }
end
end
Loading

0 comments on commit 928ad14

Please sign in to comment.