bootstrap-coa-env(*)

Dockerfile

@@ -13,7 +13,7 @@ COPY Gemfile /usr/local/Gemfile
 COPY Gemfile.lock /usr/local/Gemfile.lock
 RUN cd /usr/local && bundle install
 
-#install fly-cli
+# install fly-cli
 RUN curl -sfL "https://github.com/concourse/concourse/releases/download/v${CONCOURSE_VERSION}/fly_linux_amd64" -o /usr/local/bin/fly \
   && [ ${CONCOURSE_SHA} = $(shasum -a 256 /usr/local/bin/fly | cut -d' ' -f1) ] \
   && chmod +x /usr/local/bin/fly
@@ -24,6 +24,11 @@ RUN curl -sfL "https://codeclimate.com/downloads/test-reporter/test-reporter-lat
 RUN curl -sfL "https://raw.githubusercontent.com/ekalinin/github-markdown-toc/master/gh-md-toc" > /usr/local/bin/gh-md-toc \
   && chmod a+x /usr/local/bin/gh-md-toc
 
+# Download BOSH v2 CLI
+RUN curl -o /usr/local/bin/bosh https://s3.amazonaws.com/bosh-cli-artifacts/bosh-cli-3.0.1-linux-amd64 \
+  && echo "58e6853291c3535e77e5128af9f0e8e4303dd57e5a329aa976f197c010517975 */usr/local/bin/bosh" | shasum -a 256 -c - \
+  && chmod +x /usr/local/bin/bosh
+
 # remove old version of bundler to avoid confusion between bundler and bundle cmd
 #   bundler => old binary
 #   bundle => latest binary

Gemfile

@@ -1,6 +1,6 @@
 source 'https://rubygems.org'
 
-ruby '~>2.3'
+ruby '2.3.1'
 
 gem 'rhcl', '>= 0.1.0'
 

Readme.md

@@ -563,6 +563,79 @@ This type of release requires manual work.
   1. ensures `run-tests-for-hotfix-branch` is successful
   1. triggers `ship-hotfix` to publish the release on github
 
+## Bootstrapping a COA env
+
+### How to use it
+
+In order to quickly create an environment in which you can use the COA engine,
+you can use the "bootstrap_coa_env.rb" script. By running
+`ruby scripts/bootstrap_coa_env.rb /path/to/prereqs1.yml /path/to/prereqs2.yml ... /path/to/prereqsn.yml`
+where the prereqs YAML are files containing configuration information for the
+bootstrapping, pipelines will be created from the reference dataset data.
+
+### Prerequisites
+
+The prerequisites YAML files are expected to contain some information that will
+help the script to build the environment. You can write it all in a single file
+or in multiple files. An example file can be found at /lib/coa_env_bootstrapper/prereqs.example.yml.
+
+It can contain up to 8 main keys:
+
+* inactive_steps (optional): pass a list of steps that will be deactivated in case you wouldn't need them to run, for instance in case you have some resources already installed. You can deactivate:
+    * deploy_transiant_infra: you can deactivate this step if you already have an infrastructure with BOSH and Concourse
+    * upload_stemcell: you can deactivate this step if you don't want a new stemcell to be uploaded to the BOSH Director
+    * upload_cloud_config: you can deactivate this step if you don't want to overwrite the cloud config of the BOSH Director
+    * install_git_server: you can deactivate this step if you have the git-server deployment already deployed on the BOSH Director
+* bucc (optional): you have to pass this key unless you deactivate the deploy_transiant_infra step:
+    * bin_path: the path to your installation of the bucc project. The project can be found at <http:>https://github.com/starkandwayne/bucc</http:>
+    * cpi: the cpi you want to use for the deployment. The list of existing CPIs can be found on the bucc GitHub project
+    * cpi_specific_options: options you want to pass to the bucc CLI for the bucc deployment
+* git_server_manifest (mandatory):  a BOSH manifest to deploy the git-server deployment. The example one can be used to be deployed with VirtualBox, in any other case, you will want to adapt it for your IaaS.
+* cloud_config (optional): you have to pass this key unless you deactivate the "upload_cloud_config" step. This will be used by the BOSH CLI to upload a cloud-config to the BOSH Director
+* pipeline_credentials (mandatory): this passes a list of credentials that will by used by fly to upload the pipelines to Concourse.
+* concourse (optional): you have to pass this object if you deactivated the deploy_transiant_infra step or if you want to overwrite the BUCC Concourse and want to use another one. The options are self-explanatory.
+    * target
+    * url
+    * username
+    * password
+* bosh(optional): you have to pass this key if you deactivate the deploy_transiant_infra step or if you want to overwrite BUCC's BOSH and want to use another one. The options are self-explanatory.
+    * bosh_environment
+    * bosh_client
+    * bosh_client_secret
+    * bosh_ca_cert
+
+### Connecting to Concourse
+
+Once the script is done running, it displays information about how to connect to
+the Concourse it has installed. If you wish to display those information, you
+can run `bucc info`.
+
+### Known issues
+
+#### VMs access issues
+
+If you're using VirtualBox as a IaaS on OS X, you may have trouble connectiong
+to the VMs installed by BUCC's BOSH. For instance, when the script is trying to
+push the config repository to the Git server it had installed. In this case, run
+the `bucc routes` command to create the proper routes and enable communication
+to the VMs.
+
+#### Stemcell loading creates a timeout
+
+Some stemcells are very large and here we're downloading it manually which can
+take a lot of time if the script is downloading it from the internet. This can
+lead to some timeouts. To prevent this, you can manually upload the stemcell to
+the BOSH Director and desctivate the *upload_stemcell* step.
+
+#### Some Concourse resource won't load
+
+If you're observing a Concourse error saying
+`pq: insert or update on table "worker_resource_config_check_sessions" violates foreign key constraint "worker_resource_config_check__resource_config_check_sessio_fkey"`,
+it should resolve itself in a matter of seconds.
+
+There is another error where GitHub resources as well as Docker images won't
+load. In this case, it was sufficient to restart the VirtualBox image.
+
 # FAQ
 
 ## How to initialize a new bosh deployment template ?

ci/bootstrap-coa-env-pipeline.yml

@@ -0,0 +1,79 @@
+---
+resources:
+- name: cf-ops-automation-docker-image
+  type: docker-image
+  source:
+    repository: orangecloudfoundry/cf-ops-automation
+    username: ((dockerhub-username))
+    password: ((dockerhub-password))
+
+- name: cf-ops-automation
+  type: git
+  source:
+    uri: ((cf-ops-automation-git-uri))
+    branch: ((cf-ops-automation-git-branch))
+    skip_ssl_verification: ((cf-ops-automation-git-insecure))
+
+- name: cf-ops-automation-dependencies
+  type: git
+  source:
+    uri: ((cf-ops-automation-git-uri))
+    branch: ((cf-ops-automation-git-branch))
+    skip_ssl_verification: ((cf-ops-automation-git-insecure))
+    paths: [ "Gemfile*", "Dockerfile" ]
+
+- name: bucc
+  type: git
+  source:
+    uri: https://github.com/starkandwayne/bucc
+
+jobs:
+  - name: build-cached-image
+    plan:
+    - get: cf-ops-automation-dependencies
+      trigger: true
+    - put: cf-ops-automation-docker-image
+      params:
+        build: cf-ops-automation-dependencies
+        build_args: &docker_build_args
+          CONCOURSE_VERSION: 3.14.1
+          CONCOURSE_SHA: aeb91f5d464b71de44decbd34c6696325c14d4f569c76c1171c124e2a773b02e
+
+  - name: bootstrap_coa_env
+    plan:
+    - aggregate:
+      - get: cf-ops-automation-docker-image
+        passed: [build-cached-image]
+        trigger: true
+      - get: cf-ops-automation
+        trigger: true
+      - get: bucc
+    - task: generate_private_params_file
+      file: cf-ops-automation/concourse/tasks/generate_coa_env_bootstrap_private_prereqs.yml
+      params:
+          CONCOURSE_TARGET: ((concourse_target))
+          CONCOURSE_URL: ((concourse_url))
+          CONCOURSE_USERNAME: ((concourse_username))
+          CONCOURSE_PASSWORD: ((concourse_password))
+          CONCOURSE_INSECURE: ((concourse_insecure))
+          CONCOURSE_CA_CERT: ((concourse_ca_cert))
+          BOSH_ENVIRONMENT: ((bosh_environment))
+          BOSH_TARGET: ((bosh_target))
+          BOSH_CLIENT: ((bosh_client))
+          BOSH_CLIENT_SECRET: ((bosh_client_secret))
+          BOSH_CA_CERT: ((bosh_ca_cert))
+    - task: bootstrap_coa_env
+      attempts: 3
+      image: cf-ops-automation-docker-image
+      config:
+        platform: linux
+        inputs:
+        - name: cf-ops-automation
+        - name: private-prereqs
+        run:
+          path: ruby
+          args:
+          - cf-ops-automation/scripts/bootstrap_coa_env.rb
+          - cf-ops-automation/ci/bootstrap_coa_env/prereqs.yml
+          - private-prereqs/bosh-prereqs.yml
+          - private-prereqs/concourse-prereqs.yml

ci/bootstrap_coa_env/prereqs.yml

@@ -0,0 +1,92 @@
+inactive_steps:
+  - deploy_transiant_infra
+  - upload_stemcell
+  # - upload_cloud_config
+  # - install_git_server
+
+bucc:
+  path: bucc/bin/
+  cpi: virtualbox
+
+stemcell:
+  name: bosh-warden-boshlite-ubuntu-trusty-go_agent
+  version: "3586.25"
+  uri: https://s3.amazonaws.com/bosh-core-stemcells/warden/bosh-stemcell-3586.25-warden-boshlite-ubuntu-trusty-go_agent.tgz
+  sha: b9a44806dc1bb99b0d11d7413742f3619139da0b
+
+git_server_manifest:
+  name: git-server
+  releases:
+  - name: git-server
+    version: 3
+  instance_groups:
+  - name: git-server
+    azs: [z1]
+    instances: 1
+    jobs:
+    - release: git-server
+      name: git-server
+      properties:
+        repositories: ((repos))
+    vm_type: default
+    stemcell: default
+    persistent_disk: 10_000
+    networks:
+    - name: default
+  stemcells:
+  - alias: default
+    os: ubuntu-trusty
+    version: latest
+  update:
+    canaries: 1
+    max_in_flight: 3
+    serial: false
+    canary_watch_time: 1000-30000
+    update_watch_time: 1000-30000
+
+cloud_config:
+  azs:
+  - name: z1
+  stemcells:
+  - alias: "default"
+    os: "ubuntu-trusty"
+    version: "3586.25"
+  vm_types:
+  - name: default
+  cloud_properties: { name: random }
+  networks:
+  - name: default
+    type: manual
+    subnets:
+    - range: 10.244.10.0/24
+      gateway: 10.244.10.1
+      dns: ['10.244.5.16', '10.244.6.16']
+      reserved: ['10.244.10.2', '10.244.10.3']
+      static: ['10.244.10.4 - 10.244.10.29']
+      az: z1
+  compilation:
+    network: default
+    reuse_compilation_vms: true
+    workers: 10
+    az: z1
+    cloud_properties:
+      name: random
+
+pipeline_credentials:
+  slack-webhook: https://example.slack.com/webhook
+  slack-channel: channel
+  secrets-branch: master
+  paas-templates-branch: master
+  cf-ops-automation-uri: http://github.com/orange-cloudfoundry/cf-ops-automation/
+  cf-ops-automation-branch: master
+  cf-ops-automation-tag-filter: ""
+  iaas-type: virtualbox
+  s3-stemcell-access-key-id: ""
+  s3-stemcell-region-name: us-east-1
+  s3-stemcell-secret-key: ""
+  s3-stemcell-bucket: bosh-core-stemcells
+  stemcell-name-prefix: warden
+  stemcell-main-name: warden-boshlite-ubuntu-trusty-go_agent
+  stemcell-version: "3586.25"
+  s3-stemcell-endpoint: https://s3.amazonaws.com
+  s3-stemcell-skip-ssl-verification: false

concourse/pipelines/template/depls-pipeline.yml.erb

@@ -112,7 +112,6 @@ resources:
   source:
     bucket: ((s3-stemcell-bucket))
     region_name: ((s3-stemcell-region-name))
-    # customization is required to remove bosh prefix in stemcell name
     regexp: ((stemcell-name-prefix))((stemcell-main-name))/bosh-stemcell-(.*)-((stemcell-main-name)).tgz
     access_key_id: ((s3-stemcell-access-key-id))
     secret_access_key: ((s3-stemcell-secret-key))

concourse/pipelines/template/init-pipeline.yml.erb

@@ -122,4 +122,6 @@ jobs:
         <%= "- paas-templates-full/#{vars_file}" if vars_file.end_with?("-versions.yml") %>
         <% end %>
      <% end %>
+   <% else %>
+- name: this-is-an-empty-pipeline
 <% end %>

concourse/tasks/bootstrap_coa_env/generate_private_prereqs.rb

@@ -0,0 +1,39 @@
+#!/usr/bin/env ruby
+
+require 'yaml'
+require 'fileutils'
+
+FileUtils.mkdir_p "private-prereqs"
+
+puts "current env:", ENV.inspect
+
+concourse_prereqs_path = "private-prereqs/concourse-prereqs.yml"
+puts "Creating Concourse credentials file at '#{concourse_prereqs_path}'"
+
+concourse_prereqs = {
+  "concourse" => {
+    "concourse_target"   => ENV["CONCOURSE_TARGET"],
+    "concourse_url"      => ENV["CONCOURSE_URL"],
+    "concourse_username" => ENV["CONCOURSE_USERNAME"],
+    "concourse_password" => ENV["CONCOURSE_PASSWORD"],
+    "concourse_insecure" => ENV["CONCOURSE_INSECURE"],
+    "concourse_ca_cert"  => ENV["CONCOURSE_CA_CERT"]
+  }
+}
+
+File.write(concourse_prereqs_path, concourse_prereqs.to_yaml)
+
+bosh_prereqs_path = "private-prereqs/bosh-prereqs.yml"
+puts "Creating BOSH credentials file at '#{bosh_prereqs_path}'"
+
+bosh_prereqs = {
+  "bosh" => {
+    "bosh_environment"   => ENV["BOSH_ENVIRONMENT"],
+    "bosh_target"        => ENV["BOSH_TARGET"],
+    "bosh_client"        => ENV["BOSH_CLIENT"],
+    "bosh_client_secret" => ENV["BOSH_CLIENT_SECRET"],
+    "bosh_ca_cert"       => ENV["BOSH_CA_CERT"]
+  }
+}
+
+File.write(bosh_prereqs_path, bosh_prereqs.to_yaml)

concourse/tasks/generate_coa_env_bootstrap_private_prereqs.yml

@@ -0,0 +1,27 @@
+---
+#
+# Copyright (C) 2015-2017 Orange
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+# http://www.apache.org/licenses/LICENSE-2.0
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+platform: linux
+image_resource:
+  type: docker-image
+  source:
+    repository: orangecloudfoundry/cf-ops-automation
+
+inputs:
+- name: cf-ops-automation
+outputs:
+- name: private-prereqs
+
+run:
+  path: cf-ops-automation/concourse/tasks/bootstrap_coa_env/generate_private_prereqs.rb

docs/reference_dataset/another-world-root-depls.md

@@ -65,6 +65,9 @@ another-world-root-depls
 ### The shared files
 
 * [shared](/docs/reference_dataset/shared/shared)
+  * [certs](/docs/reference_dataset/shared/shared/certs)
+    * [internal_paas-ca](/docs/reference_dataset/shared/shared/certs/internal_paas-ca)
+      * [server-ca.crt](/docs/reference_dataset/shared/shared/certs/internal_paas-ca/server-ca.crt)
   * [meta.yml](/docs/reference_dataset/shared/shared/meta.yml)
   * [pipeline-credentials.yml](/docs/reference_dataset/shared/shared/pipeline-credentials.yml)
   * [secrets.yml](/docs/reference_dataset/shared/shared/secrets.yml)

docs/reference_dataset/config_repository/hello-world-root-depls/ci-deployment-overview.yml

@@ -1,17 +1,19 @@
 ---
 ci-deployment:
   hello-world-root-depls:
-    target_name: TO_BE_DEFINED
-    target_name: fe-int-micro-for-micro-depls
+    target_name: concourse-target
     terraform_config:
       state_file_path: hello-world-root-depls/terraform-config
     pipelines:
-      bosh-sample-generated:
+      hello-world-root-depls-generated:
         vars_files:
         - hello-world-root-depls/hello-world-root-depls-versions.yml
         - shared/concourse-credentials.yml
-      bosh-sample-cf-apps-generated:
+      hello-world-root-depls-init-generated:
+        vars_files:
+        - hello-world-root-depls/hello-world-root-depls-versions.yml
+        - shared/concourse-credentials.yml
+      hello-world-root-depls-cf-apps-generated:
         vars_files:
         - hello-world-root-depls/hello-world-root-depls-versions.yml
         - shared/concourse-credentials.yml
-