v3.0.0-rc.1

Major Security Improvement

• This release introduces 'entombment' of Vault secrets into Credhub for
  increased security in the transport of your secrets on deployment.

  In previous versions, the (( vault ... )) operators in the manifest
  fragments, your environment and ops/ YAML files would resolve locally,
  placing the secrets needed from vault into the manifest.
  Unfortunately, this would be done in clear text.

  The alternative secret store is Credhub, and a few kits still use
  Credhub to contain the deployments secrets. However, Credhub lacks
  the ability to manage your secrets as well as Vault, requiring a
  different tooling, or just simply deleting all secrets and allowing
  them to be generated again. The final issue is that becase the
  placeholder variable doesn't change, there is no visual confirmation
  that you have changed the value of a Credhub secret on deployment.

  Genesis 2.9.0 brings you the best of both worlds!

  Vault remains the single source of truth, and you can manage your
  secrets with the Genesis add-secrets, check-secrets, rotate-secrets
  and remove-secrets commands, independent of the BOSH director.

  However, when it comes time to deploy your manifest, Genesis
  determines what Vault paths are being used by the manifest, pulls them
  into memory, runs a sha1sum on the values, then determines Credhub
  path to inject each of those Vault values into based on its vault path
  and value. This results in a unique and idempotent variable that only
  changes if the value it contains changes. The value is stored at that
  path in Credhub if it doesn't already exist.

  When merging the manifest, the vault operators now resolve to that
  unique Credhub path that contains the desired value, and there are no
  longer any cleartext secrets in the manifest, the log, or even if you
  pull down the manifest using the bosh manifest command.

  This is enabled on default for the genesis deploy command, and can
  be enabled using the --entomb option for the manifest and lookup
  commands. Likewise, it can be disabled on the deploy command with
  --no-entomb option. You can also set it as a local setting in your
  ~/.genesis/config file with entomb_secrets key that can be set true
  or false. The command line options will take precedent over the
  config file.

  Note: The values are left in Credhub after the deploy to ensure any
  recreate or other BOSH operations that require the manifest to be
  intact still work. Over time, this will result in Credhub containing
  values that are no longer used, including expired X509 certificates,
  which may trigger false positives in Doomsday cert expiry monitoring
  tool. Future version of Genesis will enable cleaning out any old
  values that are no longer being used.

Major Improvements

• Vaultify credhub-based kits

  Genesis kits based on upstream deployment manifests that use credhub
  as a secret store, such as cf and cf-app-autoscalar, will now use
  vault instead automatically, no updated kit version needed.

  Existing secrets can be imported with genesis <env> add-secrets --import-from-credhub. After that, the normal genesis *-secrets
  command can be used to manage the secrets lifecycle.

  This allows certificates to be renewed as they approach expiry, while
  keeping the issuing key, which means deployments don't need to go
  through the three-step deployment that includes having the old and new
  ca certs in the transient deployments for systems that use mutual tls,
  such as nats and blobstore.

  Coupled with entombment, which pushes the vault values back into
  credhub keeps values out of cleartext while still showing what
  credentials have changed for the deploy.

Breaking Changes

• The genesis init command short form for the --link-dev-kit option
  changes from -L to -l. This is to support upcoming global --log|-L
  option.

• The deprecated --cloud-config and --runtime-config options for
  environment-scoped genesis commands have been removed. These were
  generally only used for debugging, and the functionality can still be
  accomplished with multiple -c|--config option calls. See help for
  details.

• The genesis manifest command has now changed due to a different
  understanding of what manifests represent. Instead of specifying
  manifest options, you specify a manifest type and subsets that you're
  interested in generating. By default, it will generate the complete
  manifest that would be used to deploy the environment. Use --list
  option to see available types and subsets.

Improvements

• Add exclude_paths key to kit.yml to exclude extraneous files from
  being compiled into the kit.

• Support Github Bearer Tokens by setting GITHUB_AUTH_TOKEN without
  setting GITHUB_USER environment variables.

• Refactored the CLI parser for Genesis, allowing the code to be better
  tested, and the help system to be generated in a uniform and accurate
  manner.

  • Defers loading of support libraries until needed, and only imports
    those for the chosen command.

  • genesis help now explains the context under which each command is
    applicable

  • genesis <command> -h now makes use of your terminal width for better
    readability and less scrolling.

• Genesis now gets unpacked into ~/.genesis and supports a
  ~/.genesis/config file (it is a yaml-syntax file, which will have
  expanded usage in the future)
  You can delete your ~/.geese directory once upgrading to Genesis 2.8.12)

• Genesis config file entry 'embedded_genesis' can be set to ignore,
  check, or use. It will warn you if your embedded genesis for a
  targeted repo is out of date if you set it to check, and it will use
  the embedded genesis if you set it to use.

• The genesis version command now takes arguments and a --json option
  to express different version components. See genesis version -h for
  full details.

• Added global logging levels and multiple logging streams. All genesis
  commands now support --log|-L to set the logging level to
  STDERR, as well as setting multiple logging streams to files in
  ~/.genesis/config:

  logs:
  - file: trace
    level: TRACE
    truncate: false
  

  file is relative to the directory the genesis command is run from.

  truncate: false will include any log events that happened prior to the log
  stream being enabled

  level must be one of the following, and includes all levels list
  above it:

  • NONE
  • ERROR
  • WARN
  • INFO
  • DEBUG
  • TRACE

• BOSH calls are now captured even when interactive, allowing genesis to
  determine if the call failed or was cancelled. In the future, this is
  intended to track tasks and create a waypoint to update exodus if the
  task completed after disconnecting, but for now it can detect the job
  was cancelled and not update exodus.

• Refactored the guts of Genesis to use the new logging system

• Added genesis <env> vault-paths to specify which vault paths are
  being used by the environment, and its associated kit and ops files.

• Adds --cpi option for environment scoped commands, such as manifest
  and do. Normally this is determined by querying the BOSH director,
  but this option can be used to debug situations where the BOSH
  director is not available.

• Sets and gets of vault and credhub secrets are redacted from the log.

• Major refactor to how manifests are built

  There are multiple different manifests are are built during the
  Genesis workflow: unevaluated environments for getting kit and param
  values, unredacted, unpruned, redacted, partial for various
  information gathering, unredacted-pruned and the new entombed for
  deployment, and the forthcoming vaultified (not to mention
  vaultified-entombed and vaultified-redacted).

  All of these different types of manifest have some things in common
  and some things divergent. By isolating them each into their own
  derived class with a parent for coordinating common workflows and a
  builder for facilitating building and caching support data and
  manifest, we have cleaned up the code and significantly reduced the
  number of spruce merge calls Genesis makes. This means Genesis is
  easier to code, and faster to run.

• Secrets management updates

  • check-secrets now has a simplified interface. Without any options,
    it validates all known secrets against their definitions in the kit
    and manifest. With the --exists option, it only checks that the
    secrets exist.

  • add-secrets adds an --import-from-credhub option for kits that were
    based on credhub as their secrets provider. Genesis moves those
    secrets into vault as the single-source-of-truth, which also
    improves the secrets lifecycle management (see rotate below).

  • rotate-secrets now default to renew x509 certificates instead of
    fully recreating them. What this means is that the certificates are
    regenerated with new validity periods, along with any changes to key
    usage and SANs that may have changes in the secrets definitions, but
    DOES NOT recreate the key. This means anything that checks the
    certificate's modulus will continue to match, which allows new certs
    to be deployed without having to provide both the old and new in a
    three-phase deployment. If you still want to regenerate the key,
    the --regen-x509-keys option will enable that.

  • remove-secrets has an improved interface to show what secrets being
    deleted were not created by the kit when specifying the --all
    option.

  • As always, you can specify a list of paths that you want the
    *-secrets commands to operate on, but you can also specify some
    filters:

    • path regular expressions (//ca$/ will select all paths that end
      with /ca)
      Format: '!//'
      ! will negate the matching

    • property comparisons (type=x509 will select all x509 secrets)
      Format 'property(=|!=|=|!)value'
      property can be type or any secret definition field (is_ca, fixed,
      length, etc)
      value can be a string. If the value is 'true' or 'false', the
      comparison will be based on boolean "truthiness"
      if using =~ or !~, the value will be treated as a regular
      expression, with option wrapping with / characters.

• Add --update-subjects option to rotate-secrets command to update
  certificate subjects to match the secret definition (if it had
  changed)

• Added allow_oversized_secrets option to the .genesis/config
  repository file to ignore secrets that are bigger than the secret
  definition specifies. This is useful for users that have migrated
  from cf genesis 1.x because the 2.x versions that use upstream specify
  a much smaller size.

Bug Fixes

• Fixed parsing of uncommited files in kit compilations.

• When deploying or checking an environment, the correct stemcell is now
  correctly detected when 'latest' version is requested. Note that this
  didn't impact what stemcell was actually used (always the latest
  available on the BOSH director for the given OS), but just the version
  reported by Genesis.

• Handle loading json files when genesis is called inside a while loop
  that is being fed stdin (weird edge case)

• Resolve getting hash size on older versions of Perl

• In the previous pre-release, the BOSH_NON_INTERACTIVE flag got stuck
  in the on position, so users were not prompted to continue on
  deployments, etc. This has now been rectified.

• BOSH CLI changed how its version is presented from v7.5.0 on, so this
  change allows Genesis to detect the correct BOSH version again.

Debugging Changes

• The -S option (aka --show-stack) now has a slightly different
  behavior. Specified once, it will show the full stack when a FATAL
  error has occurred. Specified twice or three times, it will show the
  current line or full stack respectively for all TRACE output.

Kit Authoring Improvements

• Kits can now use Perl modules in addition to Bash script to provide
  hook scripts. Perl modules have the advantage of running in the main
  Genesis process and having access to the entire Genesis library
  without having to call back into Genesis, resulting in a much faster
  execution time. Proof of concept performed on the BOSH genesis kit's
  blueprint script improved the execution time from 4.6 to 0.3 seconds.