This is a prerelease - please see commit messages for changes, but here is the summary of the significant changes:

Major Security Improvement

This release introduces 'entombment' of Vault secrets into Credhub for
increased security in the transport of your secrets on deployment.

In previous versions, the (( vault ... )) operators in the manifest
fragments, your environment and ops/ YAML files would resolve locally,
placing the secrets needed from vault into the manifest.
Unfortunately, this would be done in clear text.

The alternative secret store is Credhub, and a few kits still use
Credhub to contain the deployments secrets. However, Credhub lacks
the ability to manage your secrets as well as Vault, requiring a
different tooling, or just simply deleting all secrets and allowing
them to be generated again. The final issue is that becase the
placeholder variable doesn't change, there is no visual confirmation
that you have changed the value of a Credhub secret on deployment.

Genesis 3.0.0 brings you the best of both worlds!

Vault remains the single source of truth, and you can manage your
secrets with the Genesis add-secrets, check-secrets, rotate-secrets
and remove-secrets commands, independent of the BOSH director.

However, when it comes time to deploy your manifest, Genesis
determines what Vault paths are being used by the manifest, pulls them
into memory, runs a sha1sum on the values, then determines Credhub
path to inject each of those Vault values into based on its vault path
and value. This results in a unique and idempotent variable that only
changes if the value it contains changes. The value is stored at that
path in Credhub if it doesn't already exist.

When merging the manifest, the vault operators now resolve to that
unique Credhub path that contains the desired value, and there are no
longer any cleartext secrets in the manifest, the log, or even if you
pull down the manifest using the bosh manifest command.

Outdated: This is enabled on default for the genesis deploy command, and can
be enabled using the --entomb option for the manifest and lookup
commands. Likewise, it can be disabled on the deploy command with
--no-entomb option. You can also set it as a local setting in your
~/.genesis/config file with entomb_secrets key that can be set true
or false. The command line options will take precedent over the
config file.

Entombment will be applied to any environment that specifies a minimum
Genesis version of 3.0.0-rc.1 or higher. This ensures no other user can
deploy it without entombment by accident. You can still turn off entombment
by specifying genesis.entombed: false in the environment file.

Note: The values are left in Credhub after the deploy to ensure any
recreate or other BOSH operations that require the manifest to be
intact still work. Over time, this will result in Credhub containing
values that are no longer used, including expired X509 certificates,
which may trigger false positives in Doomsday cert expiry monitoring
tool. Future version of Genesis will enable cleaning out any old
values that are no longer being used.

Use the genesis remove-secrets --unused command to remove outdated
entombed secrets.
Vaultify credhub-based kits

Genesis kits based on upstream deployment manifests that use credhub
as a secret store, such as cf and cf-app-autoscalar, will now use
vault instead automatically, no updated kit version needed.

Existing secrets can be imported with genesis <env> add-secrets --import-from-credhub. After that, the normal genesis *-secrets
command can be used to manage the secrets lifecycle.

This allows certificates to be renewed as they approach expiry, while
keeping the issuing key, which means deployments don't need to go
through the three-step deployment that includes having the old and new
ca certs in the transient deployments for systems that use mutual tls,
such as nats and blobstore.

Coupled with entombment, which pushes the vault values back into
credhub keeps values out of cleartext while still showing what
credentials have changed for the deploy.

This will be enabled for all environments that specify a minimum genesis
version of 3.0.0-rc.1 or higher, but can also be explicitly disabled using
genesis.vaultify: false

You can use genesis <env> remove-secrets --unused to remove the
secrets from credhub once they are imported, so they don't become
outdated and trigger Doomsday alerts.

Breaking Changes

The genesis init command short form for the --link-dev-kit option
changes from -L to -l. This is to support upcoming global --log|-L
option.
The deprecated --cloud-config and --runtime-config options for
environment-scoped genesis commands have been removed. These were
generally only used for debugging, and the functionality can still be
accomplished with multiple -c|--config option calls. See help for
details.
The genesis manifest command has now changed due to a different
understanding of what manifests represent. Instead of specifying
manifest options, you specify a manifest type and subsets that you're
interested in generating. By default, it will generate the complete
manifest that would be used to deploy the environment. Use --list
option to see available types and subsets.
The bosh command now requires either --self or --parent to target
a bosh director deployment that was deployed by another bosh director.
If neither is specified, it will prompt you to choose one if in a
controlling terminal, or fail if not.

The --as-director|-A option has been deprecated from the bosh command,
but it is still available as an alias for --self.

If the environment is a create-env bosh director deployment, the --self
behavior is the only valid behavior, so the --self option is unnecessary,
and the --parent option is invalid. Likewise, if the environment is not
a bosh director deployment, the --parent behavior is the only valid
behavior, so the --parent option is unnecessary, and the --self option
is invalid.

Improvements

Add exclude_paths key to kit.yml to exclude extraneous files from
being compiled into the kit.
Support Github Bearer Tokens by setting GITHUB_AUTH_TOKEN without
setting GITHUB_USER environment variables.
Refactored the CLI parser for Genesis, allowing the code to be better
tested, and the help system to be generated in a uniform and accurate
manner.
- Defers loading of support libraries until needed, and only imports
  those for the chosen command.
- genesis help now explains the context under which each command is
  applicable
- genesis <command> -h now makes use of your terminal width for better
  readability and less scrolling.
Genesis now gets unpacked into ~/.genesis and supports a
~/.genesis/config file (it is a yaml-syntax file, which will have
expanded usage in the future)
You can delete your ~/.geese directory once upgrading to Genesis 2.8.12)
Genesis config file entry 'embedded_genesis' can be set to ignore,
check, or use. It will warn you if your embedded genesis for a
targeted repo is out of date if you set it to check, and it will use
the embedded genesis if you set it to use.
The genesis version command now takes arguments and a --json option
to express different version components. See genesis version -h for
full details.
Added global logging levels and multiple logging streams. All genesis
commands now support --log|-L to set the logging level to
STDERR, as well as setting multiple logging streams to files in
~/.genesis/config:
```
logs:
- file: trace
  level: TRACE
  truncate: false
```
file is relative to the directory the genesis command is run from.

truncate: false will include any log events that happened prior to the log
stream being enabled

level must be one of the following, and includes all levels list
above it:
- NONE
- ERROR
- WARN
- INFO
- DEBUG
- TRACE
BOSH calls are now captured even when interactive, allowing genesis to
determine if the call failed or was cancelled. In the future, this is
intended to track tasks and create a waypoint to update exodus if the
task completed after disconnecting, but for now it can detect the job
was cancelled and not update exodus.
Refactored the guts of Genesis to use the new logging system
Added genesis <env> vault-paths to specify which vault paths are
being used by the environment, and its associated kit and ops files.
Adds --cpi option for environment scoped commands, such as manifest
and do. Normally this is determined by querying the BOSH director,
but this option can be used to debug situations where the BOSH
director is not available.
Sets and gets of vault and credhub secrets are redacted from the log.
Major refactor to how manifests are built

There are multiple different manifests are are built during the
Genesis workflow: unevaluated environments for getting kit and param
values, unredacted, unpruned, redacted, partial for various
information gathering, unredacted-pruned and the new entombed for
deployment, and the forthcoming vaultified (not to mention
vaultified-entombed and vaultified-redacted).

All of these different types of manifest have some things in common
and some things divergent. By isolating them each into their own
derived class with a parent for coordinating common workflows and a
builder for facilitating building and caching support data and
manifest, we have cleaned up the code and significantly reduced the
number of spruce merge calls Genesis makes. This means Genesis is
easier to code, and faster to run.
Secrets management updates
- check-secrets now has a simplified interface. Without any options,
  it validates all known secrets against their definitions in the kit
  and manifest. With the --exists option, it only checks that the
  secrets exist.
- add-secrets adds an --import-from-credhub option for kits that were
  based on credhub as their secrets provider. Genesis moves those
  secrets into vault as the single-source-of-truth, which also
  improves the secrets lifecycle management (see rotate below).
- rotate-secrets now default to renew x509 certificates instead of
  fully recreating them. What this means is that the certificates are
  regenerated with new validity periods, along with any changes to key
  usage and SANs that may have changes in the secrets definitions, but
  DOES NOT recreate the key. This means anything that checks the
  certificate's modulus will continue to match, which allows new certs
  to be deployed without having to provide both the old and new in a
  three-phase deployment. If you still want to regenerate the key,
  the --regen-x509-keys option will enable that.
- remove-secrets has an improved interface to show what secrets being
  deleted were not created by the kit when specifying the --all
  option.
- remove-secrets now has an --unused flag to remove secrets from
  the vault that are no longer used in the environment's manifest, and
  from credhub that are imported into vault, or unused entombed secrets.
  
  While not necessary for the operation of the environment, cleaning up
  unused secrets can prevent Doomsday reporting false positives on expired
  secrets that aren't actually in use.
- As always, you can specify a list of paths that you want the
  *-secrets commands to operate on, but you can also specify some
  filters:
  - path regular expressions (//ca$/ will select all paths that end
    with /ca)
    Format: '!//'
    ! will negate the matching
  - property comparisons (type=x509 will select all x509 secrets)
    Format 'property(=|!=|=|!)value'
    property can be type or any secret definition field (is_ca, fixed,
    length, etc)
    value can be a string. If the value is 'true' or 'false', the
    comparison will be based on boolean "truthiness"
    if using =~ or !~, the value will be treated as a regular
    expression, with option wrapping with / characters.
Add --update-subjects option to rotate-secrets command to update
certificate subjects to match the secret definition (if it had
changed)
Added allow_oversized_secrets option to the .genesis/config
repository file to ignore secrets that are bigger than the secret
definition specifies. This is useful for users that have migrated
from cf genesis 1.x because the 2.x versions that use upstream specify
a much smaller size.
Added rudimentary markdown support for rendering release notes for
kits and genesis releases, and the kit manual. Supports tables, code
blocks, inline code, blockquotes, links, numbered and unordered lists,
and formatting.
kit-manual command (or man for short) added a --pager|-P option to
use less pager to view the manual instead of dumping it onto the screen
Addition of a new mode for specifying the target environment when
running genesis commands. This mode is activated by prefixing the
environment name with an @ symbol, then specifying enough of the
environment name to uniquely identify the desired environment file,
and optionally enough of the deployment repo name to identify the
location of the file; if no deployment repo pattern is specified, it
assumes the bosh deployment repo.

Example: genesis @useast2-p:jump deploy might be a unique enough
pattern to deploy the jumpbox environment for c-aws-useast2-prod.

Each of the environment and deployment patterns use glob-style
matching. Use can also use ^ and $ to anchor the match to the start
and/or end of the string, respectively (otherwise its assumed to be a
substring match).

When this mode is active, Genesis will search for the specified
environment file in all known deployment roots, and will prompt the
user to select the desired file if multiple matches are found.

To provide genesis with the list of deployment roots, you can set the
deployment_roots key in your ~/.genesis/config.yml file with a
list of paths, or you can set the GENESIS_DEPLOYMENT_ROOTS
environment variable to a colon-separated list of paths. It will also
search from your current working directory.

Note: Don't use ~ in the paths, as it will not be expanded. Any
relative paths will be relative to the current working directory, so
probably not what you want.
- Add genesis environments command to list all environments in the
  known deployment repositories. This command will list the
  environments, their kit, and the last deployment status. The
  environments can be grouped by the environment name or by the kit
  repository (default).

Bug Fixes

Fixed parsing of uncommitted files in kit compilations.
When deploying or checking an environment, the correct stemcell is now
correctly detected when 'latest' version is requested. Note that this
didn't impact what stemcell was actually used (always the latest
available on the BOSH director for the given OS), but just the version
reported by Genesis.
Handle loading json files when genesis is called inside a while loop
that is being fed stdin (weird edge case)
Resolve getting hash size on older versions of Perl
In the previous pre-release, the BOSH_NON_INTERACTIVE flag got stuck
in the on position, so users were not prompted to continue on
deployments, etc. This has now been rectified.
BOSH CLI changed how its version is presented from v7.5.0 on, so this
change allows Genesis to detect the correct BOSH version again.
- Fixes 'secret .. not found' errors when merging a manifest under certain
  conditions when trying to partially resolve a manifest without all the
  secrets or yaml fragments available.

Debugging Changes

The -S option (aka --show-stack) now has a slightly different
behavior. Specified once, it will show the full stack when a FATAL
error has occurred. Specified twice or three times, it will show the
current line or full stack respectively for all TRACE output.

Kit Authoring Improvements

Kits can now use Perl modules in addition to Bash script to provide
hook scripts. Perl modules have the advantage of running in the main
Genesis process and having access to the entire Genesis library
without having to call back into Genesis, resulting in a much faster
execution time. Proof of concept performed on the BOSH genesis kit's
blueprint script improved the execution time from 4.6 to 0.3 seconds.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

v3.0.0-rc.3

Major Security Improvement

Breaking Changes

Improvements

Bug Fixes

Debugging Changes

Kit Authoring Improvements