Way of Testing with Test Kitchen

[thheinen]

Detailed Description

As kitchen-inspec supports loading InSpec Input plugins now, these will also be using in the TestKitchen context. To avoid having TK instances connect to production Vault (isolation issue) or developer-run Vault (network config headache), it would be good to have a facility to mock Vault outputs. This is basically the same as the chef-vault databack fallback.

Context

Currently developing a number of Vault-connected cookbooks/InSpec tests for a customer and in need to test those properly.

Possible Implementation

If I get feedback for one of these options (or another one), I will do the coding as it's needed anyway:

JSON/YAML file exchange

Could use the upload/lifecycle mechanics to get a file onto the kitchen instance, then via new plugin config connect those and use as source for mocked inputs.

This has some usability issues, as the configuration in TK will have multiple steps.

TestKitchen introspection

Have the mock data as a databag, like chef-vault and connect via the TK API to retrieve those, combined with a TestKitchen autodetection.

This is probably best from a usability perspective and matches a known concept, but due to the lack of proper TK APIs between verifiers/provisioners, the code is borderline acceptable as it requires metaprogramming. I have done this for the inspec-chef plugin though:

• Documentation for inspec-chef
• TK detection code
• TK introspection

[james-stocks]

TK introspection seems like the better option to me, but would it be possible to let the user control whether mocking or real vault access happens?
I would wonder how many people would like to (or assume) their TK suites will access a real vault.

[thheinen]

If we added a plugin setting databag_fallback defaulting to true, assuming that using mock data in Kitchen is the common use case, sure.

Only issue with that is that kitchen-inspec does not allow plugin config yet. I had a feeling the other day, that we would hit that soon anyway.

If you're ok with it, I could whip up some implementation like this and we tackle the plugin config afterwards, as it extends from the supposed "main" case (using fallback in kitchen) to the "niche" case. As InSpec plugins in TK are pretty new, the impact should be negligible?

[james-stocks]

Having mock data be the default and needing config to signal that a real vault connection should be used sounds good.
I think it should be clearly documented though (perhaps in a totally new "Testing" section in the README of this repo).

Thanks again Thomas!