Cross-container interference

[karmix]

Changes made to /opt/chef are propagated to other containers. I haven't had a chance to dig into the problem, but it appears that the /opt/chef file system is not mounted with an overlay, so changes do not remain local to the system that made them.

This this can cause problems when working with cookbooks that do things like define chef_gem resources. Those problems are particularly difficult to debug on systems running many containers, like you might find in a pipeline, because they usually are not caused by the system under test.

Steps to Illustrate the Issue

Make sure you don't do this on a system running other kitchen-dokkencontainers. docker rm the chef containers when you are done testing to repair the chef environment in kitchen-dokken.

# Create two fresh cookbooks.
chef generate cookbook book1
chef generate cookbook book2

# Setup dokken in both cookbooks.
cat >book1/kitchen.dokken.yml <<'END_OF_DOKKEN'
driver:      { name: dokken }
transport:   { name: dokken }
provisioner: { name: dokken }
platforms:
- name: ubuntu-16.04
  driver:
    image: dokken/ubuntu-16.04
END_OF_DOKKEN
cp book1/kitchen.dokken.yml book2/

# Spin up book1, and verify that it can converge.
(cd book1 && KITCHEN_LOCAL_YAML=kitchen.dokken.yml kitchen converge)
# Multiple times, even.
(cd book1 && KITCHEN_LOCAL_YAML=kitchen.dokken.yml kitchen converge)

# Add something in book2 to break chef in its environment.
cat >book2/recipes/default.rb <<'END_OF_RECIPE'
directory '/opt/chef/embedded/bin' do
  action :delete
  recursive true
end
END_OF_RECIPE

# book2 breaks chef.
(cd book2 && KITCHEN_LOCAL_YAML=kitchen.dokken.yml kitchen converge)

# Second run fails because chef-client is missing.
(cd book2 && KITCHEN_LOCAL_YAML=kitchen.dokken.yml kitchen converge)

# Now book1 can't converge, either.
(cd book1 && KITCHEN_LOCAL_YAML=kitchen.dokken.yml kitchen converge)

# Cleanup
(cd book1 && KITCHEN_LOCAL_YAML=kitchen.dokken.yml kitchen destroy)
(cd book2 && KITCHEN_LOCAL_YAML=kitchen.dokken.yml kitchen destroy)
docker ps -a
# `docker rm` remaining chef/chef containers.

[marcparadise]

This may be because of the shared intermediary image that gets used:
https://github.com/test-kitchen/kitchen-dokken/blob/c4e683b01381aa3165ae6fdd5e
afd308a9e4c5a5/lib/kitchen/driver/dokken.rb#L146-L153