Can't install onto RDS

[boydj]

Error executing action run on resource 'execute[create postgresql plpgsql extension

With the following config in my wrapper cookbook:

default['supermarket_omnibus']['chef_server_url'] = 'xxx'
default['supermarket_omnibus']['chef_oauth2_app_id'] = 'xxx'
default['supermarket_omnibus']['chef_oauth2_secret'] = 'xxx'
default['supermarket_omnibus']['chef_oauth2_verify_ssl'] = false
default['supermarket_omnibus']['package_url'] = 'xxx'
default['supermarket_omnibus']['config']['features'] = 'tools, gravatar, collaborator_groups, github, announcement'

default['supermarket_omnibus']['config']['postgresql']['enable'] = false
default['supermarket_omnibus']['config']['database']['user']     = 'chef'
default['supermarket_omnibus']['config']['database']['name']     = 'supermarket'
default['supermarket_omnibus']['config']['database']['host']     = 'xxx.us-east-1.rds.amazonaws.com'
default['supermarket_omnibus']['config']['database']['port']     = '5432'
default['supermarket_omnibus']['config']['database']['password'] = ''

default['supermarket_omnibus']['config']['s3_bucket'] = ''

Looks like this was fixed previously here: https://github.com/chef-boneyard/omnibus-supermarket/pull/20/commits/108f144d6262d57d79903ba518d0b4f4b40f6d38v

[0mMixlib::ShellOut::ShellCommandFailed�[0m
------------------------------------�[0m
Expected process to exit with [0], but received '1'
�[0m---- Begin output of echo 'CREATE EXTENSION IF NOT EXISTS plpgsql' | psql ----
�[0mSTDOUT: 
�[0mSTDERR: /opt/supermarket/embedded/lib/ruby/gems/2.3.0/gems/mixlib-shellout-2.2.7/lib/mixlib/shellout.rb:206:in `getpwnam': can't find user for chef (ArgumentError)
�[0m    from /opt/supermarket/embedded/lib/ruby/gems/2.3.0/gems/mixlib-shellout-2.2.7/lib/mixlib/shellout.rb:206:in `uid'
�[0m    from /opt/supermarket/embedded/lib/ruby/gems/2.3.0/gems/mixlib-shellout-2.2.7/lib/mixlib/shellout/unix.rb:153:in `set_user'
�[0m    from /opt/supermarket/embedded/lib/ruby/gems/2.3.0/gems/mixlib-shellout-2.2.7/lib/mixlib/shellout/unix.rb:332:in `block in fork_subprocess'
�[0m    from /opt/supermarket/embedded/lib/ruby/gems/2.3.0/gems/mixlib-shellout-2.2.7/lib/mixlib/shellout/unix.rb:316:in `fork'
�[0m    from /opt/supermarket/embedded/lib/ruby/gems/2.3.0/gems/mixlib-shellout-2.2.7/lib/mixlib/shellout/unix.rb:316:in `fork_subprocess'
�[0m    from /opt/supermarket/embedded/lib/ruby/gems/2.3.0/gems/mixlib-shellout-2.2.7/lib/mixlib/shellout/unix.rb:93:in `run_command'
�[0m    from /opt/supermarket/embedded/lib/ruby/gems/2.3.0/gems/mixlib-shellout-2.2.7/lib/mixlib/shellout.rb:259:in `run_command'
�[0m    from /opt/supermarket/embedded/lib/ruby/gems/2.3.0/gems/chef-12.13.37/lib/chef/mixin/shell_out.rb:86:in `shell_out_command'
�[0m    from /opt/supermarket/embedded/lib/ruby/gems/2.3.0/gems/chef-12.13.37/lib/chef/mixin/shell_out.rb:39:in `shell_out'
�[0m    from /opt/supermarket/embedded/lib/ruby/gems/2.3.0/gems/chef-12.13.37/lib/chef/mixin/shell_out.rb:44:in `shell_out!'
�[0m    from /opt/supermarket/embedded/lib/ruby/gems/2.3.0/gems/chef-12.13.37/lib/chef/provider/execute.rb:62:in `block in action_run'
�[0m    from /opt/supermarket/embedded/lib/ruby/gems/2.3.0/gems/chef-12.13.37/lib/chef/mixin/why_run.rb:52:in `add_action'
�[0m    from /opt/supermarket/embedded/lib/ruby/gems/2.3.0/gems/chef-12.13.37/lib/chef/provider.rb:176:in `converge_by'
�[0m    from /opt/supermarket/embedded/lib/ruby/gems/2.3.0/gems/chef-12.13.37/lib/chef/provider/execute.rb:60:in `action_run'
�[0m    from /opt/supermarket/embedded/lib/ruby/gems/2.3.0/gems/chef-12.13.37/lib/chef/provider.rb:145:in `run_action'
�[0m    from /opt/supermarket/embedded/lib/ruby/gems/2.3.0/gems/chef-12.13.37/lib/chef/resource.rb:603:in `run_action'
�[0m    from /var/opt/supermarket/cache/cache/cookbooks/compat_resource/files/lib/chef_compat/monkeypatches/chef/runner.rb:78:in `run_action'
�[0m    from /var/opt/supermarket/cache/cache/cookbooks/compat_resource/files/lib/chef_compat/monkeypatches/chef/runner.rb:106:in `block (2 levels) in converge'
�[0m    from /var/opt/supermarket/cache/cache/cookbooks/compat_resource/files/lib/chef_compat/monkeypatches/chef/runner.rb:106:in `each'
�[0m    from /var/opt/supermarket/cache/cache/cookbooks/compat_resource/files/lib/chef_compat/monkeypatches/chef/runner.rb:106:in `block in converge'
�[0m    from /opt/supermarket/embedded/lib/ruby/gems/2.3.0/gems/chef-12.13.37/lib/chef/resource_collection/resource_list.rb:94:in `block in execute_each_resource'
�[0m    from /opt/supermarket/embedded/lib/ruby/gems/2.3.0/gems/chef-12.13.37/lib/chef/resource_collection/stepable_iterator.rb:116:in `call_iterator_block'
�[0m    from /opt/supermarket/embedded/lib/ruby/gems/2.3.0/gems/chef-12.13.37/lib/chef/resource_collection/stepable_iterator.rb:85:in `step'
�[0m    from /opt/supermarket/embedded/lib/ruby/gems/2.3.0/gems/chef-12.13.37/lib/chef/resource_collection/stepable_iterator.rb:104:in `iterate'
�[0m    from /opt/supermarket/embedded/lib/ruby/gems/2.3.0/gems/chef-12.13.37/lib/chef/resource_collection/stepable_iterator.rb:55:in `each_with_index'
�[0m    from /opt/supermarket/embedded/lib/ruby/gems/2.3.0/gems/chef-12.13.37/lib/chef/resource_collection/resource_list.rb:92:in `execute_each_resource'
�[0m    from /opt/supermarket/embedded/lib/ruby/2.3.0/forwardable.rb:184:in `execute_each_resource'
�[0m    from /var/opt/supermarket/cache/cache/cookbooks/compat_resource/files/lib/chef_compat/monkeypatches/chef/runner.rb:105:in `converge'
�[0m    from /opt/supermarket/embedded/lib/ruby/gems/2.3.0/gems/chef-12.13.37/lib/chef/client.rb:669:in `block in converge'
�[0m    from /opt/supermarket/embedded/lib/ruby/gems/2.3.0/gems/chef-12.13.37/lib/chef/client.rb:664:in `catch'
�[0m    from /opt/supermarket/embedded/lib/ruby/gems/2.3.0/gems/chef-12.13.37/lib/chef/client.rb:664:in `converge'
�[0m    from /opt/supermarket/embedded/lib/ruby/gems/2.3.0/gems/chef-12.13.37/lib/chef/client.rb:703:in `converge_and_save'
�[0m    from /opt/supermarket/embedded/lib/ruby/gems/2.3.0/gems/chef-12.13.37/lib/chef/client.rb:283:in `run'
�[0m    from /opt/supermarket/embedded/lib/ruby/gems/2.3.0/gems/chef-12.13.37/lib/chef/application.rb:302:in `block in fork_chef_client'
�[0m    from /opt/supermarket/embedded/lib/ruby/gems/2.3.0/gems/chef-12.13.37/lib/chef/application.rb:290:in `fork'
�[0m    from /opt/supermarket/embedded/lib/ruby/gems/2.3.0/gems/chef-12.13.37/lib/chef/application.rb:290:in `fork_chef_client'
�[0m    from /opt/supermarket/embedded/lib/ruby/gems/2.3.0/gems/chef-12.13.37/lib/chef/application.rb:255:in `block in run_chef_client'
�[0m    from /opt/supermarket/embedded/lib/ruby/gems/2.3.0/gems/chef-12.13.37/lib/chef/local_mode.rb:44:in `with_server_connectivity'
�[0m    from /opt/supermarket/embedded/lib/ruby/gems/2.3.0/gems/chef-12.13.37/lib/chef/application.rb:243:in `run_chef_client'
�[0m    from /opt/supermarket/embedded/lib/ruby/gems/2.3.0/gems/chef-12.13.37/lib/chef/application/client.rb:464:in `sleep_then_run_chef_client'
�[0m    from /opt/supermarket/embedded/lib/ruby/gems/2.3.0/gems/chef-12.13.37/lib/chef/application/client.rb:451:in `block in interval_run_chef_client'
�[0m    from /opt/supermarket/embedded/lib/ruby/gems/2.3.0/gems/chef-12.13.37/lib/chef/application/client.rb:450:in `loop'
�[0m    from /opt/supermarket/embedded/lib/ruby/gems/2.3.0/gems/chef-12.13.37/lib/chef/application/client.rb:450:in `interval_run_chef_client'
�[0m    from /opt/supermarket/embedded/lib/ruby/gems/2.3.0/gems/chef-12.13.37/lib/chef/application/client.rb:434:in `run_application'
�[0m    from /opt/supermarket/embedded/lib/ruby/gems/2.3.0/gems/chef-12.13.37/lib/chef/application.rb:60:in `run'
�[0m    from /opt/supermarket/embedded/lib/ruby/gems/2.3.0/gems/chef-12.13.37/bin/chef-client:26:in `<top (required)>'
�[0m    from /opt/supermarket/embedded/bin/chef-client:23:in `load'
�[0m    from /opt/supermarket/embedded/bin/chef-client:23:in `<main>'
�[0m---- End output of echo 'CREATE EXTENSION IF NOT EXISTS plpgsql' | psql ----
�[0mRan echo 'CREATE EXTENSION IF NOT EXISTS plpgsql' | psql returned 1�[0m

�[0mCookbook Trace:�[0m
---------------�[0m
/var/opt/supermarket/cache/cache/cookbooks/compat_resource/files/lib/chef_compat/monkeypatches/chef/runner.rb:78:in `run_action'
�[0m/var/opt/supermarket/cache/cache/cookbooks/compat_resource/files/lib/chef_compat/monkeypatches/chef/runner.rb:106:in `block (2 levels) in converge'
�[0m/var/opt/supermarket/cache/cache/cookbooks/compat_resource/files/lib/chef_compat/monkeypatches/chef/runner.rb:106:in `each'
�[0m/var/opt/supermarket/cache/cache/cookbooks/compat_resource/files/lib/chef_compat/monkeypatches/chef/runner.rb:106:in `block in converge'
�[0m/var/opt/supermarket/cache/cache/cookbooks/compat_resource/files/lib/chef_compat/monkeypatches/chef/runner.rb:105:in `converge'�[0m

�[0mResource Declaration:�[0m
---------------------�[0m
# In /var/opt/supermarket/cache/cache/cookbooks/omnibus-supermarket/recipes/database.rb
�[0m
�[0m 46:   execute "create postgresql #{ext} extension" do
�[0m 47:     user node['supermarket']['database']['user']
�[0m 48:     command "echo 'CREATE EXTENSION IF NOT EXISTS #{ext}' | psql"
�[0m 49:     not_if "echo '\\dx' | psql #{node['supermarket']['database']['name']} | grep #{ext}"
�[0m 50:   end
�[0m 51: end
�[0m
�[0mCompiled Resource:�[0m
------------------�[0m
# Declared in /var/opt/supermarket/cache/cache/cookbooks/omnibus-supermarket/recipes/database.rb:46:in `block in from_file'
�[0m
�[0mexecute("create postgresql plpgsql extension") do
�[0m  action [:run]
�[0m  retries 0
�[0m  retry_delay 2
�[0m  default_guard_interpreter :execute
�[0m  command "echo 'CREATE EXTENSION IF NOT EXISTS plpgsql' | psql"
�[0m  backup 5
�[0m  returns 0
�[0m  user "chef"
�[0m  declared_type :execute
�[0m  cookbook_name "omnibus-supermarket"
�[0m  recipe_name "database"
�[0m  not_if "echo '\dx' | psql supermarket | grep plpgsql"
�[0mend
�[0m
�[0mPlatform:�[0m
---------�[0m
x86_64-linux�[0m

[irvingpop]

Thanks for reporting that @boydj ! Would you mind submitting a PR to re-add those lines?

[robbkidd]

I think if these lines were to go somewhere, they would be in the omnibus-supermarket recipe that is internal to the omnibus package. I propose that this issue move to the chef/supermarket repo under the guise of "handle PG database ownership and enabling PG extensions when PG is run external to the omnibus install." The solution might be something other than flipping these resources off if omnibus'd-postgresql is disabled. For example, I can picture adding to the database recipe a .pgpass file for psql commands to pick up and use to authenticate to wherever PG is running during these configuration steps.

[robbkidd]

Looking a bit closer at the error, the specific error has to do with the expectation that the execute resource for the psql command will be run with an OS user that matches the name of a user within the PostgreSQL service who has the privileges to install extensions. This is true for the default install—supermarket OS service account and supermarket PG database user—but not necessarily true for a PG database configured elsewhere. In @boydj's case, there is a chef user in the RDS database, but that an OS user of the same name does not exist on the Supermarket host (and also maybe shouldn't).

I sense that the bug here is assuming that an OS user for the command to install extensions will match an admin DB user. We should spend a little time unpacking that.

[boydj]

I can confirm that changing the DB user to match a system user, in this case, supermarket, allows installation to complete successfully.

[robbkidd]

Thanks for the confirmation, @boydj. That helps shape our options. I see them as:

• document the current expectation—requirement, really—that the database username match the OS service account name, defaulted to supermarket (increased cognitive load on the humans performing an install)
• update the omnibus package's internal cookbook to provide the flexibility to private Supermarkets to pick the name of the database user (increased code complexity)

Are those reasonable? Are there others?