Groups entries not being created on first/automatic puppet run

[5obol]

Hi,
I'm using 1.5.1 version of the module found at forge. We've been using the module since version 1.3.3 and throughout the same problem has been present. The user details are not populated into listed groups. The only way for this to work is to logon to the box and manually run puppet agent -t, somehow standard interval runs will not execute correctly. The outcome is that users are not added to wheel group and whilst we disable standard admin users on box we loose admin access to the box.

Our setup
In hiera:

  test:
    ensure: 'present'
    uid: 5000
    gid: 5000
    comment: "Test User"
    groups: ["wheel", "users"]
    shell: "/bin/bash"
    password_max_age: "99999"
    ssh_keys:
      'Test user key': # an unique indentifier of a key
        type: "ssh-rsa"
        key: "AAAAB3NzaC1ySOMEKEY...."

First automatic puppet run, details found:

Jan  6 17:18:55 ptest-001 puppet-agent[30585]: (/Stage[main]/Profile::Packages/Package[htop]/ensure) created
Jan  6 17:18:55 ptest-001 puppet-agent[30585]: (/Stage[main]/Accounts/Accounts::User[test]/User[test]/ensure) created
Jan  6 17:18:55 ptest-001 puppet-agent[30585]: (/Stage[main]/Accounts/Accounts::User[test]/Accounts::Authorized_keys[test]/File[/home/test/.ssh]/ensure) created
Jan  6 17:18:55 ptest-001 puppet-agent[30585]: (/Stage[main]/Accounts/Accounts::User[test]/Accounts::Authorized_keys[test]/Ssh_authorized_key[Test User key]/ensure) created

Then doesn't matter how many times the puppet agent run through standard interval nothing will change. Here's log of such run from PE server:

2017-01-06 17:48:11,148 INFO  [qtp844134227-78] [puppetserver] Puppet Caching node for ptest-001
2017-01-06 17:48:16,995 INFO  [qtp844134227-305] [puppetserver] Puppet 'replace_facts' command for ptest-001 submitted to PuppetDB with UUID d3e6dd9f-7740-431a-93b4-ac929729582b
2017-01-06 17:48:17,212 INFO  [qtp844134227-305] [puppetserver] Puppet Caching node for ptest-001
2017-01-06 17:48:18,386 INFO  [qtp844134227-305] [puppetserver] Puppet Inlined resource metadata into static catalog for ptest-001 in environment feature_root_recovery in 0.05 seconds
2017-01-06 17:48:18,386 INFO  [qtp844134227-305] [puppetserver] Puppet Compiled static catalog for ptest-001 in environment feature_root_recovery in 1.14 seconds
2017-01-06 17:48:18,386 INFO  [qtp844134227-305] [puppetserver] Puppet Caching catalog for ptest-001
2017-01-06 17:48:18,735 INFO  [qtp844134227-305] [puppetserver] Puppet 'replace_catalog' command for ptest-001 submitted to PuppetDB with UUID 9be1c948-9f33-4ea4-90af-d3ae5065fad6
2017-01-06 17:48:22,339 INFO  [qtp844134227-312] [puppetserver] Puppet 'store_report' command for ptest-001 submitted to PuppetDB with UUID 11bc36ed-d460-4139-bd6a-fdf72fc4def0

When I login to the box manually and run puppet I get correct action:

[root@ptest-001 ~]# puppet agent -t
Notice: Local environment: 'production' doesn't match server specified node environment 'feature_root_recovery', switching agent to 'feature_root_recovery'.
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for ptest-001
Info: Applying configuration version '1483724617'
Notice: /Stage[main]/Accounts/Accounts::Group[test]/Group[test]/members: members changed '' to 'test'
Notice: /Stage[main]/Accounts/Accounts::Group[wheel]/Group[wheel]/members: members changed 'ec2-user' to 'test'
Notice: /Stage[main]/Accounts/Accounts::Group[users]/Group[users]/members: members changed '' to 'test'
Notice: Applied catalog in 2.61 seconds

[deric]

Are you sure that the error actually relates to this module? Is your environment correctly configured on the node? Does automatic mode fetch feature_root_recovery environment? Just curious... the first message looks suspicious.

Just to have complete information, which distribution and Puppet version do you use? I'm not sure that I'll be able to replicate such error. Automated tests are able to configure accounts in a single run on clean system (currently tested on Debian and CentOS).

[5obol]

Hi,
It's not error, it's that the module isn't appearing to do all actions it suppose to do. We have no problem with any other modules/manifests. The first message is normal and all it's say is that i overwrite default environment with a testing one.

We use Puppet Enterprise 2016.5.1 and this issue I have been observing for at least 6 months on three different versions of puppet. The problem exist on many clients.

Also, out of curiosity why is the module adding username to group that user has defined as default

test:x:1011:100:Test Account,,,:/home/test:/bin/bash
test:x:100:test

[deric]

Which OS do you use? How do you provision the OS? Is it possible that some settings are dependent on environment variables that are being initialized after login?

why is the module adding username to group that user has defined as default

You're right, that shouldn't be necessary, I'm gonna look at that.

[5obol]

Redhat 7.2 and 7.3. Latest PE agent. The behaviour is weird the group actions are never executed, doesn't matter how many times puppet agent run ( as a service/scheduled run ), other additions to manifest will be executed. The only thing that help to setup groups is run puppet agent manually.
I'm actually not 100% sure this is a bug in the module's code, perhaps it's also bug in puppet's code and we have an edge case here. I'll raise this with puppet folks.

Re, the user setup, that be fantastic. Thanks!

[5obol]

Hey,
I did spend some more time on the issue. My suspicion fell onto SELinux. Our boxes are running in enforcing mode. However I've tested permissive mode and problem remained same. Just for peace of mind I've added:

/sbin/setsebool -P puppetagent_manage_all_files 1

but that didn't fix it.

On debugging side of this, the run which fails never lists gpasswd, there's never any action around that ( i'm using the latest gpasswd ). In the manual run which is always succesful (using --debug and --trace) I can see gpasswd is invoked and users are added to the group as expected.

The scheme that appers to make the issue better is if i login to the box(attaching to the console helps?). Don't even have to execute puppet, enough I restart machine and on next agent run the policy will be applied correctly. If lets say I had restart in bootstrap script done after puppet execution that won't help, next boot and it still won't apply policy.

I think this is weird problem with Redhat and puppet agent.

[deric]

Good, just one more thing. Do you have some special settings in /etc/pam, especially /etc/pam.d/login or even /etc/login.defs?

PAM configuration is loaded upon login, so that might explain why is login needed.

The configuration puppetagent_manage_all_files does not include gpasswd, right? gpasswd is executed from puppet, I'm not sure if that's a problem for SELinux.

[5obol]

I was suspecting that maybe our CIS module is forcing something so I've created another testing box without it. Same result, the users are not added to groups.

Also correction to the above, when puppet agent is run through bootstrap script, then restart of the box is executed as part of bootstrap, then on next agent run when machine is back users are added to the group ( and I didn't have to login ).

So I guess that will be my workaround for now.

[deric]

Ok, it seems to me that the problem is related either to the OS or Puppet itself. Is it ok if we close this issue as it can't be fixed within this module?

[TheHob]

I think I've discovered what's happening here. Here were the steps I took:

1. Set up one CentOS 6 and one CentOS 7 VM.

2. Created an 'Accounts' node classification group on my PE master.

3. Created a simple profile and YAML entries for users and groups like the ones in this module's documentation.

# Accounts test profile
class profile::example::accounts {
  class { 'accounts': }
}

4. Classified my nodes into the 'Accounts' group, which includes the above profile.

5. Destroyed the nodes and re-created them with Vagrant.

6. Results were that all users were created, but group membership was not managed. It gave me this error:

Debug: /Group[friendly]: Provider groupadd does not support features manages_members; not managing attribute members

7. Added the 'deric/gpasswd' module to my Puppetfile to add the :manages_members attribute to the 'Group' puppet type.

8. Re-created the nodes.

9. Results were that upon the first puppet run, two things of note happened.

9.a. First, The gpasswd.rb provider file (meant to add :manages_members feature/attribute to Group type) gets written to the agent node before the catalog is applied:

# 'gpasswd.rb' provider is created on the agent node before the catalog is applied
# Appears to clear environment cache, but I'm not sure it's happening properly
Notice: /File[/opt/puppetlabs/puppet/cache/lib/puppet/provider/group/gpasswd.rb]/ensure: defined content as '{md5}1878f3fc91089904031dbe0534ffc012'
Debug: /File[/opt/puppetlabs/puppet/cache/lib/puppet/provider/group/gpasswd.rb]: The container /opt/puppetlabs/puppet/cache/lib will propagate my refresh event
Debug: Finishing transaction 32945140
Debug: Evicting cache entry for environment 'production'
Debug: Caching environment 'production' (ttl = 0 sec)
Debug: Loading external facts from /opt/puppetlabs/puppet/cache/facts.d
Info: Loading facts

9.b. But support for the :manages_members attribute appears not to be loaded into memory/environment cache. This is evident as the agent 'decides' later in the run that :manages_groups feature still isn't supported (even though the gpasswd.rb provider has been written to the agent and should have added the feature to the Group type.)

Debug: /Group[friendly]: Provider groupadd does not support features manages_members; not managing attribute members
Debug: Puppet::Type::Vcsrepo::ProviderSvn: file svn does not exist
Debug: Puppet::Type::Vcsrepo::ProviderHg: file hg does not exist
Debug: Puppet::Type::Vcsrepo::ProviderCvs: file cvs does not exist
Debug: Puppet::Type::Vcsrepo::ProviderBzr: file bzr does not exist
Debug: Creating default schedules
Debug: Loaded state in 0.03 seconds
Debug: User[myuser] parsed for purging Ssh_authorized_key[myawesomefirstkey]
Debug: User[myuser] parsed for purging Ssh_authorized_key[myawesomesecondkey]
Debug: Loaded transaction store file in 0.03 seconds
Info: Applying configuration version '90ac4e64a15f1578f5ef2a72e3db705d394700d3'

10. The groups were added on the first run, but membership was not managed/users were not added to the groups (verified in '/etc/group' file and with id $USER).

11. The users were added to the groups with no issue on the second puppet run because the :manages_members attribute was appropriately loaded into memory/the environment cache because gpasswd.rb was added on the previous run.

12. I implemented a workaround that appears to force the gpasswd provider/:manages_members attribute to be appropriately added to the Group type and loaded into memory ahead of attempting to manage group membership on the first run.

# accounts test
class profile::example::accounts {

  # Workaround below:
  Group { provider => 'gpasswd' }

  class { 'accounts': }
}

Conclusions:

1. The deric/accounts module's README should contain documentation about requiring deric/gpasswd or another gpasswd module if an accounts module user wants to manage group membership.

2. It seems like the environment cache is not properly being cleared when the gpasswd.rb provider file in deric/gpasswd is added to the agent node. As such the :manages_members feature appears not to be available when the group and membership resources are managed/applied on the first run.

3. Given the above, I'm not sure if this is a question of the 'correct' behavior in the puppet agent or if the workaround, Group { provider => 'gpasswd' }, should be incorporated into the module and toggled on/off in conjunction with the manage_groups parameter.
   I'm circling back with Puppet staff to see what they recommend, but I'm open to suggestions.

As it stands, adding Group { provider => 'gpasswd' } to my profile solves the problem of group membership not being managed on the first run, but may lead to others.

Note: I'm not sure if the supported accounts module on the forge suffers from the same issue as I've not tested it.

[deric]

@TheHob Thanks for your comments!

Regarding the conclusion 1. it's quite similar issue as #71. I agree, the dependency should be mentioned. In order to address #67 I'm considering moving logic from gpasswd.rb to this module and replacing gpasswd command by usermod (or switching between these two would be dependent on the undelying OS). Having external module with pretty much single file proved to be more complication than benefit. Though someone might be using alternative implementation of gpasswd (and breaking dependencies of this module). What do you think?

Ad 2. I'm not aware that we could anyhow influence Puppet's cache. Behavior might differ between Puppet versions.

Ad 3. Interesting proposal, I think it's worthy trying at least. It's quite problematic that no error or warning is shown when gpasswd provider is missing (again could be fixed by merging gpasswd module).

[TheHob]

@deric Thanks for the quick response. I'll answer inline.

1. I'm a +1 on using usermod wherever you can, as it's more widely available and can be ensured by ensuring the passwd package (at least on Debian, I think it's the same on CentOS/RHEL). I also think that your idea is a good one to toggle between the providers based on OS. It would actually kill two birds with one stone (and probably solve the issue we're discussing here completely) if you always set usermod or gpasswd as the default provider in the module based on the OS.

2. I currently work for Puppet and am going to speak with our agent engineers about the caching behavior. However, your point is valid that this should probably be resolved in the module (at least short to mid-term) as there are a lot of people using a lot of Puppet versions out there. I think always setting a default provider based on OS is the best and fastest way forward.

3. I may propose adding an error to the stdlib groupadd provider when an absent feature is invoked to Puppet engineers and/or submit a PR to stdlib.

[TheHob]

I opened a ticket for the puppet agent question. It should be publicly accessible.

https://tickets.puppetlabs.com/browse/PA-895

[deric]

@TheHob Just a quick update, in 1.6 branch I've switched to usermod, for all command but setting all members of a group:

gpasswd -M user1,user2 group

which AFAIK doesn't have equivalent in usermod. The advantage of gpasswd is that it will remove all users that are not declared to be part of the group.

Regarding this issue, I'm closing it as there's not much we can do about it.