Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow for empty AD group in LDAP sync #282

Closed
andersbogsnes opened this issue Jul 1, 2019 · 16 comments · Fixed by #336
Closed

Allow for empty AD group in LDAP sync #282

andersbogsnes opened this issue Jul 1, 2019 · 16 comments · Fixed by #336
Assignees

Comments

@andersbogsnes
Copy link

We have an issue where we mirror ldap2pg.yaml across our environments, (test, dev, prod) and have a similar naming scheme for our AD groups.

  - ldap:
      base: "base_dn_goes_here"
      filter: "(CN=${ENV}_name_of_ad_group)" # Replaced at runtime with ENV
    role:
      name: '{member.cn}'
      options: LOGIN SUPERUSER
      parent:
        - ldap_roles
        - owners
      on_unexpected_dn: warn

The problem occurs when there are no users in the AD group - something that can also occur if someone quits. ldap2pg will error out saying it has no attribute member.

As far as I can see, there is no way to have ldap2pg ignore an AD group with no members.
I have tried "on_unexpected_dn" but that does not seem to work in this case.

@bersace
Copy link
Member

bersace commented Jul 2, 2019

Hi @andersbogsnes, thanks for the report.

on_unexpected_dn is meant to tell ldap2pg how to manage recursion, not missing attribute.

Could you add a filter that exclude entry without member attribute ? It's hard for ldap2pg to distinguish a missing attribute from a typo in attribute name.

@bersace bersace self-assigned this Jul 2, 2019
@andersbogsnes
Copy link
Author

@bersace Thanks for the swift reply!

I'm not sure how that filter would work - if the filter returns empty, then it will never have a member attribute and I would get the same error, no? Or does ldap2pg handle that case differently?

@bersace
Copy link
Member

bersace commented Jul 2, 2019

Hi @andersbogsnes , having (member=*) will return no entry instead of an incomplete set. This give me an idea. Maybe ldap2pg could automatically add LDAP filter, but that would defeat the debugging of mispelled attributes.

On empty set, ldap2pg just do nothing. As usual, ldap2pg drops managed roles not returned by directory.

Is it clearer for you ?

@andersbogsnes
Copy link
Author

Just to spell it out for someone not well-versed in ldap queries :-)

So I would change my ldap section to this?

  - ldap:
      base: "base_dn_goes_here"
      filter: "(&(CN=${ENV}_name_of_ad_group)(member=*))" # Replaced at runtime with ENV
    role:
      name: '{member.cn}'
      options: LOGIN SUPERUSER
      parent:
        - ldap_roles
        - owners
      on_unexpected_dn: warn

@bersace
Copy link
Member

bersace commented Jul 2, 2019

Yep, this looks good. Thanks for the sharing !

@andersbogsnes
Copy link
Author

Thanks for the help!

@meegoSVK
Copy link

meegoSVK commented Jun 26, 2020

Hello.
Would it be possible to implement that do nothing on empty group as you mentioned earlier?

Hi @andersbogsnes , having (member=*) will return no entry instead of an incomplete set. This give me an idea. Maybe ldap2pg could automatically add LDAP filter, but that would defeat the debugging of mispelled attributes.

On empty set, ldap2pg just do nothing. As usual, ldap2pg drops managed roles not returned by directory.

Is it clearer for you ?

I wanted to use ldap2pg to synchronize our AD with several postgresql engines.
We have a policy, that we have groups that are intentionaly empty on creation.
For example we do not want to delete users from applications/databases and preserve their groups.
But when someone leaves for example for maternity leave or. longer period of time from our comapy, he becomes disabled in AD and he becomes a member of group called group_database_disabled.
Members of these groups get rejected by pg_hba.conf
host database +database_disabled 0.0.0.0/0 reject

When I got YAML like:

- ldap:
    base: OU=server01,OU=PGSQL,OU=Groups,OU=Company,DC=ad,DC=company,DC=org
    filter: "(cn=group_pgsql_*)"
    scope: sub
    joins:
      member:
        filter: "(member=*)"
  roles:
  - names:
    - '{member.sAMAccountName}'
    options: LOGIN
    parent:
    - '{cn}'
    - ldap_users
    on_unexpected_dn: fail
    comment: "User account from AD -  User {member.displayName}."

everything works fine, until an empty group is found in LDAP - ldap2pg crashes because no member is there.

Unhandled error:
Traceback (most recent call last):
  File "/home/postgres/ldap2pg/.venv/lib64/python3.6/site-packages/ldap2pg/ldap.py", line 111, in get_attribute
    joined_entries = joins[attribute]
KeyError: 'member'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/postgres/ldap2pg/.venv/lib64/python3.6/site-packages/ldap2pg/script.py", line 94, in main
    exit(wrapped_main(config))
  File "/home/postgres/ldap2pg/.venv/lib64/python3.6/site-packages/ldap2pg/script.py", line 70, in wrapped_main
    count = manager.sync(syncmap=config['sync_map'])
  File "/home/postgres/ldap2pg/.venv/lib64/python3.6/site-packages/ldap2pg/manager.py", line 240, in sync
    ldaproles, ldapacl = self.inspect_ldap(syncmap)
  File "/home/postgres/ldap2pg/.venv/lib64/python3.6/site-packages/ldap2pg/manager.py", line 129, in inspect_ldap
    on_unexpected_dn=on_unexpected_dn,
  File "/home/postgres/ldap2pg/.venv/lib64/python3.6/site-packages/ldap2pg/manager.py", line 160, in build_format_vars
    for value in get_attribute(entry, field):
  File "/home/postgres/ldap2pg/.venv/lib64/python3.6/site-packages/ldap2pg/ldap.py", line 114, in get_attribute
    raise ValueError(msg)
ValueError: Missing join result for member
Please file an issue at https://github.com/dalibo/ldap2pg/issues with full log.

Whe I tried @andersbogsnes workaround without ${ENV}_name_of_ad_group it works only when any user is not in two groups.
In that case, both groups have all members of each of them even when they are not in both in AD.

YAML:

- ldap:
    base: OU=server01,OU=PGSQL,OU=Groups,OU=Company,DC=ad,DC=company,DC=org
    filter: "(&(cn=group_pgsql_*)(member=*))"
  roles:
  - names:
    - '{member.sAMAccountName}'
    options: LOGIN
    parent:
    - '{cn}'
    - ldap_users
    on_unexpected_dn: fail
    comment: "User account from AD -  User {member.displayName}."

Example:

AD group: CN=group_pgsql_database1_ro,OU=server01,OU=PGSQL,OU=Groups,OU=Company,DC=ad,DC=company,DC=org
Members: User_01, User_02
AD group: CN=group_pgsql_database1_ro,OU=server01,OU=PGSQL,OU=Groups,OU=Company,DC=ad,DC=company,DC=org
Members: User_03, User_04, User_02
AD group: CN=group_pgsql_database1_disabled,OU=server01,OU=PGSQL,OU=Groups,OU=Company,DC=ad,DC=company,DC=org
Members: none

Result in Postgres is:

User_01  |  | {ldap_users,group_pgsql_database1_ro,group_pgsql_database1_rw}
User_02  |  | {ldap_users,group_pgsql_database1_ro,group_pgsql_database1_rw}
User_03  |  | {ldap_users,group_pgsql_database1_ro,group_pgsql_database1_rw}
User_04  |  | {ldap_users,group_pgsql_database1_ro,group_pgsql_database1_rw}
group_pgsql_database1_disabled | Cannot login | {ldap_groups}
group_pgsql_database1_ro | Cannot login | {ldap_groups}
group_pgsql_database1_rw | Cannot login | {ldap_groups}

Could you please add an option to ldap2pg, something like you alredy have -> on_unexpected_dn?
It could be an option something like on_empty_group and it could be failing or skipping.

Thank you very much in advance!

ldap2pg -V
ldap2pg 5.4
psycopg2 2.8.4 (dt dec pq3 ext lo64)
python-ldap 3.2.0
Python 3.6.8 (default, Jun 11 2019, 15:15:01) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-39)]

@bersace
Copy link
Member

bersace commented Jun 26, 2020

Hi @meegoSVK . That's fair.

Just thinking loud. What do you think of something like:

sync_map:
- ldap:
    base: ...
    on_unexpected_dn: fail
    allow_missing_attributes: [member]

This is so common that this could be the default. Maybe with a warning message.

$ ldap2pg ...
...
INFO: Query LDAP to create superusers.                               
INFO: Querying LDAP ou=groups,dc=ldap,dc=lda... (cn=dba)...
WARN: No member attribute returned for cn=group_pgsql_.... Considering an empty list.
...

What do you think of this ? Both of you :-)

@bersace bersace reopened this Jun 26, 2020
@meegoSVK
Copy link

Hi @bersace
That would be awesome!

@andersbogsnes
Copy link
Author

Looks good to me! Seems like a very useful feature, and the warning is definitely a good way to avoid foot-guns!

Is there a flag for treating warnings as errors if the user wants to run in “strict mode” ala Sphinx?

@bersace
Copy link
Member

bersace commented Jun 26, 2020

Hi @andersbogsnes treating warnings as error is a good idea. I suggest you to open an issue or maybe complete this to #286

@bersace
Copy link
Member

bersace commented Jun 26, 2020

Hu, that's odd. There ldap2pg already defaults missing attributes to [].

https://github.com/dalibo/ldap2pg/blob/master/ldap2pg/manager.py#L59

@andersbogsnes
Copy link
Author

Hi @andersbogsnes treating warnings as error is a good idea. I suggest you to open an issue or maybe complete this to #286

Created #338

@meegoSVK
Copy link

meegoSVK commented Jul 8, 2020

Hi @bersace !

Thanks for implementing missing attributes.
But it looks like I am missing something :(

I tried using allow_missing_attributes, but I always get:

[ldap2pg.script       CRITI] Missing join result for member.

Here is YAML:

sync_map:
# First, setup static roles and grants
- roles:
  - names:
    - ldap_admins
    - ldap_users
    - ldap_groups
    options: NOLOGIN
    comment: "Groups created for LDAP synchronization."

# Now query LDAP to create roles and grant them privileges by parenting.
# Create admins
- ldap:
    base: OU=PGSQL,OU=Groups,OU=My_Domain,DC=ad,DC=mydomain,DC=info
    allow_missing_attributes: []
    filter: "(cn=sgrp_pgsql_admins)"
  role:
    name: '{member.cn}'
    options: LOGIN SUPERUSER CREATEROLE CREATEDB
    parent:
    - ldap_admins
    comment: "Superuser account from AD - User {member.displayName}"
    on_unexpected_dn: warn

# Create roles from specific OU
- ldap:
    base: OU=tst_ou,OU=PGSQL,OU=Groups,OU=My_Domain,DC=ad,DC=mydomain,DC=info
    allow_missing_attributes: []
    filter: "(cn=sgrp_pgsql_*)"
  role:
    name: '{cn}'
    options: NOLOGIN
    parent:
    - ldap_groups
    comment: "Mastered by Identity management"
    on_unexpected_dn: warn

# Fill roles from specific OU with users - if any
- ldap:
    base: OU=tst_ou,OU=PGSQL,OU=Groups,OU=My_Domain,DC=ad,DC=mydomain,DC=info
    allow_missing_attributes: [member]
    filter: "(cn=sgrp_pgsql_*)"
    scope: sub
    joins:
      member:
        allow_missing_attributes: []
        filter: "(objectClass=User)"
  role:
    name: '{member.sAMAccountName}'
    options: LOGIN
    parent:
    - '{cn}'
    - ldap_users
    on_unexpected_dn: warn
    comment: "User account from AD -  User {member.displayName}."

Here is output from verbose mode:

[ldap2pg.manager       INFO] Querying LDAP OU=tst_ou,OU=PGSQL,OU... (cn=sgrp_pgs...
[ldap2pg.ldap         DEBUG] Doing: ldapsearch -x -D CN=pgsqlsynchro,OU=Special_Users,DC=ad,DC=mydomain,DC=info -W -b OU=tst_ou,OU=PGSQL,OU=Groups,OU=My_Domain,DC=ad,DC=mydomain,DC=info -s sub '(cn=sgrp_pgsql_*)' member cn
[ldap2pg.manager      DEBUG] Got 6 entries from LDAP.
[ldap2pg.manager      WARNI] Missing 'member' from CN=sgrp_pgsql_tst_ou_app_1_disabled,OU=tst_ou,OU=PGSQL,OU=Groups,OU=My_Domain,DC=ad,DC=mydomain,DC=info. Considering it as an empty list.
[ldap2pg.manager      WARNI] Missing 'member' from CN=sgrp_pgsql_tst_ou_app_2_disabled,OU=tst_ou,OU=PGSQL,OU=Groups,OU=My_Domain,DC=ad,DC=mydomain,DC=info. Considering it as an empty list.
[ldap2pg.manager      WARNI] Missing 'member' from CN=sgrp_pgsql_tst_ou_app_3_disabled,OU=tst_ou,OU=PGSQL,OU=Groups,OU=My_Domain,DC=ad,DC=mydomain,DC=info. Considering it as an empty list.
[ldap2pg.manager       INFO] Sub-querying LDAP CN=USR115036,OU=ext...
[ldap2pg.ldap         DEBUG] Doing: ldapsearch -x -D CN=pgsqlsynchro,OU=Special_Users,DC=ad,DC=mydomain,DC=info -W -b CN=USR115036,OU=external,OU=Users,OU=My_Domain,DC=ad,DC=mydomain,DC=info -s sub '(objectClass=User)' displayName sAMAccountName
[ldap2pg.manager      DEBUG] Got 1 entries from LDAP.
[ldap2pg.manager      WARNI] Missing 'member' from CN=USR115036,OU=external,OU=Users,OU=My_Domain,DC=ad,DC=mydomain,DC=info. Considering it as an empty list.
[ldap2pg.manager       INFO] Sub-querying LDAP CN=USR115020,OU=ext...
[ldap2pg.ldap         DEBUG] Doing: ldapsearch -x -D CN=pgsqlsynchro,OU=Special_Users,DC=ad,DC=mydomain,DC=info -W -b CN=USR115020,OU=external,OU=Users,OU=My_Domain,DC=ad,DC=mydomain,DC=info -s sub '(objectClass=User)' displayName sAMAccountName
[ldap2pg.manager      DEBUG] Got 1 entries from LDAP.
[ldap2pg.manager      WARNI] Missing 'member' from CN=USR115020,OU=external,OU=Users,OU=My_Domain,DC=ad,DC=mydomain,DC=info. Considering it as an empty list.
[ldap2pg.manager       INFO] Sub-querying LDAP CN=USR115015,OU=ext...
[ldap2pg.ldap         DEBUG] Doing: ldapsearch -x -D CN=pgsqlsynchro,OU=Special_Users,DC=ad,DC=mydomain,DC=info -W -b CN=USR115015,OU=external,OU=Users,OU=My_Domain,DC=ad,DC=mydomain,DC=info -s sub '(objectClass=User)' displayName sAMAccountName
[ldap2pg.manager      DEBUG] Got 1 entries from LDAP.
[ldap2pg.manager      WARNI] Missing 'member' from CN=USR115015,OU=external,OU=Users,OU=My_Domain,DC=ad,DC=mydomain,DC=info. Considering it as an empty list.
[ldap2pg.manager       INFO] Sub-querying LDAP CN=USR00125,OU=Primary...
[ldap2pg.ldap         DEBUG] Doing: ldapsearch -x -D CN=pgsqlsynchro,OU=Special_Users,DC=ad,DC=mydomain,DC=info -W -b CN=USR00125,OU=Primary,OU=Users,OU=My_Domain,DC=ad,DC=mydomain,DC=info -s sub '(objectClass=User)' displayName sAMAccountName
[ldap2pg.manager      DEBUG] Got 1 entries from LDAP.
[ldap2pg.manager      WARNI] Missing 'member' from CN=USR00125,OU=Primary,OU=Users,OU=My_Domain,DC=ad,DC=mydomain,DC=info. Considering it as an empty list.
[ldap2pg.manager       INFO] Sub-querying LDAP CN=USR00123,OU=Primary...
[ldap2pg.ldap         DEBUG] Doing: ldapsearch -x -D CN=pgsqlsynchro,OU=Special_Users,DC=ad,DC=mydomain,DC=info -W -b CN=USR00123,OU=Primary,OU=Users,OU=My_Domain,DC=ad,DC=mydomain,DC=info -s sub '(objectClass=User)' displayName sAMAccountName
[ldap2pg.manager      DEBUG] Got 1 entries from LDAP.
[ldap2pg.manager      WARNI] Missing 'member' from CN=USR00123,OU=Primary,OU=Users,OU=My_Domain,DC=ad,DC=mydomain,DC=info. Considering it as an empty list.
[ldap2pg.manager      DEBUG] Want role USR115036 in LDAP.
[ldap2pg.manager      DEBUG] Want role USR115020 in LDAP.
[ldap2pg.manager      DEBUG] Want role USR115015 in LDAP.
[ldap2pg.manager      DEBUG] Want role USR00125 in LDAP.
[ldap2pg.manager      DEBUG] Want role USR00123 in LDAP.
[ldap2pg.script       CRITI] Missing join result for member.
[ldap2pg.psql         DEBUG] Closing Postgres connexion to libpq default.

This happens when it tires to query empty group.
I know, I can make special -ldap entry for every group, but this is what I want to avoid and have "automatized".

Thanks for help, and sorry for reopening old wounds :)

@meegoSVK
Copy link

meegoSVK commented Jul 14, 2020

Hi @bersace

I found out, that querying for {member.displayName} and {member.sAMAccountName} makes the problem.

So when I change the YAML to:

# Fill roles from specific OU with users - if any
- ldap:
    base: OU=tst_ou,OU=PGSQL,OU=Groups,OU=My_Domain,DC=ad,DC=mydomain,DC=info
    allow_missing_attributes: [member]
    filter: "(cn=sgrp_pgsql_*)"
    scope: sub
    joins:
      member:
        allow_missing_attributes: []
        filter: "(objectClass=User)"
  role:
    name: '{member.cn}'
    options: LOGIN
    parent:
    - '{cn}'
    - ldap_users
    on_unexpected_dn: warn
    comment: "User account from AD -  User."

Everything works fine.
But I am missing the displayName in comment, and I have to rely on that sAMAccountName is same as CN.

@bersace
Copy link
Member

bersace commented Jul 15, 2020

@meegoSVK I have a WIP code to fix the combination of member.* in several fields to make ensure more consistency. The sub-query feature had some consequences in formating combination that I didn't spotted early.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants