Cross domain queries (same Active Directory, different "forest")

[arjan-saly-tfs]

Hi,

So, we have different domains in our organization, like:

1. ldap://domain1.company.com
2. ldap://domain2.company.com
   In Active Directory these are defined as different "forests" (I'm not an AD expert, just writing what our expert told me to....).

Regardless of which domain I'm in, logging in to services normally works.

But when querying using ldapsearch, I run into an odd problem:

• When I connect to ldap server domain1 with a user from that same domain ("domain1user"), I can query all details from groups in that same domain (domain1), but for groups that are in the other domain (domain2), I don't get member details.
• This ldapsearch connection works fine both using the -H -D switches and the -Y DIGEST-MD5 -U switches
• But when I try to connect with the same user (domain1user) on domain2 the odd stuff occurs:
  -- I can connect and search using the -H -D switches and then all works fine, but that doesn't work for ldap2pg
  -- When I connect and search using the -Y DIGEST-MD5 -U switches, like ldap2pg does, get this error:
  SASL/DIGEST-MD5 authentication started ldap_sasl_interactive_bind_s: Invalid credentials (49) additional info: 8009030C: LdapErr: DSID-0C09058A, comment: AcceptSecurityContext error, data 52e, v4563

Interesting enough, the same issue occurs the other way around. So, when using a user on domain2 ("domain2user"), all works fine on domain2, but I get the same error when trying to connect to domain1. In this case, I can query all details for domain2 groups, but don't get members from domain1 groups.

As we need to synchronize members from both domains, how do I get around this?
Do I really need to create as many AD users as I have domains? Or do you have a different solution?

Note: We're not bound to -Y DIGEST-MD5, -H -D would be fine as well, but I don't know how to configure that in ldap2pg (separate ticket, not being replied on....)

Thanks in advance!

[bersace]

Hi @arjan-saly-tfs . Interesting situation. I think that forest are more common than one may think because of enterprises bought by another.

Does some users have some accounts in the both trees ? What do you think of having two synchronisation, one for each tree of the forest ?

I need to investigate deeper in you use case.

Note: We're not bound to -Y DIGEST-MD5, -H -D would be fine as well, but I don't know how to configure that in ldap2pg (separate ticket, not being replied on....)

You mean #407 ? I'll take an eye on this too.

[arjan-saly-tfs]

Hi @bersace,
Thanks for the quick reply. As far as I know, users don't have accounts in both trees. The forests are more on a geographical base.

We considered separate synchronization for each tree with different search users. It is an last resort option, but not the a preferred one. This is because we're facing more than two trees. Not sure how many we have or need to query, at least 2 probably 3, possibly more. Also, we want to use different search users for the different DTAP-environments (Develop, Test, Acceptance, Production), as well as database uses (different clusters).
When that's multiplied by the amount of trees we have, we'll end up with a pretty large amount of accounts.

This is why I would prefer to do all synchronization using one user.

Hope you can find a solution that can do with a single user.

And yeah, I meant #407 ...

[arjan-saly-tfs]

Update: your answer provided in #407 does let me login to the two different domains using the same search user and binddn.
Still puzzles me why I can connect to the "other domain" using binddn and not using SASL.

However, now I became aware of another issue: In the config, the uri is defined, directing to domain1.
For my groups in domain2, I need to define another uri to be used for that query as it needs to be executed on domain2.

Is that possible?

This is what I already tried:

• specifying the uri's for both domains in one uri:-line, but then it uses the first one and ignores the second.
• Adding an additional set of ldap & sync_map sections for the second domain, after the first sync_map section. Then it uses he uri specified in the last ldap section.....

[bersace]

Update: your answer provided in #407 does let me login to the two different domains using the same search user and binddn. Still puzzles me why I can connect to the "other domain" using binddn and not using SASL.

It looks like SASL is bound to the domain. I'm wondering if ldap2pg could handle multiple LDAP connection like it handle a connection to each database on Postgres. I mean every parameters are the same, just the domain may vary like a database. I have absolutely no experience like this. That would be good to have simple setup to test this.

However, now I became aware of another issue: In the config, the uri is defined, directing to domain1. For my groups in domain2, I need to define another uri to be used for that query as it needs to be executed on domain2.

By URI, do you mean LDAPURI parameter ? Can you share some redacted ldap2pg.yaml ? It's hard to get it clear.

[arjan-saly-tfs]

Here's a redacted example from the ldap2pg.yml I used to clarify the uri. I meant the "uri:" parameter in the ldap: section.

For the first run on domain1, this would be the config:

#########################################
ldap:
  uri: ldap://domain1.company.com
  binddn: CN=username,OU=Users,OU=Global,DC=domain1,DC=company,DC=com
  #For SASL
  user: username
  password: somepassword
######################################### 

For the second run on domain2, this would be the config

#########################################
ldap:
  uri: ldap://domain2.company.com
  binddn: CN=username,OU=Users,OU=Global,DC=domain1,DC=company,DC=com
  #For SASL
  user: username
  password: somepassword
######################################### 

Using the same username for both.

Given new insights provided by other tickets, I wonder if binddn: is relevant here, as this issue only occurs when using SASL.

Recently found new information: When running the same config using the docker version, I get the same error but now for both configs.
So, when executing ldap2pg on the linux commandline using the SASL configuration, all works fine when connecting to the LDAP domain where the user is defined, but the error occurs when connecting to another domain. Even though the user does have access to that domain.

When executing ldap2pg from its docker container, I get the same message on both domains. Hence, even when connecting to the domain where the user is defined.

Note that this issue is very related to #406, handling referrals. If ldap2pg can handle referrals by itself, cross domain queries probably will be resolved automatically. On the other hand, this issue may be in the way of handling referrals....

[bersace]

Given new insights provided by other tickets, I wonder if binddn: is relevant here, as this issue only occurs when using SASL.

binddn is unused with SASL. That would be good if ldap2pg logs about this. You'd better not defining it when using SASL to avoid confusion.

Recently found new information: When running the same config using the docker version, I get the same error but now for both configs.

Can you share a redacted verbose output of this execution ?

So, when executing ldap2pg on the linux commandline using the SASL configuration, all works fine when connecting to the LDAP domain where the user is defined, but the error occurs when connecting to another domain. Even though the user does have access to that domain.

Note that this issue is very related to #406, handling referrals. If ldap2pg can handle referrals by itself, cross domain queries probably will be resolved automatically. On the other hand, this issue may be in the way of handling referrals....

Yes. For this issue, do you need to specify a different server for each ldapsearches ? Something like this :

sync_map:
- ldapsearch:
    uri: ldap://domain1.company.com
    basedn: ...
  roles: "{cn}"

[arjan-saly-tfs]

Recently found new information: When running the same config using the docker version, I get the same error but now for both configs.

Can you share a redacted verbose output of this execution ?

$ sudo docker run --rm -v ${PWD}/ldap2pg.emea.yml:/workspace/ldap2pg.yml -e VERBOSITY=DEBUG dalibo/ldap2pg
[ldap2pg.config       DEBUG] Trying ./ldap2pg.yml.
[ldap2pg.config        INFO] Using /workspace/ldap2pg.yml.
[ldap2pg.config       DEBUG] Read verbosity from argv.
[ldap2pg.config       DEBUG] Read version from YAML.
[ldap2pg.config       DEBUG] Read ldap:uri from YAML.
[ldap2pg.config       DEBUG] Read ldap:user from YAML.
[ldap2pg.config       DEBUG] Read ldap:password from YAML.
[ldap2pg.config       DEBUG] Read postgres:dsn from YAML.
[ldap2pg.config       DEBUG] Read postgres:databases_query from YAML.
[ldap2pg.config       DEBUG] Read postgres:owners_query from YAML.
[ldap2pg.config       DEBUG] Read postgres:managed_roles_query from YAML.
[ldap2pg.config       DEBUG] Read postgres:roles_blacklist_query from YAML.
[ldap2pg.config       DEBUG] Read postgres:schemas_query from YAML.
[ldap2pg.config       DEBUG] Read privileges from YAML.
[ldap2pg.config       DEBUG] Read sync_map from YAML.
[ldap2pg.config       DEBUG] Drop unused hidden privilege __default_execute_on_functions__
[ldap2pg.config       DEBUG] Drop unused hidden privilege __execute_on_all_functions__
[ldap2pg.config       DEBUG] Drop unused hidden privilege __global_default_execute_on_functions__
[ldap2pg.config       DEBUG] Configuration loaded.
[ldap2pg.script       DEBUG] Setting up LDAP client.
[ldap2pg.ldap         DEBUG] Ignoring: [Errno 2] No such file or directory: '/etc/openldap/ldap.conf'
[ldap2pg.ldap         DEBUG] Ignoring: [Errno 2] No such file or directory: '/etc/ldap/ldap.conf'
[ldap2pg.ldap         DEBUG] Ignoring: [Errno 2] No such file or directory: '/root/ldaprc'
[ldap2pg.ldap         DEBUG] Ignoring: [Errno 2] No such file or directory: '/root/.ldaprc'
[ldap2pg.ldap         DEBUG] Ignoring: [Errno 2] No such file or directory: '/workspace/ldaprc'
[ldap2pg.ldap         DEBUG] Read PASSWORD from env.
[ldap2pg.ldap         DEBUG] Read URI from YAML.
[ldap2pg.ldap         DEBUG] Read USER from YAML.
[ldap2pg.ldap         DEBUG] Read PASSWORD from YAML.
[ldap2pg.ldap         DEBUG] Connecting to LDAP server ldap://emea.company.com.
[ldap2pg.ldap         DEBUG] Trying SASL DIGEST-MD5 auth.
[ldap2pg.ldap         DEBUG] Authenticating: ldapwhoami -Y DIGEST-MD5 -U username -W
[ldap2pg.script       CRITI] Failed to connect to LDAP: {'desc': 'Unknown authentication method', 'errno': 22, 'info': 'SASL(-4): no mechanism available: No worthy mechs found'}

[arjan-saly-tfs]

sync_map:
- ldapsearch:
  uri: ldap://domain1.company.com>
  basedn: ...
 roles: "{cn}"

Unfortunately this doesn't seem to work:

[ldap2pg.script       CRITI] Failed to query LDAP: {'info': u'Referral:\nldap://domain2.company.com/OU=searchOU1,OU=searchOU2,DC=domain2,DC=company,DC=com', 'msgid': 4, 'msgtype': 101, 'result': 10, 'desc': u'Referral', 'ctrls': []}.

[bersace]

sync_map:
- ldapsearch:
  uri: ldap://domain1.company.com>
  basedn: ...
 roles: "{cn}"

Unfortunately this doesn't seem to work:

[ldap2pg.script       CRITI] Failed to query LDAP: {'info': u'Referral:\nldap://domain2.company.com/OU=searchOU1,OU=searchOU2,DC=domain2,DC=company,DC=com', 'msgid': 4, 'msgtype': 101, 'result': 10, 'desc': u'Referral', 'ctrls': []}.

The sample YAML was a proposition of a new uri ldapsearch parameter in YAML. Does this allow you to express what you mean by cross-domain queries ?

[arjan-saly-tfs]

Ah, OK, sorry for misunderstanding.

Does this allow you to express what you mean by cross-domain queries ?
Yes, it does.

Might want to take it one step further though by enabling to specify binddn/userid/password as well?

[bersace]

$ sudo docker run --rm -v ${PWD}/ldap2pg.emea.yml:/workspace/ldap2pg.yml -e VERBOSITY=DEBUG dalibo/ldap2pg

...

[ldap2pg.script CRITI] Failed to connect to LDAP: {'desc': 'Unknown authentication method', 'errno': 22, 'info': 'SASL(-4): no mechanism available: No worthy mechs found'}

I confirm the bug. The docker image miss some sasl2 modules. I'll patch this.

[bersace]

@arjan-saly-tfs latest docker image has SASL modules, fixing Unknown authentication method error.

[arjan-saly-tfs]

@bersace I tested the new docker image and it SASL now works fine indeed. Thanks for the quick fix!
Unfortunately with the same domain/forest limitations as I run into with the dockerless version.

[bersace]

@bersace I tested the new docker image and it SASL now works fine indeed. Thanks for the quick fix! Unfortunately with the same domain/forest limitations as I run into with the dockerless version.

That's expected. Thanks for the confirmation.

This is a new feature. I'm ok to add multi-ldap-server support in ldap2pg as long as it's the same auth for all servers.

[bersace]

That feature is a good step toward #406

[arjan-saly-tfs]

This is a new feature. I'm ok to add multi-ldap-server support in ldap2pg as long as it's the same auth for all servers.
That feature is a good step toward #406

As stated before, I'm not an AD or LDAP expert. But based on what I've learnt the past week on this subject, multi-ldap-server support mainly seems to make sense as an workaround for #406 . Especially if it is "limited" to the same auth for all servers. I do understand this limitation by the way as that'll be the situation for #406 as well.

So, if adding multi-ldap-server is really is a good and useful step towards #406 , it would be a good addition. However, if that would mean it will take you longer go reach #406, perhaps focusing on #406 would be more appropriate.

[arjan-saly-tfs]

For your additional information: This week we made the design choice to step away from existing AD groups, define dedicated AD groups for the ldap2pg sync and keep those all in one ldap domain. I have already seen that when a group in domain1 holds members from domain2, they're synchronized fine, This means that for now, there is no urgent need anymore for synchronizing from multiple ldap domains.

As I can't predict the future, I think #406 remains a useful addition for future implementations. But at this stage, the urgent need for this to be solved is gone from my end.

Thanks for bringing me this far as we now are able to setup a useful standard of synchronizing AD with PostgreSQL thanks to your tool and your help!

Please let me know if you want me to test new features like #406 !

[bersace]

So, if adding multi-ldap-server is really is a good and useful step towards #406 , it would be a good addition. However, if that would mean it will take you longer go reach #406, perhaps focusing on #406 would be more appropriate.

I need small steps to implement such big feature. Long PR is bad for managing the project. This is why I want #409 before #406. Maybe with a release in-between.

[bersace]

For your additional information: This week we made the design choice to step away from existing AD groups, define dedicated AD groups for the ldap2pg sync and keep those all in one ldap domain. I have already seen that when a group in domain1 holds members from domain2, they're synchronized fine, This means that for now, there is no urgent need anymore for synchronizing from multiple ldap domains.

This is very wise. Provide some kind of LDAP *view" server side is way better : it make synchronization much much faster and simpler to design and configure.

As I can't predict the future, I think #406 remains a useful addition for future implementations. But at this stage, the urgent need for this to be solved is gone from my end.

That's very good news for you. I can't warranty delay when contributing to ldap2pg until we have a deal. Dalibo offer 20% of my work time to contribute to the community, that's very fair but it take time.

Thanks for bringing me this far as we now are able to setup a useful standard of synchronizing AD with PostgreSQL thanks to your tool and your help!

Yep, I adopted these features in the roadmap. But expect this in a few monthes.

[bersace]

Did you try memberOf filter ?