Research how document level security works with array values

[AlexRuiz7]

Description

All our indices include the field agent.groups to implement RBAC using document-level security. The main goal is to filter documents by roles, so only users with the appropriate role that grant permissions to see certain groups of agents are effectively allowed to.

The agent.groups field is an array of string, containing the groups' IDs the agent belong to. We are not sure about how document-level security behaves on array values. We need to explore this functionality to ensure our requirements for RBAC are met.

For example, a role that grants permissions to the user to see document that belong to the agent group "group2" should be able to see documents whose agent.groups value contain that group ID, for example, agent.groups: [group1, group2, group7].

[QU3B1M]

Currently doing research and reading ElasticSearch documentation. On an initial research and superficial analysis, I suspect this requirement could be satisfied using a DSL Script query, with something like:

{
  "bool": {
    "filter": {
      "script": {
        "script": """
            return 'test_group' in doc['agent.groups'].value;
        """
      }
    }
  }
}

[QU3B1M]

Found and tested two different methods to apply a document-level security that satisfies the requirement, one approach is using TLQs, and the other with Script query.

In this POC I've used the following index

{
  "settings": {
    "index": {
      "number_of_shards": 1,
      "number_of_replicas": 1
    }
  },
  "mappings": {
    "properties": {
      "title": {
        "type": "text"
      },
      "content": {
        "type": "text"
      },
      "groups": {
        "type": "keyword"
      }
    }
  }
}

With this docs

{"title":"test_1","content":"This will have all groups","groups":["groupA","groupB","groupC"]}
{"title":"test_2","content":"This will have only group C","groups":["groupC"]}
{"title":"test_3","content":"This will have groups B and C ","groups":["groupB","groupC"]}
{"title":"test_4","content":"This will have groupA and groupC","groups":["groupA","groupC"]}
{"title":"test_5","content":"This will have groupB","groups":["groupB"]}

• Create the index

  curl -XPUT "https://localhost:9200/content" -u admin:admin --insecure -H 'Content-Type: application/json' -d'
  {
    "settings": {
      "index": {
        "number_of_shards": 1,
        "number_of_replicas": 1
      }
    },
    "mappings": {
      "properties": {
        "title": {
          "type": "text"
        },
        "content": {
          "type": "text"
        },
        "groups": {
          "type": "keyword"
        }
      }
    }
  }'

• Populate the index

  curl -XPOST "https://localhost:9200/_bulk" -u admin:admin --insecure -H 'Content-Type: application/json' -d'
  {"index":{"_index":"content"}
  {"title":"test_1","content":"This will have all groups","groups":["groupA","groupB","groupC"]}
  {"index":{"_index":"content"}
  {"title":"test_2","content":"This will have only group C","groups":["groupC"]}
  {"index":{"_index":"content"}
  {"title":"test_3","content":"This will have groups B and C ","groups":["groupB","groupC"]}
  {"index":{"_index":"content"}
  {"title":"test_4","content":"This will have groupA and groupC","groups":["groupA","groupC"]}
  {"index":{"_index":"content"}
  {"title":"test_5","content":"This will have groupB","groups":["groupB"]}
  '

Document-level security role with Term-level lookup query

This query method search for documents that contains a specific term, in this case the term is groupB

DLS Role

{
  "cluster_permissions": [
    "*"
  ],
  "index_permissions": [{
    "index_patterns": [
      "content*"
    ],
    "dls": "{\"terms_set\": {\"groups\": {\"terms\": [\"groupB\"], \"minimum_should_match_script\": {\"source\": \"1\"}}}}",
    "allowed_actions": [
      "read"
    ]
  }]
}

Response of a user with that role applied

...
"hits": [
    {"_index": "content","_id": "CoAj-JIBxjld0LZtKnDu","_score": 2.0,"_source": {"title": "test_1","content": "This will have all groups","groups": ["groupA","groupB","groupC"]}},
    {"_index": "content","_id": "DIAj-JIBxjld0LZtKnDv","_score": 2.0,"_source": {"title": "test_3","content": "This will haver groups B and C ","groups": ["groupB","groupC" ] } },
    {"_index": "content","_id": "DoB1-JIBxjld0LZtdHC0","_score": 2.0,"_source": {"title": "test_5","content": "This will have groupB","groups": ["groupB"]}}
]
...

Full evidence

• Create test user

  curl -XPUT "https://localhost:9200/_opendistro/_security/api/internalusers/testie" -u admin:admin --insecure -H 'Content-Type: application/json' -d'
  {
    "password": "TestPassword00!"
  }'
  {"status":"CREATED","message":"'testie' created."}

• Create role

  curl -XPUT "https://localhost:9200/_plugins/_security/api/roles/testrole" -u admin:admin --insecure -H 'Content-Type: application/json' -d'
  {
    "cluster_permissions": [
      "*"
    ],
    "index_permissions": [{
      "index_patterns": [
        "content*"
      ],
      "dls": "{\"terms_set\": {\"groups\": {\"terms\": [\"groupB\"], \"minimum_should_match_script\": {\"source\": \"1\"}}}}",
      "allowed_actions": [
        "read"
      ]
    }]
  }'
  {"status":"CREATED","message":"'testrole' created."}

• Map role with user

  curl -XPUT "https://localhost:9200/_plugins/_security/api/rolesmapping/testrole" -u admin:admin --insecure -H 'Content-Type: application/json' -d'
  {
    "users": [
      "testie"
    ]
  }'
  {"status":"CREATED","message":"'testrole' created."}

• Read the documents with the created user

  curl -XGET https://localhost:9200/content/_search -u testie:TestPassword00! --insecure
  {
      "took": 978,
      "timed_out": false,
      "_shards": {
          "total": 1,
          "successful": 1,
          "skipped": 0,
          "failed": 0
      },
      "hits": {
          "total": {
              "value": 3,
              "relation": "eq"
          },
          "max_score": 2.0,
          "hits": [
              {
                  "_index": "content",
                  "_id": "CoAj-JIBxjld0LZtKnDu",
                  "_score": 2.0,
                  "_source": {
                      "title": "test_1",
                      "content": "This will have all groups",
                      "groups": [
                          "groupA",
                          "groupB",
                          "groupC"
                      ]
                  }
              },
              {
                  "_index": "content",
                  "_id": "DIAj-JIBxjld0LZtKnDv",
                  "_score": 2.0,
                  "_source": {
                      "title": "test_3",
                      "content": "This will haver groups B and C ",
                      "groups": [
                          "groupB",
                          "groupC"
                      ]
                  }
              },
              {
                  "_index": "content",
                  "_id": "DoB1-JIBxjld0LZtdHC0",
                  "_score": 2.0,
                  "_source": {
                      "title": "test_5",
                      "content": "This will have groupB",
                      "groups": [
                          "groupB"
                      ]
                  }
              }
          ]
      }
  }

Document-level security role with Script Query

In this case we filter the documents by checking the field "groups" contains a value "groupA".

DLS Role

{
  "cluster_permissions": [
    "*"
  ],
  "index_permissions": [{
    "index_patterns": [
      "content*"
    ],
    "dls": "{\"bool\": {\"filter\": [{\"script\": {\"script\": \"return doc[\\\"groups\\\"].contains(\\\"groupA\\\");\"}}]}}",
    "allowed_actions": [
      "read"
    ]
  }]
}

Response of a user with that role applied

...
"hits": [
    {"_index": "content","_id": "CoAj-JIBxjld0LZtKnDu","_score": 2.0,"_source": {"title": "test_1","content": "This will have all groups","groups": ["groupA", "groupB", "groupC"]}},
    {"_index": "content","_id": "DYBy-JIBxjld0LZtrXAG","_score": 2.0,"_source": {"title": "test_4","content": "This will have groupA and groupC","groups": ["groupA", "groupC"]}}
]
...

Full evidence

• Create role

  curl -XPUT "https://localhost:9200/_plugins/_security/api/roles/testrolescript" -u admin:admin --insecure -H 'Content-Type: application/json' -d'
  {
    "cluster_permissions": [
      "*"
    ],
    "index_permissions": [{
      "index_patterns": [
        "content*"
      ],
      "dls": "{\"bool\": {\"filter\": [{\"script\": {\"script\": \"return doc[\\\"groups\\\"].contains(\\\"groupA\\\");\"}}]}}",
      "allowed_actions": [
        "read"
      ]
    }]
  }'

• Create new test user

  curl -XPUT "https://localhost:9200/_opendistro/_security/api/internalusers/testie2" -u admin:admin --insecure -H 'Content-Type: application/json' -d'
  {
    "password": "TestPassword00!"
  }'

• Map role with user

   curl -XPUT "https://localhost:9200/_plugins/_security/api/rolesmapping/testrolescript" -u admin:admin --insecure -H 'Content-Type: application/json' -d'
  {
    "users": [
      "testie2"
    ]
  }'

• Read index using the created user

  curl -XGET https://localhost:9200/content/_search -u testie2:TestPassword00! --insecure
  {
      "took": 2,
      "timed_out": false,
      "_shards": {
          "total": 1,
          "successful": 1,
          "skipped": 0,
          "failed": 0
      },
      "hits": {
          "total": {
              "value": 2,
              "relation": "eq"
          },
          "max_score": 2.0,
          "hits": [
              {
                  "_index": "content",
                  "_id": "CoAj-JIBxjld0LZtKnDu",
                  "_score": 2.0,
                  "_source": {
                      "title": "test_1",
                      "content": "This will have all groups",
                      "groups": [
                          "groupA",
                          "groupB",
                          "groupC"
                      ]
                  }
              },
              {
                  "_index": "content",
                  "_id": "DYBy-JIBxjld0LZtrXAG",
                  "_score": 2.0,
                  "_source": {
                      "title": "test_4",
                      "content": "This will have groupA and groupC",
                      "groups": [
                          "groupA",
                          "groupC"
                      ]
                  }
              }
          ]
      }
  }

[QU3B1M]

Exclusive filtering method

Using the script query, we can filter the results by excluding matches that belong to other groups apart from the one we want to filter.

The rule looks like this:

{
  "cluster_permissions": [
    "*"
  ],
  "index_permissions": [{
    "index_patterns": [
      "content*"
    ],
    "dls": "{\"bool\": {\"filter\": [{\"script\": {\"script\": \"return doc[\\\"groups\\\"]==[\\\"groupA\\\"];\"}}]}}",
    "allowed_actions": [
      "read"
    ]
  }]
}

Search returns:

curl -XGET https://localhost:9200/content/_search -u testing:TestPassword00! --insecure

{"took":3,"timed_out":false,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0},"hits":{"total":{"value":1,"relation":"eq"},"max_score":2.0,"hits":[{"_index":"content","_id":"usiSBpMB_D-360vHBCBO","_score":2.0,"_source":{"title":"test_1","content":"This will have only groupA","groups":["groupA"]}}]}}

[gdiazlo]

What if groups is an array of objects, inside other object, like:

// Object as defined in ECS plus new groups object, plus nested host object
"agent": {
  "id": "2887e1cf-9bf2-431a-b066-a46860080f56",
  "type": "endpoint",
  "version": "5.0.0",
  // nested host object, as defined in ECS
    "host": {
        "hostname": "myhost",
        "os": {
            "name": "Amazon Linux 2",
            "platform": "Linux",
        },
        "ip": "192.168.1.2"
        "architecture": "x86_64",
    }
    // new groups object not present in ECS
    "groups": [
        {"name": "My Group"},
        {"name": "Another Group"}
    ],
}