Loading TOC...
Security Guide (PDF)

MarkLogic Server 11.0 Product Documentation
Security Guide — Chapter 6

« Previous chapter
Next chapter »

Element Level Security

MarkLogic Server includes element level security, an addition to the security model that allows you to specify more complex security rules on specific elements in documents. The feature also can be applied to JSON properties in a document. Using element level security, parts of a document may be concealed from users who do not have the appropriate roles to view them. Users without appropriate permissions cannot view the secured element or JSON property using XPath expressions or queries. Element level security can conceal the XML element (along with properties and attributes) or JSON property so that it does not appear in any searches, query plans, or indexes, unless accessed by a user with a role included in query roleset.

Element level security protects elements or JSON properties in a document using a protected path, where the path to an element or property within the document is protected so that only roles belonging to a specific query roleset can view the contents of that element or property. Only users with specific roles that match the specific query roleset can view the elements or properties protected by element level security. You can set protection with element level security to conceal a document's sensitive contents in real time, and also control which contents can be viewed and/or updated by other users.

See Interactions with Other MarkLogic Features for details about using element level security with SQL and semantic queries.

Permissions on an element or property are similar to permissions defined on a document. Elements or properties may contain all supported datatypes. Search results and update built-ins will honor the permissions defined at the element level. Element level security is applied consistently across all areas of the MarkLogic Server, including reads, updates, query plans, etc.

The protected paths are in the form of XPath expressions (not fields) that specify that an XML element or JSON property is part of a protected path. You will need to install or upgrade to MarkLogic 9.0-1 or later to use element level security.

This chapter describes element level security and includes the following topics:

Understanding Element Level Security

Elements of a document can be protected from being viewed as part of a query or XPath expression, or from being updated by a user, unless that user has the appropriate role. You specify that an element is part of a protected path by adding the path to the Security database. You also then add the appropriate role to a query roleset, which is also added to the Security database.

Element level security uses query rolesets to determine which elements will appear in query results. If a query roleset does not exist with the associated role that has permissions on the path, the role cannot view the contents of that path.

A user with admin privileges can access documents with protected elements by using fn:doc to retrieve documents (instead of using a query). To see protected elements as part of query results, however, a user needs the appropriate role(s).

Example --Element Level Security

This section describes a scenario using element level security. The scenario is not meant to demonstrate the correct way to set up element level security, as your situation is likely to be unique. However, it demonstrates how element level security works and may give you ideas for how to implement your own security model. You will need access to both MarkLogic Admin Interface and Query Console. Install or upgrade to MarkLogic Server 9.0-x or later prior to starting the example.

Description:

For a MarkLogic application used by a department, certain parts of documents may be hidden so that only users with the correct role may view or update those parts of the document. Users without the proper role will not be able to see the element concealed by the protected path.

To set up the element level security for this scenario, you will follow these steps:

Create Roles

Using the Admin Interface, create the roles as follows. You will create two roles, els-role-1 and els-role-2.

  1. In the Admin UI, click Security in the left tree menu.
  2. Click Roles and then click the Create tab.
  3. On the Role Configuration page, enter the information for the first role: role name: els-role-1description: els role 1

  4. Click ok to save the role.
  5. Repeat these steps to create the second role (els-role-2, els role 2)

See Roles in the Administrator's Guide for details about creating roles.

Create Users and Assign Roles

Now create three users (els-user-1, els-user-2, and els-user-3) using the Admin UI. Assign roles to two of the users.

  1. In the Admin UI, click Security in the left tree menu.
  2. Click Users and then click Create.
  3. On the User Configuration page, enter the information for the first user: user name: els-user-1 description: ELS user 1 password: <password>

    Enter a password of your choice.

Add this user to the first role that you created (els-role-1):

  1. Scroll down the User Configuration page until you see the els-role-1 role you just created.
  2. Click the box next to els-role-1 to assign the role to the user.

  3. Click ok to save your changes.

Repeat these steps to create a second user and third user (els-user-2, ELS user 2, els-user-3, ELS user 3). Assign roles to the users as shown. ELS user 3 will not have an assigned role.

See Users in the Administrator's Guide for details on creating users.

Admin users must be added to a role in order to view the results of a query on protected paths that involve concealed elements.

Add the Documents

For our simple example, we will use three documents, two in XML and one in JSON. Use the Query Console to insert these documents into the Documents database, along with read and update permissions for els-user-1 and els-user-2:

(: run this against the Documents database :)

xquery version "1.0-ml";
import module namespace sec="http://marklogic.com/xdmp/security" 
  at "/MarkLogic/security.xqy";

xdmp:document-insert("test1.xml", 
<root>
  <bar baz="1" attr="test">abc</bar>
  <bar baz="2">def</bar>
  <bar attr="test1">ghi</bar>
</root>,
(xdmp:permission("els-role-1", "read"), xdmp:permission("els-role-2", "read"), xdmp:permission("els-role-1", "update"),
xdmp:permission("els-role-2", "update")))
,
xdmp:document-insert("test2.xml",
<root>
  <reg expr="this is a string">1</reg>
  <reg>2</reg>
</root>,
(xdmp:permission("els-role-1", "read"), xdmp:permission("els-role-2", "read"), xdmp:permission("els-role-1", "update"),
xdmp:permission("els-role-2", "update")))
,
xdmp:document-insert("test1.json", object-node { 
"foo" : 1, "bar" : "2",  "baz" : object-node 
{"bar" : array-node {3,4}, "test" : 5} 
}, 
(xdmp:permission("els-role-1", "read"), xdmp:permission("els-role-2", "read"), xdmp:permission("els-role-1", "update"),
xdmp:permission("els-role-2", "update")))

The code example adds permissions to the documents for els-role-1 and els-role-2 while inserting them into the database.

Add Protected Paths and Query Rolesets

Using the Admin UI, add the protected paths and query rolesets to the Security database. If no query rolesets are configured, a query will only match documents by the terms that are visible to everyone.

To start, check for any existing protected paths using this query in the Query Console:

(: run this query against the Security database :)

fn:collection("http://marklogic.com/xdmp/protected-paths")

This will return an empty sequence if there are no protected paths. If there are protected paths, information about those protected paths will be displayed, including the path ID, the path expression, the permissions, and roles associated with that path.

Using the Admin UI, add protected paths with permissions for els-user-2. To add the protected path from the Admin UI:

  1. Click Security in the left tree menu.
  2. Click Protected Paths and then click the Create tab.
  3. Enter the path expression for the first path (/root/bar[@baz=1]),with read permissions for els-role-2.

  4. Click ok when you are done. Since there are no namespaces in these examples, the prefix and namespace are not required for the protected path.

For examples using namespaces and prefixes as part of a protected path, see Namespaces as Part of a Protected Path.

Repeat this for two additional protected paths, test and /root/reg[fn:matches(@expr, 'is')].

The three protected paths with read permissions for els-role-2 are:

/root/bar[@baz=1]
test
/root/reg[fn:matches(@expr, 'is')]

Alternatively, you can add these protected paths with the Query Console. Use this code to add these protected paths with permissions for els-user-2 to the Security database:

(: add protected paths -> run against the Security database :)

xquery version "1.0-ml";
import module namespace sec="http://marklogic.com/xdmp/security" 
  at "/MarkLogic/security.xqy";

sec:protect-path("/root/bar[@baz=1]", (), (xdmp:permission("els-role-2", "read"))),
sec:protect-path("test", (), (xdmp:permission("els-role-2", "read"))),
sec:protect-path("/root/reg[fn:matches(@expr, 'is')]", (), (xdmp:permission("els-role-2", "read")))

=> Returns three numbers representing the protected paths

Adding, unprotecting, or changing permissions on protected paths will trigger reindexing. This reindexing will only apply to documents that include or match the paths.

Now add query rolesets for these documents. In the Query Console, run this code to add query rolesets for els-user-2:

(: run this against the Security database :)

xquery version "1.0-ml";
import module namespace sec = "http://marklogic.com/xdmp/security" 
  at "/MarkLogic/security.xqy";

let $qry := 'xdmp:database-node-query-rolesets(fn:doc(), ("all"))'
let $qry-rolesets := 
xdmp:eval($qry, (),<options xmlns="xdmp:eval">
                   <database>{xdmp:database('Documents')}</database>
                 </options>)
return
sec:add-query-rolesets($qry-rolesets)

In most cases you will want to use the helper functions (xdmp:database-node-query-rolesets and xdmp:node-query-rolesets) to create query rolesets. The helper function automatically created the query rolesets based on the protected paths you have set. See Helper Functions for Query Rolesets for more information. To understand more about query rolesets, see Query Rolesets.

You can also can add query rolesets manually with XQuery in the Query Console if you only have a few query rolesets to add. Use this code, checking to be sure you are running it against the Security database:

(: add query rolesets => run against the Security database :)

xquery version "1.0-ml";
import module namespace sec="http://marklogic.com/xdmp/security" 
  at "/MarkLogic/security.xqy";

let $roleset := sec:query-roleset("els-role-2")
return
sec:add-query-rolesets(sec:query-rolesets($roleset))
=>
Returns a unique ID representing the added query rolesets

Adding query rolesets does not trigger reindexing, since it is only used by queries.

Check for query rolesets in the Security database using the Query Console:

(: run this query against the Security database :)

fn:collection("http://marklogic.com/xdmp/query-rolesets")
=>
Returns details about query rolesets in the Security database.

There is also a collection for protected paths in the Security database:

(: run this query against the Security database :)

fn:collection("http://marklogic.com/xdmp/protected-paths")
=>
Returns details about protected paths in the Security database.

The els-role-2 can now see the elements in these paths, but the els-user-1 cannot:

test
/root/bar[@baz=1]
/root/reg[fn:matches(@expr, 'is')]

Run the Example Queries

This section includes examples in both XQuery and JavaScript. Run the following queries in the Query Console. For simplicity, the sample queries use xdmp:eval and xdmp:get-current-user (or xdmp.eval and xdmp.getCurrentUser) to execute a query in the context of each user. Different elements and properties in a document are concealed for the different roles. Notice the different types of queries, using either XQuery or JavaScript, that are used to search for content.

These examples assume that you have access permissions for both the MarkLogic Admin Interface and the Query Console.

This section contains these topics:

XQuery Examples of Element Level Security

Run these queries on the Documents database using XQuery in Query Console. First run the queries in the context of els-user-1:

(: run this against the Documents database :)

xdmp:eval(
'cts:search(fn:doc(), cts:word-query("def"), "unfiltered"),
"-----------------------------------------------------",
cts:search(fn:doc(), cts:element-attribute-word-query(xs:QName("bar"), xs:QName("attr"), "test"), "unfiltered"), 
"----------------------------------------------------",
cts:search(fn:doc(), cts:json-property-value-query("bar", "2")),
"-----------------------------------------------------",
cts:search(fn:doc(), cts:element-attribute-word-query(xs:QName("reg"), xs:QName("expr"), "is"), "unfiltered")',
(),
  <options xmlns="xdmp:eval">
    <user-id>{xdmp:user("els-user-1")}</user-id>
  </options>
  )

=>
<?xml  version="1.0" encoding="UTF-8"?>
<root>
  <bar baz="2">def</bar>
  <bar attr="test1">ghi</bar>
</root>
-----------------------------------------------------

----------------------------------------------------
{
  "foo": 1, 
  "bar": "2", 
  "baz": {
    "bar": [
     3, 
     4
    ]
  }
}
-----------------------------------------------------

Notice that in the first query, all of the documents are returned, but the elements with protected paths are missing from the content:

<bar baz="1" attr="test">abc</bar>
"test": 5
<reg expr="this is a string">1</reg>

In the second query, the document does not show up at all because the query is searching on a protected path that els-user-1 is not allowed to see (protected path /root/bar[@baz=1]).

If you are getting different results, check to see that you have set up your user roles correctly and added the query rolesets to the Security database.

Now, modify the query to use the context of the els-user-2 and run the queries again:

(: run this against the Documents database :)

xdmp:eval(
'cts:search(fn:doc(), cts:word-query("def"), "unfiltered"),
"-----------------------------------------------------",
cts:search(fn:doc(), cts:element-attribute-word-query(xs:QName("bar"), xs:QName("attr"), "test1"), "unfiltered"), 
"----------------------------------------------------",
cts:search(fn:doc(), cts:json-property-value-query("bar", "2")),
"-----------------------------------------------------",
cts:search(fn:doc(), cts:element-attribute-word-query(xs:QName("reg"), xs:QName("expr"), "is"), "unfiltered")',
(),
  <options xmlns="xdmp:eval">
    <user-id>{xdmp:user("els-user-2")}</user-id>
  </options>
  )

=>
<?xml  version="1.0" encoding="UTF-8"?>
<root>
  <bar baz="1" attr="test">abc</bar>
  <bar baz="2">def</bar>
  <bar attr="test1">ghi</bar>
</root>
-----------------------------------------------------
<?xml  version="1.0" encoding="UTF-8"?>
<root>
  <bar baz="1" attr="test">abc</bar>
  <bar baz="2">def</bar>
  <bar attr="test1">ghi</bar>
</root>
----------------------------------------------------
{
  "foo": 1, 
  "bar": "2", 
  "baz": {
    "bar": [
      3, 
      4
    ], 
    "test": 5
  }
}
-----------------------------------------------------
<?xml  version="1.0" encoding="UTF-8"?>
<root>
  <reg expr="this is a string">1</reg>
  <reg>2</reg>
</root>

This time all of the documents are returned, along with the protected elements. Notice that the one document is returned twice; two different queries find the same document.

Run the query one more time using the xdmp:eval pattern as els-user-3 and notice that none of the documents are returned because els-user-3 does not have the basic permissions to read the documents. 

(: run this against the Documents database :)

xdmp:eval(
'cts:search(fn:doc(), cts:word-query("def"), "unfiltered"),
"-----------------------------------------------------",
cts:search(fn:doc(), cts:element-attribute-word-query(xs:QName("bar"), xs:QName("attr"), "test1"), "unfiltered"), 
"----------------------------------------------------",
cts:search(fn:doc(), cts:json-property-value-query("bar", "2")),
"-----------------------------------------------------",
cts:search(fn:doc(), cts:element-attribute-word-query(xs:QName("reg"), xs:QName("expr"), "is"), "unfiltered")',
(),
  <options xmlns="xdmp:eval">
    <user-id>{xdmp:user("els-user-3")}</user-id>
  </options>
  )

=>
-----------------------------------------------------

-----------------------------------------------------

-----------------------------------------------------

Because els-user-3 does not have document level permissions, no documents are returned. You can use document level permissions along with element level security for additional security. See Combining Document and Element Level Permissions for more information.

Now unprotect the paths and run the previous query again without the protected paths to see difference in output. First unprotect the paths:

(: run this against the Security database :)

import module namespace sec="http://marklogic.com/xdmp/security" 
  at "/MarkLogic/security.xqy";

sec:unprotect-path("/root/bar[@baz=1]", ()),
sec:unprotect-path("test", ()),
sec:unprotect-path("/root/reg[fn:matches(@expr, 'is')]", ())

Adding or unprotecting protected paths will trigger reindexing. After unprotecting elements, you must wait for reindexing to finish.

Unprotecting the paths does not remove them from the database. You will still see the protected paths in the Admin UI or when you run fn:collection("http://marklogic.com/xdmp/protected-paths") against the Security database. But you will be able to see the whole document once the protected paths are unprotected, if you have document permissions for the document. See Unprotecting or Removing Paths for more details.

Look through the code examples and run the queries using the xdmp:eval pattern to change users. Run the queries in the context of the different users to better understand how the element level security logic works.

JavaScript Examples of Element Security

You can also query the documents using Server-Side JavaScript. Run these JavaScript queries, using the previous users and documents, on the Documents database in Query Console.

First run the queries in the context of els-user-1:

// run this against the Documents database

var prog1 = `cts.search(cts.wordQuery("def"), "unfiltered")`;
var prog2 = `cts.search(cts.elementAttributeWordQuery(xs.QName("bar"), xs.QName("attr"), "test1"), "unfiltered")`;
var prog3 = `cts.search(cts.jsonPropertyValueQuery("bar", "2"))`;
var prog4 = `cts.search(cts.elementAttributeWordQuery(xs.QName("reg"), xs.QName("expr"), "is"), "unfiltered")`;
var res = [];
res.push(xdmp.eval(prog1, null, {userId:xdmp.user("els-user-1")}));
res.push(xdmp.eval(prog2, null, {userId:xdmp.user("els-user-1")}));
res.push(xdmp.eval(prog3, null, {userId:xdmp.user("els-user-1")}));
res.push(xdmp.eval(prog4, null, {userId:xdmp.user("els-user-1")}));
res;
=>
[
"<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n
<root><bar baz=\"2\">def</bar>
<bar attr=\"test1\">ghi</bar>
</root>", 
"<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n
<root>
<bar baz=\"2\">def</bar>
<bar attr=\"test1\">ghi</bar>
</root>", 
  {
  "foo": 1, 
    "bar": "2", 
    "baz": {
      "bar": [
       3, 
       4
      ]
    }
  },
  null
]

Notice that all of the documents are returned, but the elements with protected paths are missing from the content:

<bar baz="1" attr="test">abc</bar>
"test": 5
<reg expr="this is a string">1</reg>

In the second query, the document does not show up at all because the query is searching on a protected path that els-user-1 is not allowed to see (protected path test).

If you are getting different results, check to see that you have set up your user roles correctly and added the query rolesets to the Security database.

Now, modify the query to use the context of the els-user-2 and run the queries again:

// run this against the Documents database

var prog1 = `cts.search(cts.wordQuery("def"), "unfiltered")`;
var prog2 = `cts.search(cts.elementAttributeWordQuery(xs.QName("bar"), xs.QName("attr"), "test1"), "unfiltered")`;
var prog3 = `cts.search(cts.jsonPropertyValueQuery("bar", "2"))`;
var prog4 = `cts.search(cts.elementAttributeWordQuery(xs.QName("reg"), xs.QName("expr"), "is"), "unfiltered")`;
var res = [];
res.push(xdmp.eval(prog1, null, {userId:xdmp.user("els-user-2")}));
res.push(xdmp.eval(prog2, null, {userId:xdmp.user("els-user-2")}));
res.push(xdmp.eval(prog3, null, {userId:xdmp.user("els-user-2")}));
res.push(xdmp.eval(prog4, null, {userId:xdmp.user("els-user-2")}));
res;
=>
[
"<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n
<root>
<bar baz=\"1\" attr=\"test\">abc</bar>
<bar baz=\"2\">def</bar>
<bar attr=\"test1\">ghi</bar>
</root>", 
"<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n
<root><bar baz=\"1\" attr=\"test\">abc</bar>
<bar baz=\"2\">def</bar>
<bar attr=\"test1\">ghi</bar>
</root>", 
  {
  "foo": 1, 
    "bar": "2", 
    "baz": {
      "bar": [
       3, 
       4
      ]
    ,
"test": 5
    }
  },
"<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n
<root>
<reg expr=\"this is a string\">1</reg>
<reg>2</reg>
</root>"
]

This time all of the documents are returned, along with the protected elements. Notice that the one document is returned twice; two different queries will find the same document.

Run the query one more time using the xdmp:eval pattern as els-user-3 and notice that none of the documents are returned because els-user-3 does not have the basic permissions to read the documents. 

// run this against the Documents database

var prog1 = `cts.search(cts.wordQuery("def"), "unfiltered")`;
var prog2 = `cts.search(cts.elementAttributeWordQuery(xs.QName("bar"), xs.QName("attr"), "test1"), "unfiltered")`;
var prog3 = `cts.search(cts.jsonPropertyValueQuery("bar", "2"))`;
var prog4 = `cts.search(cts.elementAttributeWordQuery(xs.QName("reg"), xs.QName("expr"), "is"), "unfiltered")`;
var res = [];
res.push(xdmp.eval(prog1, null, {userId:xdmp.user("els-user-3")}));
res.push(xdmp.eval(prog2, null, {userId:xdmp.user("els-user-3")}));
res.push(xdmp.eval(prog3, null, {userId:xdmp.user("els-user-3")}));
res.push(xdmp.eval(prog4, null, {userId:xdmp.user("els-user-3")}));
res;
=>
[
null, 
null, 
null, 
null
]

Because els-user-3 does not have document level permissions, no documents are returned. You can use document level permissions along with element level security for additional security. See Combining Document and Element Level Permissions for more information.

Now unprotect the paths and run the previous query again without the protected paths to see difference in output. Unprotect the paths :

//run this against the Security database

var security = require('/MarkLogic/security.xqy');
declareUpdate();

security.unprotectPath('/root/bar[@baz=1]', []);
security.unprotectPath('test', []);
security.unprotectPath('/root/reg[fn:matches(@expr, "is")]', []);

Adding, unprotecting, or changing permissions on protected paths will trigger reindexing. After unprotecting elements, you must wait for reindexing to finish.

Unprotecting the paths does not remove them from the database. You will still see the protected paths in the Admin UI or when you run fn:collection("http://marklogic.com/xdmp/protected-paths") against the Security database. But if you are els-role-1 or els-role-2, you will be able to see the whole document once the protected paths are unprotected, if you have document permissions for the document (i.e. els-role-1 and els-role-2, but not els-role-3). See Unprotecting or Removing Paths for more details.

Look through the code examples and run the queries using the xdmp.eval pattern. Run the queries in the context of the different users to better understand how the element level security logic works.

Additional Examples

This section includes additional examples to try, both in XQuery and Server-Side JavaScript, that demonstrate the concealing of elements. Using fn:doc instead of a cts query to retrieve documents, different users will be able to view (or not view) protected elements. Since there is no query involved, query rolesets are not required.

These examples make use of the users and roles set up in the earlier example. (See Example--Element Level Security for details.) The first example shows hierarchies of permissions (top-secret, secret, and unclassified) in a document. The second example shows a slightly different way of protecting content with attributes. The example queries can be done in using XQuery or JavaScript.

XQuery - Query Element Hierarchies

Use this code to insert a new document (along with permissions) into the Documents database:

(: insert document with permissions => run against Documents database :)

xquery version "1.0-ml";

xdmp:document-insert(
"hierarchy.xml", <root>
 <title>Title of the Document</title>
 <summary>Summary of document contents</summary>
 <executive-summary>Executive summary of the document contents
   <secret>Only role having "secret" can read this
     <top-secret>Only role having "top-secret" can read this
     </top-secret>
   </secret>
</executive-summary>
<content>Contents of document 
  <top-secret>Only role with "top-secret" can read this
     <secret>Only role with "secret" can read this</secret>
  </top-secret>
Unclassified content
</content>
</root>,
(xdmp:permission("els-role-1", "read"), xdmp:permission("els-role-2", "read"), 
xdmp:permission("els-role-1", "update"), xdmp:permission("els-role-2", "update")))

Add protected paths with permissions for roles to the Security database:

(: add protected paths -> run against the Security database :)

xquery version "1.0-ml";
import module namespace sec="http://marklogic.com/xdmp/security" 
  at "/MarkLogic/security.xqy";

sec:protect-path("secret", (), (xdmp:permission("els-role-2", "read"))),
sec:protect-path("top-secret", (), (xdmp:permission("els-role-1", "read")))

=>
Returns two numbers representing the protected paths

Adding, unprotecting, or changing permissions on protected paths will trigger reindexing.

Test this example in the context of the different els-users. This first query uses the context of els-user-1:

(: run this against the Documents database :)

xdmp:eval('fn:doc("hierarchy.xml")',(),
  <options xmlns="xdmp:eval">
    <user-id>{xdmp:user("els-user-1")}</user-id>
  </options>
)
=>
<root>
 <title>Title of the Document
 </title>
 <summary>Summary of document contents</summary>
 <executive-summary>Executive summary of contents 
  </executive-summary>
 <content>Contents of document 
  <top-secret>Only role with "top-secret" can read this</top-secret>
 Unclassified content</content>
</root>

The top-secret role (els-user-1) cannot see the elements marked with secret, only those that have no protected paths or marked with the protected path for top-secret. Next, run the query in the context of els-user-2:

(: run this against the Documents database :)

xdmp:eval('fn:doc("hierarchy.xml")',(),
  <options xmlns="xdmp:eval">
    <user-id>{xdmp:user("els-user-2")}</user-id>
  </options>
)
=>
<root>
 <title>Title of the Document</title>
 <summary>Summary of document contents</summary>
 <executive-summary>Executive summary of contents 
  <secret>Only role having "secret" can read this</secret></executive-summary>
 <content>Contents of document 
 Unclassified content</content>
</root>

Notice that even though in the original document there is an element secret within the top-secret contents of the document, it is a child of the top-secret element and therefore hidden to users without the top-secret role.

The els-user-1 (top-secret) cannot see the secret content unless you add the els-role-2 to els-user-1. When you add the role, els-user-1 will be able to see both the secret and top-secret elements.

If you run the query as els-user-3, the query returns an empty sequence. The els-user-3 from the previous query does not have permission to even see the document.

XQuery - Matching By Paths or Attributes

This next example shows how protected paths can be used with fn:contains and fn:matches. The example uses the same roles from the previous example, adding a new role (els-role-3).

First unprotect the protected paths from the previous example:

(: unprotect the protected paths -> run against the Security database :)

xquery version "1.0-ml"; 
import module namespace sec = "http://marklogic.com/xdmp/security" 
  at "/MarkLogic/security.xqy";
      
sec:unprotect-path("secret", ()),
sec:unprotect-path("top-secret", ())

Adding or unprotecting protected paths will trigger reindexing. After unprotecting elements, you must wait for reindexing to finish.

Create a new role els-role-3 and add els-user-3 to the role. See Create Roles and Create Users and Assign Roles for details.

Add a new document with permissions to the Documents database:

(: run this against the Documents database :)

xquery version "1.0-ml";

xdmp:document-insert(
"attributes.xml", <root>
 <title>Document Title</title>
 <summary>Summary of document contents</summary>
 <executive-summary>Executive summary of contents
  <info attr="EU">Only role with "EU" attribute can read this summary </info>
  <info attr="UK">Only role with "UK" attribute can read this summary </info>
  <info attr="US">Only role with "US" attribute can read this summary </info>
</executive-summary>
 <content>Contents of document 
  Unclassified content
  <notes> 
    <info attr="EU">Only role with "EU" attribute can read this content</info>
    <info attr="UK">Only role with "UK" attribute can read this content</info>
    <info attr="US">Only role with "US" attribute can read this content</info>
  </notes>
 </content>
</root>,
(xdmp:permission("els-role-1", "read"), xdmp:permission("els-role-2", "read"), xdmp:permission("els-role-3", "read"),
xdmp:permission("els-role-1", "update"), xdmp:permission("els-role-2", "update"), xdmp:permission("els-role-3", "update")))

Add the new protected paths with permissions for roles to the Security database:

(: add new protected paths -> run against the Security database :)

xquery version "1.0-ml";
import module namespace sec="http://marklogic.com/xdmp/security" 
  at "/MarkLogic/security.xqy";

sec:protect-path("//info[fn:matches(@attr, 'US')]", (), (xdmp:permission("els-role-1", "read"))),
sec:protect-path("//info[fn:matches(@attr, 'UK')]", (), (xdmp:permission("els-role-2", "read"), 
  xdmp:permission("els-role-3", "read"))),
sec:protect-path("//info[fn:matches(@attr, 'EU')]", (), (xdmp:permission("els-role-3", "read")))
=>
Returns three numbers representing the protected paths

Adding, unprotecting, or changing permissions on protected paths will trigger reindexing.

Notice that the protected paths include attributes in the document elements. Also note that els-role-3 has permissions for two protected paths (@attr, 'UK' and @attr, 'EU').

Run this next query, similar to the previous queries, this time looking for the attributes.xml document. First query in the context of els-user-1 who has a role that can see the US attribute:

(: run this against the Documents database :)

xdmp:eval('fn:doc("attributes.xml")',(),
  <options xmlns="xdmp:eval">
    <user-id>{xdmp:user("els-user-1")}</user-id>
  </options>
)

=>
<?xml  version="1.0" encoding="UTF-8"?>
<root>
 <title>Document Title</title>
 <summary>Summary of document contents</summary>
 <executive-summary>Executive summary of contents 
  <info attr="US">Only role having "US" attribute can read this summary</info>
 </executive-summary>
 <content>Contents of document 
  Unclassified content 
 <notes>
  <info attr="US">Only role having "US" attribute can read this content
  </info>
 </notes>
 </content>
</root>

Next modify the query to run in the context of els-user-2, who has a role that can see the UK attribute:

(: run this against the Documents database :)

xdmp:eval('fn:doc("attributes.xml")',(),
  <options xmlns="xdmp:eval">
    <user-id>{xdmp:user("els-user-2")}</user-id>
  </options>
)

=>
<?xml  version="1.0" encoding="UTF-8"?>
<root>
 <title>Document Title</title>
 <summary>Summary of document contents</summary>
 <executive-summary>Executive summary of contents 
  <info attr="UK">Only role having "UK" attribute can read this summary
  </info>
 </executive-summary>
 <content>Contents of document 
  Unclassified content 
 <notes>
  <info attr="UK">Only role having "UK" attribute can read this content</info>
 </notes>
 </content>
</root>

And finally modify the query to run in the context of els-user-3:

(: run this against the Documents database :)

xdmp:eval('fn:doc("attributes.xml")',(),
  <options xmlns="xdmp:eval">
    <user-id>{xdmp:user("els-user-3")}</user-id>
  </options>
)

=>
<?xml  version="1.0" encoding="UTF-8"?>
<root>
 <title>Document Title</title>
 <summary>Summary of document contents</summary>
 <executive-summary>Executive summary of contents 
  <info attr="EU">Only role having "EU" attribute can read this summary

  </info>
  <info attr="UK">Only role having "UK" attribute can read this summary
   
  </info>
 </executive-summary>
 <content>Contents of document 
  Unclassified content
  
 <notes>
  <info attr="EU">Only role having "EU" attribute can read this content
     
  </info>
  <info attr="UK">Only role having "UK" attribute can read this content
     
  </info>
 </notes>
 </content>
</root>

The els-user-3 has protected path permissions on both elements with the EU info attribute and the elements with the UK info attribute, so the els-user-3 can see both elements. If you are getting different results, check to be sure that you created an els-role-3 and added the els-user-3 to that role.

If you run the query in the context of the admin user, you will be able to see the entire document because the query is using fn:doc.

JavaScript - Query Element Hierarchies

You can also try these examples demonstrating concealed elements using JavaScript. Using fn:doc instead of a cts query to retrieve documents, different users will be able to view (or not view) protected elements. Since there is no query involved, query rolesets are not required.

Use this JavaScript code to insert this document (with permissions) into the Documents database:

// insert document with permissions -> run against Documents database

declareUpdate();
var perms = [xdmp.permission("els-role-1", "read"), xdmp.permission("els-role-2", "read"), 
xdmp.permission("els-role-1", "update"), xdmp.permission("els-role-2", "update")                         
];
xdmp.documentInsert(      
"hierarchy.xml", xdmp.unquote(` 
<root>
 <title>Title of the Document</title>
 <summary>Summary of document contents</summary>
 <executive-summary>Executive summary of the document contents
   <secret>Only role having "secret" can read this
     <top-secret>Only role having "top-secret" can read this
     </top-secret>
   </secret>
</executive-summary>
<content>Contents of document 
  <top-secret>Only role with "top-secret" can read this
     <secret>Only role with "secret" can read this</secret>
  </top-secret>
Unclassified content
</content>
</root>
`), {permissions: perms})

Add protected paths with permissions for roles to the Security database:

// add protected paths -> run against the Security database

declareUpdate();
var security = require('/MarkLogic/security.xqy');

security.protectPath('secret', [], [xdmp.permission("els-role-2", "read", "element")]),
security.protectPath('top-secret', [], [xdmp.permission("els-role-1", "read", "element")])
=>
Returns a number representing the protected paths

Adding, unprotecting, or changing permissions on protected paths will trigger reindexing.

Test this example in the context of the different els-users. This query uses the context of els-user-1:

// run this query against the Documents database

xdmp.eval("fn.doc('hierarchy.xml')", null,
  {
    "userId" : xdmp.user("els-user-1")
  })
=>
<?xml  version="1.0" encoding="UTF-8"?>
<root>
  <title>Title of the Document</title>
  <summary>Summary of document contents</summary>
  <executive-summary>Executive summary of the document contents
   
  </executive-summary>
  <content>Contents of document 
  
  <top-secret>Only role with "top-secret" can read this</top-secret>

  Unclassified content
  </content>
</root>

The top-secret role (els-user-1) cannot see the elements marked with secret, only those that have no protected paths or marked with the protected path for top-secret. Next, run the query in the context of els-user-2:

// run this query against the Documents database

xdmp.eval("fn.doc('hierarchy.xml')", null,
  {
    "userId" : xdmp.user("els-user-2")
  })
=>
<?xml  version="1.0" encoding="UTF-8"?>
<root>
  <title>Title of the Document</title>
  <summary>Summary of document contents</summary>
  <executive-summary>Executive summary of the document contents
   
  <secret>Only role having "secret" can read this</secret>
  </executive-summary>
  <content>Contents of document 
  
  Unclassified content
  </content>
</root>

Notice that even though in the original document, there is an element secret within the top-secret contents of the document, it is a child of the top-secret element and therefore hidden to users without the top-secret role.

The els-user-1 (top-secret) cannot see the secret content unless you add the els-role-2 to els-user-1. When you add the role, els-user-1 will be able to see both the secret and top-secret elements.

If you run the query as els-user-3, the query returns an empty sequence. The els-user-3 from the previous query does not have permission to even see the document.

JavaScript - Matching By Paths or Attributes

This next example shows how protected paths can be used with fn.contains and fn.matches. The example uses the same roles from the previous example, adding a new role (els-role-3).

First unprotect the protected paths from the previous example:

// unprotect protected paths -> run against the Security database

declareUpdate();
var security = require('/MarkLogic/security.xqy');

security.unprotectPath('secret', []),
security.unprotectPath('top-secret', [])

Adding, unprotecting, or changing permissions on protected paths will trigger reindexing.

Create a new role els-role-3 and add els-user-3 to the role. See Create Roles and Create Users and Assign Roles for details.

Add a new document to the Documents database:

// insert document and permissions -> run this against the Documents database

declareUpdate();
var perms = [xdmp.permission("els-role-1", "read"), xdmp.permission("els-role-2", "read"), 
xdmp.permission("els-role-3", "read"), xdmp.permission("els-role-1", "update"), 
xdmp.permission("els-role-2", "update"), xdmp.permission("els-role-3", "update")  
];
xdmp.documentInsert(
"attributes.xml", xdmp.unquote(`
<root>
  <title>Document Title</title>
  <summary>Summary of document contents</summary>  
  <executive-summary>Executive summary of contents
    <info attr="EU">Only role with "EU" attribute can read this summary </info>
    <info attr="UK">Only role with "UK" attribute can read this summary </info>
    <info attr="US">Only role with "US" attribute can read this summary </info>
  </executive-summary>
  <content>Contents of document
  Unclassified content
    <notes> 
    <info attr="EU">Only role with "EU" attribute can read this content</info>
    <info attr="UK">Only role with "UK" attribute can read this content</info>
    <info attr="US">Only role with "US" attribute can read this content</info>
    </notes>
  </content>
</root>
`), {permissions: perms})

Add the new protected paths with permissions for roles to the Security database:

// add new protected paths -> run against the Security database

declareUpdate();
var security = require('/MarkLogic/security.xqy');

security.protectPath("//info[fn:matches(@attr, 'US')]", [],[xdmp.permission("els-role-1","read", "element")]),
security.protectPath("//info[fn:matches(@attr, 'UK')]", [],[xdmp.permission("els-role-2", "read", "element"), 
  xdmp.permission("els-role-3", "read", "element")]),
security.protectPath("//info[fn:matches(@attr, 'EU')]", [],
  [xdmp.permission("els-role-3", "read", "element")])

=>
Returns one number representing the protected paths

Adding or changing permissions on protected paths will trigger reindexing.

Run the same queries as before, first in the context of els-user-1, who has a role that can see the US attribute:

// run this query against the Documents database

xdmp.eval("fn.doc('attributes.xml')", null, 
  {
    "userId" : xdmp.user("els-user-1")
  });
=>
<?xml  version="1.0" encoding="UTF-8"?>
<root>
  <title>Document Title</title>
  <summary>Summary of document contents</summary>
  <executive-summary>Executive summary of contents

  <info attr="US">Only role with "US" attribute can read this summary</info>
  </executive-summary>
  <content>Contents of document 
  Unclassified content
  
  <notes>
  <info attr="US">Only role with "US" attribute can read this content</info>
  </notes></content>
</root>

Next modify the query to run in the context of els-user-2,who has a role that can see the UK attribute

// run this query against the Documents database

xdmp.eval("fn.doc('attributes.xml')", null, 
  {
    "userId" : xdmp.user("els-user-2")
  });
=>
<?xml  version="1.0" encoding="UTF-8"?>
<root>
  <title>Document Title</title>
  <summary>Summary of document contents</summary>
  <executive-summary>Executive summary of contents

  <info attr="UK">Only role with "UK" attribute can read this summary</info></executive-summary>
  <content>Contents of document 
  Unclassified content
  
  <notes>
  <info attr="UK">Only role with "UK" attribute can read this content</info>
  </notes></content>
</root>

And finally modify the query to run in the context of els-user-3:

// run this query against the Documents database

xdmp.eval("fn.doc('attributes.xml')", null, 
  {
    "userId" : xdmp.user("els-user-3")
  });
=>
<?xml  version="1.0" encoding="UTF-8"?>
<root>
  <title>Document Title</title>
  <summary>Summary of document contents</summary>
  <executive-summary>Executive summary of contents
  
  <info attr="EU">Only role with "EU" attribute can read this summary</info>
  <info attr="UK">Only role with "UK" attribute can read this summary</info>
  </executive-summary>
  <content>Contents of document 
  Unclassified content 
  <notes>
  <info attr="EU">Only role with "EU" attribute can read this content</info>
  <info attr="UK">Only role with "UK" attribute can read this   content</info>
  </notes></content>
</root>

The els-user-3 has protected path permissions on both elements with the EU info attribute and the elements with the UK info attribute. So that user can see both elements.

If you run the query in the context of the admin user, you will be able to see the entire document because the query is using fn.doc.

Configuring Element Level Security

Configuring element level security includes setting up protected paths and creating query rolesets, then adding them to the Security database. This section covers the steps you will need to follow to configure element level security. As an overview, you will need to do the following:

  • Set up roles
  • Create users and assign roles
  • Add or update documents with permissions for users
  • Add protected paths for elements in documents, by inserting the protected paths into the Security database
  • Add the query rolesets to the Security database

Configuring the query rolesets is a task for the administrator. There are two helper functions to help configure query rolesets. The helper function xdmp:database-node-query-rolesets is used for querying documents already in your database to discover existing query rolesets, while xdmp:node-query-rolesets is used to query for protected paths in documents as they are being loaded into the database. See APIs for Element Level Security for more information. You can configure element level security using the Admin UI, using XQuery, or by using REST.

The number of protected paths that you set on a single element may impact performance. One or two protected paths on an element will have no discernable impact (less than 5% in our testing), 10 or so protected paths may have some impact (around 10%), but setting 100 or so protected paths on a single element will cause severe and noticeable impact on performance.

This section covers these topics:

Protected Paths

You can define permissions on an element in the same way that you define permissions on a document. Element level security works by specifying an indexable path to an element (or JSON property) and configuring permissions on that path - creating a protected path.

For performance and security reasons, you can only use a subset of XPath for defining protect paths. For details, see Element Level Security in the XQuery and XSLT Reference Guide.

The section contains these topics:

Examples of Protected Paths

This table shows some examples of protected paths.

Protected Path Permissions Result
/foo/bar (role1, read) Element bar is readable by role1 but concealed for all other roles. No mention of other permissions means that others can update or insert content for this element.
/foo/bar

(role1, read)

(role2, read)

 Element bar is readable by role1 or role2 but concealed for all other roles. No mention of other permissions means that others can update or insert content for this element.
/foo/bar

(role1, read)

(role1, update)

 Element bar is readable by role1 but concealed for all other roles. Role1 can update the element. No mention of insert permissions means that others can insert content for this element.
/foo/bar[@attr= test]

(role1, read)

(role1, update)

 Same as above except that it only applies to a bar element if the element has an attribute attr with the value test. No mention of insert permissions means that others can insert content for this element.
bar (role1, read) This is the simplest path. Element bar is readable by role 1, but concealed for all other roles. This applies to all bar elements. No mention of other permissions means that others can update or insert content for this element.
/root/reg[fn:matches(@expr, 'is')] (role1, read) (role1, update) Elements that match the regular express for 'is will be readable by role 1, but concealed for all other roles. Role 1 can update the element. No mention of insert permissions means that others can insert content for this element.

For more about update permissions with element level security, see the table in the section Document and Element Level Permissions Summary.

Defining element level security protection (protected paths) on reserved elements or properties (for example, alerting, thesaurus, and so on) may cause undefined behavior.

The path is an XPath expression, not a field.

Namespaces as Part of a Protected Path

Both namespaces and prefixes can be used as part of a protected path. For instance this simple example uses the namespace ex as part of the protected path:

(: add protected paths -> run against the Security database :)

xquery version "1.0-ml";
import module namespace sec = "http://marklogic.com/xdmp/security" 
  at "/MarkLogic/security.xqy";

declare namespace ex = "http://marklogic.com/example";

let $role := "role-4"
return 
  sec:protect-path(
    "/ex:envelope/ex:instance/employee/salary", 
    (let $prefix := "ex",$namespace-uri := 
      "marklogic.com/example"
    return
    sec:security-path-namespace($prefix, $namespace-uri)), 
  (xdmp:permission($role, "read"))
  )

For simple cases, you can also specify a namespace as part of a protected path when configuring protected paths in the Admin UI.

You can also specify a namespace when using the helper functions xdmp:database-node-query-rolesets and xdmp:node-query-rolesets. See page Helper Functions for Query Rolesets for more info.

Unprotecting or Removing Paths

Unprotecting protected paths does not remove them from the database, it removes the permissions, which disables the protection. You will still see the unprotected paths in the Admin UI. The unprotected paths can also be seen by running fn:collection("http://marklogic.com/xdmp/protected-paths") against the Security database, in the Query Console.

Removing protected paths is a two step process. First you must unprotect the path, and then you can remove it.

You must first unprotect a path before removing it to trigger the reindexer. Since query rolesets changes don't require reindexing, there is no need for the separate step of unprotecting before removing a query roleset.

To unprotect a protected path:

  1. Navigate to Protected Path Configuration by clicking Security and then Protected Paths in the left tree menu.
  2. Click on the name of the protected path you want to unprotect.
  3. On the Protected Path Configuration page there are two buttons; an unprotect button and a delete button (greyed out).

  4. Click the unprotect button.
  5. Click ok to save the changes.

When you have unprotected the protected paths, you'll see the protected paths on the Summary page, but no permissions are associated with the paths.

To remove a path, you will need to first unprotect the path. See Unprotecting or Removing Paths

  1. After unprotecting the path, go back to the Protected Path Configuration page. Notice that the delete button is now available and the unprotect button is greyed out.

  2. Click the delete button to remove this protected path.
  3. Click ok to confirm and save your changes.

The deleted path no longer appears on the Summary page of protected paths.

Adding, unprotecting, or changing permissions for protected paths will trigger reindexing of the relevant databases. Having too many unprotected paths for your database can affect performance. Best practice is to delete unprotected paths when you no longer need them. Be sure to let the reindexing triggered by unprotecting finish before you delete the paths.

Performance Considerations With Protected Paths

The fewer protected paths that you have in your documents, the better performance you will have with element level security. One way to reduce the number of protected paths is to group information. If you have the ability to control the schema of your documents, you can group information that you want to protect under one element and then protect that element.

In this example, an insurance company has a schema that groups policy information to control access to the information, making it easier to protect client information and policy information by role (US Read, ID_Read, Compliance, and Risk):

"policy": {
"access": "US Read", 
"client": {
   "access": "ID_Read", 
   "name": "Paul", 
   "address": "999 Broadway St", 
   "phone": "323-344-1555", 
   "country": "US", 
   "ssn4digits": "5664"
     }
, 
"clientSSN": {
   "access": "Compliance",  
   "ssn": "999-999-5664"
     }
, 
"clientIncome": {
   "access": "Risk", 
   "income": "44,4444"
     }
, 
"info": {
   "access": "Risk", 
   "propertyType": "Home", 
   "premium": 432, 
   "assetValue": 750000, 
   "currency": "Dollar"
     }
}

Different users would be able to see different parts of the data: the Call Center might have the ID_Read role, the Financial Risk Researcher might have the Risk role, and a Compliance Admin might have the ID_Read, Risk, and Compliance roles. Each of these would all need to have the US Read role as well.

If you don't have control of the schema and your document data is in various formats, you can leverage Entity Services as a way to improve performance. You can use entity services to create an entity that groups multiple elements under a single node and then use a single protected path on that node. See Introduction to Entity Services in the Entity Services Developer's Guide for information about creating an entity that links to the source document and protecting both.

Query Rolesets

What are query rolesets and what do they do? This section describes query rolesets and how they are used with element level security. It contains these topics:

How Query Rolesets Work

When you add a document into MarkLogic, it parses the document and puts terms (or keys) into the universal index. Later when you run a query, the query side needs to know what terms to find in the universal index. In element level security, the terms are combined with permissions in the index. Existing query rolesets are automatically used by the query to figure out which terms to use, based on the role(s) of the user running the query. Each query can include multiple query rolesets. If no query rolesets are configured, a query will only match documents using the terms that are visible to everyone.

Let's use an example. Say you have a protected path defined as the following:

sec:protect-path("/root/bar[@baz=1]", (), (xdmp:permission("els-role-2", "read")))

And then you ingest a document like this:

<root>
  <bar baz=1>Hello</bar>
</root>

When MarkLogic parses the document, it sees that the word Hello is inside the element <bar> that matches the protected path definition (since bar is under root and has an attribute baz=1). So instead of simply putting the term Hello into the universal index, it combines the term Hello and the permission information in the protected path (in this case, basically the role name els-role-2) into one term and puts this new term into the universal index.

Suppose then you run a search with a query cts:word-query("Hello") with a user that has the els-role-2 role. The query must know this new term to find the document. The query already knows the word Hello but how would it know the permission information in the protected path?

This is where the query rolesets are used. You configure query rolesets (with just els-role-2 in this example) and then the query compares that query roleset with the caller's role. If the caller's role matches the query rolesets, the query will combine that information with the word Hello to generate the term, which matches the term put into the universal index by MarkLogic.

There are three ways to configure query rolesets:

This last method of manually creating query rolesets works for simple examples and cases where there are not many protected paths. If you have a single protected path that matches an element like one in the examples above (with no overlaps), use a simple rule to create the query roleset in the Admin UI. See Add Protected Paths and Query Rolesets for details

The two helper functions; xdmp:database-node-query-rolesets and xdmp:node-query-rolesets, can help with configuring more complex query rolesets, either for documents already stored in MarkLogic or while documents are being added. MarkLogic leaves query rolesets configuration (creating and inserting the query rolesets into the Security database) to the administrator.

Query rolesets are made up of roles. There can be any number of roles in a roleset, as long as there are no duplicates. There can be multiple query rolesets in a database.

Query rolesets are required for element level security to work. You may ask why not just get the query rolesets information automatically from the protected paths when you configure sec:protect-path to avoid the manual configuration of query rolesets. For this simple example this seems practical, but in the real world it is not uncommon to have multiple protected paths that match the same node or element. Some use cases will have 1000s of protected paths but only 100s of query rolesets. The indexer side of MarkLogic often needs to combine multiple query rolesets to create the term.

There is no way for the query side to derive that information from the protected path configuration, since whether a node element matches a protected path is based on the value of the node. And the query side doesn't know the value of a node. There is no way for the query side to know what subsets of all the configured protected paths need to be taken into consideration when creating the query term. Since enumerating all possible combinations of the roles used in all protected paths is not practical, MarkLogic leaves query rolesets configuration (creating and inserting the query rolesets into the Security database) to the administrator.

Parent/Child Relationships in Query Rolesets

You might have a document where one user has permissions for an element that is the child of a parent element, for which that user does not have permissions. For example, there might be a simple document like this:

<root>
 <content>Contents of document 
  <top-secret>Only role with "top-secret" can read this
    <secret>Only role with "secret" can read this</secret>
  </top-secret>
 Unclassified content
 </content>
</root>

This document might have these protected paths:

sec:protect-path("secret", (), (xdmp:permission("els-role-2", "read"))),
sec:protect-path("top-secret", (), (xdmp:permission("els-role-1", "read")))

A user with permissions on only the protected path for secret can't see secret content unless the user also had permissions for the protected path for top-secret because the secret node is a child of the top-secret parent node.

Overlapping Protected Paths

Consider a more complex case with multiple paths matching the same node. Suppose you have a document like this:

<root>
  <foo a=1 b=2 c=3>Hello</foo>
</root>

It is possible to define three different protected paths that all match the foo element, overlapping each other:

sec:protect-path("/root/foo[@a=1]", (), (xdmp:permission("els-role-1", "read")))
sec:protect-path("/root/foo[@b=2]", (), (xdmp:permission("els-role-2", "read")))
sec:protect-path("/root/foo[@c=3]", (), (xdmp:permission("els-role-3", "read")))

MarkLogic will still create just one term for Hello, which is the combination of the word and the query rolesets ((els-role-1),(els-role-2),(els-role-3)).

As a side note, in the above example the query rolesets is ((els-role-1),(els-role-2),(els-role-3)), which is different from simply (els-role-1,els-role-2,els-role-3).

In MarkLogic 9.0-2 query rolesets have been simplified and optimized. Existing documents with query rolesets configured in 9.0-1 will still be protected in 9.0-2. To take advantage of the optimization however, you need to reindex your documents and regenerate your query rolesets using the helper functions (APIs for Element Level Security). It is highly recommended that you reindex any protected documents already in your database and regenerate your query rolesets, since documents may be reindexed by another operation, which may cause a mismatch between the documents and the query rolesets. See Algorithm That Determines Which Query Rolesets to Use for examples and more details.

This is what the query rolesets hierarchy looks like for ((els-role-1),(els-role-2),(els-role-3)); three query rolesets and three roles:

This is what the query rolesets hierarchy looks like for (els-role-1,els-role-2,els-role-3); one query roleset and three roles:

If you only have one protected path that matches foo in the above example but with three roles, like this:

sec:protect-path("//foo", (), (
xdmp:permission("els-role-1", "read"),
xdmp:permission("els-role-2", "read"), 
xdmp:permission("els-role-3", "read")))

Then (els-role-1,els-role-2,els-role-3) would be the proper query roleset to use. To configure the former ((els-role-1),(els-role-2),(els-role-3)), you would call:

(:run against the Security database :)
xquery version "1.0-ml";
import module namespace sec="http://marklogic.com/xdmp/security" 
  at "/MarkLogic/security.xqy";
  
let $roleset1 := sec:query-roleset(("els-role-1"))
let $roleset2 := sec:query-roleset(("els-role-2"))
let $roleset3 := sec:query-roleset(("els-role-3"))
return sec:add-query-rolesets(sec:query-rolesets(($roleset1,$roleset2,$roleset3)))

To configure the latter (els-role-1,els-role-2,els-role-3), you can simply call:

(:run against the Security database :)
xquery version "1.0-ml";
import module namespace sec="http://marklogic.com/xdmp/security" 
  at "/MarkLogic/security.xqy";
  
let $roleset1 := sec:query-roleset(("els-role-1","els-role-2","els-role-3"))
return
sec:add-query-rolesets(sec:query-rolesets($roleset1))

When you are starting to configure and use element level security, the two query rolesets helper functions, xdmp:database-node-query-rolesets and xdmp:node-query-rolesets can simplify the process of setting up your query rolesets. These functions can be used for configuring query rolesets either for documents in the database, or for documents during ingestion. See Helper Functions for Query Rolesets for more information.

Protected Path Sets

A protected path set is a way to allow multiple protected paths covering the same element, with both AND and OR relationships between the permissions. This enables multiple arbitrary security marking for an element.

A protected path set is an optional string that represents the name of a set is associated with a protected path. A path that has no set name can be seen as a degenerated form of a set. The diagram below shows how permissions from paths in the same set are ORed, while permissions between sets are ANDed.

The set information (the name) is simply a tag on the protected path definition, not a separate document in the Security database.

Consider the following element:

<foo classification="TS" releasableTo="USA GBR AUS">

Using protected paths, MarkLogic element level security allows multiple protected paths covering the same element with an AND relationship among their permissions. This models a multiple security markings (for example @classification and @releasableTo) situation well. For the element above, two protected paths may be defined:

//foo[@classification="TS"]    ("Role_TS", "read")
//foo[@releasableTo="USA GBR AUS"] (("Role_USA", "read"), ("Role_GBR","read"), ("Role_AUS","read"))

Note that the value of @releasableTo is a list of country codes, with each mapping to a role. A user with any of the country roles is allowed to read the element. The challenge is that a list can contain an arbitrary combination of country codes (total 200+). The above approach would require a user to define one protected path for each of the possible combinations, which may lead to a very large number of protected paths.

Note that defining the following protected paths won't satisfy the requirement because the permissions among the paths are ANDed, not ORed.

//foo[fn:contains(@releasableTo, "USA")] ("Role_USA", "read")
//foo[fn:contains(@releasableTo, "GBR")] ("Role_GBR", "read")
//foo[fn:contains(@releasableTo, "AUS")] ("Role_AUS", "read")

The following example shows the benefit of the path set concept more clearly. Consider the following elements to be protected:

<foo classification="TS" releasableTo="USA">
<foo classification="TS" releasableTo="GBR">
<foo classification="TS" releasableTo="AUS">
<foo classification="TS" releasableTo="USA GBR">
<foo classification="TS" releasableTo="GBR AUS">
<foo classification="TS" releasableTo="USA AUS">
<foo classification="TS" releasableTo="USA GBR AUS">

Without using protected path sets, the following protected paths would need to be defined to protect the elements above:

//foo[@classification="TS"] ("Role_TS", "read")
//foo[@releasableTo="USA"]  ("Role_USA", "read")
//foo[@releasableTo="GBR"]  ("Role_GBR","read")
//foo[@releasableTo="AUS"]  ("Role_AUS","read")
//foo[@releasableTo="USA GBR"] (("Role_USA", "read"), ("Role_GBR","read"))
//foo[@releasableTo="GBR AUS"] (("Role_GBR","read"),  ("Role_AUS","read"))
//foo[@releasableTo="USA AUS"] (("Role_USA", "read"),  ("Role_AUS","read"))
//foo[@releasableTo="USA GBR AUS"] (("Role_USA", "read"), ("Role_GBR","read"), ("Role_AUS","read"))

With protected path sets, only these protected paths are needed:

//foo[@classification="TS"]    ("Role_TS", "read")
//foo[fn:contains(@releasableTo, "USA")] ("Role_USA", "read")   "SetReleasableTo"
//foo[fn:contains(@releasableTo, "GBR")] ("Role_GBR", "read")   "SetReleasableTo"
//foo[fn:contains(@releasableTo, "AUS")] ("Role_AUS", "read")    "SetReleasableTo"

The total number of protected paths required for the @releasableTo attribute is reduced from 7 to 3 using the SetReleasableTo protected path set.

In real world systems, the total number of possible country codes for these examples are more than 200, which leads to millions of possible combinations. So with protected path sets, the number of required protected paths can be reduced from millions to just a couple of hundred for the @releasableTo use case.

Helper Functions for Query Rolesets

In order to search for query rolesets, you find out which query rolesets are configured for protected paths for a document already in the database. You can also discover if query rolesets are required for proper querying of a document being loaded into the database. Element level security includes two built-ins that can be used to discover existing protected paths in documents. The xdmp:database-node-query-rolesets built-in is used for querying documents already in the database, while xdmp:node-query-rolesets is used to query for protected paths in documents that are being loaded into the database. Given a node, these functions will return a list of the query rolesets for any protected paths, as long as the user of the built-ins has sufficient privileges and permissions. Usually these function are called by an admin user.

For xdmp:database-node-query-rolesets, the built-in returns a sequence of query rolesets that are required for proper querying of any given database nodes where element level security is in place on a document already in the database. 

xquery version "1.0-ml";
import module namespace sec = "http://marklogic.com/xdmp/security" 
  at "/MarkLogic/security.xqy";

(: run this against the Security database :)

let $qry := 'xdmp:database-node-query-rolesets(fn:doc("/example.xml"), ("all"))'
let $qry-rolesets := 
xdmp:eval($qry, (),<options xmlns="xdmp:eval">
                   <database>{xdmp:database(YOUR_DB_NAME)}</database>
                 </options>)
return
sec:add-query-rolesets($qry-rolesets)

=>
<query-rolesets xml:lang="zxx" xmlns="http://marklogic.com/xdmp/security">
 <query-roleset>
  <role-id>12006351629398052509
  </role-id>
 </query-roleset>
</query-rolesets>

To find the name of this role ID, use this query in the Query Console:

xquery version "1.0-ml";
import module namespace sec = "http://marklogic.com/xdmp/security" 
  at "/MarkLogic/security.xqy";
  
sec:get-role-names((12006351629398052509))
=>
<sec:role-name xmlns:sec="http://marklogic.com/xdmp/security">els-role-2</sec:role-
name>

The unconfigured option for xdmp:database-node-query-rolesets will return only those query rolesets that are not configured, meaning these query rolesets are not in the Security database yet (you have not configured them yet). The all option returns all query rolesets, even if they are already configured.

You can find existing or yet-to-be-configured query rolesets for documents being loaded into the database using xdmp:node-query-rolesets. This built-in returns a sequence of query rolesets that are required for proper querying with element level security if the node is inserted into the database with the given document-insert options. This built-in also comes with the unconfigured option and the all option, and works the same as the xdmp:database-node-query-rolesets built-in.

A typical workflow would call this function and add each query rolesets through the sec:add-query-rolesets function before inserting the document into the database, so that the document can be correctly queried with element level security as soon as the document is inserted.

xdmp:node-query-rolesets(
    "/example.xml",
    <foo>aaa</foo>,
    <options xmlns="xdmp:document-insert">
      <permissions>
{xdmp:permission("role1","read"),xdmp:permission("role2","read")}
      </permissions>
    </options>)

To run this built-in you need to have the security role privileges.

Query for Protected Paths on a Document

You can use this XQuery code as a model to customize. The code sample searches for the protected paths associated with foo.xml. 

xquery version "1.0-ml";
import module namespace sec = "http://marklogic.com/xdmp/security" 
  at "/MarkLogic/security.xqy";

declare function local:get-role-name($p as element(sec:permission)) {
  element sec:permission {
    $p/*,
    sec:get-role-names($p/sec:role-id)
  }
};

let $doc := xdmp:eval('fn:doc("foo.xml")', (), <options xmlns="xdmp:eval"><database>{xdmp:database("Documents")}</database></options>)
for $p in fn:collection(sec:protected-paths-collection())/sec:protected-path
let $path := 
   xdmp:with-namespaces(
       for $ns in $p//sec:path-namespace
       return ($ns/sec:prefix/fn:string(.), $ns/sec:namespace-uri/fn:string(.)),
       xdmp:value("$doc" || $p/sec:path-expression/fn:string()))
return
  if (fn:exists($path)) then
    element sec:protected-path {
      $p/* except $p/sec:permissions,
      element sec:permissions {
        $p/sec:permissions/sec:permission ! local:get-role-name(.)
      }
    }
  else 
    ()

You will only be able to see the protected paths for elements that you as the user would have permission to see. For example if you had role1 and the protected path was associated with role2, role1 would not be able to see those paths.

Related functionality is the all-query-rolesets-fragment-count element returned from xdmp:forest-counts. This number tells the caller how many fragments are indexed with a certain query-rolesets. If the number is 0 (across all databases), then query-rolesets is no longer in use.

Configure Element Level Security in the Admin UI

Protected paths and query rolesets for element level security can be configured from the Admin UI. The steps to configure users and roles for element level security are the same as described in Create Roles and Create Users and Assign Roles. To test the examples, add the sample documents using Query Console, as described in Add the Documents.

Add a Protected Path

To add a protected path for element level security:

  1. Click Protected Paths in the left tree menu.
  2. Click the Create tab.

  3. Enter the information for the protected path: the path expression, the prefix and namespace, and the role name and capabilities for the permissions.
  4. Click more permissions to add additional permissions to this protected path.
  5. Click ok when you are done.

Add a Query Roleset

To add a query roleset for element level security, using the Admin UI:

  1. Click Security in the left tree menu.
  2. Click Protected Paths and then click the Create tab.

  3. Add the roles (els-role-1 and els-role-2) for the query roleset, separated by commas.
  4. Click more items to add additional comma-separated query rolesets.
  5. Click ok when you are done.

    An administrator must define query rolesets.

Configure Element Level Security With XQuery

To configure element level security, you'd follow the same series of steps that you used for the earlier example. (See Example--Element Level Security.)

  • Set up roles
  • Create users and assign roles
  • Insert documents with permissions
  • Add the query rolesets to the Security database
  • Add protected paths for elements in documents, by inserting the protected paths into the Security database

Using XQuery for Query Rolesets

Use the xdmp:database-node-query-rolesets helper function with the sec:add-query-rolesets command to set up query rolesets using XQuery.

For example:

xquery version "1.0-ml";
import module namespace sec = "http://marklogic.com/xdmp/security" 
  at "/MarkLogic/security.xqy";

(: run this against the Security database :)

let $qry := 'xdmp:database-node-query-rolesets(fn:doc("/example.xml"), ("all"))'
let $qry-rolesets := 
xdmp:eval($qry, (),<options xmlns="xdmp:eval">
                   <database>{xdmp:database('Documents')}</database>
                 </options>)
return
sec:add-query-rolesets($qry-rolesets)

To manually set up just a few query rolesets, use the sec:add-query-rolesets command using XQuery. 

(: add a few query rolesets => run against the Security database :)

xquery version "1.0-ml";
import module namespace sec="http://marklogic.com/xdmp/security" 
  at "/MarkLogic/security.xqy";

let $roleset := sec:query-roleset("new-role")
return
sec:add-query-rolesets(sec:query-rolesets(($roleset))

Using XQuery for Protected Paths

Use the sec:protect-path command to set up your protected paths.

For example:

(: add protected paths -> run against the Security database :)

xquery version "1.0-ml";
import module namespace sec="http://marklogic.com/xdmp/security" 
  at "/MarkLogic/security.xqy";

sec:protect-path("secret", (), (xdmp:permission("els-role-2", "read"))),
sec:protect-path("top-secret", (), (xdmp:permission("els-role-1", "read")))

This example uses a second parameter to set a protected path on the example path namespace. 

(: add protected paths -> run against the Security database :)

xquery version "1.0-ml";
import module namespace sec = "http://marklogic.com/xdmp/security" 
  at "/MarkLogic/security.xqy";

declare namespace ex = "http://marklogic.com/example";

let $role := "executive"
return 
  sec:protect-path(
    "/ex:envelope/ex:instance/employee/salary", 
    (let $prefix := "ex",$namespace-uri := 
      "marklogic.com/example"
    return
    sec:security-path-namespace($prefix, $namespace-uri), 
  (xdmp:permission($role, "read"))
  )

Configure Element Level Security With REST

You can also use the REST Management APIs to configure element level security. The REST properties endpoint is available to create query rolesets and protected paths:

GET /manage/v2/security/properties

Using REST for Query Rolesets

The following XML and JSON examples show what is returned from GET (or used as payload to PUT) when using REST for query rolesets.

This example uses a GET with the response payload in XML:

$ curl -GET --anyauth -u admin:admin 
  -H "Accept:application/xml,Content-Type:application/xml"
  http://localhost:8002/manage/v2/security/properties

This returns:

<security-properties xmlns="http://marklogic.com/manage">
    <query-rolesets>
      <query-roleset>
             <role>432432534053458236326</role>
             <role>454643243253405823326</role>
      </query-roleset>
      <query-roleset>
             <role>124325333458236346123</role>
             <role>124233432432534058213</role>
      </query-roleset>
     </query-rolesets>
</security-properties>

Here is the same example with a JSON reponse payload:

$ curl -GET --anyauth -u admin:admin 
  -H "Accept:application/json,Content-Type:application/json"
  GET:/manage/v2/security/properties

This returns:

{
    "queryRoleset": [
          [
               432232321212123100000,
               432432534053458200000
          ],
          [
               124325333458236346123,
               124233432432534058213
          ]
     ]
}

The REST Management APIs will accept both role names and role IDs in configuring query rolesets with PUT.

The following are example payloads for POST or PUT calls for managing query rolesets.

JSON Example
{
  "role-name": ["manage-admin","rest-writer"]
}
XML Example
<query-roleset-properties xmlns="http://marklogic.com/manage/query-roleset/properties">
  <query-roleset>
    <role-name>rest-reader</role-name>
  </query-roleset>
</query-roleset-properties>

Using REST for Protected Paths

The following XML and JSON examples show what is returned from GET (or used as payload to PUT) when using REST for query rolesets.

This example uses a GET with the reponse payload in XML:

$ curl -GET --anyauth -u admin:admin \
  -H "Accept:application/xml,Content-Type:application/xml" \
  http://localhost:8002/manage/v2/security/properties

This returns:

<security-properties xmlns="http://marklogic.com/manage">
  <protected-paths>
<protect-path>
  <path-namespaces>
      <path-namespace>
          <prefix>ml</prefix>
          <namespace-uri>marklogic.com</namespace-uri>
      </path-namespace>
  </path-namespaces>
      <path-expression>/ml:foo/ml:bar</path-expression>
      <permissions>
        <permission>
          <role-name>user1</role-name>
          <capability>read</capability>
        </permission>
      </permissions>
    </protected-path>
  </protect-paths>
</security-properties>

Here is the same example with a JSON reponse payload:

$ curl -GET --anyauth -u admin:admin \ 
  -H "Accept:application/json,Content-Type:application/json" \
  http://localhost:8002/manage/v2/security/properties

This returns:

"protected-path": [
{
  "path-namespace": [
      {
         "prefix" : "ml",
         "namespace-uri":  "marklogic.com"
      }
   ]
      "path-expression": "/some/path",
      "permissions": [
        {
          "role-name": "user1",
          "capability": "read"
        }
      ]
    }
  ]
}

When DELETE is used, a force=true url param will force deletion of in use protected paths.

To specify an options element namespace in a JSON REST payload, you will need to define an options-ns key to set the namespace.

Combining Document and Element Level Permissions

This section describes how document level and element level permissions interact when both are applied to a document. At the element level read, insert, and node-update permissions can be used as part of the protected path definition.

At the element level, the update and node-update capabilities are equivalent.

This section contains the following topics:

Document Level Security and Indexing

The document level security (document permissions with read capability) interacts with the element level security and affects:

  • The indexing of protected elements and whether index keys are combined with query rolesets
  • Whether protected embedded triples are indexed

During indexing, the element level security of every node in the document is compared to the document's protection. For a given node in the document, the permissions on every matching protected path are compared to the document's permissions. When all matching protected paths are determined to be weaker than the document's protection, the element's protection is considered to be weaker. In this case, the query rolesets for the matching protected paths are not used when indexing the current node. An embedded triple with weaker protection on all of its nodes (subject, predicate and object), is extracted.

How is the element level protection determined to be weaker? In the absence of compartment security, a higher number of roles implies weaker permission because it means more accessibility. More roles in this case doesn't mean the total number of roles. It means that one set of roles is a superset of the other. The smaller set (the subset) is considered stronger because it is more restrictive. Roles are OR'ed by default. If the document is permitted to be accessed by more roles than the element (the element is more restrictive because there are more limitations on access), then the element security is considered to be stronger than the document security. In such a case, the element security is given higher precedence and the element is protected (i.e. the element is more restrictive). The fewer the number of contained or embedded roles, the more restrictive the permissions.

In situations where neither is stronger or it is unclear whether the document security or element security is stronger, the element level is always considered stronger. Only Read capability is checked when comparing the document's permissions to the element's permissions.

Note that there is no flattening of roles (inheritance of permissions) with element level security. Using the helper functions, described in APIs for Element Level Security can facilitate both discovering existing query rolesets and applying them as part of ingestion.

Combination Security Example

More roles does not mean the total number of roles. It means that one set of roles is a superset of the other. The smaller set of roles is considered stronger. Consider the following examples:

Note that in example 1, element level protection is more restrictive that the document level protection. With compartment security, it's more complicated. The security level that has the most compartments wins, because more compartments means that access is more restrictive.

When element security is weaker than the document security, MarkLogic will index the content based on the document level security. MarkLogic lets the document level security protect it.

If the element is considered stronger, then content won't be visible without the correct query rolesets. If the element is weaker, then MarkLogic will return the element as part of a query (with the correct document level permissions).

Node Update Capabilities

Node update capabilities allow you to update document content by node. At the document level xdmp:document-delete and xdmp:document-insert can still be used if you have update capabilities, but node-update provides a finer control when combined with element level security. The node-update capability exists at the document level and at the element level. At the document level, if you have the node-update capability you can call xdmp:node-replace and xdmp:node-delete to modify nodes in a document, but not xdmp:document-delete or xdmp:document-insert. All of the node update built-ins take element level permissions into consideration.

Note that node-update, just like insert, can be seen as a subset of update, meaning that if a role has the update capability, it automatically gets the node-update capability as well.

If you have the update capability at the document level, you can call xdmp:document-insert, xdmp:document-delete, and all node-update functions. When you have the update capability at the document level, the element level security for update will not be checked, it is effectively turned off. If you have the node-update capability, you can only call all node-update functions for that node.

Updates With Element Level Security

You can update content in documents when protected paths have been defined with element level security. Both document level and element level permissions will apply to the content (compartment level permissions may apply as well - see Interactions With Compartment Security for details). With the appropriate permissions, you can use insert and node-update at the element level to modify content containing protected paths. These capabilities take all element level permissions into consideration.

You can also protect document property nodes with element level security. With the node-update/insert capability, you can call xdmp:document-add-properties, xdmp:document-remove-properties, xdmp:document-set-property, or xdmp:document-set-properties. See Document and Element Level Permissions Summary for details.

Node Update and Node Insert at the Element Level

The node-update capability at the element level enables to you replace and delete nodes with xdmp:node-replace and xdmp:node-delete. The insert capability enables you to call xdmp:insert-node-before, xdmp:node-insert-after, and xdmp:node-insert-child.

At the element level, the update and node-update capabilities are equivalent.

Here are some simple examples using the xdmp:insert-node-before, xdmp:insert-node-after, and xdmp:node-replace functions at the element level. These examples assume that both roles have document insert/node-update permissions as well as read permissions for the document and that the query rolesets are configured correctly.

Say that you have a document with these nodes:

<root>
  <foo>hello</foo>
  <bar>World</bar>
</root>

There are two roles; role1 with both read and update permissions on the <foo> node, and role2 with read and node-insert permissionson the <root> node:

<foo>,("role1", "read"),("role2", "read"),("role1", "update")
<root>,("role1", "read"),("role2", "read"),("role2", "insert")

The protected paths look like this: 

sec:protect-path("//foo", (), (
xdmp:permission("role1", "read"),("role1", "update"),"role2", "read")) 
sec:protect-path("//root", (), (
xdmp:permission("role1", "read"),("role2", "read"),("role2", "insert"))

The insert and update permissions check the ancestors of a node as well. See Document and Element Level Permissions Summary for details.

(: insert a new document :)
xdmp:document-insert("/example.xml",
<root>
  <foo>hello</foo>
  <bar>World</bar>
</root>
(xdmp:permission("role1", "read"), xdmp:permission("role2", "read"),
xdmp:permission("role1", "node-update"),("role1", "insert"), xdmp:permission("role2", "node-update"),("role2", "insert")));

As role2, use xdmp:node-insert-before to add a node to the document:

(: add a baz node before the foo node :)
xdmp:node-insert-before(fn:doc("/example.xml")/root/foo,
    <baz>Greetings</baz>); 
(: view the revised document :)
fn:doc("/example.xml")

=>
<root>
  <baz>Greetings</baz>
  <foo>hello</foo>
  <bar>World</bar>
</root>

As role1 you can use xdmp:node-replace to change the <bar> node.

xdmp:node-replace(doc("/example.xml")/root/foo,<foo>Hello</foo>));
doc("/example.xml");
fn:doc("/example.xml") 
=>
<root>
  <baz>Greetings</baz>
  <foo>Hello</foo>
  <bar>World</bar>
</root>

If you are using a user to other than role1 do these same operations, a permission denied exception will be thrown.

Document and Element Level Permissions Summary

This table describes the permissions required to add, remove, or modify content at the document and element level.

Function Signature Document and Element Level Permissions
xdmp:node-replace($old,$new)

Document: node-update is required

Element: $old and all its ancestors, as well as descendants are checked for update/node-update
xdmp:node-delete($old)

Document: node-update is required

Element: $old and all its ancestors as well as descendants are checked for update/node-update

xdmp:node-insert-before($sibling,$new)

Document: insert is required

Element: all ancestors of $sibling are checked for insert
xdmp:node-insert-after($sibling,$new)

Document: insert is required

Element: all ancestors of $sibling are checked for insert
xdmp:node-insert-child($parent,$new)

Document: insert is required

Element $parent and all its ancestors are checked for insert
xdmp:document-add-properties($uri, $props)

Document: node-update is required

Element: the properties root* is checked for insert
xdmp:document-set-property($uri,$prop)

Document: node-update is required

Element:

IF the property to be set doesn't exist, THEN the properties root is checked for insert;

  ELSE

 a.) the properties root* is checked for update/node-update

  b.) the property nodes) and all their descendants are checked for update/node-update
xdmp:document-set-properties($uri, $props)

Document: node-update is required

Element:

IF there is no properties fragment THEN the properties root is checked for insert;

  ELSE

  a.) the properties root* is checked for update/node-update

  b.) all existing property nodes and all their descendants are checked for update/node-update
xdmp:document-remove-properties($uri, $property-names)

Document: node-update is required

Element:

  a.) the properties root* is checked for update/node-update

  b.) all property nodes to be removed and all their descendants are checked for update/node-update

* The properties root is the root of the properties node of a document, not the individual properties contained in the properties node. The properties root is the first line in this diagram:

<prop:properties xmlns:prop="http://marklogic.com/xdmp/property">
   <prop1>. . .</prop1>
   <prop2>. . .</prop2>
   .
   .
   .
   <propN>. . .</propN>
</prop:properties>

See Interactions With Compartment Security for more about combining element level security with compartment security.

Node Update and Document Permissions Expanded

These examples expand on the interactions of element level security and document permissions. This section contains these examples:

Unexpected Behavior with Permissions

In this example the role has the necessary document-level permissions. The example has to do with the element level, protected path permissions. Say you have a document (example.xml) with these nodes:

<foo>
  <bar>
</foo>

For this example role1 has both read and update permissions on the <foo> node, and update permissions on the <bar> node, but no read permissions on the <bar> node:

<foo>, ("role1", "read"), ("role1", "update")
<bar>, ("role1", "update")

It is assumed for these examples that all of the query rolesets are already configured correctly.

If role1 calls this xdmp:node-replace query:

xquery version "1.0-ml"; 

xdmp:node-replace(doc("/example.xml")/foo, <foo><baz>Hello</baz></foo>);

The query will succeed, because role1 has update permissions on /foo.

If role1 calls this xdmp:node-replace query on /bar: 

xquery version "1.0-ml"; 

xdmp:node-replace(doc("/example.xml")/foo/bar, <baz>Hello</baz>);

The expression /foo/bar will return an empty sequence because role1 cannot read the bar element. Hence the node-replace call will effectively be a no-op, because xdmp:node-replace was asked to replace nothing with something.

Different Permissions on the Same Node

Multiple roles can have different permissions on the same node. Some interactions between roles may be unexpected. For example, if you have a document with two nodes <foo> and <bar>. The <bar> node is a child of the <foo> node. 

<foo>
  <bar>

You have two roles; role1 with both read and update permissions on the <foo> node, and role2 with read permissions on the <bar> node:

<foo>, ("role1", "read"), ("role1", "node-update")
<bar>, ("role2", "read")

At the element level, the update and node-update functions are equivalent.

The protected paths for this document would look like this:

sec:protect-path("//foo", (), (
xdmp:permission("els-role-1", "read"),("role1", "node-update")) 

sec:protect-path("//foo/bar", (), (
xdmp:permission("role2", "read"))

With these protected paths, role1 cannot read the <bar> node. But because role1 has update permissions on the parent node (<foo>), role1 can overwrite the <bar> node, even though it cannot read it.

To prevent this, add node-update permissions to the <bar> node. The permissions would now look like this:

<foo>, ("role1", "read"), ("role1", "node-update")
<bar>, ("role2", "read"), ("role2", "node-update")

The presence of the node-update permission on the <bar> node prevents role1 from being able to update and overwrite the <bar> node (the child node of the <foo> node).

This happens because node permissions are checked separately; first there's a check for protected paths for read. Then there is a check for protected paths for update. If no update is found for /foo/bar, then role1 is allowed to update <bar>. If there is a protected path for updating <bar>, then role1 is not allowed to update <bar>.

A More Complex Example

To expand even more on the node-update example with added document permissions, you could have roles with both protected paths and document permissions.

Say you have a document with these nodes:

<foo>
  <bar>
<baz>

At the document level, there are these permissions:

("role1", "read"), ("role1", "node-update")
("role2", "read"), ("role2", "node-update")
("role3", "read"), ("role3", "update")

At the element level, there are these permissions for protected paths:

<foo>, ("role1", "read"), ("role1", "node-update")
<bar>, ("role2", "read"), ("role2", "node-update")

In this example:

  • role1 cannot update (or override) <bar> because at the element level role2 has <bar> protected path permissions
  • role3 can override everything because at the document level it has update capability, but can only read <baz> which has no protected paths.

APIs for Element Level Security

This section includes the following topics:

XQuery APIs

These built-in functions are available to help manage element level security:

With the appropriate permissions, protected path content can be modified using these node update APIs:

These two helper functions can be used to search for protected paths:

REST Management APIs

The REST Management APIs provide the same functionality as the XQuery APIs covered in XQuery APIs for both protected paths and query rolesets.

REST Management APIs for Protected Paths

These REST Management APIs can be used for adding, modifying, or deleting protected paths.

GET /manage/v2/protected-paths

POST /manage/v2/protected-paths

GET /manage/v2/protected-paths/{id|name}

DELETE /manage/v2/protected-paths/{id|name}

GET /manage/v2/protected-paths/{id}/properties

PUT /manage/v2/protected-paths/{id}/properties

REST Management APIs for Query Rolesets

These REST Management APIs are available for managing query rolesets:

GET /manage/v2/query-rolesets

POST /manage/v2/query-rolesets

GET:/manage/v2/query-rolesets/{id|name}

DELETE:/manage/v2/query-rolesets/{id|name}

GET:/manage/v2/query-rolesets/{id|name}/properties

PUT:/manage/v2/query-rolesets/{id|name}/properties

Algorithm That Determines Which Query Rolesets to Use

In MarkLogic 9.0-1, if the path permissions on a node are weaker (as defined in Document Level Security and Indexing) than the document level permissions or its parent node's permissions, the path level permissions will be ignored as far as query rolesets definition is concerned.

A child node will still inherit its parent's query rolesets.

In MarkLogic 9.0-2, the set of query rolesets for a given node (after inheritance from ancestors) will be compacted based on the weaker permissions defined in Document Level Security and Indexing. If a query roleset in the set is weaker than any other query rolesets in the set, that weaker roleset will be removed.

For example:

Roles: role-1, role-2, role-3

Document: 

<foo>Hello<bar>World</bar>,</foo>

with ((role-1, read), (role-2, read), (role-3, read))

Protected Paths: 

//foo (role-1, read), (role-2, read)
//bar (role-1, read)

In MarkLogic 9.0-1, the query rolesets for the bar node is ((role-1, role-2), (role-1)), but in 9.0-2 it is simplified (compacted) to ((role-1)).

If any query roleset in the above set is weaker than the document level permissions, it will be omitted too.

Here is another example:

Roles: role-1, role-2, role-3

Document: 

<foo><bar>Hello</bar></foo>

with (role-1, read)

Protected Paths: 

/foo/bar (role-1, read), (role-2, read)
//bar (role-3, read)

In 9.0-1, the query rolesets for the bar node is ((role-1, role-2), (role-3)), but in 9.0-2 it is simplified (compacted) to ((role-3)) because (role-1, role-2) is weaker than the document level permissions.

Interactions With Compartment Security

You can add an extra level of protection to any content concealed by protected paths by using compartment security in conjunction with element level security. Compartment security adds a finer granularity of protection for content because a user must have the appropriate role and belong to the appropriate compartment to view the concealed content. For more about compartment security see Compartment Security.

A compartment is a name associated with a role. The compartment name is used as an additional check when determining a user's authority to access, modify, or create documents. If compartment security is not used, permissions are checked using OR semantics. For example, if a document has read permissions for role1 and read permissions for role2, without compartment security, a user who has either role1 or role2 can read that document.

If any permission on a document has a compartment, then the user must have that compartment in order to access any of the capabilities, even if the capability is not the one with the compartment. Access to a document requires a permission in each compartment for which there is a permission on the document, regardless of the capability of the permission. So if there is read permission for role compartment1, there must also be an update permission for some role in compartment1 (but not necessarily the same role).

If compartment security is used, then the permissions are checked using AND semantics for each compartment. If the document has compartment permissions for both compartment1 and compartment2, a role must be associated with both compartments to view the document. If two roles have different compartments associated with them (for example compartment1 and compartment2) , a user must have role1 and role2 access the document.

This is in addition to checking the OR semantics for each non-compartmented role, as well as a non-compartmented role that has a corresponding permission on the document. If compartment security is used along with element level security, a user must have both the appropriate compartment security and the appropriate role to view protected content.

Because element level security follows the same role based authorization model, compartment security checks are be done in the same way at the element level. The only difference is that when calculating compartments needed at the element level, only those permissions with the capability being requested (for example read) are checked.

Here is an example using these three roles:

  • role0 (with no compartment)
  • role1 (with compartment1)
  • role2 (with compartment2)

These permissions have been set on the document:

(role0, read), (role1, read), and (role2, update)

With these permissions set on the document, a user with both role1 and role0 cannot perform a read operation. This is because one of the permissions mentions role2, even though it is not for read. In fact, with these permissions at the document level, no one (except for admin) would be able to read the document.

If the above permissions are set for an element, a user with both role1 and role0 will be able to read the element, because element level security checks read, update, and insert permissions separately, based on the operation requested.

Permission checks at the document and element levels are performed independently.

Compartment Security and Indexing

Using more compartments means stronger security because compartments are AND'ed. The roles within the same compartment are OR'ed. When a document or element is protected by more compartments, this implies stricter access. Roles without compartments are OR'ed amongst themselves and then AND'ed with compartment roles. The general rules are:

  • If an element is protected by more compartments than the document's, the element level protection is considered stronger.
  • Within the same compartment, if the element is protected for fewer roles, the element level protection is stronger.
  • There are situations where the weaker/stronger protection cannot be clearly determined. In this case, element level security is always considered to be stronger.

See Node Update and Document Permissions Expanded and Combination Security Example for more about security protection and indexing. For more information about compartment security, see Compartment Security.

Interactions with Other MarkLogic Features

The element level security feature is an index-level feature that is implemented in the universal index, the geospatial index, the bitemporal index, the range index, and the triple index. Features that use a single lexicon (values, elements, element values, sum-aggregation, and so forth.) will work with element level security, as well as operations that make use of the triple index.

Query operations that rely on the triple index (such as SPARQL, SQL, the new version of MarkLogic ODBC, and the Optic API) are supported by element level security. Element-level security can be leveraged by semantic graphs and SQL. In semantics, individual triples can be protected. In SQL, this allows you to enable column-level security by protecting specific columns in a Template (TDE). See Node Update and Document Permissions Expanded for details.

This section describes interactions with these MarkLogic features:

Lexicon Calls

For simple lexicons like values or words, this feature is similar to cts queries (see Others). However, lexicon calls that involve co-occurrences will only work with unprotected values (range-index based SQL implementation has the same problem).

Fragmentation

The indexer in MarkLogic doesn't know the full path when working on child fragments of a parent document, because the indexer indexes the child fragments first before it indexes the parent. Because of this element level security and fragmentation don't work well together, although fragmentation will still work on documents that don't have any protected elements.

Any new document with matching fragmentation and protected elements will be rejected. Either an XDMP-PARENTLINK or an XDMP-FRAGMENTPROTECTEDPATH error will be thrown. When element level security and fragmentation both apply simultaneously to an existing document (already in the database), a reindexing error will be thrown, causing reindexing to stop. User must either remove/fix the matching element level security path or the matching fragmentation element.

For example, if a protected path that ends with baz is added (/foo/bar/baz) and if a fragment root is configured for baz, any document containing node baz (even under a different path /A/B/C/baz) will error out with XDMP-PARENTLINK when the document is inserted or reindexed.

SQL on Range-Index Based Views

SQL that is based on Range-Index views will only work with values that are not protected by element level security.

UDFs (including UDF-based aggregate built-ins)

UDFs that operate on a single range index will work with element level security. This includes the most commonly used aggregate functions like cts:sum-aggregate, cts:stddev, and so on. UDFs that apply to more than one range index will only work with unprotected values.

Reverse Indexes

Similar to the case for triples (see SPARQL), if an element that contains a cts:query matches a protected path of any role, or any part of the cts:query matches any role, the query won't be added into the reverse index unless the document's security is stronger than the element security on the element. See Node Update and Document Permissions Expanded for details. A cts:reverse-query that would normally find a document containing a matching cts:query will no longer match once the embedded cts:query (or its children) is protected by element level security that is stronger than the document's security.

SPARQL

If a sem:triple is inside an element that is concealed for any role and the element level security is stronger than the document security, it will not be put into the triple index. If the triple itself or its subject, predicate, or object is protected, it will not be put into the triple index, unless the document security is stronger than the element level security protection. In some scenarios, where the document's security is stronger than the element security on a triple, the protected triple will be added to the triple index. This is because the document's security already covers the protected element. The information will be protected at the document level. See Node Update and Document Permissions Expanded for details.

Alerting and QBFR

Each target in a QBFR (Query Based Flexible Replication) configuration is associated with a user and a query. A target can only get documents that match the query and that the user is allowed to access. In QBFR, some flows must use the privileged user to run queries because the process needs to figure out what documents will be deleted from a target. Internally, alerting uses reverse queries to determine the set of matching rules for a given document or node. The matching rules are then used to trigger the appropriate action for the target user of each matching rule.

There is a two pass rule matching approach; first the rule matching runs against the full version of the document, then for each matching rule, a second match test is performed using the version of the document that the target user of the rule is allowed to see.

Now, a rule that matches hello will not trigger the action if the target user cannot see hello due to element level security protection. Using element level security, MarkLogic Server will deliver a redacted version of the document, based on element level security configuration of protected paths and the user's role.

When using element level security with Alerting and QBFR, if a query contains a NOT clause, you may see false negatives. What this means is documents might not be replicated when the alerting rule contains a cts:not-query due to the false negatives.

mlcp

When you use mlcp to ingest files from MarkLogic 9 or later to another MarkLogic 9 or later instance, the protected paths and node-update permissions will be preserved.

If you use mlcp to export a database archive that includes documents with the node-update permission, and then import the archive into MarkLogic 8.0-6 or earlier, the behavior is undefined. If you import the archive in MarkLogic 8.0-7 or a later version of MarkLogic 8, the node-update permission is silently discarded.

Similarly, if you use mlcp to copy documents from MarkLogic 9 or later to MarkLogic 8.0-6 or earlier, the behavior is undefined. If your copy destination is MarkLogic 8.0-7 or a later version of MarkLogic 8, the node-update permission is silently discarded.

XCC

If you use XCC to insert a document with the node-update permission into MarkLogic 8.0-6 or earlier, the behavior is undefined.

If you use XCC to insert a document with the node-update permission into MarkLogic 8.0-7 or a later version of MarkLogic 8, the node-update permission is silently discarded.

These restrictions apply to using Session.insertContent with a Content object whose ContentCreateOptions include the ContentCapability.NODE_UPDATE capability.

Bitemporal

Do not protect system axis for bitemporal queries when using element level security.

Others

A key concept to support cts queries with element level security is query rolesets. A query roleset is simply a list of roles. When indexing, MarkLogic takes query roleset information into consideration and essentially partitions indexes based on query rolesets. All queries (except for composite ones like and-query) will look into indexes for different query rolesets based on the caller's role and logically OR the results. See Query Rolesets for more about query rolesets.

There are special rules for cts queries, phrase breaks, field values, geo element pairs, auditing and term-queries when the elements involved are protected.

  • cts queries - Positions are always calculated based on the original (full) document, prior to any concealing. This implies that the distances calculated based on indexes will be larger than what appears in the concealed document.
  • Phrase breaks - When indexing, any element that is protected is considered a phrase break. Consider the this example: <foo>1<bar>2 3</bar>4</foo>. If bar is protected by any protected path, then it is considered a phrase break regardless whether a phrase through is defined on it. So in the example, 2 3 is still a phrase, but 1 2 or 3 4 is not. 1 4 is not a phrase either.
  • Fields - For an XML document, field values or field range values are sometimes calculated by concatenating elements included in the field. If those elements don't have the same rolesets (permissions), concatenating can cause leaking of information. MarkLogic server will treat this as a misconfiguration and log a warning. The query result on such a field is undefined.
  • Geo element pair with inconsistent permissions - Similar to the field case above, if permissions on the two elements (or JSON properties) of the geo pair are not consistent (or either of the two elements has different permissions from the parent node), MarkLogic server will treat it as a misconfiguration and log a warning. The query result is undefined in this case.
  • Auditing -
    1. For the document-read event, if the node involved has any element concealed, the string concealed will be reported in the event. Here is an example:
      2016-10-18 15:45:29.886 event=document-read; type=concealed; uri=foo.json; database=Documents; success=true;
    2. When a node or properties update built-in call is rejected due to the lack of element-level permissions, the no-permission event will be reported. This is very similar to how the event is used when such a call is rejected due to the lack of document-level permissions.
  • term-query - Element level security won't prevent a malicious user from getting a term key through xdmp:plan from a different MarkLogic deployment, then passing that to a cts:term-query to find out information she is not supposed to see on the current MarkLogic deployment. The solution is to add a new execute privilege term-query to protect cts:term-query. For backward compatibility, this privilege will only be checked when element level security is in use (i.e., when at least one protected path is configured).

Rolling Upgrades

For rolling upgrades, configuration API calls (as well as Admin GUIs) will throw an error when a rolling upgrade (from a release that doesn't support element level security) has not yet completed and been committed across the cluster. Document inserts (or set-permissions) with the new node-update capability will be rejected if the effective version is not 9.0-1 or above.

« Previous chapter
Next chapter »