Application Developer's Guide — Chapter 16

Terms Used in this Chapter

• REST stands for Representational State Transfer, which is an architecture style that, in the context of monitoring MarkLogic Server, describes the use of HTTP to make calls between a monitoring application and monitor host.
• A Rewriter interprets the URL of the incoming request and rewrites it to an internal URL that services the request. A rewriter can be implemented as an XQuery module as described in this chapter, or as an XML file as described in xWeb Services.
• An Endpoint is an XQuery module on MarkLogic Server that is invoked by and responds to an HTTP request.

Overview of the REST Library

The REST Library consists of a set of XQuery functions that support URL rewriting and endpoint validation and a MarkLogic REST vocabulary that simplifies the task of describing web service endpoints. The REST vocabulary is used to write declarative descriptions of the endpoints. These descriptions include the mapping of URL parts to parameters and conditions that must be met in order for the incoming request to be mapped to an endpoint.

The REST Library contains functions that simplify:

• Creating a URL rewriter for mapping incoming requests to endpoints
• Validating that applications requesting resources have the necessary access privileges
• Validating that incoming requests can be handled by the endpoints
• Reporting errors

The REST vocabulary allows you to use same description for both the rewriter and the endpoint.

When you have enabled RESTful access to MarkLogic Server resources, applications access these resources by means of a URL that invokes an endpoint module on the target MarkLogic Server host.

The REST library does the following:

1. Validates the incoming HTTP request.
2. Authorizes the user.
3. Rewrites the resource path to one understood internally by the server before invoking the endpoint module.

If the request is valid, the endpoint module executes the requested operation and returns any data to the application. Otherwise, the endpoint module returns an error message.

The API signatures for the REST Library are documented in the MarkLogic XQuery and XSLT Function Reference. For additional information on URL rewriting, see Setting Up URL Rewriting for an HTTP App Server.

A Simple XQuery Rewriter and Endpoint

This section describes a simple rewriter script that calls a single endpoint.

Navigate to the /<MarkLogic_Root>/bill directory and create the following files with the described content.

Create a module, named requests.xqy, with the following content:

xquery version "1.0-ml";
module namespace     requests="http://marklogic.com/appservices/requests";
import module namespace rest = "http://marklogic.com/appservices/rest"
    at "/MarkLogic/appservices/utils/rest.xqy";
declare variable $requests:options as element(rest:options) :=
  <options xmlns="http://marklogic.com/appservices/rest">
    <request uri="^/(.+)$" endpoint="/endpoint.xqy">
      <uri-param name="play">$1.xml</uri-param>
    </request>
  </options>;

Create a module, named url_rewriter.xqy, with the following content:

xquery version "1.0-ml";
import module namespace rest = "http://marklogic.com/appservices/rest"
     at "/MarkLogic/appservices/utils/rest.xqy"; 
import module namespace requests =   "http://marklogic.com/appservices/requests"
     at "requests.xqy";
rest:rewrite($requests:options)

Create a module, named endpoint.xqy, with the following content:

xquery version "1.0-ml";
import module namespace rest = "http://marklogic.com/appservices/rest"
    at "/MarkLogic/appservices/utils/rest.xqy";
import module namespace requests =   "http://marklogic.com/appservices/requests" at "requests.xqy";
let $request := $requests:options/rest:request
                  [@endpoint = "/endpoint.xqy"][1]
let $map  := rest:process-request($request) 
let $play := map:get($map, "play") 
return
  fn:doc($play)

Enter the following URL, which uses the bill App Server created in Preparing to Run the Examples:

http://localhost:8060/macbeth

The rest:rewrite function in the rewriter uses an options node to map the incoming request to an endpoint. The options node includes a request element with a uri attribute that specifies a regular expression and an endpoint attribute that specifies the endpoint module to invoke in the event the URL of an incoming request matches the regular expression. In the event of a match, the portion of the URL that matches (.+) is bound to the $1 variable. The uri-param element in the request element assigns the value of the $1 variable, along with the .xml extension, to the play parameter.

<rest:options xmlns="http://marklogic.com/appservices/rest">
  <rest:request uri="^/(.+)" endpoint="/endpoint.xqy">
    <uri-param name="play">$1.xml</uri-param>
  </rest:request>
</rest:options>

In the example rewriter module above, this options node is passed to the rest:rewrite function, which outputs a URL that calls the endpoint module with the parameter play=macbeth.xml:

/endpoint.xqy?play=macbeth.xml

The rest:process-request function in the endpoint locates the first request element associated with the endpoint.xqy module and uses it to identify the parameters defined by the rewriter. In this example, there is only a single parameter, play, but for reasons described in Extracting Multiple Components from a URL, when there are multiple request elements for the same endpoint, then the request element that extracts the greatest number of parameters from a URL should be listed in the options node ahead of those that extract fewer parameters.

let $request := $requests:options/rest:request
                  [@endpoint = "/endpoint.xqy"][1]

The rest:process-request function in the endpoint uses the request element to parse the incoming request and return a map that contains all of the parameters as typed values. The map:get function extracts each parameter from the map, which is only one in this example.

let $map  := rest:process-request($request) 
let $play := map:get($map, "play") 

Notes About Rewriter Match Criteria

The default behavior for the rewriter is to match the request against all of the criteria: URI, accept headers, content-type, conditions, method, and parameters. This assures that no endpoint will ever be called except in circumstances that perfectly match what is expected. It can sometimes, however, lead to somewhat confusing results. Consider the following request node:

<request uri="/path/to/resource" endpoint="/endpoint.xqy">
  <param name="limit" as="decimal"/>
</request>

An incoming request of the form:

/path/to/resource?limit=test

does not match that request node (because the limit is not a decimal). If there are no other request nodes which match, then the request will return 404 (not found).

That may be surprising. Using additional request nodes to match more liberally is one way to address this problem. However, as the number and complexity of the requests grows, it may become less attractive. Instead, the rewriter can be instructed to match only on specific parts of the request. In this way, error handling can be addressed by the called module.

The match criteria are specified in the call to rest:rewrite. For example:

rest:rewrite($options, ("uri", "method"))

In this case, only the URI and HTTP method will be used for the purpose of matching.

The criteria allowed are:

• uri = match on the URI
• accept = match on the accept headers
• content-type = match on the content type
• conditions = match on the conditions
• method = match on the HTTP method
• params = match on the params

The request must match all of the criteria specified in order to be considered a match.

The options Node

The REST Library uses an options node to map incoming requests to endpoints. The options node must be declared as type element(rest:options) and must be in the http://marklogic.com/appservices/rest namespace.

Below is a declaration of a very simple options node:

declare variable $options as element(rest:options) :=
  <rest:options xmlns="http://marklogic.com/appservices/rest">
    <rest:request uri="^/(.+)" endpoint="/endpoint.xqy">
      <uri-param name="play">$1.xml</uri-param>
    </rest:request>
  </rest:options>;

The table below summarizes all of the possible elements and attributes in an options node. On the left are the elements that have attributes and/or child elements. Attributes for an element are listed in the Attributes column. Attributes are optional, unless designated as '(required)'. Any first-level elements of the element listed on the left are listed in the Child Elements column. The difference between the user-params="allow" and user-params="allow-dups" attribute values is that allow permits a single parameter for a given name, and allow-dups permits multiple parameters for a given name.

options

user-params = ignore
            | allow
            | allow-dups
            | forbid

request

0..n

request

uri=string

endpoint=string

user-params = ignore
            | allow
            | allow-dups
            | forbid

uri-param

param

http

auth

function

accept

user-agent

and

or

0..n

0..n

0..n

0..n

0..n

0..n

0..n

0..n

0..n

uri-param

name=string (required)

as=string

param

name=string (required)

as=string

values=string

match=string

default=string

required = true
         | false

repeatable = true
           | false

pattern = <regex>

http

method = string (required)

user-params = ignore
            | allow
            | allow-dups
            | forbid

param

auth

function

accept

user-agent

and

or

0..n

0..n

0..n

0..n

0..n

0..n

0..n

auth

privilege

kind

0..n

0..n

function

ns=string (required)

apply=string (required)

at=string (required)

Validating options Node Elements

You can use the rest:check-options function to validate an options node against the REST schema. For example, to validate the options node defined in the requests.xqy module described in A Simple XQuery Rewriter and Endpoint, you would do the following:

xquery version "1.0-ml";
import module namespace rest = "http://marklogic.com/appservices/rest"
    at "/MarkLogic/appservices/utils/rest.xqy";
import module namespace requests =   "http://marklogic.com/appservices/requests" at "requests.xqy";
rest:check-options($requests:options)

An empty sequence is returned if the options node is valid. Otherwise an error is returned.

You can also use the rest:check-request function to validate request elements in an options node. For example, to validate all of the request elements in the options node defined in the requests.xqy module described in A Simple XQuery Rewriter and Endpoint, you would do the following:

xquery version "1.0-ml";
import module namespace rest = "http://marklogic.com/appservices/rest"
    at "/MarkLogic/appservices/utils/rest.xqy";
declare option xdmp:mapping "false";
rest:check-request($requests:options/rest:request)

An empty sequence is returned if the request elements are valid. Otherwise an error is returned.

Before calling the rest:check-request function, you must set xdmp:mapping to false to disable function mapping.

Extracting Multiple Components from a URL

An options node may include one or more request elements, each of which may contain one or more uri-param elements that assign parameters to parts of the request URL. The purpose of each request element is to detect a particular URL pattern and then call an endpoint with one or more parameters. Extracting multiple components from a URL is simply a matter of defining a request element with a regular expression that recognizes particular URL pattern and then binding the URL parts of interest to variables.

For example, you want expand the capability of the rewriter described in A Simple XQuery Rewriter and Endpoint and add the ability to use a URL like the one below to display an individual act in a Shakespeare play:

http://localhost:8060/macbeth/act3

The options node in requests.xqy might look like the one below, which contains two request elements. The rewriter employs a first-match rule, which means that it tries to match the incoming URL to the request elements in the order they are listed and selects the first one containing a regular expression that matches the URL. In the example below, if an act is specified in the URL, the rewriter uses the first request element. If only a play is specified in the URL, there is no match in the first request element, but there is in the second request element.

The default parameter type is string. Non-string parameters must be explicitly typed, as shown for the act parameter below. For more information on typing parameters, see Parameter Types.

<options>
  <request uri="^/(.+)/act(\d+)$" endpoint="/endpoint.xqy">
    <uri-param name="play">$1.xml</uri-param>
    <uri-param name="act" as="integer">$2</uri-param>
  </request>
  <request uri="^/(.+)/?$" endpoint="/endpoint.xqy">
    <uri-param name="play">$1.xml</uri-param>
  </request>
</options>

When an act is specified in the incoming URL, the first request element binds macbeth and 3 to the variables $1 and $2, respectively, and then assigns them to the parameters named, play and act. The URL rewritten by the rest:rewrite function looks like:

/endpoint.xqy?play=macbeth.xml&act=3

The following is an example endpoint module that can be invoked by a rewriter that uses the options node shown above. As described in A Simple XQuery Rewriter and Endpoint, the rest:process-request function in the endpoint uses the request element to parse the incoming request and return a map that contains all of the parameters as typed values. Each parameter is then extracted from the map by means of a map:get function. If the URL that invokes this endpoint does not include the act parameter, the value of the $num variable will be an empty sequence.

The first request element that calls the endpoint.xqy module is used in this example because, based on the first-match rule, this element is the one that supports both the play and act parameters.

xquery version "1.0-ml";
import module namespace rest = "http://marklogic.com/appservices/rest"
    at "/MarkLogic/appservices/utils/rest.xqy";
import module namespace requests =   "http://marklogic.com/appservices/requests" at "requests.xqy";
  let $request := $requests:options/rest:request
                    [@endpoint = "/endpoint.xqy"][1]
  let $map  := rest:process-request($request) 
  let $play := map:get($map, "play") 
  let $num  := map:get($map, "act") 
return
  if (empty($num))
  then   
    fn:doc($play)
  else 
    fn:doc($play)/PLAY/ACT[$num]

Handling Errors

The REST endpoint library includes a rest:report-error function that performs a simple translation of MarkLogic Server error markup to HTML. You can invoke it in your modules to report errors:

try {
  let $params := rest:process-request($request)
  return
    ...the non-error case...
} catch ($e) {
  rest:report-error($e)
}

If the user agent making the request accepts text/html, a simple HTML-formatted response is returned. Otherwise, it returns the raw error XML.

You can also use this function in an error handler to process all of the errors for a particular application.

Handling Redirects

As shown in the previous sections of this chapter, the URL rewriter translates the requested URL into a new URL for dispatching within the server. The user agent making the request is totally unaware of this translation. As REST Librarys mature and expand, it is sometimes useful to use redirection to respond to a request by telling the user agent to reissue the request at a new URL.

For example, previous users accessed the macbeth play using the following URL pattern:

http://localhost:8060/Shakespeare/macbeth

You want to redirect the URL to:

http://localhost:8060/macbeth

The user can tell that this redirection happened because the URL in the browser address bar changes from the old URL to the new URL, which can then be bookmarked by the user.

You can support such redirects by adding a redirect.xqy module like this one to your application:

xquery version "1.0-ml";
import module namespace rest="http://marklogic.com/appservices/rest"
     at "/MarkLogic/appservices/utils/rest.xqy";
import module namespace requests =
  "http://marklogic.com/appservices/requests" at "requests.xqy";
(: Process requests to be handled by this endpoint module. :)
let $request := $requests:options/rest:request
                    [@endpoint = "/redirect.xqy"][1]
let $params  := rest:process-request($request)
(: Get parameter/value map from request. :)
let $query  := fn:string-join(
    for $param in map:keys($params)
    where $param != "__ml_redirect__"
    return
      for $value in map:get($params, $param)
      return
        fn:concat($param, "=", fn:string($value)),
  "&")
(: Return the name of the play along with any parameters. :)
let $ruri   := fn:concat(map:get($params, "__ml_redirect__"),
                      if ($query = "") then ""
                      else fn:concat("?", $query))
(: Set response code and redirect to new URL. :)
return
  (xdmp:set-response-code(301, "Moved permanently"),
   xdmp:redirect-response($ruri))

In the options node in requests.xqy, add the following request elements to perform the redirect:

<request uri="^/shakespeare/(.+)/(.+)" endpoint="/redirect.xqy">
  <uri-param name="__ml_redirect__">/$1/$2</uri-param>
</request>
<request uri="^/shakespeare/(.+)" endpoint="/redirect.xqy">
  <uri-param name="__ml_redirect__">/$1</uri-param>
</request>

Your options node should now look like the one shown below. Note that the request elements for the redirect.xqy module are listed before those for the endpoint.xqy module. This is because of the first-match rule described in Extracting Multiple Components from a URL.

<options xmlns="http://marklogic.com/appservices/rest">
  <request uri="^/shakespeare/(.+)/(.+)" endpoint="/redirect.xqy">
    <uri-param name="__ml_redirect__">/$1/$2</uri-param>
  </request>
  <request uri="^/shakespeare/(.+)" endpoint="/redirect.xqy">
    <uri-param name="__ml_redirect__">/$1</uri-param>
  </request>
  <request uri="^/(.+)/act(\d+)" endpoint="/endpoint.xqy">
    <uri-param name="play">$1.xml</uri-param>
    <uri-param name="act" as="integer">$2</uri-param>
  </request>
  <request uri="^/(.+)$" endpoint="/endpoint.xqy">
    <uri-param name="play">$1.xml</uri-param>
  </request>
</options>

You can employ as many redirects as you want through the same redirect.xqy module by changing the value of the __ml_redirect__ parameter.

Handling HTTP Verbs

A request that doesn't specify any verbs only matches HTTP GET requests. If you want to match other verbs, simply list them by using the http element with a method attribute in the request element:

<request uri="^/(.+?)/?$" endpoint="/endpoint.xqy">
  <uri-param name="play">$1.xml</uri-param>
  <http method="GET"/>
  <http method="POST"/>
</request>

This request will match (and validate) if the request method is either an HTTP GET or an HTTP POST.

The following topics describe use cases for mapping requests with verbs and simple endpoints that service those requests:

• Handling OPTIONS Requests
• Handling POST Requests

xquery version "1.0-ml";
import module namespace rest="http://marklogic.com/appservices/rest"
  at "/MarkLogic/appservices/utils/rest.xqy";
import module namespace requests =
  "http://marklogic.com/appservices/requests" at "requests.xqy";
(: Process requests to be handled by this endpoint module. :)
let $request := $requests:options/rest:request
                  [@endpoint = "/options.xqy"][1]
(: Get parameter/value map from request. :)
let $params  := rest:process-request($request)
let $uri     := map:get($params, "__ml_options__")
let $accept  := xdmp:get-request-header("Accept")
let $params  := map:map()
(: Get request element that matches the specified URL. :)
let $request := rest:matching-request($requests:options, 
                                      $uri,
                                      "GET",
                                      $accept,
                                      $params)
(: If URL is '/', return options node. Otherwise, return request
   element that matches the specified URL. :)
return
  if ($uri = "/")
  then
    $requests:options
  else
    $request

<request uri="^(.+)$" endpoint="/options.xqy" user-params="allow">
  <uri-param name="__ml_options__">$1</uri-param>
  <http method="OPTIONS"/>
</request>

xdmp:http-options("http://localhost:8011/",
<options xmlns="xdmp:http">
  <authentication method="digest">
    <username>name</username>
    <password>password</password>
  </authentication>
</options>)

xdmp:http-options("http://localhost:8011/shakespeare/macbeth",
<options xmlns="xdmp:http">
  <authentication method="digest">
    <username>name</username>
    <password>password</password>
  </authentication>
</options>)

<request uri="^/shakespeare/(.+)" endpoint="/redirect.xqy"
  xmlns="http://marklogic.com/appservices/rest">
    <uri-param name="__ml_redirect__">/$1</uri-param>
</request>

<request uri="^(.+)$" endpoint="/options.xqy" user-params="allow">
  <uri-param name="__ml_options__">$1</uri-param>
  <http method="OPTIONS"/>
</request>

xquery version "1.0-ml";
import module namespace rest="http://marklogic.com/appservices/rest"
       at "/MarkLogic/appservices/utils/rest.xqy";
import module namespace requests =
  "http://marklogic.com/appservices/requests" at "requests.xqy";
(: Process requests to be handled by this endpoint module. :)
let $request := $requests:options/rest:request
                  [@endpoint = "/post.xqy"][1]
(: Get parameter/value map from request. :)
let $params  := rest:process-request($request)
let $posturi := map:get($params, "_ml_post_")
let $type    := xdmp:get-request-header('Content-Type')
(: Obtain the format of the content. :)
let $format := 
  if ($type = 'application/xml' or ends-with($type, '+xml'))
  then
    "xml"
  else
    if (contains($type, "text/"))
    then "text"
    else "binary"
(: Insert the content of the request body into the database. :)
let $body := xdmp:get-request-body($format)
return
  (xdmp:document-insert($posturi, $body),
  concat("Successfully uploaded: ", $posturi, "
"))

<request uri="^/post/(.+)$" endpoint="/post.xqy">
  <uri-param name="_ml_post_">$1</uri-param>
  <http method="POST"/>
</request>

let $document:= xdmp:quote(
  <html>
    <title>My Document</title>
    <body> 
      This is my document.
    </body>
  </html>)
return
  xdmp:http-post("http://localhost:8011/post/mydoc.xml",
    <options xmlns="xdmp:http">
      <authentication method="digest">
        <username>name</username>
        <password>password</password>
      </authentication>
      <data>{$document}</data>
      <headers>
        <content-type>text/xml</content-type>
      </headers>
    </options>)

Defining Parameters

This section details the uri-param and param elements in a request element. The topics in this section are:

• Parameter Types
• Supporting Parameters Specified in a URL
• Required Parameters
• Default Parameter Value
• Specifying a List of Values
• Repeatable Parameters
• Parameter Key Alias
• Matching Regular Expressions in Parameters with the match and pattern Attributes

  <request uri="^/(.+)/act(\d+)$" endpoint="/endpoint.xqy">
    <uri-param name="play">$1.xml</uri-param>
    <uri-param name="act" as="integer">$2</uri-param>
  </request>

http://host:port/url-path?param=value

http://localhost:8011/macbeth/act4?scene=2

<request uri="^/(.+)/act(\d+)$" endpoint="/endpoint.xqy">
  <uri-param name="play">$1.xml</uri-param>
  <uri-param name="act" as="integer">$2</uri-param>
  <param name="scene" as="integer" match="(.+)">$1</param>
</request>

xquery version "1.0-ml";
import module namespace rest = "http://marklogic.com/appservices/rest"
    at "/MarkLogic/appservices/utils/rest.xqy";
import module namespace requests =
  "http://marklogic.com/appservices/requests" at "requests.xqy";
let $request := $requests:options/rest:request
                 [@endpoint = "/requests.xqy"][1]
let $map  := rest:process-request($request) 
let $play := map:get($map, "play") 
let $num  := map:get($map, "act")
let $scene := map:get($map, "scene") 
return
    if (empty($num))
    then
      fn:doc($play)
    else if (empty($scene))
    then 
      fn:doc($play)/PLAY/ACT[$num]
    else
      fn:doc($play)/PLAY/ACT[$num]/SCENE[$scene]

<param name="scene" as="integer" match="(.+)" required="true">
  $1
</param>

<param name="scene" as="integer" match="(.+)" default="1">
  $1
</param>

<param name="scene" as="integer" values="1|2|3" default="1"/>

<param name="css" repeatable="true"/>

http://whatever/endpoint?a=b&c[]=d&c[]=e

<param name="format" match="^application/(.*?)(\+xml)?$">
    $1
</param>

<param name="test" values="foo|bar" match="^(.+)$">
    baz-$1
</param>

<param name="test" values="baz-foo|baz-bar" match="^(.+)$">
    baz-$1
</param>

pattern='xmlns:.+'
pattern='val[0-9]+'

Adding Conditions

You can add conditions, either in the body of the request, in which case they apply to all verbs, or within a particular verb. For example, the request element below contains an auth condition for the POST verb and a user-agent condition for both GET and POST verbs.

<request uri="^/slides/(.+?)/?$" endpoint="/slides.xqy">
    <uri-param name="play">$1.xml</uri-param>
    <http method="GET"/>
    <http method="POST">
      <auth>
        <privilege>http://example.com/privs/editor</privilege>
        <kind>execute</kind>
      </auth>
    </http>
    <user-agent>ELinks</user-agent>
</request>

With this request, only users with the specified execute privilege can POST to that URL. If a user without that privilege attempts to post, this request won't match and control will fall through to the next request. In this way, you can provide fallbacks if you wish.

In a rewriter, failing to match a condition causes the request not to match. In an endpoint, failing to match a condition raises an error.

The topics in this section are:

• Authentication Condition
• Accept Headers Condition
• User Agent Condition
• Function Condition
• And Condition
• Or Condition
• Content-Type Condition

<auth>
  <privilege>privilege-uri</privilege>
  <kind>kind</kind>
</auth>

<request uri="^/post/(.+)$" endpoint="/post.xqy">
  <uri-param name="_ml_post_">$1</uri-param>
  http method="POST">
    <auth>
      <privilege>
         http://marklogic.com/xdmp/privileges/infostudio
      </privilege>
      <kind>execute</kind>
    </auth>
  </http>
</request>

<accept>application/json</accept>

<user-agent>ELinks</user-agent>

<function ns="http://example.com/module" 
          apply="my-function" 
          at="utils.xqy"/>

<and>
  ...conditions...
</and>

<and>
  <accept>text/html</accept>
  <user-agent>ELinks</user-agent>
</and>

<or>
  ...conditions...
</or>

<or>
  <accept>text/html</accept>
  <accept>text/plain</accept>
</or>

Preparing to Run the Examples

Before you can run the examples in this chapter, you must perform the following steps:

• Load the Example Data
• Create the Example App Server

xquery version "1.0-ml";
import module namespace ooxml= "http://marklogic.com/openxml" 
          at "/MarkLogic/openxml/package.xqy";
xdmp:set-response-content-type("text/plain"),
let $zip-file := 
 xdmp:document-get("http://www.ibiblio.org/bosak/xml/eg/shaks200.zip")
return for $play in ooxml:package-uris($zip-file)
  where fn:contains($play , ".xml") 
    return (let $node := xdmp:zip-get ($zip-file, $play)
      return xdmp:document-insert($play, $node) )