MarkLogic Server 11.0 Product Documentation

REST Application Developer's Guide — Chapter 1

Preparation

Before beginning this walkthrough, you should have the following software installed:

• MarkLogic Server, version 6.0-1 or later
• curl, a command line tool for issuing HTTP requests, or an equivalent tool.

Though the examples rely on curl, you can use any tool or library capable of sending HTTP requests. If you are not familiar with curl or do not have curl on your system, see Introduction to the curl Tool.

To create the input documents used by the walkthrough:

1. Create a text file named one.xml with the following contents:

   <one>
     <child>The noble Brutus has told Caesar was ambitious</child>
   </one>

2. Create a text file named two.json with the following contents:

   {
     "two": {
       "child": "I come to bury Caesar, not to praise him."
     }
   }

Choose a REST API Instance

You must have a REST API instance to use the REST Client API. A REST API instance is an HTTP App Server specially configured to service HTTP requests against the API. For details, see What Is an Instance?.

Each REST API instance can only host a single application. You cannot share the modules database across multiple REST API instances.

When you install MarkLogic Server 8 or later, the App Server on port 8000 can be used as a REST API instance. This instance is attached to the Documents database. The examples in this walkthrough and the remainder of this guide use the REST API instance on port 8000.

You can also create a REST API instance on a different port, attached to a different database. For details, see Creating an Instance.

Load Documents Into the Database

This procedure loads sample content into the database associated with your REST API instance using the /documents service. The /documents service allows you to create, read, update and delete documents in the database.

To load the sample documents into the database:

1. Navigate to the directory containing the sample documents you created in Preparation.
2. Execute the following command to load one.xml into the database with the URI /xml/one.xml:

   $ curl --anyauth --user user:password -X PUT -d@'./one.xml' \
       -H "Content-type: application/xml" \
       'http://localhost:8000/LATEST/documents?uri=/xml/one.xml'

   The URL tells the /documents service to create an XML document with database URI /xml/one.xml (uri=...) from the contents in the request body. If the request succeeds, the service returns status 201 (Document Created).

3. Execute the following command to load two.json into the database with the URI /json/two.json:

   $ curl --anyauth --user user:password -X PUT -d@'./two.json' \
     -H "Content-type: application/json" \
     'http://localhost:8000/LATEST/documents?uri=/json/two.json'

4. Optionally, use Query Console to explore the database. The database should contain 2 documents, /xml/one.xml and /json/two.json.

To learn more about the document manipulation features of the documents service, see Manipulating Documents.

Search the Database

The REST Client API provides several query services. This procedure uses the search service to perform a simple string query search of the database, finding documents containing caesar. For details, see Using and Configuring Query Features.

To search the database:

1. Execute the following command to send a search request to the instance, requesting matches to the search string caesar. Results are returned as XML by default.

   $ curl --anyauth --user user:password \
       'http://localhost:8000/LATEST/search?q=caesar'

2. Examine the XML search results returned in the response body. Notice that there are two matches, one in each document.

   <search:response snippet-format="snippet" total="2" start="1" ...>
     <search:result index="1" uri="/xml/one.xml" ...>
       <search:snippet>
         <search:match path="fn:doc(&quot;/xml/one.xml&quot;)/one/child">The noble Brutus has told <search:highlight>Caesar</search:highlight> was ambitious</search:match>
       </search:snippet>
     </search:result>
     <search:result index="2" uri="/json/two.json" path="fn:doc(&quot;/json/two.json&quot;)" score="2048" confidence="0.283107" fitness="0.235702">
       <search:snippet>
         <search:match path="fn:doc(&quot;/json/two.json&quot;)/*:json/*:two/*:child">I come to bury <search:highlight>Caesar</search:highlight>, not to praise him.</search:match>
       </search:snippet>
     </search:result>
     <search:qtext>caesar</search:qtext>
     <search:metrics>...</search:metrics>
   </search:response>

3. Run the search command again, generating JSON output by an Accept header:

   $ curl --anyauth --user user:password \
       -H "Accept: application/json" \
       'http://localhost:8000/LATEST/search?q=caesar'
   {
     "snippet-format": "snippet",
     "total": 2,
     "start": 1,
     "page-length": 10,
     "results": [
       {
         "index": 1,
         "uri": "\/xml\/one.xml",
         "path": "fn:doc(\"\/xml\/one.xml\")",
         "score": 2048,
         "confidence": 0.283107,
         "fitness": 0.235702,
         "matches": [
           {
             "path": "fn:doc(\"\/xml\/one.xml\")\/one\/child",
             "match-text": [
               "The noble Brutus has told ",
               {
                 "highlight": "Caesar"
               },
               " was ambitious"
             ]
           }
         ]
       },
       {
         "index": 2,
         "uri": "\/json\/two.json",
         "path": "fn:doc(\"\/json\/two.json\")",
         "score": 2048,
         "confidence": 0.283107,
         "fitness": 0.235702,
         "matches": [
           {
             "path": "fn:doc(\"\/json\/two.json\")\/*:json\/*:two\/*:child",
             "match-text": [
               "I come to bury ",
               {
                 "highlight": "Caesar"
               },
               ", not to praise him."
             ]
           }
         ]
       }
     ],
     "qtext": "caesar",
     "metrics": { ... }
   }

Additional query features allow you to search using structured queries or Query By Example (QBE), to search by JSON property and value or XML element and element attribute values, and to search and analyze lexicons and range indexes. You can also define search options to tailor your search and results. For details, see Using and Configuring Query Features.

Tear Down the REST API Instance

If you are using the pre-configured REST API instance on port 8000, skip this step. If you are using a REST API instance on another port that you created to walk through the examples, then you can tear it down following these instructions.

This procedure uses the rest-apis service on port 8002 to remove a REST Client API instance. By default, removing the instance leaves the content and modules databases associated with the instance intact, but in this example we remove them by using the include request parameter.

Tearing down a REST API instance causes a server restart.

Follow this procedure to remove your instance and associated content and modules databases:

1. Run the following shell command to remove the instance and database. Change the instance-name to the name you used when you created your instance.

   $ curl --anyauth --user user:password -X DELETE \
     'http://localhost:8002/LATEST/rest-apis/instance-name?include=content&include=modules'

2. Navigate to the Admin Interface in your browser to confirm removal of the databases and App Server:

   http://localhost:8001

Basic Security Requirements

The user with which you make a REST Client API request must have appropriate privileges for the content accessed by the request, such as permission to read or update documents in the target database.

In addition, the user must use one or more of the pre-defined roles listed below, or the equivalent privileges. The role/privilege requirement for each REST Client API operation is listed in the MarkLogic REST API Reference. The capabilities of each role in the table is subsumed in the roles below it.

To restrict access on a per-user basis, you should use custom roles, rather than assigning users to the pre-defined rest-reader and rest-writer roles. For details, see Controlling Access to Documents and Other Artifacts.

Some operations require additional privileges, such as using a database other than the default database associated with the REST API and using the /eval and /invoke services. These requirements are detailed elsewhere in Security Requirements.

Controlling Access to Documents and Other Artifacts

In MarkLogic 10.0-1, when inserting documents the REST API assigns permissions based only on the default permissions configured for the user and role. For further information see Change in Default rest-reader and rest-writer Permissions in the Release Notes.

• If you use the convenience rest-writer role to write documents, the documents will be readable by the convenience rest-reader role and writable by the convenience rest-writer role.
• If you use your own role with the rest-writer privilege to write documents, the documents will be writable and readable by roles specified by the default permissions of your own role.
• If those default roles have both the appropriate permission on the document and also the rest-reader or rest-writer privileges, those default roles will be able to execute the read or write operation with the REST API.

To enable users to create and update documents using the REST API yet restrict access, use custom roles with the rest-reader and rest-writer execute privileges and suitable default permissions, rather than relying on the pre-defined rest-reader and rest-writer roles.

The rest-reader and rest-writer privileges grant users permission to execute REST API code for reading and writing documents, while the default permissions controls access to a document whether it is through the REST API or through other code running on MarkLogic Server. For details, see the Security Guide.

The rest-extension-user role enables users to access resource service extension methods. This role is implicit in the other pre-defined roles, but you need to explicitly include it if you're defining custom roles for users that should also be able to use extensions.

For example, suppose you have two groups of users, A and B. Both can create documents using the REST API, but Group A users should not be able to read documents created by Group B, and vice versa. You can implement these restrictions in the following way:

1. Create a GroupA security role.
2. Assign the rest-reader and rest-writer execute privileges to the GroupA role. Use the privileges, not the base roles. That is, assign these privileges to the role:

   http://marklogic.com/xdmp/privileges/rest-reader
   http://marklogic.com/xdmp/privileges/rest-writer

3. If you also want to enable the execution of REST resource extensions, assign the rest-extension-user role to the GroupA role. Note that the rest-extension-user role provides a base role, not a privilege.
4. Give the GroupA role suitable default permissions. For example, set the default permissions of the role to update and read.
5. Assign the GroupA role to the appropriate users.
6. Repeat Steps 1-3 for a new GroupB role and assign GroupB to the appropriate users.

Now, users with the GroupA role can create documents with the REST API and read or update them, but users with the GroupB role have no access to documents created by GroupA. Similarly, users with the GroupB role can create documents and read or update them, but users with the GroupA role have no access to documents created by GroupB users. A user with the default rest-reader role, however, can read documents created by both GroupA and GroupB users.

Other security configurations are possible. For more details, see the Security Guide.

Evaluating Requests Against a Different Database

Most methods support a database request parameter that enables the request to be evaluated against a content database other than the default database associated with the REST API instances. Only users with the http://marklogic.com/xdmp/privileges/xdmp-eval-in (xdmp:eval-in) or equivalent privilege can use this feature.

If you want to enable this capability, you must create a role that enables xdmp:eval-in, in addition to appropriate mix of rest-* roles.

For details about roles and privileges, see the Security Guide.

Evaluating or Invoking Server-Side Code

You can evaluate ad-hoc queries and pre-installed modules on MarkLogic Server using the /eval and /invoke services, respectively. These services require special privileges, such as xdmp-eval, instead of the normal REST API roles like rest-reader and rest-writer.

For details, see the following:

• Evaluating an Ad-Hoc Query or POST:/v1/eval
• Evaluating a Module Installed on MarkLogic Server or POST:/v1/invoke.

Addressing a Resource

A resource address takes the form of a URL that includes a host name and a port number:

http://host:port/version/resource/

The host and port must reference a host running MarkLogic Server with a REST Client API instance running on that port. A REST Client API instance is served by an HTTP App Server. For details, see Creating an Instance.

A resource address always includes the API version in URL. For details, see Specifying the REST API Version.

You can optionally include parameters in a resource address as follows:

http://host:port/version/resource?param=value&param=value

For details, see Specifying Parameters in a Resource Address.

Specifying the REST API Version

To guarantee stable behavior of the REST Client API as new versions are released, each resource address in the REST Client API includes a version number. The examples in this chapter show the version as LATEST or simply version. The version has the format:

v#

Where # is the version number. For example, in the initial version of the API, the current version number is 1, so you can access the /documents service using the following URL:

http://localhost:8000/v1/documents

You can use LATEST to reference the current version, without regard to the actual version number. For example:

http://localhost:8000/LATEST/documents

The current version number is v1.

The version number is only updated when resource addresses and/or parameters have changed. It is not updated when resource addresses and/or parameters are added or removed.

Specifying Parameters in a Resource Address

Resource services accept request parameters to tailor behavior or control input and output format.

To specify multiple parameters, use the '?' sign before the first parameter and the '&' sign before any additional parameters:

http://host:port/version/resource?param1=value&param2=value....

Some resources only accept parameter values as URL-encoded form data in the request body. Such requests require an input content MIME type of x-www-form-urlencoded. You can use the curl option --data-url-encode to set such parameters to a properly encoded value. For details, see Introduction to the curl Tool.

See the MarkLogic REST API Reference for a list of parameters available with each resource.

Introduction to the curl Tool

curl is a command line tool for sending HTTP requests. You are not required to use curl with the REST Client API. You can use any tool or library capable of sending HTTP requests. However, since all the examples in this guide use curl, this section introduces you to the most relevant options.

If you do not have curl, you can download a copy from http://curl.haxx.se/download.html, or use an equivalent tool. This section provides a brief overview of the curl command line options used in this guide. For details, see the curl man page or the online documentation at http://curl.haxx.se/docs/.

The curl command line is of the form:

curl options URL

The options most often used in the examples in this guide are summarized in the table below.

--anyauth

--user username:password

-X http_method

-d data

--data-binary data

--data-urlencode data

-H headers

-i

For example, the following command sends a POST request with the contents of the file my-body.json in the request body, and specifies the Content-type as application/json:

$ curl --anyauth --user me:mypassword -X POST -d @my-body.json \
    -H "Content-type: application/json" \
    http://localhost:8000/LATEST/config/query/my-options

When reading data for a POST or PUT request body with curl, you must use --data-binary rather than -d if you need to preserve newlines in the data. For example, use --data-binary when uploading SPARQL or a JavaScript module that uses line-oriented comments (// a comment).

Modifying the Example Commands for Windows

The command line examples in this guide use Unix command line syntax, usable from either a Unix or Cygwin command line. If you are the Windows command interpreter, Cmd.exe, use the following guidelines to modify the example commands for your environment:

• Omit the $ character at the beginning of the command. This is the Unix default prompt, equivalent to > in Windows.
• For aesthetic reasons, long example command lines are broken into multiple lines using the Unix line continuation character '\'. Remove the line continuation characters and place the entire command on one line, or replace the line continuation characters with the Windows equivalent, '^'.
• Replace arguments enclosed in single quotes (') with double quotes ("). If the single-quoted string contains embedded double quotes, escape the inner quotes.
• Escape any unescaped characters that have special meaning to the Windows command interpreter.

General Content Type Guidelines

The following guidelines apply to specifying input and output content type for most requests:

• Document content: Rely on the MarkLogic Server MIME type mapping defined for the URI extension.
• Non-document data: Set the request Content-type and/or Accept headers. In most cases, this means setting the header(s) to application/xml or application/json.

The installation-wide MarkLogic Server MIME type mappings define associations between MIME type, URI extensions, and document format. For example, the default mappings associate the MIME type application/pdf and the pdf URI extension with the binary document format. You can view, change, and extend the mappings in the Mimetypes section of the Admin Interface or using the XQuery functions admin:mimetypes-get and admin:mimetypes-add.

As long as your documents have URI extensions with MIME type mappings and you set the HTTP Content-type and/or Accept headers consistent with your data, these guidelines are all you need. For situations that do not fit this model, see Details on Content Type Determination.

Details on Content Type Determination

This section provides a detailed description of how content type is determined. This information is useful for requests that do not conform to the guidelines in General Content Type Guidelines. For example, you might need this deeper understanding in the following situations:

• Reading or writing documents that have no URI extension or an unrecognized URI extension.
• Reading or writing document content and non-document data in the same request, such as reading a document and its metadata in a single request.
• Creating requests that have both input and output, such as a POST /LATEST/search request that has a query in the POST body and search results in the response.
• Requesting non-document data through a browser. Browsers often do not give you full control over the HTTP headers.

The table below summarizes how input and output content type is determined, depending on type of data and the request context (input or output). The content type sources in the third column are listed from highest to lowest precedence. For example, for input document content, the URI extension mapping is used if possible; the Content-type header is only used if there is no mapping available.

The format request parameter is supported by most REST API methods that accept or produce non-document data. You can set it to one of a limited set of values, usually xml or json; see the API documentation for individual methods for allowed values.

Requests which accept or produce multipart data behave asymmetrically because the Content-type header (multipart input) or Accept header (multipart output) must be multipart/mixed. On input, you can use the part Content-type header to indicate the non-document data format in a given part, but on output you can only use the format parameter to request a specific output format. A multi-document write using POST /LATEST/documents is an example of an operation with multipart input. Reading a document and its metadata in a single request is an example of an operation with multipart output. On such a read, you can use the format parameter to specify the metadata format.

Example: Inserting and Reading a Document

This example demonstrates how the general content type guidelines apply to document content. The example relies on the pre-defined MIME type mapping between the json URI extension and the MIME type application/json.

The following command inserts a JSON document into the database with URI example.json. Because of the MIME type mapping, a JSON document is created, whether or not you specify application/json in the request Content-type header.

$ curl --anyauth --user user:password -X PUT -d '{"key":"value"}' \
  -i -H "Content-type: anything" \   http://host:port/LATEST/documents?uri=example.json

The following command reads the document just inserted. Whether or not you set the Accept header to application/json, MarkLogic Server sets the response Content-type header to application/json because the URI extension is json.

$ curl --anyauth --user user:password -X GET -i \
  http://host:port/LATEST/documents?uri=example.json
...
HTTP/1.1 200 OK
vnd.marklogic.document-format: json
Content-type: application/json; charset=utf-8
...
{"key":"value"}

If the URI has no extension or there is no MIME type mapping defined for the extension, MarkLogic Server falls back on sources such as the HTTP Content-type header for input and the document type or Accept header for output. For details, see Details on Content Type Determination.

Example: Inserting and Reading Metadata

This example illustrates how the general content type guidelines apply to non-document data. The example inserts and reads document metadata.

The following command inserts metadata for a document. Assume the file ./mymatadata contains a JSON representation of document metadata. The request Content-type header tells MarkLogic Server to interpret the metadata in the request body as JSON.

$ curl --anyauth --user user:password -X PUT -d @./mymetadata \
  -H "Content-type: application/json" \
  'http://host:port/LATEST/documents?uri=anything&category=metadata'

For a complete example, PUT:/v1/documents or see Adding Metadata.

The following command reads the metadata for a document. The Accept header tells MarkLogic Server to return the metadata as XML.

$ curl --anyauth --user user:password -X GET \
  -H "Accept: application/xml" \
  'http://host:port/LATEST/documents?uri=anything&category=metadata'

For a complete example, see GET:/v1/documents or Retrieving Metadata About a Document.

If you cannot control the Content-type header for input or the Accept header for output, you can use the format request parameter. For details, see Details on Content Type Determination.

http://host:port/LATEST/documents?uri=anything&category=metadata&format=json

Example: Documents With No or Unknown URI Extension

This example illustrates how the output content type is determined when reading a document with no URI extension or a URI extension that has no MIME type mapping.

The following command inserts a text document into the database at a URI that has no extension. Since there is no extension, MarkLogic Server uses the MIME type mapping defined for text/plain in the request Content-type header to determine the document type.

curl --anyauth --user user:password -X PUT -i \
  -d '{ "key" : "value" }' -H "Content-type: text/plain" \
  http://host:port/LATEST/documents?uri=no-extension

If you leave off the Content-type header or set it to value for which there is no MIME type mapping, a binary document is created because binary is the default document type when there is no extension or MIME type mapping.

The following command reads the document inserted above. The response Content-type header is text/plain because the root node of the document is a text node.

curl --anyauth --user user:password -X GET -i \
  http://host:port/LATEST/documents?uri=no-extension

For binary documents, the root document node type is too generic for most applications. You can use the Accept header to coerce the response Content-type header to a specific MIME type. The content is unaffected. For example, if you read a binary document with no extension that you know actually contains PDF, then the following command returns the document with a Content-type header of application/pdf.

curl --anyauth --user user:password -X GET -i \
  -H "Accept: application/pdf" \
  http://host:port/LATEST/documents?uri=no-extension

If you cannot control the Accept header, then the response Content-type for a binary document is application/x-unknown-content-type by default.

Example: Mixing Document and Non-Document Data

This example describes how content type is determined for requests that include both document and non-document data as input or as output. In this example, the non-document data is metadata for a document.

The following example command inserts an XML document and its metadata into the database. Assume the file multipart-body contains a multipart/mixed POST body with a part for the metadata and a part for the content. For a complete example, see Loading Content and Metadata Using a Multipart Message.

curl --anyauth --user user:password -X PUT -d @./multipart-body \
  -i -H "Content-type: multipart/mixed; boundary=BOUNDARY" \
  http://host:port/LATEST/documents?uri=example.xml

The metadata format is derived from the Content-type header on the metadata part in the POST body. The document content type is derived from the URI extension of .xml. If the document URI did not have an extension, the document content type would be derived from the Content-type header on the document part in the POST body.

The following example command reads a JSON document and its metadata. The response is multipart data, with one part containing the metadata and one part containing the document. Since the Accept header must be multipart/mixed in this case, the format parameter is used to request the metadata as JSON.

curl --anyauth --user user:password -X GET -i \
  -H "Accept: multipart/mixed; boundary=BOUNDARY" \
  'http://host:port/LATEST/documents?uri=example.json?format=json'

In the response, the Content-type header for the metadata part is set to application/json because of the format parameter value. The Content-type header for the document part is set to application/json because the document URI is .json. If the document URI had no extension, the Content-type header for the document part would still be application/json as long as root node of the document indicates a JSON document.

For a complete example of reading a document and its metadata, see Retrieving Content and Metadata in a Single Request.