MarkLogic Server 11.0 Product Documentation

REST Application Developer's Guide — Chapter 8

Writing Transformations

This section covers the following topics related to authoring transform functions:

• Guidelines for Writing Transforms
• Writing JavaScript Transformations
• Writing XQuery Transformations
• Writing XSLT Transformations
• Expected Input and Output
• Reporting Errors
• Context Map Keys
• Controlling Transaction Mode

function yourTransformName(context, params, content) 
{
    ...
};

exports.transform = yourTransformName;

context

params

content

const mutableDoc = content.toObject();

http://marklogic.com/rest-api/transform/yourTransformName

declare function yourNS:transform(
  $context as map:map,
  $params as map:map,
  $content as document-node())
as document-node()

$context

$params

$content

<xsl:stylesheet version="2.0"
    xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
    xmlns:example="http://marklogic.com/rest-api/example/transform"
    xmlns:map="http://marklogic.com/xdmp/map">
<xsl:param name="context" as="map:map"/>
<xsl:param name="params"  as="map:map"/>
<xsl:template match="/*">
  <xsl:copy>
    <xsl:attribute name='{(map:get($params,"name"),"transformed")[1]}'
      select='(map:get($params,"value"),"UNDEFINED")[1]'/>
...

declare function example:transform(
    $context as map:map,
    $params  as map:map,
    $input   as document-node()
) as document-node()
{
    (: get 'input-types' to use in content negotiation :)
    let $input-types := map:get($context,"input-types")
    let $negotiate :=
        if ($input-types = "application/xml")
        then () (: process, insert/update :)
        else fn:error((),"RESTAPI-SRVEXERR",
          ("415","tsk tsk","whoops"))
    return document { "Done"}  (: may return a document node :)
};

HTTP/1.1 415 tsk tsk
Content-type: application/xml
Server: MarkLogic
Set-Cookie: SessionID=714070bdf4076536; path=/
Content-Length: 62
Connection: close
{
  "message": "js-example: response with invalid 400 status",
  "statusCode": 415,
  "body": {
    "errorResponse": {
      "statusCode": 415,
      "status": "tsk tsk",
      "messageCode": "RESTAPI-SRVEXERR",
      "message": "whoops"
    }
  }
}

inputType

uri

acceptTypes

outputType

declare %rapi:transaction-mode("the-mode") function yourNS:transform(...)

declare %rapi:transaction-mode("query") function example:transform(
  $context as map:map,
  $params as map:map,
  $content as document-node())
as document-node()
{...};

Installing Transformations

This section covers the following topics:

• Basic Installation
• Including Transform Metadata

http://host:port/version/config/transforms/yourTransformName

xquery version "1.0-ml";
module namespace example = 
  "http://marklogic.com/rest-api/transform/yourTransformName";
...

# Windows users, see Modifying the Example Commands for Windows 
$ curl --anyauth --user user:password -X PUT -i \
    --data-binary @"./trans-ex.sjs" \
    -H "Content-type: application/vnd.marklogic-javascript" \
    'http://localhost:8000/LATEST/config/transforms/js-example'

http://host:port/version/config/transforms/example?provider=MarkLogic%20Corporation

trans:paramName=paramTypeAndCardinality

http://host:port/version/config/transforms/example?trans:reviewer=string\?

Applying Transformations

All REST Client API services that support transformations use the following mechanism for specifying application of a transformation:

1. Specify the name of an installed transform as the value of the transform request parameter:

   transform=transformName

2. If the transform expects parameters, specify each parameter with a request parameter of the form:

   trans:paramName=value

For example, to apply a transform during document insertion, you can send a PUT request to the /documents service with a transform request parameter. Assuming the transform called example expects one parameter named reviewer, the PUT request URL might look like the following:

http://host:port/version/documents?uri=/doc/theDoc.xml&transform=example&trans:reviewer=me

You could apply the same transformation during document retrieval by sending a GET request to the /documents service with the same URL.

For a complete example, see one of the following topics:

• JavaScript Example: Adding a JSON Property During Ingestion
• XSLT Example: Adding an Attribute During Ingestion
• XQuery Example: Adding an Attribute During Ingestion

Discovering Transformations

Use GET:/v1/config/transforms to discover the name, interface, and other metadata about installed transformations. Send a GET request to the /config/transforms service of the form:

http://host:port/version/config/transforms

MarkLogic Server returns a summary of the installed transforms in XML or JSON. The default summary format is XML. Use the Accept header or format request parameter to select the output content type. For details, see Controlling Input and Output Content Type.

By default, this request rebuilds the transform metadata each time it is called to ensure the metadata is up to date. If you find this refresh makes discovery take too long, you can disable the refresh by setting the refresh request parameter to false. Disabling the refresh can result in this request returning inaccurate information, but it does not affect the freshness or availability of the implementation of any transforms.

The following example requests an XML report:

# Windows users, see Modifying the Example Commands for Windows 
$ curl --anyauth --user user:password -X GET \
    -H "Accept: application/xml" \
    http://localhost:8000/LATEST/config/transforms
...
HTTP/1.1 200 Transform List Retrieved
Content-type: application/xml
Server: MarkLogic
Content-Length: 348
Connection: close

<?xml version="1.0" encoding="UTF-8"?>
<rapi:transforms xmlns:rapi="http://marklogic.com/rest-api">
  <rapi:transform>
    <rapi:name>add-attr-xsl</rapi:name>
    <rapi:title/>
    <rapi:version/>
    <rapi:provider-name>MarkLogic Corporation</rapi:provider-name>
    <rapi:description/>
    <rapi:transform-parameters>
      <rapi:parameter>
        <rapi:parameter-name>value</rapi:parameter-name>
        <rapi:parameter-type>string?</rapi:parameter-type>
      </rapi:parameter>
      <rapi:parameter>
        <rapi:parameter-name>name</rapi:parameter-name>
        <rapi:parameter-type>string?</rapi:parameter-type>
      </rapi:parameter>
    </rapi:transform-parameters>
    <rapi:transform-source>
      /v1/transforms/add-attr
    </rapi:transform-source>
  </rapi:transform>
</rapi:transforms>

Retrieving the Implementation of a Transformation

To retrieve the XQuery code or XSLT stylesheet that implements a transformation, send a GET request to the /config/transforms/{name} service of the form:

http://host:port/version/config/transforms/transformName

Where transformName is the name under which the extension is installed.

Set the request Accept header to one of the following MIME types. The type must match the implementation language of the transform.

• XQuery: application/xquery
• JavaScript: application/vnd.marklogic-javascript
• XSLT: application/xslt+xml

MarkLogic Server returns the XQuery library module or XSLT stylesheet that implements the transform in the response body. For example:

# Windows users, see Modifying the Example Commands for Windows 
$ curl --anyauth --user user:password -X GET \
    -H "Accept: application/xquery" \
    http://localhost:8000/LATEST/config/transforms/example
...
xquery version "1.0-ml";
module namespace example="http://marklogic.com/rest-api/transform/example";
declare function example:transform( ...

JavaScript Example: Adding a JSON Property During Ingestion

This example demonstrates a transformation that adds a property to a JSON document. The first example in this section adds a property with a fixed name and a value corresponding to the current dateTime. The second example adds a property corresponding to a name and value passed in as transform parameters.

When used as a read transform, the following transform adds a property named readTimestamp to the content returned to the client. When used as a write transform, it adds a property named writeTimestamp to the document stored in the database. In both cases, the value of the property is the dateTime at the time of the operation. If the input document is not JSON, the content is unchanged on both read and write.

Follow these steps to exercise the transform:

1. Copy the following code into a file called trans-ex.sjs.

   function insertTimestamp(context, params, content)
   {
     if (context.inputType.search('json') >= 0) {
       const result = content.toObject();
       if (context.acceptTypes) {                 /* read */
         result.readTimestamp = fn.currentDateTime();
       } else {                                   /* write */
         result.writeTimestamp = fn.currentDateTime();
       }
       return result;
     } else {
       /* Pass thru for non-JSON documents */
       return content;
     }
   };
   
   exports.transform = insertTimestamp;

2. Send a PUT:/v1/config/transforms/{name} request to install the transform with the name js-example. For example:

   # Windows users, see Modifying the Example Commands for Windows 
   $ curl --anyauth --user user:password -X PUT -i \
       --data-binary @"./trans-ex.sjs" \
       -H "Content-type: application/vnd.marklogic-javascript" \
       'http://localhost:8000/LATEST/config/transforms/js-example'

3. Optionally, verify installation of the transformation using GET:/v1/config/transforms. For example:

   $ curl --anyauth --user user:password -X GET \
       -H "Accept: application/json" \
       http://localhost:8000/LATEST/config/transforms
   { "transforms": {
       "transform": [
         {
           "name": "js-example",
           "source-format": "javascript",
           "transform-parameters": "",
           "transform-source": "/v1/config/transforms/example"
         }
   ] }

4. Send a PUT:/v1/documents request to insert a document into the database, using the transform to add the writeTimestamp property. For example:

   $ curl --anyauth --user user:password -X PUT -d'{"name":"value"}' \
       -H "Content-type: application/json" \
       'http://localhost:8000/LATEST/documents?uri=/transforms/example.json&transform=js-example'

5. Send a GET:/v1/documents request to retrieve the inserted document. Note the added writeTimestamp attribute. For example:

   $ curl --anyauth --user user:password -X GET \
       -H "Accept: application/json" \
       http://localhost:8000/LATEST/documents?uri=/transforms/example.json
   { "key":"value", 
     "writeTimestamp":"2014-11-25T16:59:00.459241-08:00"
   }

6. Send a second GET:/v1/documents request to retrieve the inserted documents and apply the transform as a read transform. Note the added readTimestamp property. For example:

   $ curl --anyauth --user user:password -X GET \
       -H "Accept: application/json" \
       'http://localhost:8000/LATEST/documents?uri=/transforms/example.json&transform=js-example'
   { "key":"value", 
     "writeTimestamp":"2014-11-25T16:59:00.459241-08:00",
     "readTimestamp":"2014-11-25T17:38:28.417881-08:00"
   }

   The writeTimestamp is unchanged since it is part of the document. The readTimestamp is an artifact of the read, rather than part of the document. It will change each time you apply the transform during the read.

The following function is an alternative transform that demonstrates passing in transform parameters. For each parameter passed in via trans:name=value, the transform inserts a JSON property with the same name and value. The transform is applicable to both read and write operations. As before, non-JSON documents are unchanged.

function insertProperty(context, params, content)
{
  if (context.inputType.search('json') >= 0 && params) {
    const result = content.toObject();
    Object.keys(params).forEach(function(key) {
      result[key] = params[key];
    });
    return result;
  } else {
    // Pass thru unchanged
    return content;
  }
};

exports.transform = insertProperty;

The following example commands demonstrate using this transform on a read operation, assuming the transform is installed with the name ex-params:

$ curl --anyauth --user user:password -X GET \
    -H "Accept: application/json" -i \
    'http://localhost:8000/LATEST/documents?uri=/transforms/example.json&transform=ex-params&trans:prop1=42&trans:prop2=dog'
...
{ "key":"value", 
  "writeTimestamp":"2014-11-25T16:59:00.459241-08:00", 
  "prop1":"42", 
  "prop2":"dog" 
}

You can specify the same parameter more than once. For example, if the request parameter prop1 is supplied more than once, the value of prop1 is an array.

http://localhost:8000/LATEST/documents?...&trans:prop1=42&trans:prop1=43
==>
{ ..., 
  "prop1":["42", "43"]
}

XQuery Example: Adding an Attribute During Ingestion

This example demonstrates using a transformation to add an attribute to the root node of an XML document during ingestion. The name and value of the added attribute are passed in as parameters to the transform function. You can use the same transform for document retrieval by applying it to a GET request instead of a PUT.

The transformation XQuery module shown below reads the attribute name and value from parameters, and then makes a copy of the input document, adding the requested attribute to the root node. The transformation cannot perform direct transformations on the input XML, such as calling xdmp:node-replace, because the document has not yet been inserted into the database when the transform function runs.

xquery version "1.0-ml";
module namespace example =
  "http://marklogic.com/rest-api/transform/add-attr";

declare function example:transform(
  $context as map:map,
  $params as map:map,
  $content as document-node()
) as document-node()
{
  if (fn:empty($content/*)) then $content
  else
    let $value := (map:get($params,"value"),"UNDEFINED")[1]
    let $name := (map:get($params, "name"), "transformed")[1]
    let $root  := $content/*
    return document {
      $root/preceding-sibling::node(),
      element {fn:name($root)} {
        attribute { fn:QName("", $name) } {$value},
        $root/@*,
        $root/node()
      },
      $root/following-sibling::node()
    }
};

To exercise this example:

1. Cut and paste the code above into a file called add-attr.xqy.
2. Make a PUT request to the /config/transforms/{name} service to install the transform as add-attr, accepting 2 string parameters called name and value. For example:

   # Windows users, see Modifying the Example Commands for Windows 
   $ curl --anyauth --user user:password -X PUT -d@"./add-attr.xqy" \
       -H "Content-type: application/xquery" \
       'http://localhost:8000/LATEST/config/transforms/add-attr?trans:name=string\?&trans:value=string\?'

3. Optionally, verify installation of the transformation by making a GET request to the /config/transforms service. For example:

   $ curl --anyauth --user user:password -X GET -i \
       -H "Accept: application/xml" \
       http://localhost:8000/LATEST/config/transforms
   ...
   HTTP/1.1 200 Transform List Retrieved
   Content-type: application/xml
   Server: MarkLogic
   Content-Length: 348
   Connection: close
   
   <?xml version="1.0" encoding="UTF-8"?>
   <rapi:transforms xmlns:rapi="http://marklogic.com/rest-api">
     <rapi:transform>
       <rapi:name>add-attr</rapi:name>
       <rapi:source-format>xquery</rapi:source-format>
       <rapi:title/>
       <rapi:version/>
       <rapi:provider-name/>
       <rapi:description/>
       <rapi:transform-parameters>
         <rapi:parameter>
           <rapi:parameter-name>value</rapi:parameter-name>
           <rapi:parameter-type>string?</rapi:parameter-type>
         </rapi:parameter>
         <rapi:parameter>
           <rapi:parameter-name>name</rapi:parameter-name>
           <rapi:parameter-type>string?</rapi:parameter-type>
         </rapi:parameter>
       </rapi:transform-parameters>
       <rapi:transform-source>
         /v1/transforms/add-attr
       </rapi:transform-source>
     </rapi:transform>
   </rapi:transforms>

4. Make a PUT request to the /documents service to insert a document into the database, using the add-attr transform to add the attribute example with a value of new. For example:

   $ curl --anyauth --user user:password -X PUT -d"<data/>" \
       'http://localhost:8000/LATEST/documents?uri=/doc/transformed.xml&transform=add-attr&trans:name=example&trans:value=new'

5. Make a GET request to the /documents service to retrieve the inserted document. Note the added example attribute. For example:

   $ curl --anyauth --user user:password -X GET \
       http://localhost:8000/LATEST/documents?uri=/doc/transformed.xml
   <?xml version="1.0" encoding="UTF-8"?>
   <data example="new"/>

XSLT Example: Adding an Attribute During Ingestion

This example demonstrates using a transformation to add an attribute to the root node of an XML document during ingestion. The name and value of the added attribute are passed in as parameters to the stylesheet. You can use the same transform for document retrieval by applying it to a GET request instead of a PUT.

The transformation stylesheet shown below reads the attribute name and value from parameters, and then makes a copy of the input document, adding the requested attribute to the root node.

<xsl:stylesheet version="2.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
    xmlns:example="http://marklogic.com/rest-api/example/transform"
    xmlns:map="http://marklogic.com/xdmp/map">
<xsl:param name="context" as="map:map"/>
<xsl:param name="params"  as="map:map"/>
<xsl:template match="/*">
    <xsl:copy>
        <xsl:attribute name='{(map:get($params,"name"),"transformed")[1]}'
            select='(map:get($params,"value"),"UNDEFINED")[1]'/>
        <xsl:copy-of select="@*"/>
        <xsl:copy-of select="node()"/>
    </xsl:copy>
</xsl:template>
</xsl:stylesheet>

To exercise this example:

1. Create a REST Client API instance on port 8003. For instructions, see Creating an Instance.
2. Cut and paste the stylesheet above into a file called add-attr.xsl.
3. Make a PUT request to the /config/transforms/{name} service to install the transform as add-attr-xsl, accepting 2 string parameters called name and value. For example:

   # Windows users, see Modifying the Example Commands for Windows 
   $ curl --anyauth --user user:password -X PUT -d@"./add-attr.xsl" \
       -H "Content-type: application/xslt+xml" \
       'http://localhost:8000/LATEST/config/transforms/add-attr-xsl?trans:name=string\?&trans:value=string\?'

4. Optionally, verify installation of the transformation by sending a GET:/v1/config/transforms request. For example:

   $ curl --anyauth --user user:password -X GET \
       -H "Accept: application/xml" \
       http://localhost:8000/LATEST/config/transforms
   ...
   HTTP/1.1 200 Transform List Retrieved
   Content-type: application/xml
   Server: MarkLogic
   Content-Length: 348
   Connection: close
   
   <?xml version="1.0" encoding="UTF-8"?>
   <rapi:transforms xmlns:rapi="http://marklogic.com/rest-api">
     <rapi:transform>
       <rapi:name>add-attr-xsl</rapi:name>
       <rapi:source-format>xslt</rapi:source-format>
       <rapi:title/>
       <rapi:version/>
       <rapi:provider-name/>
       <rapi:description/>
       <rapi:transform-parameters>
         <rapi:parameter>
           <rapi:parameter-name>value</rapi:parameter-name>
           <rapi:parameter-type>string?</rapi:parameter-type>
         </rapi:parameter>
         <rapi:parameter>
           <rapi:parameter-name>name</rapi:parameter-name>
           <rapi:parameter-type>string?</rapi:parameter-type>
         </rapi:parameter>
       </rapi:transform-parameters>
       <rapi:transform-source>
         /v1/transforms/add-attr
       </rapi:transform-source>
     </rapi:transform>
   </rapi:transforms>

5. Send a PUT:/v1/documents request to insert a document into the database, using the add-attr-xsl transform to add the attribute example with a value of new. For example:

   $ curl --anyauth --user user:password -X PUT -d"<data/>" \
       'http://localhost:8000/LATEST/documents?uri=/doc/transformed.xml&transform=add-attr-xsl&trans:name=example&trans:value=new'

6. Send a GET:/v1/documents request to retrieve the inserted document. Note the added example attribute. For example:

   $ curl --anyauth --user user:password -X GET \
       http://localhost:8000/LATEST/documents?uri=/doc/transformed.xml
   <?xml version="1.0" encoding="UTF-8"?>
   <data example="new"/>

XQuery Example: Modifying Document Type During Ingestion

This example is a write transform that converts JSON input into XML, inserting an XML document into the database. It also demonstrates modifying the context map to change the document URI.

When you insert a document with a PUT or POST request to the /documents service and specify a transform function name, the MIME type in the Content-type request header takes precedence over the URI extension in determining the format of the data in the request body.

The following transform takes advantage of this behavior to accept JSON input and convert it to XML. The transform also modifies the URI to match, if necessary: If the URI has a .json URI suffix, the URI is modified to have a .xml suffix by modifying the value of the uri key in the $context map.

xquery version "1.0-ml";
module namespace xqjson2xml =
  "http://marklogic.com/rest-api/transform/xqjson2xml";

import module namespace json="http://marklogic.com/xdmp/json"
 at "/MarkLogic/json/json.xqy";


declare function xqjson2xml:transform(
  $context as map:map,
  $params as map:map,
  $content as document-node()
) as document-node()
{
  if (fn:contains(map:get($context, "input-type"), "json"))
  then (
    map:put($context, "output-type", "application/xml"),
    let $uri := map:get($context, "uri")
    return
      if (fn:ends-with($uri, ".json")) then 
        map:put($context, "uri", 
                fn:concat(fn:substring-before($uri, ".json"), ".xml"))
      else (),
    document {json:transform-from-json($content)}
  )
  else $content
};

To exercise this example:

1. Cut and paste the code above into a file called json2xml.xqy.
2. Make a PUT request to the /config/transforms/{name} service to install the transform as xqjson2xm. For example:

   # Windows users, see Modifying the Example Commands for Windows 
   $ curl --anyauth --user user:password -X PUT -d@"./json2xml.xqy" \
       -H "Content-type: application/xquery" \
       'http://localhost:8000/LATEST/config/transforms/xqjson2xml'

3. Optionally, verify installation of the transformation by making a GET request to the /config/transforms service. For example:

   $ curl --anyauth --user user:password -X GET -i \
       -H "Accept: application/xml" \
       http://localhost:8000/LATEST/config/transforms

4. Make a PUT request to the /documents service to insert a document into the database with a .xml extension, using the xqjson2xml transform. Note the request payload is JSON, but an XML document is inserted into the database.

   $ curl --anyauth --user user:password -i -X PUT \
       -d'{"a": "b"}' -H "Content-type: application/json" \
       'http://localhost:8000/LATEST/documents?uri=/doc/j2x.xml&transform=xqjson2xml'

5. Make a GET request to the /documents service to retrieve the inserted XML document. For example:

   $ curl --anyauth --user user:password -X GET \
       'http://localhost:8000/LATEST/documents?uri=/doc/j2x.xml'
   ...
   HTTP/1.1 200 OK
   vnd.marklogic.document-format: xml
   Content-type: application/xml; charset=utf-8
   Server: MarkLogic
   Content-Length: 134
   Connection: Keep-Alive
   Keep-Alive: timeout=5
   
   <?xml version="1.0" encoding="UTF-8"?>
   <json type="object" xmlns="http://marklogic.com/xdmp/json/basic"><a type="string">b</a></json>

6. Make another PUT request to the /documents service to insert a document into the database, using the xqjson2xml transform, but this time with a URI with a .json suffix. The transform modifies both the document type and the URI extension.

   $ curl --anyauth --user user:password -i -X PUT \
       -d'{"c": "d"}' -H "Content-type: application/json" \
       'http://localhost:8000/LATEST/documents?uri=/doc/j2x_2.json&transform=xqjson2xml'

7. Make a GET request to the /documents service to retrieve the inserted XML document, using the modified URI. For example:

   $ curl --anyauth --user user:password -X GET \
       'http://localhost:8000/LATEST/documents?uri=/doc/j2x_2.xml'
   ...
   HTTP/1.1 200 OK
   vnd.marklogic.document-format: xml
   Content-type: application/xml; charset=utf-8
   Server: MarkLogic
   Content-Length: 134
   Connection: Keep-Alive
   Keep-Alive: timeout=5
   
   <?xml version="1.0" encoding="UTF-8"?>
   <json type="object" xmlns="http://marklogic.com/xdmp/json/basic"><c type="string">d</c></json>

Note that you can write a similar transformation in JavaScript.