MarkLogic 9 Product Documentation

Release Notes — Chapter 4

Documents Created as JSON With MarkLogic 7 REST API or mlcp Must Be Converted to Native JSON

In previous versions of MarkLogic, you can load JSON documents into MarkLogic using either the REST Client API or the Java Client API. When you do so, the documents are transformed and stored as XML, but are still queryable as and returned as JSON. In MarkLogic 8 this XML facade is no longer needed, and the REST and Java Client APIs in MarkLogic 8 do not do the translation to the XML facade anymore. Therefore, if you have any document that were loaded as JSON in MarkLogic 7 and earlier, you must convert those document to native JSON in order to query them as JSON using the REST API.

To help with the conversion, MarkLogic supplies a set of conversion scripts. These scripts do not handle every case, and for any case it does not handle you will have to convert the documents some other way. Because the conversion scripts do not handle all cases, it is very important to do a backup of your database before attempting the conversion. The scripts are located in the Samples/migrate-scripts directory under the MarkLogic installation directory (/opt/MarkLogic on Linux, c:/Program Files/MarkLogic on Windows, and ~/Library/MarkLogic on Mac OS). The scripts require bash and curl, and on Windows platforms they also require cygwin.

For more details, see the <marklogic-dir>/Samples/migrate-scripts/README file.

Generally, the conversion scripts perform the following:

• Converts the documents to native JSON.
• Updates index configurations to reference the JSON content.
• Updates existing saved search options. This includes changing references to the JSON basic namespace to reference the new JSON content and changes occurrences of json-key to json-property.
• Updates any alerting rules that reference the JSON content.

The scripts do not upgrade your application code. The types of things you will need to change in your application include:

• Modify client code to update structured queries, combined queries, and query options that reference json-key and anything in the http://marklogic.com/xdmp/json/basic namespace. For more details see Search, Java, REST: json-key Is Now json-property in Options and Structured Query.
• Rewrite and reinsert any server-side code (for example, transformations or custom constraints) that operated over the internal XML representation of your JSON documents. For details see Java and REST: Transforms and Extensions That Manipulate JSON Must Be Rewritten.
• Modify client code that relies on the /v1/keyvalues endpoint for key/value searches over JSON.
• Modify client code to update patch specifications over JSON. For details see Java and REST: New Restrictions on Patching JSON Content.
• Review index settings and queries over document properties, update as needed (because you can no longer have JSON properties).

The basic steps to upgrade your MarkLogic 7 or earlier JSON to native JSON in MarkLogic 8 are as follows:

1. Backup the database in which your JSON documents exist.
2. Make copies and edit the connection details and other information in the various configuration files in the Samples/migrate-scripts/conf directory. This files have details about your configuration and index settings.
3. Run the Samples/migrate-scripts/migrate script.
4. Test your results. Make sure the index changes that the scripts made match your newly converted JSON data. It is especially important to review path and fields indexes to make sure they are including the same content in the converted JSON is they were previously.

If you have problems upgrading your application and you have an active maintenance contract, you can contact MarkLogic Technical Support.

json:unquotedString Primitive No Longer Available

MarkLogic 7 had a primitive to convert an XQuery string to an unquoted String called json:unquotedString. In MarkLogic 8, that function is no longer available because it is no longer needed, as MarkLogic 8 has much more extensive support for JSON. For details on working with JSON in MarkLogic, see Working With JSON in the Application Developer's Guide.

xdmp:to-json and json:transform-to-json Now Returns a document-node()

In MarkLogic 8, the xdmp:to-json function returns a document-node(); previously, it returned a string. If you have code that expects a JSON string, you might need to modify your code to perform XPath on the document-node() to get the JSON node (which will serialize into a string); depending on what your code does, you might or might not need to do this. For example:

(: 7.0 :)
xdmp:to-json(("a",fn:false()))
=> ["a",false]
(: 8.0 :)
xdmp:to-json(("a",fn:false()))/node()
=> ["a",false]

The json:transform-to-json function uses xdmp:to-json, so it also returns a document-node() in MarkLogic 8.

Search, Java, REST: json-key Is Now json-property in Options and Structured Query

All occurrences of json-key in query options and structured query are now json-property. If you use the constructs listed below to search JSON documents by key/property name, you will need to modify your query options (search:options, in XML) or structured queries.

The following query options are affected. For details, see search:search or Appendix: Query Options Reference in the Search Developer's Guide.

• container-constraint
• extract-metadata
• range-constraint
• sort-order
• value-constraint
• word-constraint

The following structured query components are affected. For more details, see the structured query Syntax Reference in the Search Developer's Guide.

• container-query
• range-query
• value-query
• word-query

The corresponding Java Client API structured query builder method name has also changed. StructuredQueryBuilder.JSONKey is now StructuredQueryBuilder.JSONProperty.

Java and REST: Specifying a Language for JSON Documents is Deprecated

Previously, you could specify a language when ingesting JSON documents. This was only possible because JSON documents were represented internally as XML.

This parameter is now deprecated and will be ignored when present. This affects the following interfaces:

• Java: JSONDocumentManager.setLanguage and JSONDocumentManager.getLanguage are deprecated. Calling setLanguage has no effect.
• REST: The lang request parameter of PUT /v1/documents is deprecated and will be ignored.
• REST: The lang request parameter of POST /v1/documents (all variants) is deprecated and will be ignored.

Java and REST: Default Path Language for JSON Document Patches is Now XPath

This section applies to applications that use the Java Client API or REST Client API patch (partial update) feature on JSON documents.

Previously, JSONPath was the default path language for identifying the target of update operations in a JSON patch. XPath is now the default path language for both XML and JSON patches.

Use of JSONPath is now deprecated. To convert your JSON patches to use XPath expressions instead of JSONPath, see Traversing JSON Documents Using XPath in the Application Developer's Guide.

To continue using JSONPath, you can explicitly override the default path language in one of the following ways:

• For a raw JSON patch, include a pathlang property as the sibling of the top level patch property. For example:

  { "pathlang": "jsonpath",
    "patch": [ ... ] }

  Raw patches are used by POST:/v1/documents and can be used with the Java Client API method DocumentManager.patch.

• When using the Java Client API, use DocumentPatchBuilder.pathLanguage. to set the path language to JSONPath, as shown in the following example:

  DocumentPatchBuilder patchBldr = docMgr.newPatchBuilder();
  patchBldr.pathLanguage(PathLanguage.JSONPATH);

Java and REST: New Restrictions on Patching JSON Content

The native JSON document model imposes some new restrictions on partial updates to JSON documents. Therefore, some patch operations that were previously supported will now be rejected or produce different results. For details, see Limitations of JSON Path Expressions in the REST Application Developer's Guide and Traversing JSON Documents Using XPath in the Application Developer's Guide.

For example, you cannot construct patch path expressions that address anonymous nodes. In the JSON document model, object nodes and array nodes are anonymous. The name in a property name-value pair addresses the value(s), not the containing node.

This means you cannot use last-child position to insert a new property or value under the root node of a document or in an array because the parent node is anonymous and cannot be selected by the context expression of the insert operation. Similarly, you cannot address an array node that is nested inside another array ([1, [2, 3], 4]) because it is unnamed.

You can no longer replace an entire property (name-value pair) in a single patch replace operation for the same reason. To replace a property, you must delete it and then insert a new one.

Java and REST: Transforms and Extensions That Manipulate JSON Must Be Rewritten

This section applies to REST Client API and Java Client API applications that use content transformations, resource service extensions, and other server-side code to manipulate the XML representation of JSON documents.

Previously, content transformations, resource service extensions, and other server-side code manipulated JSON content as XML in the http://marklogic.com/xdmp/json/basic namespace. Now, such code must operate on JSON document nodes instead of XML.

For example, previously, the JSON data { "key": "value" } was represented in the database as XML of the following form, and this is what your server-side transforms and extensions worked with:

<json type="object" xmlns="http://marklogic.com/xdmp/json/basic">
  <key type="string">value</key>
</json>

Thus, to access the value of the key property in XQuery, you could use a path expression like this following:

$someDocument/json:json/json:key

With a native JSON document you reference the same data using the following path expression:

$someDocument/key

You should understand the native JSON document model before rewriting your code. For details, see Working With JSON in the Application Developer's Guide.

Java and REST: JSON Array Items and Property Values No Longer Distinguishable in QBE

This change may affect applications that use QBE to search JSON documents using the Java Client API or REST Client API.

Previously, you could construct a QBE that included a word or value query explictly scoped to an array item or the . Now, it is not possible to distinguish between an array item and a property value.

For example, given a document with the following context:

{ "notArray": "blue", "array": ["azure", "blue"] }

Previously, the following QBE would only match the occurence of the value "blue" in the "array" property. Now it matches both the occurrence in "array" and the occurrence in "notArray".

{"$query": [ {"$value": ["blue"]} ] }

This is because array nodes are unnamed in the native JSON document model, so they cannot be explicitly identified in a QBE.

Field Range Query and Field Value Query on JSON May Behave Differently

This difference only applies to applications using field range queries or field value queries on JSON documents.

JSON and XML are not indexed in exactly the same way. Some of the indexing differences affect the behavior of field range queries and field value queries over JSON. Since JSON documents were previously stored as XML, this means your field range queries and field value queries over JSON may behave differently.

For example, previously you could construct a field value query for John Smith that would match the following document by defining a field on the name property that excluded the middle property.

{ "name": {
    "first": "John",
    "middle": "NMI",
    "last": "Smith"
}

This was possible because the document was represented as XML and the text nodes in the field were concatenated together so that the field value in the above document was John Smith. In native JSON documents, this concatenation does not occur, and the values of the equivalent field are John and Smith. To get the same effect now, you would have to use a construct such as a near query.

For more details, see Creating Indexes and Lexicons Over JSON Documents and How Field Queries Differ Between JSON and XML in the Application Developer's Guide.

Changed Function: sem:sparql

In MarkLogic 8, the signature of sem:sparql has changed for the last parameters. The fourth parameter is now a sem:store*, where previously there where two parameters at the end, one to specify a cts:query and another to specify the forest ID. The old signature is still available, but is deprecated. If you have any code from MarkLogic 7 that uses the fourth or fifth arguments to sem:sparql, it will still work in MarkLogic 8, but you should migrate that code to use the new signature using sem:store as the fourth parameter.

Changed Function: sem:sparql-values

In MarkLogic 8, sem:sparql-values now serializes a string as a cts:word-query when used as part of an argument. In other words, string values will be passed as cts:query arguments. This is a change from MarkLogic 7.

Changed Function: sem:sparql-values

The sem:sparql-values function no longer accepts forest-id as an option. This is an incompatibility with MarkLogic 7 functionality.

Deprecated Function: sem:sparql-triples

The sem:sparql-triples function has been deprecated in favor of sem:in-memory-store in MarkLogic 8. See documentation for details - Querying Triples in Memory in the Semantics Developer's Guide.

Changed Behavior: Graphs

In MarkLogic 8, graph documents containing metadata are created when triples are ingested, whether they are ingested using SPARQL Update, mlcp, or SPARQL endpoints over REST. These named graphs inherit the permissions of the user, unless specified as part of the ingest process. The graph permissions are stored along with other metadata in the graph document.

When loading triples with mlcp, if the output-permissions parameter is set when loading RDF, the graph will inherit the default permissions just as it would in a sem:sparql-update operation. If the output_collection parameter is set when loading RDF, the graph document is only created for the first collection specified (because a managed triple can only belong to one graph).

In an upgraded system, graph metadata for managed triples created in MarkLogic 7 will be created the first time you add triples to the graph or modify triples in the graph. The graph will either have the user's permissions or the permissions specified as part of the operation. Make sure the document permissions of documents containing managed triples created in MarkLogic 7 are passed into sem:sparql-update as default-permissions to ensure the graph metadata created is consistent with the permissions for existing managed triples created in MarkLogic 7.

Must Upgrade to Java Client API v3.0

You cannot use earlier versions of the Java Client API with MarkLogic 8. Update your application to use version 3.0 or later.

REST API Instance Must Use the Declarative Rewriter on the App Server

In MarkLogic 7, App Servers that are REST API Instances used a different URL rewriter than in MarkLogic 8. In MarkLogic 8, the App Server is configured to use the declarative rewriter. If you had not modified anything in your REST API Instance App Server setup, the upgrade to MarkLogic 8 will reconfigure your App Server to use the new rewriter. If, however, you have modified something in setup to use a different rewriter, then you will have to make similar changes to the new setup (or consider not using those changes in the REST API Instance). For details on the declarative rewriter, see xWeb Services in the Application Developer's Guide.

Default Transaction Mode for the POST Method of Resource Service Extensions is Now Query

In MarkLogic 7, the POST method of a resource service extension was always executed in update mode. In MarkLogic 8, POST methods in a single-statement transaction are executed in query mode. However, within a multi-statement transaction, they are in update mode. The new way is safer because, generally speaking, if a function is not doing an update, it is much more efficient for it to run as a query.

If you have a resource service extension that requires the transaction mode to be update, you need to modify the extension code to add an annotation to the function to explicitly force it into update mode. For example, to modify an extension that is a GET, add an annotation like the following to your get function:

declare %rapi:transaction-mode("update") function testrstxn:get(
   $context, $params) { 
<Your code goes here>
}; 

The annotation %rapi:transaction-mode("update") forces the function to run as an update. For details, see Controlling Transaction Mode in the REST Application Developer's Guide.

REST API: Empty Bulk Read by Query Now Returns 200 Status

Previously, performing a bulk read by retrieving all documents that match a query would return a 404 Not Found response status if no documents matched the query. As of MarkLogic 8, such a request returns a 200 OK status with an empty response body.

This change applies to the following methods, when the Accept header is set to multi-part/mixed and the request does not ask for a search result summary in addition to the matching documents.

• GET /v1/search
• POST /v1/search
• GET /v1/qbe
• POST /v1/qbe

For more details on these interfaces, see Reading Multiple Documents Matching a Query in the REST Application Developer's Guide.

Error Reporting Format and Detail Changes

The changes in this section might affect your application if either of the following is true:

• Your REST or Java client application directly manipulates error details returned by MarkLogic through a REST API instance. This is unlikely for Java applications because the receive such errors as Java exceptions.
• Your application includes content transformations or resource service extensions that report errors to the client.

MarkLogic 8 introduces the following incompatible changes to error reporting for users of the

• Error Format Defaults to JSON and is a REST Instance Creation Property
• Error Detail Element and Property Names Have Changed
• Use RESTAPI-SRVEXERR to Report Errors from Transforms and Extensions

Deprecated Interface: Keyvalue Queries

This topic applies to applications that use the Java Client API class KeyValueQueryDefinition or the REST Client API method GET:/v1/keyvalue. These interfaces are now deprecated.

You can use Query By Example (QBE) or structured query to perform the same kind of search.

For example, to search for a JSON property named author with the value Mark Twain, use a QBE such as the following:

{ "$query": { "author": "Mark Twain" } }

The following is a similar search for an XML element:

<q:qbe xmlns:q="http://marklogic.com/appservices/querybyexample">
  <q:query>
    <author>Mark Twain</author>
  </q:query>
</q:qbe>

With structured query, use value-query or container-query.

For details, see the following references:

• Searching Using Query By Example in the Search Developer's Guide.
• Searching Using Structured Queries in the Search Developer's Guide.
• RawQueryByExampleDefinition or StructuredQueryBuilder in the Java Client API javadoc.
• GET /v1/search or POST /v1/search in the MarkLogic REST API Reference.

Transaction ID Format Has Changed

Previously the transaction ids created using DatabaseClient.openTransaction (Java) or POST /v1/transactions (REST) were of the form hostId_transactionId. The hostId segment has now been dropped.

As long as your application treats the transaction id as a black box, this change is transparent.

A Transaction Can No Longer Be Shared Across Users

This change only affects Java Client API and REST Client API applications that use multi-statement transactions and share the resulting transaction id across multiple users.

Previously, it was possible to create a multi-statement transaction as one MarkLogic user and then perform operations within the transaction as another user. This is no longer possible. Now, all operations within a transaction must be performed as the same user who created the transaction.

Java: QBE Search Results No Longer Automatically Match the Query Format

Previously, using QueryManager.search with a Query By Example (QBE) automatically returned results in the same format as the query. That is, XML results were returned for an XML QBE, and JSON results were returned for a JSON QBE. Now, you must explicitly request JSON.

For example, if the qbe variable in the following statement contains a JSON QBE, then previously you would receive JSON results. Now, you will receive XML by default, instead.

queryMgr.search(qbe.newStringHandle()).get();

To achieve the same result as before, explicitly set the format on the result handle to JSON. For example:

queryMgr.search(qbe.newStringHandle().withFormat(Format.JSON)).get();

search:parse Output is Now Unannotated cts:query XML

Previously, search:parse produced XML representing an annotated cts:query that could be passed directly to search:unparse. As of MarkLogic 9, the default output from search:parse does not include annotations, so it cannot be passed to search:unparse.

You can get annotated output from search:parse in MarkLogic 9 using the $output parameter as shown below:

search:parse("myQueryText", options, "cts:annotated-query")

Deprecated Option: extract-metadata

The extract-metadata query option is now deprecated. Use extract-document-data instead. For details, see search:search or Extracting a Portion of Matching Documents in the Search Developer's Guide.

The new extract-document-data option does not support extracting specific metadata properties, but properties are available in other ways, such as using xdmp:document-properties or by fetching all properties metadata through one of the client APIs.

Deprecated Functions: search:unparse, search:remove-constraint

The search:unparse and search:remove-constraint functions are now deprecated. If you need to deconstruct a query, modify it, and put it back together, use structured query or cts:query with search:resolve instead.

For example, if you previously did something similar to the following:

let $ctsquery := search:parse("myQueryString", $options)
(: ...modify $ctsquery... :)
return search:search(search:unparse($ctsquery), $options)

Then you can achieve the same result doing the following:

let $ctsquery := search:parse("myQueryString", $options)
(: ...modify $ctsquery... :)
return search:resolve($ctsquery, $options)

To generate a structured query instead of a cts:query, set the third parameter of search:parse to "search:query":

search:parse("myQueryString", $options, "search:query")

Structured Query: locks-query and properties-query Renamed

The following structured query elements have been renamed to more accurately reflect their purpose:

• locks-query is now locks-fragment-query
• properties-query is now properties-fragment-query

For details, see locks-fragment-query and properties-fragment-query in the Search Developer's Guide.

sort-order Query Option Requires an Index

If you use the sort-order query option to sort search results by something other than score, such as an XML element, JSON property, or field, then the database configuration must include a range index on the entity used for a sort key. Failing to create such an index causes SEARCH-BADORDERBY to be thrown when searching. Previously, the index requirement was not enforced.

For details, see sort-order in the Search Developer's Guide.

Terms Matched by additional-query Are Highlighted in Snippets

Previously, the Search API documentation stated that terms matched by the query specified in an additional-query query option were not highlighted in search result snippets. This is no longer the case.

You should expect any terms matched by the additional-query option to be included in highlighted sections of snippets.

xdmp.multipartDecode Now Returns a JSON Payload for Headers

In 8.0-4, the Server-Side JavaScript xdmp.multipartDecode function returns the headers (in the first item of the returned ValueIterator) as a JSON array; previously, it was returned as an XML element. If you have code that is expecting the XML element, you need to either rewrite your code to parse the JSON array or use the XQuery version (xdmp:multipart-decode).

In JavaScript, Some Thesaurus and Spelling Function Have Different Return Type

In 8.0-4, there are changes to the thesaurus and spelling function modules to make them more friendly to JavaScript users. If you have JavaScript code that imports these XQuery libraries, then the following functions now return JavaScript Object by default:

• spell.makeDictionary
• thsr.lookup
• thsr.queryLookup

For each of these function, you can set a new optional parameter to change its output. By default, these functions return XML output in XQuery and JavaScript Objects in JavaScript. If you have existing JavaScript code that uses these functions, you either need to add the new option to your code specifying the XML output or rework your code to accept the returned JavaScript Object. For details on these functions, see the API documentation for each function. In XQuery, the functions behave the same way they did in previous versions by default, but now allow you to specify the optional parameter to output JavaScript Objects instead of XML, if you so choose.

xdmp.databaseRestoreStatus Now Returns an Object

In 8.0-4, the Server-Side JavaScript API xdmp.databaseRestoreStatus now returns an Object; previously, it returned an Array. The Array that was previously returned is now the value of the "forest" key, and there is also a "status" key containing the current status of the restore. The xdmp.databaseBackupStatus also contains this new "status" key, but the return type is not changed. Similarly, the XQuery counterparts to these APIs (xdmp:database-backup-status and xdmp:database-restore-status) also contain information about the status, but their signature has not changed. If you have code that relies on any of the old behavior, you will need to modify that code to work with the changed output.

Serialization Error Code Changes

In 8.0-4, the names of some error exception codes for serialization, as well as the message text, have changed as shown in the following table:

SER-DOCTYPESYSTEM2

SER-DOCTYPESYSTOPLEVTXT

SER-DOCTYPESYSTEM3

SER-DOCTYPESYSMULTIELEMROOT

SER-STANDALONE2

SER-STANDALONETOPLEVTXT

SER-STANDALONE3

SER-STANDALONEMULTIELEMROOT

SER-STANDALONE4

SER-STANDALONEOMITXMLDEC

If you have code that does a try/catch looking for one of the old exceptions or the old text, or if you have tests that use the old exceptions as keys, you will have to rewrite that code to look for the new exception.

Change to Required Java Version

The following tools and libraries that depend on a Java Runtime Environment (JRE) now require at least Java 7, rather than Java 6:

• mlcp
• MarkLogic Connector for Hadoop
• Java Client API
• XCC for Java (XCC/J)

Deprecated mlcp Command Line Options

The -aggregate_uri_id and -delimited_uri_id command line options are now deprecated. Use the more general -uri_id instead.

REST APIs That Have JSON or XML Payloads Cannot Have Empty Payloads

Starting in 8.0-4, any of the REST APIs that specifies a JSON or and XML content-type for its payload cannot have an empty payload. Previously, it allowed an empty payload. REST calls that specify a JSON or XML content-type with an empty payload throw a MANAGE_EMPTYPAYLOAD exception beginning in 8.0-4. For example, POST /admin/v1/init previously allowed an empty payload with a JSON or XML content, but requires the payload in 8.0-4.

If you have code that does not send a payload, then you must either add an empty JSON or XML document as the payload or change the content-type to one that allows an empty payload (for example, text/plain).

Geospatial Namespace and Data Version Changes

The following changes have been made to some of the geospatial built-in and library functions in 8.0-4:

• GML and KML Library Modules Moved to a New Namespace
• Some Built-In Geospatial Functions Moved to geo Namespace
• Older GML and KML Versions Deprecated

xquery version "1.0-ml";
import module namespace kml =
  "http://earth.google.com/kml/2.0" at
    "/MarkLogic/geospatial/kml.xqy";

kml:box(
  <kml:LatLongBox>...</kml:LatLongBox>)

xquery version "1.0-ml";
import module namespace geokml =
  "http://marklogic.com/geospatial/kml"
  at "/MarkLogic/geospatial/kml.xqy";
declare namespace kml =
  "http://www.opengis.net/kml/2.2";

geokml:box(
  <kml:LatLongBox>...</kml:LatLongBox>)

xquery version "1.0-ml";
import module namespace gml =
  "http://www.opengis.net/gml"
    at "/MarkLogic/geospatial/gml.xqy";

gml:box(
  <gml:Envelope>...</gml:Envelope>)

xquery version "1.0-ml";
import module namespace geogml =
  "http://marklogic.com/geospatial/gml"
    at "/MarkLogic/geospatial/gml.xqy";
declare namespace gml =
    http://www.opengis.net/gml/3.2";

geogml:box(
  <gml:Envelope>...</gml:Envelope>)

GML
Old: gml:geospatial-query($regions, $options, $weight)
New: geogml:geospatial-query(
       $regions, $options, $weight, "http://www.opengis.net/gml")

KML
Old: kml:geospatial-query($regions, $options, $weight)
New: geokml:geospatial-query(
       $regions, $options, $weight, "http://earth.google.com/kml/2.0")

spell.suggestDetailed, xdmp.filesystemDirectory, and xdmp.encodingLanguageDetect Now Return ValueIterator

In 8.0-3, the spell.suggestDetailed, xdmp.filesystemDirectory, and xdmp.encodingLanguageDetect functions now return results in a ValueIterator. In 8.0-2 and earlier, they returned results in an Array. If you have any code that expects an Array, you must rework that code to accept a ValueIterator; for example, you can use xdmp.arrayValues to convert the ValueIterator result into an Array result. For details on the ValueIterator JavaScript type, see ValueIterator in the JavaScript Reference Guide.

xdmp.databaseRestoreStatus Now Returns an Array

In 8.0-3, the xdmp.databaseRestoreStatus function now returns a JavaScript Array. Previously, it returned an ArrayNode. If you have any code that expects the ArrayNode, you must rework that code to accept an Array.

xdmp.gssServerNegotiate Now Returns a JavaScript Object

In 8.0-3, the xdmp.gssServerNegotiate function (used for kerberos GSS authentication) returns a JavaScript object. Previously, it returned an XML element.

Use of String Transaction Ids in Node.js To Be Deprecated

The Node.js interfaces that open, manipulate, or pass around transaction ids now accept either a simple id (as before) or a transaction object. You obtain a transaction object rather than an id by passing true in for the new withState parameter of DatabaseClient.transactions.open. For example:

// old forms: return a transaction id, for backward compatibility
db.transactions.open();
db.transactions.open({timeLimit: limit, transactionName: name});

// new forms: return transaction object; preferred.
db.transactions.open(true);
db.transactions.open({withState: true, ...});

In a future release, the default of withState will be changed to true and the use of transaction ids will be deprecated.

If you do not modify your code to use transaction objects rather than ids, the session affinity required by multi-statement transactions is not guaranteed to be properly preserved in all cases.

For details, see Managing Transactions in the Node.js Application Developer's Guide.

CDH 4.3 is No Longer a Supported Hadoop Distribution

The MarkLogic features and tools that rely on Hadoop no longer support CDH 4.3. Instead, use one of the newer supported versions of Hadoop.

This change affects MarkLogic Content Pump (mlcp), the MarkLogic Connector for Hadoop, and forest storage on HDFS.

Changes to How MarkLogic Locates Java and Hadoop Libraries for HDFS Forest Storage

When you use HDFS for forest storage, MarkLogic must be able to find a suitable Java installation and Hadoop HDFS libraries. The algorithm for where MarkLogic looks has changed substantially as of 8.0-3.

You should now make the Hadoop libraries available to your MarkLogic hosts using one of the new Hadoop HDFS client bundles. You must unpack one of these bundles under /opt, /usr, or /space. For details, see HDFS Storage in the Query Performance and Tuning Guide.

MarkLogic now locates a suitable Java installation using a different algorithm. You might need to either change the path to your JDK or set JAVA_HOME in the MarkLogic startup environment. You can now make JAVA_HOME available to MarkLogic in a way that is preserved across upgrades, using /etc/marklogic.conf. For details, see HDFS Storage in the Query Performance and Tuning Guide.

Array Input Differences in fn.distinctValues, fn.subsequence, and Other Functions

In 8.0-2, the Server-Side JavaScript functions fn.distinctValues and fn.subsequence behave differently from 8.0-1 if you pass in an array. In 8.0-1, these functions will extract the values out of the array to make multiple inputs, one for each item in the array. In 8.0-2, these functions treat the array as a single item. Therefore, to get the same behavior in 8.0-2, you need to call xdmp.arrayValues on the array before you pass it into these functions. For example:

fn.distinctValues([1, 1, 2]);
// returns 1, 2 in 8.0-1
// returns [1, 1, 2] in 8.0-2

To modify the above code in 8.0-2 to return the same answer as in 8.0-1:

fn.distinctValues(xdmp.arrayValues([1, 1, 2]));
// returns 1, 2 in 8.0-2

The nature of the change is that there are fewer times when MarkLogic coerces an array into its values in 8.0-2 than there were in 8.0-1. The newer behavior is more natural to JavaScript developers.

In addition to fn.distinctValues and fn.subsequence, this applies to any JavaScript function that takes a ValueIterator as input to a parameter, including: fn.exactlyOne, fn.head, fn.insertBefore, fn.reverse, fn.unordered, fn.empty, fn.remove, fn.reverse, fn.zeroOrOne, fn.oneOrMore, fn.deepEqual, fn.count, cts.contains, xdmp.setSessionField, xdmp.setServerField, and others.

The Second Parameters of xdmp.eval, xdmp.invoke, xdmp.xqueryEval, and xdmp.spawn Now Take a Single Object

In 8.0-2, the vars parameter to the Server-Side JavaScript functions xdmp.eval, xdmp.invoke, xdmp.xqueryEval, and xdmp.spawn has been simplified so it only takes a single Object. If you have any code that you used in 8.0-1 that passes the external variables (vars parameter) as an array of Objects, as an array of stings, or as a ValueIterator, you must rewrite that code in 8.0-2 so it passes a single Object.

extract-document-data Results Now Inline By Default

The query option extract-document-data previously caused search:search and search:resolve to return a sequence consisting of the search:response and the extracted documents. Now, this option returns the extracted documents embedded in the search:response as search:extracted elements.

When you use the option with the REST Client API, the /v1/search service returns the extracted content inside the search response if the Accept header MIME type is application/xml or application/json. The extracted content is still returned as individual documents when the Accept header MIME type is multipart/mixed (a multi-document read).

For details, see Extracting a Portion of Matching Documents in the Search Developer's Guide.

XCC v8.0-2 May Require Config Change When Used with Older Versions of MarkLogic

This change affects XCC applications that set the transaction mode (Session.setTransactionMode) or transaction time limit (Session.setTransactionTimeout) and use XCC v8.0-2 or later with MarkLogic Server 8.0-1 or earlier.

If your XCC application meets the above conditions, you must set the property xcc.txn.compatible to true. If you do not do so, an exception is raised if you set the transaction time limit or set the transaction mode to a value other than Session.TransactionMode.AUTO.

You can set this system property on the java command line with an argument of the following form:

java -Dxcc.txn.compatible=true

You can also set the property programmatically by calling System.setProperty.

Client APIs: JavaScript Extension and Transform Error Reporting Convention Change

The following change affects applications that use the REST Client API, Java Client API, or Node.js Client API and that raise RESTAPI-SRVEXERR from a server-side JavaScript extension, transform, or custom snippeter, or other custom code.

In MarkLogic Server 8.0-1, when calling fn.error to raise RESTAPI-SRVEXERR,error detail such as the response status code and status text are passed to fn.error as an array. Starting with MarkLogic Server 8.0-2, the error details must be passed as a sequence. You can use xdmp.arrayValues to convert the array to a sequence.

For example, if you previously had an fn.error call in an extension similar to the following:

fn.error(null, 'RESTAPI-SRVEXERR', 
         [statusCode, statusMsg, body])

Then you should now wrap the array that is the 3rd argument to fn.error in a call to xdmp.arrayValues, similar to the following:

fn.error(null, 'RESTAPI-SRVEXERR', 
         xdmp.arrayValues([statusCode, statusMsg, body]))

Some JavaScript Built-In Functions that Returned XML Structures Now Return JSON Structures

The following APIs return JSON output in 8.0-2.

• cts.plan
• cts.relevanceInfo
• xdmp.zipManifest
• xdmp.userExternalSecurity
• xdmp.userLastLogin
• xdmp.databasePathNamespaces
• cts.distinctiveTerms
• cts.cluster
• cts.train
• cts.classify
• cts.thresholds

In 8.0-1, these APIs returned the same XML structures that their XQuery counterparts return. If you have any JavaScript code that relies on the XML structures, you will need to modify that code to use the new JSON output.

HDP No Longer a Supported Hadoop Platform

Hortonworks Data Platform (HDP) is no longer a supported Hadoop distribution for use with the MarkLogic Connector for Hadoop or the distributed mode of MarkLogic Content Pump (mlcp).

Java API: ContentVersionRequest Property Deprecated

The REST server configuration property ContentVersionRequest is now deprecated. If you currently use com.marklogic.admin.ServerConfigurationManager.setContentVersionRequest() and the related ServerConfiguration.Policy type, you should modify your application to use com.marklogic.admin.ServerConfigurationManager.setUpdatePolicy() and ServerConfigurationManager.UpdatePolicy instead. You should also change calls to getContentVersionRequest() into call to getUpdatePolicy().

The table below shows the correspondence between Policy values and UpdatePolicy values. The behavior is unchanged with respect to these values.

NONE

MERGE_METADATA

REQUIRED

VERSION_REQUIRED

OPTIONAL

VERSION_OPTIONAL

REST API: content-versions Property Deprecated

The REST instance configuration property content-versions is now deprecated, in favor of the new update-policy property.

If you currently set content-versions, you should modify your application to use the update-policy configuration property instead. For example, if you set this property using PUT /v1/config/properties or PUT /v1/config/properties/content-versions, then you should now use PUT /v1/config/properties or PUT /v1/config/properties/update-policy to set update-policy instead.

Similarly, you should modify any code that reads the configuration properties to expect update-policy instead of content-versions.

The table below shows the correspondence between content-versions values and update-policy values. The behavior is unchanged with respect to these values.

none

merge-metadata

required

version-required

optional

version-optional

REST API: JSON documents Cannot be Retrieved as XML

In previous versions, you could use the REST Client API to retrieve the internal XML representation of a JSON document using GET /v1/documents and specifying XML in the format request parameter or application/xml in the Accept header.

As of MarkLogic 7.0-3, JSON documents are always returned as JSON. You can still examine the internal representation of a JSON document using XQuery or the Query Console database explorer.

Change to JSON Output from the REST API

The JSON output from the manage REST resource addresses with the metrics view has been changed and you may need to change any custom clients that consume this JSON data.

Changes to the MarkLogic Connector for Hadoop API

com.marklogic.mapreduce.MarkLogicDocument is now an interface instead of a class. The previous functionality of MarkLogicDocument is provided by the new class com.marklogic.mapreduce.DatabaseDocument.

Modify your code and job configuration to use DatabaseDocument instead of MarkLogicDocument.

XQuery HTTP Client Built-In Functions Now Require a Privilege

The HTTP client XQuery APIs (xdmp:http-delete, xdmp:http-get, xdmp:http-post, xdmp:http-put, and so on) now require a privilege. Previously, these functions did not require a privilege. To support these privileges, the following privileges are added to MarkLogic 9:

• http://marklogic.com/xdmp/privileges/xdmp-http-get
• http://marklogic.com/xdmp/privileges/xdmp-http-head
• http://marklogic.com/xdmp/privileges/xdmp-http-options
• http://marklogic.com/xdmp/privileges/xdmp-http-delete
• http://marklogic.com/xdmp/privileges/xdmp-http-post
• http://marklogic.com/xdmp/privileges/xdmp-http-put

Additionally, the network-access role has been added, which contains all of these privileges.

If you have code that accesses these functions with a user that does not have the admin role, you will have to add these privileges (or the network-access role) to the set of privileges inherited by your users. For details on adding privileges, see Security Administration in the Administrator's Guide. For details on security, see Security Guide.

HTTP Client Functions Are Now HTTP 1.1 Compliant

The XQuery HTTP client functions (xdmp:http-delete, xdmp:http-get, xdmp:http-post, xdmp:http-put, and so on) now fully implement HTTP 1.1, including the use of connection keep-alives and built-in decoding of chunked transfer encoded responses. In most cases, this will not cause any incompatible behavior, but if your code tried to perform some work to do HTTP 1.1 features such as taking a chunked response as a large binary and decoding the chunks in XQuery, then that code might behave differently in MarkLogic 7. If you have such code, you might need to rework it in MarkLogic 7.

xdmp:get-request-username and xdmp:get-request-user Changes

The xdmp:get-request-username and xdmp:get-request-user functions have changed slightly in MarkLogic 7.

In previous releases, xdmp:get-request-username always returned the user in the Authorization HTTP header; in MarkLogic 7, if you are using application-level authentication, then it returns the user from the last successful call to xdmp:login (when using any other authentication scheme, it is unchanged from previous releases and returns the user in the Authorization HTTP header).

In previous releases, xdmp:get-request-user returned the ID of the current user; in MarkLogic 7, if you are using application-level authentication, then it returns the user ID from the last successful call to xdmp:login (when using any other authentication scheme, it now returns the user ID corresponding to the user in the Authorization HTTP header). If you want the old behavior of returning the ID of the current user, use the new function xdmp:get-current-userid.

If you have any code that uses the xdmp:get-request-username or xdmp:get-request-user functions, you might need to change it to work with the current behavior.

Specifying a Forest Now Only Works With Strict Locking

The various loading APIs (for example, xdmp:document-load and xdmp:document-insert) have an option to specify the forest in which a document is loaded. In MarkLogic 7, those forest options are only available with the locking parameter on the database set to strict (the default is fast). If you try to specify forest placement with fast locking, it will throw an XDMP-PLACEKEYSLOCKING exception. Previously, this operation would be allowed.

In most cases, it is not recommended to specify a placekey, as MarkLogic does a good job of distributing data. If you want to continue to use the forest placekeys when loading or updating, you must change your locking parameter to strict or change your code to no longer use forest placekeys. You can also consider using Tiered Storage to control what data goes into which forests. For details on Tiered Storage, see Tiered Storage in the Administrator's Guide.

Custom Dictionaries for Japanese and Chinese Languages Need to be Re-saved

In MarkLogic 7, there is a change to the way custom dictionaries are stored for languages that use a unified cd (Japanese and both Simplified and Traditional Chinese). The custom dictionaries for those languages now have separate files for tokenization and for stemming. If you are using these languages and you have created a custom dictionary, you must re-save the custom dictionary in MarkLogic 7, which will save it in the new format.

For example, if you have a Japanese language custom dictionary, run the following XQuery program to re-save the custom dictionary:

import module namespace
   cdict="http://marklogic.com/xdmp/custom-dictionary"
   at "/MarkLogic/custom-dictionary.xqy";

cdict:dictionary-save("ja", cdict:dictionary-read("ja"))

This will re-save the dictionary to the new format.

Default Attributes on XML Copy Changes

In MarkLogic 7, the behavior of default attributes has changed when you are copying XML nodes that have an in-scope schema with default attributes. In MarkLogic 6, the default attributes were applied in the copied node. In MarkLogic 7, the default attributes are still applied in the data model of the copied node, but they will only be serialized if default-attributes=yes is set in the serialization options (for example, the xdmp:output option). This is true for copied nodes in both XQuery (for example, see below) and XSLT (for example, xsl:copy).

The following shows an example using the XHTML schema, which is always in-scope.

xquery version "1.0-ml";
declare option xdmp:output "default-attributes=no";
(: the above xdmp:output declaration is the default :)

<x>{xdmp:unquote('<html xmlns="http://www.w3.org/1999/xhtml">
    <body>
       <p>hello</p> 
    </body>
</html>
')}</x>
=>
In MarkLogic 7 returns: (no default attributes on html element)

<x>
  <html xmlns="http://www.w3.org/1999/xhtml">
    <body>
       <p>hello</p> 
    </body>
  </html>
</x>

In MarkLogic 6 returns: (added default attribute version 
                         on html element)
<x>
  <html version="-//W3C//DTD XHTML 1.1//EN"
        xmlns="http://www.w3.org/1999/xhtml">
    <body>
       <p>hello</p> 
    </body>
  </html>
</x>

If the xdmp:output option is set to default-attributes=yes, then the attributes will continue to be serialized in MarkLogic 7. Furthermore, if the copied node is used in another schema that has different default values for the default attributes, then the resulting default value for those attributes comes from the copied node, not the new schema. This change is not likely to impact very many applications, and the new behavior is what most people would expect, but if you have code that relies on the old behavior, you will need to explicitly set the xdmp:output option is set to default-attributes=yes.

Serialization of Alerting, Reverse, and Path Range Queries Change

The serialization of path range index queries has changed to use cts:path-expression rather than cts:path as the element name. This affects not only stored path range index queries, but also stored Alerting queries and reverse queries. All such queries must be transformed to the new serialization and reloaded.

The following XSLT stylesheet makes the required change:

<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
                xmlns:cts="http://marklogic.com/cts"
                version="2.0">
  <xsl:template match="cts:path">
    <xsl:element name="cts:path-expression">
      <xsl:copy-of select="namespace::*|@*|node()"/>
    </xsl:element>
  </xsl:template>
</xsl:stylesheet>

You can apply the stylesheet to affected documents using a query similar to the following:

xquery version "1.0-ml";
for $query in fn:collection("my-stored-queries")
return xdmp:node-replace($query, xdmp:xslt-eval($stylesheet, $query)

Java and REST Client API Incompatibilities

Unless otherwise noted, the following changes can affect applications using either the REST or Java Client API:

• Content Transformations on JSON Documents Operate on XML
• Resource Service Extensions Moved to Modules Database on Upgrade
• JSON Key Name Change for System Properties
• Java Batch Example Package Name Change

{"properties":{
  "last-modified":"value"
}}

{"properties":{
  "$ml.prop":{
    "last-modified":"value"
  }
}}

Namespace Change for Properties Persisted Using JSON

This topic applies to REST and Java Client API applications that insert or update document properties using JSON and either of the following are true:

• You define indexes based on JSON property keys.
• Your application includes a content transformation, resource extension, or other code that that manipulates JSON property metadata in its XML representation.

In MarkLogic 6, inserting or updating user-defined document properties using JSON stored the properties as XML elements in the namespace http://marklogic.com/json. In MarkLogic 7, such properties are stored as elements in the namespace http://marklogic.com/xdmp/json/basic. If you insert or update a property using JSON using MarkLogic 7, the new namespace is used.

If you convert properties in the old namespace to the new namespace, then all pre-existing and future properties will use the new namespace. If you do not perform such a conversion, then you should adapt your indexes and queries to accommodate both namespaces because pre-existing properties will be in the MarkLogic 6 namespace while new or updated properties will be in the MarkLogic 7 namespace.

After upgrading to MarkLogic 7, use one of the following solutions to adapt your content and application to this change:

• Modify Pre-Existing Propertie
• Modify Queries to Use Both Namespaces
• Create a Field That Spans Both Namespaces

<options xmlns="http://marklogic.com/appservices/search">
  <constraint name="my-prop">
    <range type="xs:string">
      <element name="my-property"
        ns="http://marklogic.com/json" />
      <fragment-scope>properties</fragment-scope>
    </range>
  </constraint>
</options>

<options xmlns="http://marklogic.com/appservices/search">
  <constraint name="my-prop">
    <range type="xs:string">
      <element name="my-property"
        ns="http://marklogic.com/xdmp/json/basic" />
      <fragment-scope>properties</fragment-scope>
    </range>
  </constraint>
</options>

{"search": {
  "options": {
    "constraint": [
      {
        "name": "old-prop",
        "range": {
          "type": "xs:string",
          "element": {
            "name": "my-property",
            "ns": "http://marklogic.com/json"
          },
          "fragment-scope": "properties"
        }
      },
      {
        "name": "new-prop",
        "range": {
          "type": "xs:string",
          "element": {
            "name": "my-property",
            "ns": "http://marklogic.com/xdmp/json/basic"
          },
          "fragment-scope": "properties"
        }
      }
    ]
  },
  "query" : {
    "or-query" : {
      "queries" : [
        {"range-constraint-query": {
          "constraint-name": "old-prop",
          "value": "the value"
        }},
        {"range-constraint-query": {
          "constraint-name": "new-prop",
          "value": "the value"
        }}
    ]}
  }
}}

{ "search": {
 "options": {
   "constraint": {
     "name": "my-prop",
     "range": {
       "type": "xs:string",
       "element": {
         "name": "my-property",
         "ns": "http://marklogic.com/json"
       },
       "fragment-scope": "properties"
  }}},
  "query" : {
    "range-constraint-query": {
      "constraint-name": "my-prop",
      "value": "the value"
  }}
}}

{"search": {
  "options": {
    "constraint": {
      "name": "my-prop",
      "range": {
        "type": "xs:string",
        "field": { 
          "name": "my-new-property" 
        },
        "fragment-scope": "properties"
  }}},
  "query" : {
    "range-constraint-query": {
      "constraint-name": "my-prop",
      "value": "the value"
  }}
}}

mlcp Incompatibilities

MarkLogic Content Pump (mlcp) version 1.1 includes the following changes that potentially affect compatibility for users of mlcp version 1.0-*.

• Compressed Input Default URI Includes Input Filename
• Default Character Encoding Changed to UTF-8

/path/inside/zip/filename

/compressed-file-path/path/inside/zip/filename 

-content-encoding system

REST Management API Version Incremented to v2

The REST Management API version has been incremented to v2. The v1 services are no longer available. If you send a request using a v1 URL, MarkLogic Server responds with status code 410 (Gone) and a MANAGE-UNSUPPORTEDVERSION error.

Requests that use LATEST as the version when constructing requests will continue to work. However, you may need to make other changes due to the behavior changes between v1 and v2.

The incompatibilities between v1 and v2 are detailed in the remainder of this section

• View Parameter Required Instead of Path Steps
• JSON Output Includes Units
• Element/Key Name Changes in Status Views
• Changes to the Management API Plugins
• Changes to the Packaging API

GET /manage/version/databases/{id|name}/status
GET /manage/version/databases/{id|name}?view=status

GET /manage/version/databases/{id|name}?view=status

/manage/v1/clusters/{id|name}/config

/manage/v2/clusters/{id|name}?view=config

/manage/v1/clusters/{id|name}/status

/manage/v2/clusters/{id|name}?view=status

/manage/v1/databases/{id|name}/config

/manage/v2/databases/{id|name}?view=config

/manage/v1/databases/{id|name}/counts

/manage/v2/databases/{id|name}?view=counts

/manage/v1/databases/{id|name}/edit

/manage/v2/databases/{id|name}?view=edit

/manage/v1/databases/{id|name}/status

/manage/v2/databases/{id|name}?view=status

/manage/v1/forests/{id|name}/config

/manage/v2/forests/{id|name}?view=config

/manage/v1/forests/{id|name}/counts

/manage/v2/forests/{id|name}?view=counts

/manage/v1/forests/{id|name}/edit

/manage/v2/forests/{id|name}?view=edit

/manage/v1/forests/{id|name}/status

/manage/v2/forests/{id|name}?view=status

/manage/v1/groups/{id|name}/config

/manage/v2/groups/{id|name}?view=config

/manage/v1/groups/{id|name}/counts

/manage/v2/groups/{id|name}?view=counts

/manage/v1/groups/{id|name}/edit

/manage/v2/groups/{id|name}?view=edit

/manage/v1/groups/{id|name}/status

/manage/v2/groups/{id|name}?view=status

/manage/v1/hosts/{id|name}/config

/manage/v2/hosts/{id|name}?view=config

/manage/v1/hosts/{id|name}/counts

/manage/v2/hosts/{id|name}?view=counts

/manage/v1/hosts/{id|name}/edit

/manage/v2/hosts/{id|name}?view=edit

/manage/v1/hosts/{id|name}/status

/manage/v2/hosts/{id|name}?view=status

/manage/v1/servers/{id|name}/config

/manage/v2/servers/{id|name}?view=config

/manage/v1/servers/{id|name}/edit

/manage/v2/servers/{id|name}?view=edit

/manage/v1/servers/{id|name}/status

/manage/v2/servers/{id|name}?view=status

"elapsed-time" : "0.023453"

"elapsed-time" : { "units" : "sec", "value" : "0.023453" }

Assets/plugins/marklogic/manage/v1

/v1/list/package/appserver={name} (GET)

/v2/servers/{name}?view=package (GET)

/v2/packages/{pkgname} (POST)

/v1/list/package/database={name} (GET)

/v2/databases/{name}?view=package (GET)

/v2/packages/{pkgname} (POST)

/v1/package/compare (GET)

/v2/packages/{pkgname}?view=differences (GET)

/v1/package/install (POST)

/v2/packages/{pkgname}/install (POST)

/v1/package-tickets/revert (POST)

/v2/tickets/{ticketnumber}/revert (PUT)

Changes to the Configuration Manager

The Configuration Manager Packaging feature has been more tightly integrated with the Configuration Manager. Rather than exporting configurations to an XML file, as in MarkLogic 6, configurations in MarkLogic 7 are now exported, as XML, to a ZIP file.

You can import a configuration saved by MarkLogic 6 into MarkLogic 7. However, you cannot import a configuration saved in MarkLogic 7 into MarkLogic 6.

xdmp:plan Now Requires a Privilege

In MarkLogic 7, the xdmp:plan function requires a privilege (http://marklogic.com/xdmp/privileges/xdmp-plan). Previously, it did not require a privilege. If you have any code that calls xdmp:plan that is run by non-privileged users, you will need to add the privilege to a role that the users are granted (or to a role that they already have).

fn:analyze-string Now Returns Output in a Different Namespace

In MarkLogic 7, the fn:analyze-string function returns an XML node in the http://www.w3.org/2005/xpath-functions namespace, which is the namespace specified by the working draft of the latest XQuery specification. Previously, this function was not fully specified and it returned XML in the http://www.w3.org/2009/xpath-functions/analyze-string namespace. If you have code that is expecting output in the old namespace, you will need to change that code to expect the new namepsace.