Loading TOC...
Release Notes (PDF)

MarkLogic Server 11.0 Product Documentation
Release Notes
— Chapter 4

Known Incompatibilities with Previous Releases

Most applications implemented on previous versions of MarkLogic will run either without modifications or with very minor modifications in MarkLogic 11. This section summarizes product changes that can cause compatibility issues for applications developed using previous releases and includes the following topics:

Incompatibilities (External)

No NVidia GPUs drivers available for CentOS 8

Currently there are no NVidia GPU drivers available on CentOS 8. Azure running CentOS 8 will not be able to use GPUs. See https://docs.microsoft.com/en-us/azure/virtual-machines/extensions/hpccompute-gpu-linux/ for more information.

The libnsl.so.1 library is required for CentOS 8

For CentOS 8, MarkLogic has a dependency on libnsl.so.1(). You must install this library when using CentOS 8. You can either rely on yum to pull in the dependency automatically, or install it manually.

DHF mlUpdateIndexes fails when database index json payload has language tag

Entity Services in MarkLogic 10.0-2 has a change whereby a language tag is added to database properties. In Data Hub, the mlUpdateIndexes fails when database index json payload has language tag. The workaround is to delete the language tag, and you should be able to update the indexes using mlUpdateIndexes gradle task.

Incompatibilities between 10.0-5 and 10.0-4

Change to Calling SUM or AVG on String-Typed Column

As of MarkLogic Server 10.0-5, in SQL, SPARQL, and Optic, XDMP-EXPR is thrown when calling SUM() or AVG() on a string-typed column argument.

Entity Services Features Deprecated

The XML representation of an entity type has been deprecated in 10.0-5.

The following Entity Services functions have been deprecated in 10.0-5 because DHF provides the equivalent functionality. These functions will not be supported in MarkLogic 11.

JavaScript XQuery
es.databasePropertiesGenerate es:database-properties-generate
es.extractionTemplateGenerate es:extraction-template-generate
es:optional
es.piiGenerate es:pii-generate
es.schemaGenerate es:schema-generate
es.searchOptionsGenerate es:search-options-generate

Hadoop Connector Removed

The Hadoop Connector was deprecated starting with MarkLogic 10.0-4 and is removed from the product in MarkLogic 10.0-5.

Ops Director Deprecated

The Ops Director feature is deprecated in MarkLogic 10.0-5.

Reindexing After 10.0-5 Upgrade

A fix made in 10.0-5 may cause reindexing after the server is upgraded to 10.0-5. If you have a server configuration where a named field in a database is configured for value searches and stemming is turned off, while the database itself has stemming is turned on, reindexing will occur after the upgrade to 10.0-5 is completed.

Incompatibilities between 10.0-4 and 10.0-3

mlcp Distributed Mode Deprecated

mlcp distributed mode has been deprecated for 10.0-4.

HDFS Support Deprecated

HDFS support has been deprecated in MarkLogic 10.0-3.

Priority to Determine MarkLogic License Key

MarkLogic prioritizes the license key value from the following locations in the following order. The version number determines which locations take precedence.

The mlcmd.conf location is only considered when MarkLogic is running on Amazon Web Services (AWS).

10.0-4:

  1. Admin Interface (server.xml)
  2. /etc/marklogic.conf
  3. mlcmd.conf (AWS CloudFormation template user data)

10.0-3:

  1. mlcmd.conf (AWS CloudFormation template user data)
  2. /etc/marklogic.conf
  3. Admin Interface (server.xml)

Entity Services Features Deprecated

The following Entity Services functions were deprecated in 10.0-4 because DHF provides the equivalent functionality. These functions will not be supported in MarkLogic 11.

JavaScript XQuery
es.instanceConverterGenerate es:instance-converter-generate
es.instanceFromDocument es:instance-from-document
es.instanceGetAttachments es:instance-get-attachments
es.instanceJsonFromDocument es:instance-json-from-document
es.instanceXmlFromDocument es:instance-xml-from-document
es.modelFromXml es:model-from-xml
es.modelGetTestInstances es:model-get-test-instances
es.modelToXml es:model-to-xml
es:add-attachments
es:copy-attachments
es:serialize-attachments
es:init-instance
es:init-source
es:init-translation-source
es.versionTranslatorGenerate es:version-translator-generate
es:extract-array
es:with-namespace

Incompatibilities between 10.0-3 and 10.0-2

Privilege escalation allowing execution of xdmp:data-directory()

Certain privileged built-in functions, for example xdmp:data-directory and xdmp:list-cache-size, that return static environmental data may be prematurely optimized out with the results in-lined as literals in code passed to xdmp:invoke-function or xdmp:spawn-function.

The in-lining occurs in the outer environment not the inner environment, so the execution privileges checked are of the outer environment not the inner environment. As a result, even if the inner environment does not have privileges to execute the built-in functions, no exception is thrown when the optimized code is run. This issue is addressed in 10.0-3.

The API trgr:triggers-change-modules-database() Only Has Two Parameters

The API trgr:triggers-change-modules-database() only takes two parameters in 10.0-3.

1. the old database as xs:anyAtomicType. This can be an ID unsignedLong or a name xs:string

2. the new database as xs:anyAtomicType. This can be a database's ID unsignedLong or a database's name xs:string.

For more details and sample code, see our API documentation.

Configuration Manager Removed

The Configuration Manager tool is deprecated starting with MarkLogic 9.0-5 and removed from MarkLogic Server in version 10.0-3.

Incompatibilities between 10.0-2 and 10.0-1

Module and Library Removed

Beginning with version 10.0-2, the mustache.xqy module and library existing under the folder MarkLogic/Modules/MarkLogic/mustache/ is no longer available.

Encryption Change

Data encrypted by the embedded KMS in MarkLogic 10.0-2 cannot be decrypted using versions prior to MarkLogic 10.0-2. Exporting the internal wallet requires a passphrase that is now encrypted by argon2, where previously the files were encrypted with crypt. Additionally, backups encrypted with only a passphrase in MarkLogic 10.0-2 cannot be restored in versions prior to 10.0-2 for the same reason.

Incompatibilities Between 10.0-1 and MarkLogic 9

Change to Admin Role

In MarkLogic 9, a user with the admin role was automatically able to see template driven extraction (TDE) views that were read-protected. In MarkLogic 10, a user with the admin role cannot see TDE views unless explicitly granted read access.

XQuery 0.9-ml Deprecated

The XQuery 0.9-ml dialect has been deprecated and will eventually no longer be supported. MarkLogic recommends that you port any legacy 0.9-ml XQuery code to either enhanced XQuery dialect (1.0-ml) or strict XQuery dialect (1.0). For more information, see Porting 0.9-ml XQuery Code to Enhanced 1.0-ml in the XQuery and XSLT Reference Guide.

Template Driven Extraction (TDE) functions are also affected by this deprecation. The function cts:valid-index-path may throw error messages if you do not upgrade your code to XQuery dialect 1.0-ml or 1.0.

JavaScript: Calling the seal() Function on MarkLogic Wrapped Objects Now Raises an Error

The following code will now throw an error because fn.doc("/doc-1.json").toObject()[0].root.d is a MarkLogic wrapped object.

var r = fn.doc("/doc-1.json").toObject()[0].root.d
  r.beforeSeal = "hello"
    var before =  r.beforeSeal;
      Object.seal(r)
=>
Error running JavaScript request: TypeError: Cannot seal

Change in Default rest-reader and rest-writer Permissions

To enable the creation of a document by a user who does not have rest-reader or rest-writer privileges, the following backward-incompatible changes were made to the REST API to improve security:

Prior to MarkLogic 10.0-1, when inserting documents, the REST API assigned permissions based on the default permissions configured for the user and role but also assigned read permissions to the rest-reader role and assigned update permissions to the rest-writer role. As a result, any user with the rest-reader role had permission to read to read all documents and any user with the rest-writer role had permission to update all documents.

In MarkLogic 10.0-1, when inserting documents, the REST API assigns permissions based only on the default permissions configured for the user and role. As a result, it is possible to adopt a security model in the REST API where no role has access to all documents.

In other words MarkLogic 9, when you wrote documents with the REST API using PUT v1/documents, the documents had the union of the following permissions:

In MarkLogic 10, when you write documents with the REST API using PUT v1/documents, the documents have the union of the following permissions:

What has changed is that the rest-reader and rest-writer convenience roles no longer have any permissions on a document unless one of the following is true:

Since the rest-writer convenience role default permissions grant reader permission to the rest-reader role and update permissions to the rest-writer role, documents written by a user who has the rest-writer convenience role are readable by users with the rest-reader role and writable by users with the rest-writer role.

To override this backward incompatibility, you must modify the user role creating documents to give default permissions to rest-writer and rest-reader.

Certificate Authority Handling Changed During Upgrade

When you upgrade to MarkLogic Server 10, the CAs defined by External Securities are merged into their appropriate App Server's CA list. This consolidates the functionality of where an App Server admin defines CAs to accept as the signer of client certificates.

Changed Functions

The following functions have been changed as of MarkLogic 10:

For more information, see External Certificate User Authentication in the Security Guide.

Deprecated Functions

The following functions have been deprecated as of MarkLogic 10 and will be removed in a future release:

Please use the following functions instead:

MarkLogic 9 Incompatibilities

This section describes the incompatibilities between MarkLogic 8 and MarkLogic 9. This is here just for convenience; for the MarkLogic 9 Release Notes, see docs.marklogic.com/9.0/guide/relnotes. The following are the incompatibilities:

JavaScript: ValueIterator Replaced By Sequence

The ValueIterator interface used to represent sequences of value in MarkLogic 8 has been replaced by the new Sequence interface. A Sequence is a JavaScript Iterable object. All functions which previously operated on or returned a ValueIterator now use a Sequence instead.

In many cases, this change is transparent to your code. However, code that depends on the following ValueIterator properties and methods must be changed:

For more details, see Sequence in the JavaScript Reference Guide and Sequence in the MarkLogic Server-Side JavaScript Function Reference.

Database Stemming is Off, Word Searches On By Default

In MarkLogic 9 and later, when you create a new database, the stemmed searches property is off by default. In MarkLogic 8 and earlier, the default is basic.

In MarkLogic 9 and later, when you create a new database, word searches are enabled by default. In MarkLogic 8 and earlier releases, word searches were disabled by default.

To achieve the pre-MarkLogic 9 default behavior, configure your database to turn off stemmed searches and set word searches to true.

These changes only affect databases you create after upgrading to MarkLogic 9. Databases that exist when you upgrade will retain their previous settings.

Collection Lexicon and Triple Index Enabled by Default

In MarkLogic 9, a fresh install of MarkLogic 9 will enable the triple index and collection lexicon for all databases. Databases that exist when you upgrade to MarkLogic 9 will retain their previous settings. Any new databases created after upgrading to MarkLogic 9 will have the triple index and collection lexicon enabled.

XCC .NET API No Longer Available

The XCC .NET interfaces have been removed from MarkLogic. Current users of XCC .NET are encouraged to use the REST Client API to create equivalent interfaces. For details, see the REST Application Developer's Guide.

The XCC Java Library continues to be available.

Changes in Semantic Query Behavior

MarkLogic 9 introduced the following changes to the behavior of semantic queries:

Triple Index and SPARQL Engine Changes

The triple index and SPARQL engine were changed in MarkLogic 9. Now as part of a query, two triples are considered equal if their subjects, predicates, and objects compare as equal with the SPARQL = operator.

Previously, these triples would be treated as two different, non-identical triples:

sem:triple(xs:anyURI("http://a"), xs:anyURI("http://b"), xs:anyURI("http://c")),
sem:triple("http://a", "http://b", "http://c").

In MarkLogic 9 and later, values of type xs:string and xs:anyURI can compare equal if they have the same lexical form according to the SPARQL = operator. This functionality is called D-entailment (D as in datatype). The two triples in the example are now considered to be the same, and the de-duplication process removes the duplicate. If you return frequencies, you would see the second triple show up as one extra in the frequencies.

Also note that the query may return either sem:triple(xs:anyURI("http://a"), xs:anyURI("http://b"), xs:anyURI("http://c")), or sem:triple("http://a", "http://b", "http://c") as the result since they are considered equal.

Forest IDs Removed From sem:sparql Function

The forest ID options for the sem:sparql function, which were part of MarkLogic 7, were removed in MarkLogic 8, but their functionality was kept for backwards compatibility. In MarkLogic 9, the forest ID option will no longer work with sem:sparql.

Triple Count Increased After Inserting Same Data Twice

During a rolling upgrade, while the cluster is in a mixed mode (not all hosts committed to the new version yet) the triple count of semantic triples may be increased after inserting the same data twice. During a rolling upgrade, the MarkLogic 9 and later triple index is not able to return triples in the correct order for MarkLogic 8 semantics. For this situation to occur, a user would need to have multiple triples that are identical except for the types of the values.

Database max merge size Now Defaults to 48 GB

The database parameter max merge size defaults to 48 GB (49152 MB) for new databases created in MarkLogic 9 and later. Previously, the default was 32 GB. Existing databases will keep whatever max merge size setting is configured after upgrading to MarkLogic 9. The new setting reflects advances in storage systems and should be appropriate for most databases.

Changes to Range Index Reference Resolution

In MarkLogic 9 and later, index reference resolution in range index construction, query constructors, and lexicon operations succeed in some cases that would have thrown an XDMP-RIDXNOTFOUND error in previous releases. If the index reference contains enough information to unambiguously match an existing index, MarkLogic will do so. Previously, MarkLogic assumed default values for some missing index attributes, resulting in an error.

Code that previously succeeded will continue to do so. Some code that would previously have gotten an error will no longer do so.

For example, suppose you define a geospatial element range index on element xs:QName("coords") with type long-lat-point and coordinate system wgs84, and then refer to the index by just the element QName (cts:element-geospatial-values(xs:QName("coord")). Previously, the reference would have been an error because the default point type (point) would have been assumed. Now, the reference resolves correctly as long as there is not a second geospatial element range index on the same QName with different coordinate system or point type.

When an ambiguous range index reference is detected, MarkLogic 9 throws one of the new exceptions XDMP-RIDXAMBIGUOUS (range index) or XDMP-GIDXAMBIGUOUS (geospatial index).

Default Stemming and Tokenization Libraries Changed for Most Languages

In MarkLogic 9 and later, the default tokenization and stemming libraries have been changed for all languages (except English) tokenization. Consequently, some tokenization and stemming behavior changed between MarkLogic 8 and MarkLogic 9. In most cases, stemming and tokenization will be more precise in MarkLogic 9 and later.

If you upgrade to MarkLogic 9 or later from an earlier version of MarkLogic, your installation will continue to use the legacy stemming and tokenization libraries as the language baseline. Any fresh installation of MarkLogic will use the new libraries. You can change the baseline configuration using admin:cluster-set-language-baseline.

Changing the baseline requires a cluster-wide restart and a reindex to avoid stemming and tokenization anomalies.

Use of the legacy libraries is deprecated. These libraries will be removed from MarkLogic in a future release.

Unless you use the legacy language baseline, reindexing is required for content in the following languages:

For other languages (except English), you might be able to avoid incompatibilities depending on the nature of your queries, but reindexing is still strongly recommended.

Tokenization and stemming are significantly different for Japanese. Tokenization is significantly different for Chinese (both simplified and traditional). The impact on other languages is more nuanced, but should lead to better results, overall. You might observe some relevance score changes on stemmed searches due to the higher degree of precision. If you require low-level details about the impact on a specific language and you have an active maintenance contract, you can contact MarkLogic Technical Support.

For more details on incompatibilities related to the changes to stemming and tokenization, see the Knowledge Base article entitled MarkLogic Server v9 Tokenization and Stemming from MarkLogic Technical Support, available at the following URL:

https://help.marklogic.com/knowledgebase/article/View/484

For more information about the new tokenization and stemming support, see Default Stemming and Tokenization Libraries Changed for Most Languages.

SQL DESCRIBE No Longer Supported by xdmp:sql

In MarkLogic 9, the SQL DESCRIBE function was no longer supported. MarkLogic does support DESCRIBE queries over ODBC, but not from xdmp:sql().

Application-Specific Logging

MarkLogic 9 and later provide application-specific access to logging. Instead of logging all errors to ErrorLog.txt, the xdmp:log function now writes to an app-server-specific log file - for example 8000_ErrorLog.txt or 8020_ErrorLog.txt. Anything event running on the task server is logged to TaskServer_ErrorLog.txt. The ErrorLog.txt file is now for MarkLogic system logging only.

Splitting the log files in this manner enables customer log files (which could potentially contain confidential information) to be separated from system log files. This helps ensure the privacy of customer data when they interact with Support and use the new Telemetry feature.

In a UNIX environment you can watch the multiple log files using something like this:

tail -f /path/file1 /path/file2 etc.

Change to Classification of Some Special Symbol Tokens

The classification of some symbols changed for purposes of tokenization as of MarkLogic 9, including the symbols in the following table. These changes can affect search results.

Symbols Old Classification New Classification
spacing accents (5E, 60, A8, AF, B4, B8) punctuation diacritic
copyright (A9) registered (AE) degree (B0) punctuation symbol
Spanish masculine/feminine ordinals (AA, BA) punctuation diacritic
superscript numbers (B2, B3, B9) punctuation diacritic
micro (B5) punctuation greek
fractions (BC, BD, BE) punctuation symbol

Change to xdmp:user-last-login

The userid parameter has been removed from xdmp:user-last-login. In MarkLogic 8 the xdmp:user-last-login took one parameter (userid). If the userid was different from the current user, the function returned an empty sequence. If the userid was the same as that of the current user, it only returned the last login information.

In MarkLogic 9 the function does not take a parameter. The xdmp:user-last-login function only returns the last login information for the current user.

Changed Interfaces for xdmp:document-insert and xdmp:document-load

These two functions, xdmp:document-insert and xdmp:document-load, significantly changed interfaces in MarkLogic 9. The interfaces will still work for purposes of backward compatibility, but are no longer documented.

search:parse Returns a Different Type for cts:query Output Format

In MarkLogic 8, search:parse returned the XML serialization of a cts:query by default. You could also explicitly select this return type by specifying cts:query as the value of the $output parameter.

In MarkLogic 9 and later, if you explicitly specify cts:query as the value of the $output parameter, you will now get a cts:query object instead of a serialized query. The default return type from search:parse is unchanged.

If your code explicitly sets the output type to cts:query and passes the output of search:parse straight through to functions such as search:resolve or cts:search, no code changes are required.

If your code explicitly sets the output type to cts:query and then transforms or traverses the output query XML, then you must change your code to use the new schema-element(cts:query) for the $output parameter of search:parse.

For example, the following call generates the XML serialization of a cts:query in MarkLogic 9 and later:

search:parse("cat AND dog", (), "schema-element(cts:query)")

Default Client API Search Behavior Change on Port 8000

The defined default search behavior for the Java, Node.js, and REST Client APIs is unfiltered search unless you override the default with your own options. Prior to MarkLogic 9, this default behavior was not honored when you used the Client APIs to interact with MarkLogic through the pre-defined App Server on port 8000.

As of MarkLogic 9, searches via the Client APIs on port 8000 default to unfiltered search. To get the old behavior, use query options that explicitly specify filtered search.

JSON Property Scope and Container Queries Match Array Items Differently

In MarkLogic 8, the query constructed by the XQuery function cts:json-property-scope-query and the Server-Side JavaScript function cts.jsonPropertyScopeQuery matched if its criteria query matched anywhere within the configured scope. In MarkLogic 9 and later, the behavior changed for certain JSON property scope queries where the scope property value is an array of objects.

The behavior has changed for searches that have the following characteristics:

Given this setup, in MarkLogic 9 and later, the query only matches if all the sub-queries of the and-query occur in the same array item. In MarkLogic 8, the query matches even when the and-query matches occur in different array items. All other forms of JSON property scope query are unchanged.

This change makes it possible to construct a JSON property scope query that constrains matches to occurrences with a single array item. In MarkLogic 8, this was not possible.

This change also affects the container-query element of a structured query and QBE container queries.

The following example query is of the form affected by this change. It finds co-occurrences of prop1 with value value1 and prop2 with value2 within the scope of root. (The and-query criteria can be arbitrarily complex.)

Language Example Query
XQuery
cts:json-property-scope-query(
  "root", 
  cts:and-query((
    cts:json-property-value-query("prop1", "value1"),
    cts:json-property-value-query("prop2", "value2")
  ))
)
Server-Side JavaScript
cts.jsonPropertyScopeQuery(
  'root',
  cts.andQuery([
    cts.jsonPropertyValueQuery('prop1', 'value1'),
    cts.jsonPropertyValueQuery('prop2', 'value2')
  ])
);

If you search the following documents with the previous query shown, you get the results shown. The JSON properties that match the and-query criteria are shown in bold.

Sample Document MarkLogic 8 MarkLogic 9
// (1) criteria met in different array items
{"root": [
  { "prop1": "value1", "prop2": "v" },
  { "prop1": "v", "prop2": "value2" }
]}
Match No match
// (2) criteria met in the same array item
{"root": [
  { "prop1": "value1", "prop2": "value2" },
  { "prop1": "v", "prop2": "v" }
]}
Match Match
// (3) criteria met in a child property
{"root": {
  "child": [
    { "prop1": "value1", "prop2": "v" },
    { "prop1": "v", "prop2": "value2" }
  ]
}}
Match Match

Document (1) does not match in MarkLogic 9 and later because both and-query criteria are not satisfied in the same array item. Document (2) matches in MarkLogic 9 and later because both and-query criteria are satisfied in the same array item. The Document (3) results are unaffected because the value of the root property is not an array of objects.

You can restore the MarkLogic 8 behavior by using an and-query of multiple JSON property scope queries. For example, the following query matches Document (1) in MarkLogic 9 and later:

Language Example Query
XQuery
cts:and-query((
  cts:json-property-scope-query(
    "root", cts:json-property-value-query("prop1", "value1")),
  cts:json-property-scope-query(
    "root", cts:json-property-value-query("prop2", "value2"))
))
Server-Side JavaScript
cts.andQuery([
  cts.jsonPropertyScopeQuery(
    'root', cts.jsonPropertyValueQuery('prop1', 'value1')),
  cts.jsonPropertyScopeQuery(
    'root', cts.jsonPropertyValueQuery('prop2', 'value2'))
]);

To ensure the modified queries match only when the and-query matches occur in the same instance of the scope property (root in the above examples), wrap it in JSON property scope query on the parent property. For example:

Language Example Query
XQuery
cts:json-property-scope-query("parent-of-root",
  cts:and-query((
    cts:json-property-scope-query(
      "root", cts:json-property-value-query("prop1", "value1")),
    cts:json-property-scope-query(
      "root", cts:json-property-value-query("prop2", "value2"))
)))
Server-Side JavaScript
cts.jsonPropertyScopeQuery('parentOfRoot',
  cts.andQuery([
    cts.jsonPropertyScopeQuery(
      'root', cts.jsonPropertyValueQuery('prop1', 'value1')),
    cts.jsonPropertyScopeQuery(
      'root', cts.jsonPropertyValueQuery('prop2', 'value2'))
]));

REST Client API Incompatibilities

MarkLogic 9 introduced the following backward incompatible changes to the REST Client API.

keyvalue Service Removed

The previously deprecated /keyvalue is no longer available. That is, the method GET /v1/keyvalue is no longer available.

To perform similar operations, use the /search service with a structured query or combined query, or the /qbe service.

Collections in Request Parameters are OR Related

When you specify multiple collections through the collection request parameter of the following methods, they are now OR related:

The new behavior is consistent with the semantics of cts:collection-query and the structured query collection-query.

To get AND semantics, construct your own and-query of collection URIs as part of a structured or combined query, or by defining a constraint in your query options.

The new behavior differs from previous behavior in the following way:

Default value of Document Management repair parameter changed

The default value of the repair parameter in the Document Management PUT /v1/documents function has changed in the following way:

Java Client API Incompatibilities

MarkLogic 9 introduces the following backwards incompatible changes to the Java Client API:

Java Client API: Removal of Deprecated Interfaces

Most of the previously deprecated packages, interfaces, classes, and methods of the Java Client API have been removed. The following table lists what has been removed and provides guidance for modifying your code.

In the following table, c.m.c. is an abbreviation for com.marklogic.client..

Removed Package, Class, or Method Alternative
c.m.c.admin.ServerConfigurationManager:
  • getContentVersionRequests
  • setContentVersionRequests
Use the following equivalent methods instead.
  • getUpdatePolicy
  • setUpdatePolicy
c.m.c.admin.TransformExtensionsManager:
  • writeXqueryTransform
  • writeXSLTransform

Overloads of the listed methods that accept a paramTypes map have been removed.

Use one of the overloads that does not accept parameter metadata. This metadata is not required to install or use a transform.

c.m.c.admin.config

This package, its sub-packages, and all contained classes, interfaces, and types have been removed. This includes the QueryOptions and QueryOptionsBuilder classes.

Create query options using your preferred JSON or XML libraries, read options from a file, or build options as a string. Read and write options using appropriate handles, such as DOMHandle, JacksonHandle, FileHandle, or StringHandle.

c.m.c.document.JSONDocumentManager:
  • getLanguage
  • setLanguage
Remove your usage. Language specifications are not supported for JSON. Calls to these methods were ignored in MarkLogic 8.
c.m.c.io.QueryOptionsHandle

Create query options using your preferred JSON or XML libraries, read options from a file, or build options as a string. Read and write options using appropriate handles, such as DOMHandle, JacksonHandle, FileHandle, or StringHandle.

This class is no longer needed with the removal of QueryOptions and QueryOptionsBuilder.

c.m.c.query:
  • KeyValueDefinition
  • KeyLocator
  • ElementLocator
The key-value search capability has been removed. Build equivalent queries using a StructuredQueryDefinition (JSON property query or element query) or QBE.
c.m.c.query.QueryManager:
  • newKeyValueDefinition
  • newElementLocator
  • newKeyLocator
The key-value search capability has been removed. Build equivalent queries using a StructuredQueryDefinition (JSON property query or element query) or QBE.
c.m.c.io.SearchHandle.forceDOM Remove your usage. This setting was ignored in MarkLogic 8.
c.m.c.query.StructuredQueryBuilder:
  • FragmentScope.DOCUMENT
Use FragmentScope.DOCUMENTS.
StructuredQueryBuilder (nested classes):
  • AndNotQuery
  • AndQuery
  • BoostQuery
  • CollectionConstraintQuery
  • CollectionQuery
  • CustomConstraintQuery
  • DirectoryQuery
  • DocumentFragmentQuery
  • DocumentQuery
  • ElementConstraintQuery
  • GeospatialConstraintQuery
  • LocksQuery
  • NearQuery
  • NotQuery
  • OrQuery
  • OrQuery
  • PropertiesConstraintQuery
  • PropertiesQuery
  • RangeConstraintQuery
  • TermQuery
  • ValueConstraintQuery
  • WordConstraintQuery
Continue to construct structured queries using StructuredQueryDefinition methods, as before. The return type from the query builder methods is always StructuredQueryDefinition now. For example, StructuredQueryDefinition.and() used to return an AndQuery, but now returns a StructuredQueryDefinition.
StructuredQueryBuilder.elementConstraint StructuredQueryBuilder.containerConstraint
Java Client API: JAR File Name and Maven Artifact ID Change

The JAR file for the Java Client API is now named marklogic-client-api-version.jar. Previously, the name was java-client-api-version.jar. For example, the JAR file name for version 3.0.6 is java-client-api-3.0.6.jar, but the JAR file name for version 4.0.1 is marklogic-client-api-4.0.1.jar.

The Maven artifact ID for the Java Client API is now marklogic-client-api instead of java-client-api. Update your build dependency configuration files, such as pom.xml or build.gradle, accordingly.

Logging Turned Off by Default

The Logback library is no longer included in the Java Client API distribution. Logging is now off by default. To re-enable logging, include Logback or another slf4j JAR in your classpath.

Node.js Client API Incompatibilities

The following incompatible changes had been made to the Node.js Client API:

Changes to Return Value of documents.remove

The method DatabaseClient.documents.remove previously returned an object with both a "uri" property and a "uris" property that served the same purpose. The "uri" property has been removed. The return value now has the following form:

{ "uris":[uri1, uri2, ...], "removed":true }

The value of the "uris" property is now always an array. Previously, if you passed in just a single string as input, remove returned just the URI string, rather than an array of one item.

Transaction Creation Returns an Object by Default

Previously, DatabaseClient.transactions.open returned a transaction id string by default, and you could use the withState parameter to request a transaction object instead. The use of the string form was deprecated as of MarkLogic 8.

As of MarkLogic 9 and Node.js Client API v2.0.0, DatabaseClient.transactions.open defaults to returning a transaction object.

To force the previous behavior, set withState to false when creating a transaction. This setting is deprecated and will be removed in a future release.

Default Search Result Slice is Zero-Based

Previously, the slice clause on search (queryBuilder.slice) accepted a one-based starting position and a page length:

slice(oneBasedStart,PageLength)

As of MarkLogic 9 and Node.js Client API v2.0.0, queryBuilder.slice clause behaves like Array.prototype.slice. That is, it takes a zero-based starting position and the (zero-based) position after the last result to be retrieved. For example, the following slice call returns the first 5 results:

... .slice(0,5) ...

Also, you could previously use slice(0) to suppress the return of result documents and just retrieve an abbreviated summary. Now, you must include both the start and end positions for the same effect. For example: slice(0,0).

To restore the legacy behavior, use marklogic.setSliceMode. Note, however, that this form is deprecated and will be removed in a later release.

The semantics of valuesBuilder.slice are unchanged. This function still accepts as 1-based starting position and a page length.

Geospatial Region Accessors Can Now Return Double Values

The introduction of support for double precision coordinates in MarkLogic 9 means that some geospatial operations may return different results than in the past.

Previously, geospatial region accessor functions such as the XQuery function cts:point-latitude or the Server-Side JavaScript function cts.pointLatitude always returned float values. As of MarkLogic 9, these functions can return either single or double precision values, depending on the governing coordinate system. If you do not use a double precision coordinate system, you should not notice a difference.

The following functions are affected.

XQuery Function JavaScript Function
cts:point-latitude 
cts.pointLatitude 
cts:point-longitude 
cts.pointLongitude 
cts:box-west 
cts.boxWest 
cts:box-east 
cts.boxEast 
cts:box-south 
cts.boxSouth 
cts:box-north 
cts.boxNorth 

For more details, see How Precision Affects Geospatial Operations in the Search Developer's Guide.

User-Defined Function Plugins Must Be Recompiled

The version of MarkLogic's C++ User-Defined Function (UDF) interface has been incremented to accommodate the following changes:

You must recompile your UDF plugins for use with MarkLogic 9. UDFs from an earlier version of MarkLogic will not work in a mixed cluster prior to committing the new version. During a rolling upgrade, you must finish the upgrade and then recompile your UDFs for use with MarkLogic 9.

SLES 12 No Longer Supported

In MarkLogic 9, the SUSE Linux Enterprise Server operating system is not supported. If you are using this discontinued platform, you will need to migrate your environment to a supported platform. For details on supported platforms, see Supported Platforms.

Solaris No Longer Supported

In MarkLogic 9, the Solaris operating system is not supported. If you are using this discontinued platform, you will need to migrate your environment to a supported platform. For details on supported platforms, see Supported Platforms.

Nagios Plugin No Longer Supported

Support for the Nagios monitoring plugin has been discontinued in MarkLogic 9.

You can create your own MarkLogic monitoring integration using the MarkLogic REST Management API; for details see the Monitoring MarkLogic Guide.

MarkLogic does not endorse or support any particular 3rd party monitoring integrations. However, the MarkLogic open source community includes integrations with 3rd party monitoring platforms such as New Relic and App Dynamics. For more details see:

Application Builder and Information Studio No Longer Available

The Application Builder and Information Studio applications have been removed from MarkLogic. The info and infodev APIs remain, but they are deprecated; for details, see info and infodev APIs Deprecated.

An Application Builder application deployed on MarkLogic 8 will continue to work after the upgrade to MarkLogic 9. We do not support any mechanism to support redeploying such an app against a MarkLogic 9 or later node. The methods and modules needed for this process have been removed.

Porting an Application Builder application running in MarkLogic 7 to MarkLogic 9 is not supported by MarkLogic Server. If you have an active maintenance contract, you can contact MarkLogic Technical Support for guidance in porting this application.

Admin Interface No Longer Selects a Default Schemas Database

As of MarkLogic 9, you can create a database without an associated schemas database. Earlier versions of MarkLogic required you to specify a schemas database when creating a new database. This change has no impact on users creating databases through the Admin API. However, this change affects the database creation page of the Admin Interface as follows:

The Admin Interface no longer automatically selects the pre-defined Schemas database as the schemas database when you create a database through the UI. Instead, you must explicitly select a schemas database or else you will create a database with no associated schemas database.

Depending on the uses to which you put the database, some operations might fail if there is no associated schemas database. For example, features such as temporal document management and template driven extraction require a schemas database to be associated with the content database.

Internal Security ON with External Security Object Behavior Change

This section describes some changes related to external security support. You should be aware of these changes, but they do not introduce incompatibilities or necessitate a change to your application.

In MarkLogic 9 and later, you can log into port 7001 on an LDAP account when internal security on appserver port 7001 is set to true or false. In previous versions of MarkLogic, you can only log into port 7001 on an LDAP account if internal security on appserver port 7001 is set to false.

In MarkLogic 9 and later, you can assign multiple external security objects to an App Server. When there are multiple external security objects assigned, a MarkLogic user is authenticated and assigned to an external security based on the order in which the external security objects are assigned. In previous version of MarkLogic, you could only assign one external security object to an App Server.

If internal security is enabled for an App Server, then when a user attempts to authenticate with MarkLogic, MarkLogic first checks to see if the user is in the security database. If so, then MarkLogic verifies the credentials against the security database. When this verification fails, the behavior of MarkLogic 8 and earlier versions differs from the behavior of MarkLogic 9 and later as follows:

Thus, when internal security is enabled, there can be cases where a login will succeed in MarkLogic 9 that would have failed with earlier versions. The behavior is unchanged if there are no external security objects assigned to the App Server or internal security is disabled.

REST Management API Changes in MarkLogic 9

In MarkLogic 9, the following REST Management API methods were deleted and their functionality was moved elsewhere. The following table lists the deleted methods and the new alternative.

Endpoint Deleted in 9.0 Alternative
DELETE /manage/v2/credentials DELETE /manage/v2/credentials/properties
GET /manage/v2/credentials GET /manage/v2/credentials/properties
PUT /manage/v2/credentials PUT /manage/v2/credentials/properties
GET /manage/v2/databases/{id|name}/sub-databases GET /manage/v2/databases
POST /manage/v2/databases/{id|name}/sub-databases POST /manage/v2/databases
GET /manage/v2/databases/{id|name}/super-databases GET /manage/v2/databases
POST /manage/v2/databases/{id|name}/super-databases POST /manage/v2/databases
DELETE /manage/v2/databases/{sub-id-or-name}/super-databases/{super-id-or-name} DELETE /manage/v2/databases/{id|name}
GET /manage/v2/databases/{sub-id-or-name}/super-databases/{super-id-or-name} GET /manage/v2/databases/{id|name}/properties
DELETE /manage/v2/databases/{super-id-or-name}/sub-databases/{sub-id-or-name} DELETE /manage/v2/databases/{id|name}
GET /manage/v2/databases/{super-id-or-name}/sub-databases/{sub-id-or-name} GET /manage/v2/databases/{id|name}/properties

Configuration Packaging Format Incompatibilities

Configuration packages created with the Packaging REST API and/or with the Configuration Packaging XQuery library module are not compatible with the configuration format introduced in MarkLogic release 9.0-5.

These configuration packages cannot be used with the CMA REST API and/or with the Configuration Management API XQuery/JavaScript library modules.

For more details, see the following sections:

Configuration Management API (CMA) XQuery and JavaScript Libraries

MarkLogic 9.0-5 introduced library functions for configuration management.

The configuration management functions can be used to:

Use the following functions for configuration management:

XQuery Server-Side JavaScript
cma:generate-config cma.generateConfig
cma:apply-config cma.applyConfig

For more details, see the MarkLogic XQuery and XSLT Function Reference and the MarkLogic Server-Side JavaScript Function Reference.

Configuration Management API (CMA) REST Endpoints

MarkLogic 9.0-5 introduced a REST API for configuration management: Configuration Management API (CMA).

The Configuration Management API is a RESTful API that allows retrieving, generating, and applying configurations for MarkLogic clusters, databases, and application servers.

Use the following endpoints for configuration management:

Endpoint Description
GET /manage/v3 This endpoint enables retrieving configuration of an individual resource, a set of resources, or a full cluster. It also enables generating new configurations from scenarios.
POST /manage/v3

This endpoint enables applying named configurations to MarkLogic resources, overriding parameters and setting options.

The configurations may be applied to an individual resource, a set of resources, or a full cluster.

For more details, see the MarkLogic REST API Reference.

Java Client API 4.1.1 Incompatibilities

Version 4.1.1 of the Java Client API introduced the following incompatibilities with earlier Java Client API versions:

Load Balancer Configuration for DMDSDK Jobs

This change does not affect you if you do not connect to MarkLogic through a load balancer or if you do not use the Data Movement SDK (DMSDK) feature of the Java Client API.

You must change the configuration of any DatabaseClient objects used to connect to MarkLogic through a load balancer on behalf of a Data Movement SDK job. The following changes apply:

For more details, see Working with a Load Balancer in the Java Application Developer's Guide.

MarkLogic SQL ORDER-BY Keyword

Starting in MarkLogic 9.0-9, ORDER-BY ordering on NULL is set to NULLS LAST. This changes default behavior for nulls in query results and therefore may cause backwards incompatibility for certain queries (unless the Optic Nulls Smallest On trace event is turned on).

Changes to Accepted XML Character Set

As of MarkLogic 9.0-6, parsing of XML documents with an XML declaration that explicitly specifies XML version 1.1 (version="1.1") enforces the XML 1.1 character set. Consequently, you can now create content containing characters not accepted by XML 1.0.

Characters in the XML 1.1 restricted character ranges must be given as character entities. This enforcement applies to the following character ranges:

The following character ranges that were previously disallowed are now accepted.

Serializing content that contains characters allowed by XML 1.1 but not XML 1.0 can cause problems for client layers that cannot handle the XML 1.1 character set. You can control whether to serialize XML as version 1.0 or 1.1 by setting the output version in your App Server output options, using methods such as the following:

Minimum Required Version of HDP is 2.6

As of MarkLogic 9.0-5, if you are using the Hortonworks Data Platform (HDP) with the following MarkLogic technologies, you must use HDP v2.6.

Reindex Recommended for Geospatial Region Indexes

This change affects only database configuration that define a geospatial region index.

MarkLogic 9.0-5 introduces support for a tolerance option on geospatial region queries. As a consequence, if you use geospatial region indexes, your region queries might not be accurate until you reindex. This is applicable whether or not you take advantage of the new tolerance option.

Geospatial Region Query Results Might Differ

As of MarkLogic 9.0-5, MarkLogic uses the coordinate system default tolerance when evaluating a geospatial region query. Previously, MarkLogic used the coordinate system minimum tolerance.

Consequently, some region queries might not give the same results after upgrading to MarkLogic 9.0-5. Use the tolerance option to adjust your region queries, if necessary.

To learn more about tolerance, see Understanding Tolerance in the Search Developer's Guide.

return-query Option Output Format Change

This change only affects users of the REST, Java, and Node.js Client APIs who use the return-query query option and generate a search response summary in JSON.

In prior versions of MarkLogic, the return-query query option returned a serialized JSON structured query in the query property of the search response in JSON, but a serialized cts:query in an XML search response. As of MarkLogic 9.0-5, return-query always generates a serialized cts:query. The serialized JSON representation can be passed to cts:query or cts.query to reconstruct an in-memory query object on MarkLogic, just as you can with the XML serialization.

This change will not affect you if you do not use a JSON search response, do not use return-query, or do not have code that depends on the serialization produced by return-query.

Redaction: Deterministic Masking Values Differ

MarkLogic 9.0-4 introduced support for salting in the generation of masking values by the mask-deterministic redaction function. This means the same text redacted with mask-deterministic in earlier versions of MarkLogic will not produce the same masking value by default when redacted using MarkLogic 9.0-4.

To preserve the previous behavior, modify your mask-deterministic rules to set the salt and extend-salt options to an empty value.

For more details, see mask-deterministic in the Application Developer's Guide.

Incompatibilities Between 9.0-2 and 9.0-3

The following incompatibilities exist between MarkLogic 9.0-2 and MarkLogic 9.0-3:

Changes to Authentication Behavior with Client Certificate

MarkLogic Server 9.0-3 authenticates using both a client certificate and a username/password. This provides a greater level of security, by requiring that the user provide a client certificate that matches the specified user. In MarkLogic 9.0-2, if the password is incorrect, but the user has the correct client certificate, and the ssl require client certificate is false, authentication will succeed. In MarkLogic 9.0-3 if the password is incorrect, but the user has the correct client certificate and ssl require client certificate is false, authentication fails.

The ssl require client certificate is an appserver configuration. With the appserver, there are different authentication options: basic, digest, application-level, certificate, and kerberos-ticket. This change only applies to basic, digest or application-level authentication. This behavior only happens when ssl require client certificate is set to false

In previous versions, MarkLogic would accept username/password for any internal user, once MarkLogic verified that certificate was signed by the selected CA (certificate authority). MarkLogic 9.0-1 and 9.0-2 authentication is restricted to an internal user with a matching common name (CN). In MarkLogic 9.0-3, the username does not need to match the common name (CN).

The behavior in MarkLogic 9.0-3 is the same as it is in MarkLogic 8.0. For more information, see the Certificate topic in the Security Guide.

XCC ContentSource.newSession Interface Change

The XCC method ContentSource.newSession has been extended to include a new overload that accepts a char[] value for the password parameter, rather than a String, to enable more secure handling of passwords.

As a consequence of this change, passing null in the password parameter of newSession now results in a compile time error because the compiler cannot determine the correct overload without type information. Use a typecast to disambiguate your call.

Document Digest Authorization Behavior Changed in 9.0-3

Prior to MarkLogic 9.0-3, the server did not check the nonce for digest authentication. In MarkLogic 9.0-3, the server checks the nonce, verifies that it is a valid nonce, and verifies that the URI in the Authorization header is same as the request URI.

1-click AMIs, new compatible CloudFormation, and additional upgrade procedures

To support 1-click deployment in AWS Marketplace, MarkLogic 9 AMIs have data volume pre-configured (device on /dev/sdf). To be compatible with 1-click AMIs, new CloudFormation templates are released on https://developer.marklogic.com/products/cloud/aws. To upgrade existing clusters, new 1-click compatible CloudFormation templates are required.

If custom templates or scripts are used, additional steps may be required to handle blank data volume that is part of the new marketplace AMIs. Please find more details about the upgrade procedure on https://developer.marklogic.com/products/cloud/aws.

map:new Retains Keys with Empty Values

As of MarkLogic 9.0-4, the map:new function retains any input key-value pair whose value is an empty sequence. Previous versions of MarkLogic 9 and versions of MarkLogic 8 prior to 8.0-7 discarded such key-value pairs.

For example:

map:keys(map:new(map:entry("noval",())))

(: Now returns "noval". Previously, returned an empty sequence. :)

The mlcp Option -tolerate_errors is Ignored

The -tolerate_errors option of the mlcp import command is deprecated. As of MarkLogic 9.0-2, mlcp ignores this option and always behaves as if -tolerate_errors is set to true. The option will be removed in a future release.

Changes to jsearch.facets Output Structure

In previous versions of MarkLogic, calling jsearch.facets always produced a JSON object for each facet, with object properties of the form facetValue:count. This structure prevented proper sorting of facet values.

As of MarkLogic 9.0-2, the structure of each facet is an array of arrays instead of a JSON object if and only if you include an explicit orderBy clause in your facet definition. If you do not use orderBy, the output is unchanged.

For more details, see Sorting Facet Values with OrderBy in the Search Developer's Guide.

Array Type is Preserved in x509 Certificate with Array-Valued Properties

In MarkLogic 9.0-1, if you use xdmp.x509CertificateGenerate to generate a certificate, and the configuration object includes array-valued properties, the array values were encoded as a single string. As of MarkLogic 9.0-2, the array type is preserved. This change applies to any Relative Distinguished Names (RDNs) within a Distinguished Name (DN), such as the issuer and subject DNs.

For example, in the following snippet, the issuer.organizationName property has an array value.

var certObj = {
  version: "2",
  serialNumber: "BA0195369CD6B679",
  issuer: {
    countryName: "US",
    stateOrProvinceName: "CA",
    localityName: "San Carlos",
    organizationName: ["MarkLogic", "Mark Logic"],
    organizationalUnitName: "Eng",
    emailAddress: "jdonner@marklogic.com",
    commonName: "JGD Certificate Authority",
  },...
};
var privateKey = ...;
xdmp.x509CertificatExtract(
  xdmp.x509CertificateGenerate(certObj, privateKey)
);

If you round trip the generated certificate through xdmp.x509CertificateExtract, you will see the following output for issuer.organizationName in MarkLogic 9.0-1 vs. MarkLogic 9.0-2.

// Marklogic 9.0-1
organizationName: "[\"MarkLogic\", \"Mark Logic\"]"

// MarkLogic 9.0-2 and later
organizationName: ["MarkLogic", "Mark Logic"]

If you do not have a certificate containing a multi-valued property, you will not notice any difference in behavior.

Node.js Client API: valuesBuilder.slice is Now Zero-Based

Previously, the slice clause on values queries (valuesBuilder.slice) accepted a one-based starting position and a page length:

slice(oneBasedStart,PageLength)

As Node.js Client API v2.0.3, the valuesBuilder.slice clause behaves like Array.prototype.slice. That is, it takes a zero-based starting position and the (zero-based) position after the last result to be retrieved. For example, the following slice call returns the first 5 results:

... .slice(0,5) ...

To restore the legacy behavior, use marklogic.setSliceMode. Note, however, that this form is deprecated and will be removed in a later release.

Changes to xdmp:update XQuery Prolog Option

Previously, setting the XQuery prolog option xdmp:update to false caused MarkLogic to automatically detect whether a module should be evaluated as an update or a query transaction.

As of MarkLogic 9.0-2, setting the option to false tells MarkLogic to treat the code as a query transaction. This could cause your program to get an error if you explicitly set the option to false and your code performs an update operation.

The new auto option value is equivalent to the previous behavior of false.

For related changes, see the following topics:

Java Client API 4.0.2 Ignores HttpClientConfigurator

As of version 4.0.2, the Java Client API uses OkHttp as its HTTP client for communicating with MarkLogic over HTTP. This change should be transparent to most applications, but imposes the following backward incompatibility on applications that customize their HTTP configuration:

Attaching a configurator based on HttpClientConfigurator to a DatabaseClientFactory object no longer has any effect on the HTTP configuration. Use the new com.marklogic.client.extra.okhttpclient.OkHttpClientConfigurator interface instead. The HttpClientConfigurator interface is deprecated and will be removed in a future release.

MarkLogic 8 Incompatibilities

This section describes the incompatibilities between MarkLogic 7 and MarkLogic 8. This is here just for convenience; for the MarkLogic 8 Release Notes, see docs.marklogic.com/8.0/guide/relnotes. The following are the incompatibles:

JSON Related Incompatibilities

MarkLogic 11 includes Native JSON support. In MarkLogic 7, there was support for JSON via a set of libraries to convert between JSON and XML. If you are using the MarkLogic 7 JSON support, you will have to migrate your code to use the native JSON support. This will end up being more efficient, but will require you to do some minor code changes. This section lists the incompatibilities related to working with JSON documents:

Documents Created as JSON With MarkLogic 7 REST API or mlcp Must Be Converted to Native JSON

In previous versions of MarkLogic, you can load JSON documents into MarkLogic using either the REST Client API or the Java Client API. When you do so, the documents are transformed and stored as XML, but are still searchable as and returned as JSON. In MarkLogic 8 this XML facade is no longer needed, and the REST and Java Client APIs in MarkLogic 8 do not do the translation to the XML facade anymore. Therefore, if you have any document that were loaded as JSON in MarkLogic 7 and earlier, you must convert those document to native JSON in order to query them as JSON using the REST API.

To help with the conversion, MarkLogic supplies a set of conversion scripts. These scripts do not handle every case, and for any case it does not handle you will have to convert the documents some other way. Because the conversion scripts do not handle all cases, it is very important to do a backup of your database before attempting the conversion. The scripts are located in the Samples/migrate-scripts directory under the MarkLogic installation directory (/opt/MarkLogic on Linux, c:/Program Files/MarkLogic on Windows, and ~/Library/MarkLogic on Mac OS). The scripts require bash and curl, and on Windows platforms they also require Cygwin.

For more details, see the <marklogic-dir>/Samples/migrate-scripts/README file.

Generally, the conversion scripts perform the following:

The scripts do not upgrade your application code. The types of things you will need to change in your application include:

The basic steps to upgrade your MarkLogic 7 or earlier JSON to native JSON in MarkLogic 8 are as follows:

  1. Backup the database in which your JSON documents exist.
  2. Make copies and edit the connection details and other information in the various configuration files in the Samples/migrate-scripts/conf directory. This files have details about your configuration and index settings.
  3. Run the Samples/migrate-scripts/migrate script.
  4. Test your results. Make sure the index changes that the scripts made match your newly converted JSON data. It is especially important to review path and fields indexes to make sure they are including the same content in the converted JSON is they were previously.

If you have problems upgrading your application and you have an active maintenance contract, you can contact MarkLogic Technical Support.

json:unquotedString Primitive No Longer Available

MarkLogic 7 had a primitive to convert an XQuery string to an unquoted String called json:unquotedString. In MarkLogic 8, that function is no longer available because it is no longer needed, as MarkLogic 8 has much more extensive support for JSON. For details on working with JSON in MarkLogic, see Working With JSON in the Application Developer's Guide.

xdmp:to-json and json:transform-to-json Now Returns a document-node()

In MarkLogic 8, the xdmp:to-json function returns a document-node(); previously, it returned a string. If you have code that expects a JSON string, you might need to modify your code to perform XPath on the document-node() to get the JSON node (which will serialize into a string); depending on what your code does, you might or might not need to do this. For example:

(: 7.0 :)
xdmp:to-json(("a",fn:false()))
=> ["a",false]
(: 8.0 :)
xdmp:to-json(("a",fn:false()))/node()
=> ["a",false]

The json:transform-to-json function uses xdmp:to-json, so it also returns a document-node() in MarkLogic 8.

Search, Java, REST: json-key Is Now json-property in Options and Structured Query

All occurrences of json-key in query options and structured query are now json-property. If you use the constructs listed below to search JSON documents by key/property name, you will need to modify your query options (search:options, in XML) or structured queries.

The following query options are affected. For details, see search:search or Appendix: Query Options Reference in the Search Developer's Guide.

The following structured query components are affected. For more details, see the structured query Syntax Reference in the Search Developer's Guide.

The corresponding Java Client API structured query builder method name has also changed. StructuredQueryBuilder.JSONKey is now StructuredQueryBuilder.JSONProperty.

Java and REST: Specifying a Language for JSON Documents is Deprecated

Previously, you could specify a language when ingesting JSON documents. This was only possible because JSON documents were represented internally as XML.

This parameter is now deprecated and will be ignored when present. This affects the following interfaces:

Java and REST: Default Path Language for JSON Document Patches is Now XPath

This section applies to applications that use the Java Client API or REST Client API patch (partial update) feature on JSON documents.

Previously, JSONPath was the default path language for identifying the target of update operations in a JSON patch. XPath is now the default path language for both XML and JSON patches.

Use of JSONPath is now deprecated. To convert your JSON patches to use XPath expressions instead of JSONPath, see Traversing JSON Documents Using XPath in the Application Developer's Guide.

To continue using JSONPath, you can explicitly override the default path language in one of the following ways:

Java and REST: New Restrictions on Patching JSON Content

The native JSON document model imposes some new restrictions on partial updates to JSON documents. Therefore, some patch operations that were previously supported will now be rejected or produce different results. For details, see Limitations of JSON Path Expressions in the REST Application Developer's Guide and Traversing JSON Documents Using XPath in the Application Developer's Guide.

For example, you cannot construct patch path expressions that address anonymous nodes. In the JSON document model, object nodes and array nodes are anonymous. The name in a property name-value pair addresses the value(s), not the containing node.

This means you cannot use last-child position to insert a new property or value under the root node of a document or in an array because the parent node is anonymous and cannot be selected by the context expression of the insert operation. Similarly, you cannot address an array node that is nested inside another array ([1, [2, 3], 4]) because it is unnamed.

You can no longer replace an entire property (name-value pair) in a single patch replace operation for the same reason. To replace a property, you must delete it and then insert a new one.

Java and REST: Transforms and Extensions That Manipulate JSON Must Be Rewritten

This section applies to REST Client API and Java Client API applications that use content transformations, resource service extensions, and other server-side code to manipulate the XML representation of JSON documents.

Previously, content transformations, resource service extensions, and other server-side code manipulated JSON content as XML in the http://marklogic.com/xdmp/json/basic namespace. Now, such code must operate on JSON document nodes instead of XML.

For example, previously, the JSON data { "key": "value" } was represented in the database as XML of the following form, and this is what your server-side transforms and extensions worked with:

<json type="object" xmlns="http://marklogic.com/xdmp/json/basic">
  <key type="string">value</key>
</json>

Thus, to access the value of the key property in XQuery, you could use a path expression like this following:

$someDocument/json:json/json:key

With a native JSON document you reference the same data using the following path expression:

$someDocument/key

You should understand the native JSON document model before rewriting your code. For details, see Working With JSON in the Application Developer's Guide.

Java and REST: JSON Array Items and Property Values No Longer Distinguishable in QBE

This change may affect applications that use QBE to search JSON documents using the Java Client API or REST Client API.

Previously, you could construct a QBE that included a word or value query explicitly scoped to an array item or the property value. Now, it is not possible to distinguish between an array item and a property value.

For example, given a document with the following context:

{ "notArray": "blue", "array": ["azure", "blue"] }

Previously, the following QBE would only match the occurrence of the value "blue" in the "array" property. Now it matches both the occurrence in "array" and the occurrence in "notArray".

{"$query": [ {"$value": ["blue"]} ] }

This is because array nodes are unnamed in the native JSON document model, so they cannot be explicitly identified in a QBE.

Field Range Query and Field Value Query on JSON May Behave Differently

This difference only applies to applications using field range queries or field value queries on JSON documents.

JSON and XML are not indexed in exactly the same way. Some of the indexing differences affect the behavior of field range queries and field value queries over JSON. Since JSON documents were previously stored as XML, this means your field range queries and field value queries over JSON may behave differently.

For example, previously you could construct a field value query for John Smith that would match the following document by defining a field on the name property that excluded the middle property.

{ "name": {
    "first": "John",
    "middle": "NMI",
    "last": "Smith"
}

This was possible because the document was represented as XML and the text nodes in the field were concatenated together so that the field value in the above document was John Smith. In native JSON documents, this concatenation does not occur, and the values of the equivalent field are John and Smith. To get the same effect now, you would have to use a construct such as a near query.

For more details, see Creating Indexes and Lexicons Over JSON Documents and How Field Queries Differ Between JSON and XML in the Application Developer's Guide.

Semantics Incompatibilities

MarkLogic 8 introduces a number of new and changed Semantic features. This section describes those and includes the following changes that might cause incompatibilities:

Changed Function: sem:sparql

In MarkLogic 8, the signature of sem:sparql has changed for the last parameters. The fourth parameter is now a sem:store*, where previously there where two parameters at the end, one to specify a cts:query and another to specify the forest ID. The old signature is still available, but is deprecated. If you have any code from MarkLogic 7 that uses the fourth or fifth arguments to sem:sparql, it will still work in MarkLogic 8, but you should migrate that code to use the new signature using sem:store as the fourth parameter.

Changed Function: sem:sparql-values

In MarkLogic 8, sem:sparql-values now serializes a string as a cts:word-query when used as part of an argument. In other words, string values will be passed as cts:query arguments. This is a change from MarkLogic 7.

Changed Function: sem:sparql-values

The sem:sparql-values function no longer accepts forest-id as an option. This is an incompatibility with MarkLogic 7 functionality.

Deprecated Function: sem:sparql-triples

The sem:sparql-triples function has been deprecated in favor of sem:in-memory-store in MarkLogic 8. See documentation for details - Querying Triples in Memory in the Semantic Graph Developer's Guide.

Changed Behavior: Graphs

In MarkLogic 8, graph documents containing metadata are created when triples are ingested, whether they are ingested using SPARQL Update, mlcp, or SPARQL endpoints over REST. These named graphs inherit the permissions of the user, unless specified as part of the ingest process. The graph permissions are stored along with other metadata in the graph document.

When loading triples with mlcp, if the output-permissions parameter is set when loading RDF, the graph will inherit the default permissions just as it would in a sem:sparql-update operation. If the output_collection parameter is set when loading RDF, the graph document is only created for the first collection specified (because a managed triple can only belong to one graph).

In an upgraded system, graph metadata for managed triples created in MarkLogic 7 will be created the first time you add triples to the graph or modify triples in the graph. The graph will either have the user's permissions or the permissions specified as part of the operation. Make sure the document permissions of documents containing managed triples created in MarkLogic 7 are passed into sem:sparql-update as default-permissions to ensure the graph metadata created is consistent with the permissions for existing managed triples created in MarkLogic 7.

REST and Java Client API Incompatibilities

This section covers the incompatibilities for the REST API between MarkLogic 7 and MarkLogic 8. that are not related to JSON. If you work with JSON documents using the REST Client API or Java Client API, you should also see JSON Related Incompatibilities.

This section covers the following incompatibilities:

Must Upgrade to Java Client API v3.0

You cannot use earlier versions of the Java Client API with MarkLogic 8. Update your application to use version 3.0 or later.

REST API Instance Must Use the Declarative Rewriter on the App Server

In MarkLogic 7, App Servers that are REST API Instances used a different URL rewriter than in MarkLogic 8. In MarkLogic 8, the App Server is configured to use the declarative rewriter. If you had not modified anything in your REST API Instance App Server setup, the upgrade to MarkLogic 8 will reconfigure your App Server to use the new rewriter. If, however, you have modified something in setup to use a different rewriter, then you will have to make similar changes to the new setup (or consider not using those changes in the REST API Instance). For details on the declarative rewriter, see Creating a Declarative XML Rewriter to Support REST Web Services in the Application Developer's Guide.

Default Transaction Mode for the POST Method of Resource Service Extensions is Now Query

In MarkLogic 7, the POST method of a resource service extension was always executed in update mode. In MarkLogic 8, POST methods in a single-statement transaction are executed in query mode. However, within a multi-statement transaction, they are in update mode. The new way is safer because, generally speaking, if a function is not doing an update, it is much more efficient for it to run as a query.

If you have a resource service extension that requires the transaction mode to be update, you need to modify the extension code to add an annotation to the function to explicitly force it into update mode. For example, to modify an extension that is a GET, add an annotation like the following to your get function:

declare %rapi:transaction-mode("update") function testrstxn:get(
   $context, $params) { 
<Your code goes here>
}; 

The annotation %rapi:transaction-mode("update") forces the function to run as an update. For details, see Controlling Transaction Mode in the REST Application Developer's Guide.

REST API: Empty Bulk Read by Query Now Returns 200 Status

Previously, performing a bulk read by retrieving all documents that match a query would return a 404 Not Found response status if no documents matched the query. As of MarkLogic 8, such a request returns a 200 OK status with an empty response body.

This change applies to the following methods, when the Accept header is set to multi-part/mixed and the request does not ask for a search result summary in addition to the matching documents.

For more details on these interfaces, see Reading Multiple Documents Matching a Query in the REST Application Developer's Guide.

Error Reporting Format and Detail Changes

The changes in this section might affect your application if either of the following is true:

MarkLogic 8 introduces the following incompatible changes to error reporting for users of the

Error Format Defaults to JSON and is a REST Instance Creation Property

In previous versions, the default format for error messages returned by the REST Client API was XML, and you could change it by setting the error-format instance configuration property. As of MarkLogic 8, the default error format for new REST instances is JSON. You can now specify the format when you create the REST instance, and you can subsequently change it using the Admin Interface, admin:appserver-set-default-error-format, or the REST Management API.

This change has the following implications:

To set the error format when creating an instance, set the error-format configuration property. For details, see Creating an Instance in the REST Application Developer's Guide.

Error Detail Element and Property Names Have Changed

The error detail returned by the REST Client API has changed in the following ways. For examples of the new format, see Error Reporting in the REST Application Developer's Guide.

Use RESTAPI-SRVEXERR to Report Errors from Transforms and Extensions

Resource service extensions and transformations previously reported errors to the client using RESTAPI-EXTNERR and could specify a response payload in JSON or XML. The expected error response content type was controlled by a caller supplied parameter. This parameter is now ignored, and you should use RESTAPI-SRVEXERR instead of RESTAPI-EXTNERR. Your payload must be compatible with the MIME type expected by the caller, which can vary, so it is best to restrict the error detail to text. For details, see Reporting Errors in the REST Application Developer's Guide.

Deprecated Interface: Keyvalue Queries

This topic applies to applications that use the Java Client API class KeyValueQueryDefinition or the REST Client API method GET:/v1/keyvalue. These interfaces are now deprecated.

You can use Query By Example (QBE) or structured query to perform the same kind of search.

For example, to search for a JSON property named author with the value Mark Twain, use a QBE such as the following:

{ "$query": { "author": "Mark Twain" } }

The following is a similar search for an XML element:

<q:qbe xmlns:q="http://marklogic.com/appservices/querybyexample">
  <q:query>
    <author>Mark Twain</author>
  </q:query>
</q:qbe>

With structured query, use value-query or container-query.

For details, see the following references:

Transaction ID Format Has Changed

Previously the transaction ids created using DatabaseClient.openTransaction (Java) or POST:/v1/transactions (REST) were of the form hostId_transactionId. The hostId segment has now been dropped.

As long as your application treats the transaction id as a black box, this change is transparent.

A Transaction Can No Longer Be Shared Across Users

This change only affects Java Client API and REST Client API applications that use multi-statement transactions and share the resulting transaction id across multiple users.

Previously, it was possible to create a multi-statement transaction as one MarkLogic user and then perform operations within the transaction as another user. This is no longer possible. Now, all operations within a transaction must be performed as the same user who created the transaction.

Java: QBE Search Results No Longer Automatically Match the Query Format

Previously, using QueryManager.search with a Query By Example (QBE) automatically returned results in the same format as the query. That is, XML results were returned for an XML QBE, and JSON results were returned for a JSON QBE. Now, you must explicitly request JSON.

For example, if the qbe variable in the following statement contains a JSON QBE, then previously you would receive JSON results. Now, you will receive XML by default, instead.

queryMgr.search(qbe.newStringHandle()).get();

To achieve the same result as before, explicitly set the format on the result handle to JSON. For example:

queryMgr.search(qbe.newStringHandle().withFormat(Format.JSON)).get();

Document Library Services (DLS) Repositories Need To Perform A Bulk Upgrade Operation

MarkLogic 8 includes an enhancement to Document Library Services to make it significantly faster for large DLS repositories. This enhancement requires some metadata changes to the documents under DLS control.

If you have any DLS repositories created in MarkLogic 7 or earlier, you must first set compatibility mode for your repository, and then upgrade the documents in the repository, and finally set the repository to upgraded. The upgrade process will touch all of the documents under DLS control, so it will take a while, depending on the size of your DLS application. If you do not perform at least the first part of this upgrade, DLS functions might produce incorrect results in MarkLogic 8.

Because this is an upgrade that touches a large number of documents, MarkLogic strongly recommends that you first back up your database and that you thoroughly test your process on a development system before upgrading your production DLS repository.

To upgrade existing DLS repositories, perform the following steps:

  1. Back up your database containing the documents under DLS control.
  2. As either a user with the admin role, set compatibility mode for your DLS repository by running the following XQuery against your DLS database (for example, in Query Console):
    xquery version "1.0-ml";
    
    import module namespace dls = "http://marklogic.com/xdmp/dls" 
      at "/MarkLogic/dls.xqy";
    
    dls:set-upgrade-status(fn:false())
  3. As a user with the admin role, run the following XQuery against your DLS database (for example, in Query Console):
    xquery version "1.0-ml";
    (: This starts a task which will run for a time proportional to 
       the number of documents you have under DLS control. The
       function returns immediately though. It is safe to rerun
       this function if it is stopped or fails for any reason 
       such as a system restart. :)
    
    import module namespace dls = "http://marklogic.com/xdmp/dls" 
      at "/MarkLogic/dls.xqy";
    
    dls:start-upgrade()

    If you stop the upgrade (for example, if the server is restarted or if there are errors in the upgrade that you have fixed), you can restart the upgrade at any time by running the above query.

  4. You can check the progress at any time by looking at the upgrade-task-status.xml document, ad in the following XQuery:
    xquery version "1.0-ml";
    (: 
      this document is updated every few minutes to show the 
      progress of the upgrade
    :)
    fn:doc("http://marklogic.com/dls/upgrade-task-status.xml")
  5. You can check the progress at any time by running the following XQuery:
    xquery version "1.0-ml";
    
    import module namespace dls = "http://marklogic.com/xdmp/dls" 
      at "/MarkLogic/dls.xqy";
    
    dls:latest-validation-results()
  6. The dls:latest-validation-results output has an element names dls:validation-status. When the value of that element is completed, the process is complete.
  7. When you are satisfied that the process has completed (for example, if the previous step shows the process is complete), the set the upgrade status to true by running the following, either as a user with the admin role or a user with the dls-admin role:
    xquery version "1.0-ml";
    
    import module namespace dls = "http://marklogic.com/xdmp/dls" 
      at "/MarkLogic/dls.xqy";
    
    dls:set-upgrade-status(fn:true())

Once the process is successful, you can use DLS as usual.

It is also possible to run DLS in compatibility mode without running the upgrade on the repository (by not running the upgrade portion of the above procedure), but MarkLogic strongly recommends performing this upgrade procedure. If you have any problems or questions and you have an active maintenance contract, you can contact MarkLogic Technical Support.

Linux Now Requires Red Hat 6

MarkLogic 8 Linux platforms now require Red Hat 6, and will no longer work on Red Hat 5; they will fail to install on Red Hat 5. So if you are running MarkLogic 7 on Red Hat 5, you will have to migrate that environment to Red Hat 6 in order to use MarkLogic 8.

mlsql On Linux No Longer Ships With Server

On Linux platforms, the mlsql utility no longer is packaged with the MarkLogic Server rpm binary. To get mlsql on Linux now, you must install the ODBC Driver for Linux. For details on the ODBC Driver for Linux, see Configuring the ODBC Driver on Linux in the SQL Data Modeling Guide.

Cyrillic Tokenization Changes

The tokenization rules for Cyrillic script has changed such that mixed letter and number tokens are handled consistently with Latin letter and number tokens. That is, the following now tokenizes as a single token (previously it was two tokens):

&#x401;1 (Cyrillic A + the digit 1) 

If you have Cyrillic content in your database, you should reindex the database or reload the Cyrillic content so that it is properly tokenized and indexed in accordance with the new rules. If you do not trigger a retokenization of any existing Cyrillic content, certain queries may behave inconsistently between old content and newly loaded content.

Application Builder Applications Must Be Re-Deployed in MarkLogic 8

Applications deployed with Application Builder in MarkLogic 7 and earlier will not work correctly in MarkLogic 8 until you fully re-deploy them.

If you have any applications built using Application Builder, you must remove the code from the modules database of the deployed application and then re-deploy the application. Specifically, for each Application Builder application that is upgraded from MarkLogic 7 or earlier:

  1. Back up the application in case you need to restore it to its previous state.
  2. In the modules database for the application, either clear the database or delete the following directories:

    /application

    /Default

    /marklogic.rest.resource

    /marklogic.rest.transform

    As well as the document at the following URI

    /index.html

  3. Go into Application Builder and re-deploy the application.
  4. If you have extended your application with any customizations, re-deploy those customizations.
  5. Test your re-deployed application.

Application Builder and Information Studio Links Removed

The links in the navigation bar from Query Console and other tools no longer have links to Application Builder and Information Studio. To use these applications, enter the URL directly (for example, http://localhost:8000/appservices).

Search API Incompatibilities

The following incompatible changes have been made to the Search API:

search:parse Output is Now Unannotated cts:query XML

Previously, search:parse produced XML representing an annotated cts:query that could be passed directly to search:unparse. As of MarkLogic 11, the default output from search:parse does not include annotations, so it cannot be passed to search:unparse.

You can get annotated output from search:parse in MarkLogic 11 using the $output parameter as shown below:

search:parse("myQueryText", options, "cts:annotated-query")
Deprecated Option: extract-metadata

The extract-metadata query option is now deprecated. Use extract-document-data instead. For details, see search:search or Extracting a Portion of Matching Documents in the Search Developer's Guide.

The new extract-document-data option does not support extracting specific metadata properties, but properties are available in other ways, such as using xdmp:document-properties or by fetching all properties metadata through one of the client APIs.

Deprecated Functions: search:unparse, search:remove-constraint

The search:unparse and search:remove-constraint functions are now deprecated. If you need to deconstruct a query, modify it, and put it back together, use structured query or cts:query with search:resolve instead.

For example, if you previously did something similar to the following:

let $ctsquery := search:parse("myQueryString", $options)
(: ...modify $ctsquery... :)
return search:search(search:unparse($ctsquery), $options)

Then you can achieve the same result doing the following:

let $ctsquery := search:parse("myQueryString", $options)
(: ...modify $ctsquery... :)
return search:resolve($ctsquery, $options)

To generate a structured query instead of a cts:query, set the third parameter of search:parse to "search:query":

search:parse("myQueryString", $options, "search:query")
Structured Query: locks-query and properties-query Renamed

The following structured query elements have been renamed to more accurately reflect their purpose:

For details, see locks-fragment-query and properties-fragment-query in the Search Developer's Guide.

sort-order Query Option Requires an Index

If you use the sort-order query option to sort search results by something other than score, such as an XML element, JSON property, or field, then the database configuration must include a range index on the entity used for a sort key. Failing to create such an index causes SEARCH-BADORDERBY to be thrown when searching. Previously, the index requirement was not enforced.

For details, see sort-order in the Search Developer's Guide.

Locks and Properties Query Built-In Functions Renamed

The following built-in functions related to locks and document properties queries have been renamed to more accurately reflect their purpose.

xdmp:uri-content-type Of an XML Document Now Returns application/xml, Can Affect CPF Applications

In 8.0, the xdmp:uri-content-type function returns application/xml. In 7.0, it returns text/xml. Where MarkLogic previously returned text/xml for an XML document, it now returns application/xml. If you have applications that are expecting text/xml, you will either need to change the application to accept application/xml or you will have to modify your program to send a content type of text/xml (for example, using xdmp:set-response-content-type). MarkLogic will still accept text/xml for XML documents (in addition to application/xml). Similarly, JSON documents return application/json but MarkLogic accepts either application/json or text/json.

As a consequence of this change, if you have a CPF application that relies on the mimetype text/xml to identify an XML document, you must change that application to instead rely on application/xml. In the case of a CPF application, you will have to modify your CPF pipelines and change any text/xml mimetypes (that were referring to XML documents) to application/xml. If you are using the default pipelines, then reinstalling CPF for the database should correct this incompatibility. Otherwise, those CPF applications will not trigger the change actions for XML documents. Similar changes are required for CPF applications that rely on text/json to identify JSON documents; they need the Mimetype changed to application/json.

xdmp:function Signature Change

To allow for anonymous functions in Server-Side JavaScript, the first argument of the xdmp:function built-in function takes an xs:QName? (zero or 1 QNames) in 8.0; previously, it took an xs:QName (exactly one QName). If you have code that does not allow for zero or 1 QNames, you will need to modify that code.

Also, if you are relying on function mapping with xdmp:function, you can no longer use function mapping with it (because it no longer takes a singleton). If you have code that function maps with xdmp:function, you will need to rewrite it to not use function mapping (by calling xdmp:function in the return of a FLOWR statement for each item in your sequence of QNames, for example).

Incompatibilities Between 8.0-5 and 8.0-6

The following incompatibilities exist between MarkLogic 8.0-5 and MarkLogic 8.0-6:

Terms Matched by additional-query Are Highlighted in Snippets

Previously, the Search API documentation stated that terms matched by the query specified in an additional-query query option were not highlighted in search result snippets. This is no longer the case.

You should expect any terms matched by the additional-query option to be included in highlighted sections of snippets.

Incompatibilities Between 8.0-3 and 8.0-4

There are a few incompatibilities made to the Server-Side JavaScript implementation in 8.0-3. The following are the incompatibilities:

xdmp.multipartDecode Now Returns a JSON Payload for Headers

In 8.0-4, the Server-Side JavaScript xdmp.multipartDecode function returns the headers (in the first item of the returned ValueIterator) as a JSON array; previously, it was returned as an XML element. If you have code that is expecting the XML element, you need to either rewrite your code to parse the JSON array or use the XQuery version (xdmp:multipart-decode).

In JavaScript, Some Thesaurus and Spelling Function Have Different Return Type

In 8.0-4, there are changes to the thesaurus and spelling function modules to make them more friendly to JavaScript users. If you have JavaScript code that imports these XQuery libraries, then the following functions now return JavaScript Object by default:

For each of these function, you can set a new optional parameter to change its output. By default, these functions return XML output in XQuery and JavaScript Objects in JavaScript. If you have existing JavaScript code that uses these functions, you either need to add the new option to your code specifying the XML output or rework your code to accept the returned JavaScript Object. For details on these functions, see the API documentation for each function. In XQuery, the functions behave the same way they did in previous versions by default, but now allow you to specify the optional parameter to output JavaScript Objects instead of XML, if you so choose.

xdmp.databaseRestoreStatus Now Returns an Object

In 8.0-4, the Server-Side JavaScript API xdmp.databaseRestoreStatus now returns an Object; previously, it returned an Array. The Array that was previously returned is now the value of the "forest" key, and there is also a "status" key containing the current status of the restore. The xdmp.databaseBackupStatus also contains this new "status" key, but the return type is not changed. Similarly, the XQuery counterparts to these APIs (xdmp:database-backup-status and xdmp:database-restore-status) also contain information about the status, but their signature has not changed. If you have code that relies on any of the old behavior, you will need to modify that code to work with the changed output.

Serialization Error Code Changes

In 8.0-4, the names of some error exception codes for serialization, as well as the message text, have changed as shown in the following table:

Old Error Code New Error Code
SER-DOCTYPESYSTEM2
SER-DOCTYPESYSTOPLEVTXT
SER-DOCTYPESYSTEM3
SER-DOCTYPESYSMULTIELEMROOT
SER-STANDALONE2
SER-STANDALONETOPLEVTXT
SER-STANDALONE3
SER-STANDALONEMULTIELEMROOT
SER-STANDALONE4
SER-STANDALONEOMITXMLDEC

If you have code that does a try/catch looking for one of the old exceptions or the old text, or if you have tests that use the old exceptions as keys, you will have to rewrite that code to look for the new exception.

Change to Required Java Version

The following tools and libraries that depend on a Java Runtime Environment (JRE) now require at least Java 8, rather than Java 7:

Deprecated mlcp Command Line Options

The -aggregate_uri_id and -delimited_uri_id command line options are now deprecated. Use the more general -uri_id instead.

REST APIs That Have JSON or XML Payloads Cannot Have Empty Payloads

Starting in 8.0-4, any of the REST APIs that specifies a JSON or and XML content-type for its payload cannot have an empty payload. Previously, it allowed an empty payload. REST calls that specify a JSON or XML content-type with an empty payload throw a MANAGE_EMPTYPAYLOAD exception beginning in 8.0-4. For example, POST /admin/v1/init previously allowed an empty payload with a JSON or XML content, but requires the payload in 8.0-4.

If you have code that does not send a payload, then you must either add an empty JSON or XML document as the payload or change the content-type to one that allows an empty payload (for example, text/plain).

Geospatial Namespace and Data Version Changes

The following changes have been made to some of the geospatial built-in and library functions in 8.0-4:

GML and KML Library Modules Moved to a New Namespace

The GML library module is now in a different namespace, and the library module no longer uses the same namespace as GML and KML data.

Module Old Module Namespace New Module Namespace
GML http://www.opengis.net/gml http://marklogic.com/geospatial/gml
KML http://earth.google.com/kml/2.0 http://marklogic.com/geospatial/kml

You must update your module import declarations in XQuery or require statements JavaScript to use the new namespace. In addition, since the module and the XML data of the same format no longer use the same namespace, you may need to change the namespace prefix you use for the module.

The following code snippets demonstrate the required changes for KML and GML in XQuery. The namespace URI in the module import declaration is changed to use the new URI, and the module namespace prefix is changed to distinguish names in the module from names in the data.

Old New
xquery version "1.0-ml";
import module namespace kml =
  "http://earth.google.com/kml/2.0" at
    "/MarkLogic/geospatial/kml.xqy";

kml:box(
  <kml:LatLongBox>...</kml:LatLongBox>)
xquery version "1.0-ml";
import module namespace geokml =
  "http://marklogic.com/geospatial/kml"
  at "/MarkLogic/geospatial/kml.xqy";
declare namespace kml =
  "http://www.opengis.net/kml/2.2";

geokml:box(
  <kml:LatLongBox>...</kml:LatLongBox>)
xquery version "1.0-ml";
import module namespace gml =
  "http://www.opengis.net/gml"
    at "/MarkLogic/geospatial/gml.xqy";

gml:box(
  <gml:Envelope>...</gml:Envelope>)
xquery version "1.0-ml";
import module namespace geogml =
  "http://marklogic.com/geospatial/gml"
    at "/MarkLogic/geospatial/gml.xqy";
declare namespace gml =
    http://www.opengis.net/gml/3.2";

geogml:box(
  <gml:Envelope>...</gml:Envelope>)
Some Built-In Geospatial Functions Moved to geo Namespace

The following geospatial built-in functions are now deprecated and will be removed in a future release. You should use the corresponding built-in function with a geo prefix instead. For example, use geo:distance instead of cts:distance in XQuery, and use geo.distance instead of cts.distance in Server-Side JavaScript.

XQuery Server-Side JavaScript
cts:approx-center cts.approxCenter
cts:arc-intersection cts.arcIntersection
cts:bearing cts.bearing
cts:box-intersects cts.boxIntersects
cts:circle-intersects cts.circleIntersects
cts:complex-polygon-contains cts.complexPolygonContains
cts:complex-polygon-intersects cts.complexPolygonIntersects
cts:destination cts.destination
cts:distance cts.distance
cts:parse-wkt cts.parseWkt
cts:polygon-contains cts.polygonContains
cts:polygon-intersects cts.polygonIntersects
cts:region-contains cts.regionContains
cts:region-intersects cts.regionIntersects
cts:shortest-distance cts.shortestDistance
cts:to-wkt cts.toWkt
Older GML and KML Versions Deprecated

As of MarkLogic 8.0-4, the default KML data version is 2.2 and the default GML data version is 3.2. Older versions are deprecated. You should convert your data.

To continue querying older versions of data with geokml:geospatial-query or geogml:geospatial-query, specify the namespace URI of the older version in your query constructor. For example:

GML
Old: gml:geospatial-query($regions, $options, $weight)
New: geogml:geospatial-query(
       $regions, $options, $weight, "http://www.opengis.net/gml")

KML
Old: kml:geospatial-query($regions, $options, $weight)
New: geokml:geospatial-query(
       $regions, $options, $weight, "http://earth.google.com/kml/2.0")

Incompatibilities Between 8.0-2 and 8.0-3

There are a few incompatibilities made to the Server-Side JavaScript implementation in 8.0-3. The following are the incompatibilities:

spell.suggestDetailed, xdmp.filesystemDirectory, and xdmp.encodingLanguageDetect Now Return ValueIterator

In 8.0-3, the spell.suggestDetailed, xdmp.filesystemDirectory, and xdmp.encodingLanguageDetect functions now return results in a ValueIterator. In 8.0-2 and earlier, they returned results in an Array. If you have any code that expects an Array, you must rework that code to accept a ValueIterator; for example, you can use xdmp.arrayValues to convert the ValueIterator result into an Array result. For details on the ValueIterator JavaScript type, see ValueIterator in the JavaScript Reference Guide.

xdmp.databaseRestoreStatus Now Returns an Array

In 8.0-3, the xdmp.databaseRestoreStatus function now returns a JavaScript Array. Previously, it returned an ArrayNode. If you have any code that expects the ArrayNode, you must rework that code to accept an Array.

xdmp.gssServerNegotiate Now Returns a JavaScript Object

In 8.0-3, the xdmp.gssServerNegotiate function (used for kerberos GSS authentication) returns a JavaScript object. Previously, it returned an XML element.

Use of String Transaction Ids in Node.js To Be Deprecated

The Node.js interfaces that open, manipulate, or pass around transaction ids now accept either a simple id (as before) or a transaction object. You obtain a transaction object rather than an id by passing true in for the new withState parameter of DatabaseClient.transactions.open. For example:

// old forms: return a transaction id, for backward compatibility
db.transactions.open();
db.transactions.open({timeLimit: limit, transactionName: name});

// new forms: return transaction object; preferred.
db.transactions.open(true);
db.transactions.open({withState: true, ...});

In a future release, the default of withState will be changed to true and the use of transaction ids will be deprecated.

If you do not modify your code to use transaction objects rather than ids, the session affinity required by multi-statement transactions is not guaranteed to be properly preserved in all cases.

For details, see Managing Transactions in the Node.js Application Developer's Guide.

CDH 4.3 is No Longer a Supported Hadoop Distribution

The MarkLogic features and tools that rely on Hadoop no longer support CDH 4.3. Instead, use one of the newer supported versions of Hadoop.

This change affects MarkLogic Content Pump (mlcp), the MarkLogic Connector for Hadoop, and forest storage on HDFS.

Changes to How MarkLogic Locates Java and Hadoop Libraries for HDFS Forest Storage

When you use HDFS for forest storage, MarkLogic must be able to find a suitable Java installation and Hadoop HDFS libraries. The algorithm for where MarkLogic looks has changed substantially as of 8.0-3.

You should now make the Hadoop libraries available to your MarkLogic hosts using one of the new Hadoop HDFS client bundles. You must unpack one of these bundles under /opt, /usr, or /space. For details, see HDFS Storage in the Query Performance and Tuning Guide.

MarkLogic now locates a suitable Java installation using a different algorithm. You might need to either change the path to your JDK or set JAVA_HOME in the MarkLogic startup environment. You can now make JAVA_HOME available to MarkLogic in a way that is preserved across upgrades, using /etc/marklogic.conf. For details, see HDFS Storage in the Query Performance and Tuning Guide.

Incompatibilities Between 8.0-1 and 8.0-2

There are a few incompatible changes made to the Server-Side JavaScript implementation in 8.0-2. The following are the incompatibilities:

Array Input Differences in fn.distinctValues, fn.subsequence, and Other Functions

In 8.0-2, the Server-Side JavaScript functions fn.distinctValues and fn.subsequence behave differently from 8.0-1 if you pass in an array. In 8.0-1, these functions will extract the values out of the array to make multiple inputs, one for each item in the array. In 8.0-2, these functions treat the array as a single item. Therefore, to get the same behavior in 8.0-2, you need to call xdmp.arrayValues on the array before you pass it into these functions. For example:

fn.distinctValues([1, 1, 2]);
// returns 1, 2 in 8.0-1
// returns [1, 1, 2] in 8.0-2

To modify the above code in 8.0-2 to return the same answer as in 8.0-1:

fn.distinctValues(xdmp.arrayValues([1, 1, 2]));
// returns 1, 2 in 8.0-2

The nature of the change is that there are fewer times when MarkLogic coerces an array into its values in 8.0-2 than there were in 8.0-1. The newer behavior is more natural to JavaScript developers.

In addition to fn.distinctValues and fn.subsequence, this applies to any JavaScript function that takes a ValueIterator as input to a parameter, including: fn.exactlyOne, fn.head, fn.insertBefore, fn.reverse, fn.unordered, fn.empty, fn.remove, fn.reverse, fn.zeroOrOne, fn.oneOrMore, fn.deepEqual, fn.count, cts.contains, xdmp.setSessionField, xdmp.setServerField, and others.

The Second Parameters of xdmp.eval, xdmp.invoke, xdmp.xqueryEval, and xdmp.spawn Now Take a Single Object

In 8.0-2, the vars parameter to the Server-Side JavaScript functions xdmp.eval, xdmp.invoke, xdmp.xqueryEval, and xdmp.spawn has been simplified so it only takes a single Object. If you have any code that you used in 8.0-1 that passes the external variables (vars parameter) as an array of Objects, as an array of stings, or as a ValueIterator, you must rewrite that code in 8.0-2 so it passes a single Object.

extract-document-data Results Now Inline By Default

The query option extract-document-data previously caused search:search and search:resolve to return a sequence consisting of the search:response and the extracted documents. Now, this option returns the extracted documents embedded in the search:response as search:extracted elements.

When you use the option with the REST Client API, the /v1/search service returns the extracted content inside the search response if the Accept header MIME type is application/xml or application/json. The extracted content is still returned as individual documents when the Accept header MIME type is multipart/mixed (a multi-document read).

For details, see Extracting a Portion of Matching Documents in the Search Developer's Guide.

XCC v8.0-2 May Require Config Change When Used with Older Versions of MarkLogic

This change affects XCC applications that set the transaction mode (Session.setTransactionMode) or transaction time limit (Session.setTransactionTimeout) and use XCC v8.0-2 or later with MarkLogic Server 8.0-1 or earlier.

If your XCC application meets the above conditions, you must set the property xcc.txn.compatible to true. If you do not do so, an exception is raised if you set the transaction time limit or set the transaction mode to a value other than Session.TransactionMode.AUTO.

You can set this system property on the java command line with an argument of the following form:

java -Dxcc.txn.compatible=true

You can also set the property programmatically by calling System.setProperty.

Client APIs: JavaScript Extension and Transform Error Reporting Convention Change

The following change affects applications that use the REST Client API, Java Client API, or Node.js Client API and that raise RESTAPI-SRVEXERR from a server-side JavaScript extension, transform, or custom snippeter, or other custom code.

In MarkLogic Server 8.0-1, when calling fn.error to raise RESTAPI-SRVEXERR,error detail such as the response status code and status text are passed to fn.error as an array. Starting with MarkLogic Server 8.0-2, the error details must be passed as a sequence. You can use xdmp.arrayValues to convert the array to a sequence.

For example, if you previously had an fn.error call in an extension similar to the following:

fn.error(null, 'RESTAPI-SRVEXERR', 
         [statusCode, statusMsg, body])

Then you should now wrap the array that is the 3rd argument to fn.error in a call to xdmp.arrayValues, similar to the following:

fn.error(null, 'RESTAPI-SRVEXERR', 
         xdmp.arrayValues([statusCode, statusMsg, body]))
Some JavaScript Built-In Functions that Returned XML Structures Now Return JSON Structures

The following APIs return JSON output in 8.0-2.

In 8.0-1, these APIs returned the same XML structures that their XQuery counterparts return. If you have any JavaScript code that relies on the XML structures, you will need to modify that code to use the new JSON output.

MarkLogic 7 Incompatibilities

MarkLogic 11 allows you to upgrade either from MarkLogic 5, MarkLogic 6, or MarkLogic 7. If you are upgrading from 4.2, you must first upgrade to at least MarkLogic 5, and there are some known incompatibilities between 4.2 and 5.0 that are documented in the 5.0 Release Notes. If you are upgrading from MarkLogic 7, you can skip this section. For convenience, the incompatibilities between MarkLogic 6 and MarkLogic 7 are repeated here, and are as follows:

Incompatibilities Between MarkLogic 7.0-3 and 7.0-2

This section describes the incompatibilities between MarkLogic 7.0-3 and 7.0-2.

HDP No Longer a Supported Hadoop Platform

Hortonworks Data Platform (HDP) is no longer a supported Hadoop distribution for use with the MarkLogic Connector for Hadoop or the distributed mode of MarkLogic Content Pump (mlcp).

Java API: ContentVersionRequest Property Deprecated

The REST server configuration property ContentVersionRequest is now deprecated. If you currently use com.marklogic.admin.ServerConfigurationManager.setContentVersionRequest() and the related ServerConfiguration.Policy type, you should modify your application to use com.marklogic.admin.ServerConfigurationManager.setUpdatePolicy() and ServerConfigurationManager.UpdatePolicy instead. You should also change calls to getContentVersionRequest() into call to getUpdatePolicy().

The table below shows the correspondence between Policy values and UpdatePolicy values. The behavior is unchanged with respect to these values.

If you used Policy Value... Then use UpdatePolicy Value...
NONE
MERGE_METADATA
REQUIRED
VERSION_REQUIRED
OPTIONAL
VERSION_OPTIONAL
REST API: content-versions Property Deprecated

The REST instance configuration property content-versions is now deprecated, in favor of the new update-policy property.

If you currently set content-versions, you should modify your application to use the update-policy configuration property instead. For example, if you set this property using PUT /v1/config/properties or PUT /v1/config/properties/content-versions, then you should now use PUT /v1/config/properties or PUT /v1/config/properties/update-policy to set update-policy instead.

Similarly, you should modify any code that reads the configuration properties to expect update-policy instead of content-versions.

The table below shows the correspondence between content-versions values and update-policy values. The behavior is unchanged with respect to these values.

If you used Policy Value... Then use UpdatePolicy Value...
none
merge-metadata
required
version-required
optional
version-optional
REST API: JSON documents Cannot be Retrieved as XML

In previous versions, you could use the REST Client API to retrieve the internal XML representation of a JSON document using GET /v1/documents and specifying XML in the format request parameter or application/xml in the Accept header.

As of MarkLogic 7.0-3, JSON documents are always returned as JSON. You can still examine the internal representation of a JSON document using XQuery or the Query Console database explorer.

Float Precision Greater in 7.0-3

In MarkLogic 7.0-3, the numeric precision of a float has increased. For example:

xs:float(3435.99884)
(: Returns 3436 in 7.0-2, 
   returns 3435.999 in 7.0-3 :)

In most cases, this will not cause an incompatibility, as it is just returning a more precise number. But if you have application logic that relies on the old behavior, you will have to make changes to account for the greater precision.

Incompatibilities Between MarkLogic 7.0-2 and 7.0-1

This section describes the incompatibilities between MarkLogic 7.0-1 and 7.0-2.

Change to JSON Output from the REST API

The JSON output from the manage REST resource addresses with the metrics view has been changed and you may need to change any custom clients that consume this JSON data.

Changes to the MarkLogic Connector for Hadoop API

com.marklogic.mapreduce.MarkLogicDocument is now an interface instead of a class. The previous functionality of MarkLogicDocument is provided by the new class com.marklogic.mapreduce.DatabaseDocument.

Modify your code and job configuration to use DatabaseDocument instead of MarkLogicDocument.

Incompatibilities Between MarkLogic 7.0-1 and MarkLogic 6

XQuery HTTP Client Built-In Functions Now Require a Privilege

The HTTP client XQuery APIs (xdmp:http-delete, xdmp:http-get, xdmp:http-post, xdmp:http-put, and so on) now require a privilege. Previously, these functions did not require a privilege. To support these privileges, the following privileges are added to MarkLogic 11:

Additionally, the network-access role has been added, which contains all of these privileges.

If you have code that accesses these functions with a user that does not have the admin role, you will have to add these privileges (or the network-access role) to the set of privileges inherited by your users. For details on adding privileges, see Security Administration in the Administrator's Guide. For details on security, see Security Guide.

HTTP Client Functions Are Now HTTP 1.1 Compliant

The XQuery HTTP client functions (xdmp:http-delete, xdmp:http-get, xdmp:http-post, xdmp:http-put, and so on) now fully implement HTTP 1.1, including the use of connection keep-alives and built-in decoding of chunked transfer encoded responses. In most cases, this will not cause any incompatible behavior, but if your code tried to perform some work to do HTTP 1.1 features such as taking a chunked response as a large binary and decoding the chunks in XQuery, then that code might behave differently in MarkLogic 7. If you have such code, you might need to rework it in MarkLogic 7.

xdmp:get-request-username and xdmp:get-request-user Changes

The xdmp:get-request-username and xdmp:get-request-user functions have changed slightly in MarkLogic 7.

In previous releases, xdmp:get-request-username always returned the user in the Authorization HTTP header; in MarkLogic 7, if you are using application-level authentication, then it returns the user from the last successful call to xdmp:login (when using any other authentication scheme, it is unchanged from previous releases and returns the user in the Authorization HTTP header).

In previous releases, xdmp:get-request-user returned the ID of the current user; in MarkLogic 7, if you are using application-level authentication, then it returns the user ID from the last successful call to xdmp:login (when using any other authentication scheme, it now returns the user ID corresponding to the user in the Authorization HTTP header). If you want the old behavior of returning the ID of the current user, use the new function xdmp:get-current-userid.

If you have any code that uses the xdmp:get-request-username or xdmp:get-request-user functions, you might need to change it to work with the current behavior.

Specifying a Forest Now Only Works With Strict Locking

The various loading APIs (for example, xdmp:document-load and xdmp:document-insert) have an option to specify the forest in which a document is loaded. In MarkLogic 7, those forest options are only available with the locking parameter on the database set to strict (the default is fast). If you try to specify forest placement with fast locking, it will throw an XDMP-PLACEKEYSLOCKING exception. Previously, this operation would be allowed.

In most cases, it is not recommended to specify a placekey, as MarkLogic does a good job of distributing data. If you want to continue to use the forest placekeys when loading or updating, you must change your locking parameter to strict or change your code to no longer use forest placekeys. You can also consider using Tiered Storage to control what data goes into which forests. For details on Tiered Storage, see Tiered Storage in the Administrator's Guide.

Custom Dictionaries for Japanese and Chinese Languages Need to be Re-saved

In MarkLogic 7, there is a change to the way custom dictionaries are stored for languages that use a unified cd (Japanese and both Simplified and Traditional Chinese). The custom dictionaries for those languages now have separate files for tokenization and for stemming. If you are using these languages and you have created a custom dictionary, you must re-save the custom dictionary in MarkLogic 7, which will save it in the new format.

For example, if you have a Japanese language custom dictionary, run the following XQuery program to re-save the custom dictionary:

import module namespace
   cdict="http://marklogic.com/xdmp/custom-dictionary"
   at "/MarkLogic/custom-dictionary.xqy";

cdict:dictionary-save("ja", cdict:dictionary-read("ja"))

This will re-save the dictionary to the new format.

Default Attributes on XML Copy Changes

In MarkLogic 7, the behavior of default attributes has changed when you are copying XML nodes that have an in-scope schema with default attributes. In MarkLogic 6, the default attributes were applied in the copied node. In MarkLogic 7, the default attributes are still applied in the data model of the copied node, but they will only be serialized if default-attributes=yes is set in the serialization options (for example, the xdmp:output option). This is true for copied nodes in both XQuery (for example, see below) and XSLT (for example, xsl:copy).

The following shows an example using the XHTML schema, which is always in-scope.

xquery version "1.0-ml";
declare option xdmp:output "default-attributes=no";
(: the above xdmp:output declaration is the default :)

<x>{xdmp:unquote('<html xmlns="http://www.w3.org/1999/xhtml">
    <body>
       <p>hello</p> 
    </body>
</html>
')}</x>
=>
In MarkLogic 7 returns: (no default attributes on html element)

<x>
  <html xmlns="http://www.w3.org/1999/xhtml">
    <body>
       <p>hello</p> 
    </body>
  </html>
</x>

In MarkLogic 6 returns: (added default attribute version 
                         on html element)
<x>
  <html version="-//W3C//DTD XHTML 1.1//EN"
        xmlns="http://www.w3.org/1999/xhtml">
    <body>
       <p>hello</p> 
    </body>
  </html>
</x>

If the xdmp:output option is set to default-attributes=yes, then the attributes will continue to be serialized in MarkLogic 7. Furthermore, if the copied node is used in another schema that has different default values for the default attributes, then the resulting default value for those attributes comes from the copied node, not the new schema. This change is not likely to impact very many applications, and the new behavior is what most people would expect, but if you have code that relies on the old behavior, you will need to explicitly set the xdmp:output option is set to default-attributes=yes.

Serialization of Alerting, Reverse, and Path Range Queries Change

The serialization of path range index queries has changed to use cts:path-expression rather than cts:path as the element name. This affects not only stored path range index queries, but also stored Alerting queries and reverse queries. All such queries must be transformed to the new serialization and reloaded.

The following XSLT stylesheet makes the required change:

<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
                xmlns:cts="http://marklogic.com/cts"
                version="2.0">
  <xsl:template match="cts:path">
    <xsl:element name="cts:path-expression">
      <xsl:copy-of select="namespace::*|@*|node()"/>
    </xsl:element>
  </xsl:template>
</xsl:stylesheet>

You can apply the stylesheet to affected documents using a query similar to the following:

xquery version "1.0-ml";
for $query in fn:collection("my-stored-queries")
return xdmp:node-replace($query, xdmp:xslt-eval($stylesheet, $query)
Java and REST Client API Incompatibilities

Unless otherwise noted, the following changes can affect applications using either the REST or Java Client API:

Content Transformations on JSON Documents Operate on XML

This topic applies to REST and Java Client API applications that include custom content transformations for JSON documents.

JSON documents are stored in MarkLogic Server as XML. In MarkLogic 6, content transformation functions were invoked on JSON documents after conversion from JSON to XML. In MarkLogic 7, content transformations are applied before this conversion.

If you created a content transformation for MarkLogic 6 that expects JSON input, you must modify your implementation to expect XML in the http://marklogic.com/xdmp/json/basic namespace instead of JSON text. For details on the XML representation, see Working With JSON in Application Developer's Guide.

Transformations invoked during ingestion must similarly be modified to produce XML that conforms to the internal representation for JSON documents. Transformations invoked during document retrieval can either produce conforming XML or JSON text as when generating JSON output.

For details on the new expectations, see Expected Input and Output in REST Application Developer's Guide.

Resource Service Extensions Moved to Modules Database on Upgrade

This topic applies to REST and Java Client API applications running on a REST API instance created with MarkLogic 6.0-1 or later that use resource service extensions.

REST API instances created with MarkLogic Server version 6.0-1 stored resource service extensions in the Extensions database. REST API instances created with MarkLogic Server version 6.0-2 or later install resource service extensions in the modules database associated with the instance.

When you upgrade from MarkLogic 6 to MarkLogic 7 or later, any resource service extensions stored in the Extensions database on behalf of a 6.0-1 REST API instance are automatically migrated from the Extensions database to the modules database for you.

If your resource extension has dependent libraries or other assets that you installed in the Extensions database, you should migrate them by re-installing them using the new /ext service. For details, see Managing Dependent Libraries and Other Assets in REST Application Developer's Guide.

JSON Key Name Change for System Properties

This topic applies to REST applications that reference document properties using the JSON representation.

When you retrieve document properties in JSON format, protected system properties such as last-modified are now enclosed in an object with the key $ml.prop. In MarkLogic 6, such properties were immediate children of the properties container. User-defined property naming is unchanged.

The example output below shows the change in how the last-modified property value is returned by a request of the form GET /v1/documents?uri=your-uri&category=properties&format=json:

MarkLogic 6 MarkLogic 7
{"properties":{
  "last-modified":"value"
}}
{"properties":{
  "$ml.prop":{
    "last-modified":"value"
  }
}}
Java Batch Example Package Name Change

The Java example application in the package com.marklogic.client.example.batch is now in the package com.marklogic.client.example.extension. If your application references any classes or interfaces from this package, you must change your package imports and recompile.

Namespace Change for Properties Persisted Using JSON

This topic applies to REST and Java Client API applications that insert or update document properties using JSON and either of the following are true:

In MarkLogic 6, inserting or updating user-defined document properties using JSON stored the properties as XML elements in the namespace http://marklogic.com/json. In MarkLogic 7, such properties are stored as elements in the namespace http://marklogic.com/xdmp/json/basic. If you insert or update a property using JSON using MarkLogic 7, the new namespace is used.

If you convert properties in the old namespace to the new namespace, then all pre-existing and future properties will use the new namespace. If you do not perform such a conversion, then you should adapt your indexes and queries to accommodate both namespaces because pre-existing properties will be in the MarkLogic 6 namespace while new or updated properties will be in the MarkLogic 7 namespace.

After upgrading to MarkLogic 7, use one of the following solutions to adapt your content and application to this change:

Modify Pre-Existing Properties

Use a query similar to the one in this section to change the namespace of all properties in the old namespace. The example query uses xdmp:spawn to perform the property update in batches, thereby avoiding overly large transactions.

The procedure below assumes you have an App Server, such as a REST API instance, associated with your content database. If you are not familiar with Query Console, see Query Console User Guide.

The following procedure walks you through installing a transform query in the modules root of an App Server attached to an affected database, and then running the query to modify a specific property.

  1. Save the following transform query to a file, such as update-props.xqy. You will changed the bold text in the next step.
    xquery version "1.0-ml";
    
    declare namespace prop   = "http://marklogic.com/xdmp/property";
    declare namespace old-ns = "http://marklogic.com/json";
    
    declare default function namespace
       "http://www.w3.org/2005/xpath-functions";
    declare option xdmp:mapping "false";
    
    declare function local:transform(
        $elem as element()
    ) as element()
    {
        let $elem-name :=
            if (exists($elem/self::old-ns:*))
            then QName("http://marklogic.com/xdmp/json/basic",
                       local-name($elem))
            else node-name($elem)
        return element {$elem-name} {
            $elem/@*,
            for $child in $elem/node()
            return
                typeswitch ($child)
                case element() return local:transform($child)
                default return $child
            }
    };
    
    let $max   := 100
    let $batch :=
        subsequence(
            cts:search(collection(),
                cts:properties-query(
                    cts:element-query(xs:QName("old-ns:my-prop"),
                                      cts:and-query(()))
                    )
                ),
            1,
            $max
            )/document-uri(.)
    return
        if (empty($batch)) then ()
        else (
            for $doc-uri in $batch
            let $current-props  := xdmp:document-properties($doc-uri) /
                prop:properties/(* except prop:last-modified)
            let $modified-props := $current-props/local:transform(.)
            return xdmp:document-set-properties($doc-uri, $modified-props),
            if (count($batch) lt $max) then ()
            else xdmp:spawn("/some/path/update-props.xqy")
            )
  2. Modify the saved query to match your environment by making the following changes:
    1. Change occurrences of my-prop to the name of an affected property in your content.
    2. Change the module path in the xdmp:spawn call to the path where you will install the query in your modules root.
  3. Install the saved file in the modules database or modules root of your App Server. Install the module under the path you chose in Step 2b.
    1. To install the modules database of a REST API instance using the REST API, see Managing Dependent Libraries and Other Assets in REST Application Developer's Guide.
    2. To install in the modules database of a REST API instance using Java API, see Managing Dependent Libraries and Other Assets in Java Application Developer's Guide.
    3. To install in the modules database using XQuery, run a query similar to the following in Query Console, after modifying the file system path and database URI. The file must be accessible to MarkLogic Server. Run the query with the modules database as the content source.
      xquery version "1.0-ml";
      xdmp:document-load(
        "/filesystem/path/update-props.xqy",
        <options xmlns="xdmp:document-load">
          <uri>/some/path/update-props.xqy</uri>
        </options>
        )
  4. Run the transform query using Query Console.
    1. Create a new query in Query Console with the following contents:
      xquery version "1.0-ml";
      xdmp:invoke("/my.domain/update-props.xqy")
    2. Modify the module URI in the xdmp:spawn call to the URI under which you installed the module in Step 3.
    3. In the Query Console Content Source dropdown, select the source that corresponds to your content database and the modules database in which you installed the transform query in Step 3.
    4. Click the Run button to perform the transformation.
  5. If your database is large, you might exceed the maximum number of spawned queries. If this happens, wait for the previous spawns to complete, and then run the query again.

You must also modify any range indexes, queries, or query options for this property that depend on the old namespace. Change occurrences of http://marklogic.com/json to http://marklogic.com/xdmp/json/basic.

For example, if in MarkLogic 6 you defined an element range index over the property my-property in the namespace http://marklogic.com/json, modify the your index configuration to use the namespace URI http://marklogic.com/xdmp/json/basic.

Similarly, if you have queries or constraint definitions using the old namespace, change them to use the new namespace. The table below contains an example of how to modify a properties constraint to use the new namespace:

Version Example
MarkLogic 6
<options xmlns="http://marklogic.com/appservices/search">
  <constraint name="my-prop">
    <range type="xs:string">
      <element name="my-property"
        ns="http://marklogic.com/json" />
      <fragment-scope>properties</fragment-scope>
    </range>
  </constraint>
</options>
MarkLogic 7
<options xmlns="http://marklogic.com/appservices/search">
  <constraint name="my-prop">
    <range type="xs:string">
      <element name="my-property"
        ns="http://marklogic.com/xdmp/json/basic" />
      <fragment-scope>properties</fragment-scope>
    </range>
  </constraint>
</options>
Modify Queries to Use Both Namespaces

You can modify your queries and query options to accommodate both namespaces. For example, if your query options already include a constraint on the old namespace, add a constraint for the new namespace, and then use an OR query to apply both constraints.

The following example uses a JSON structured or-query to demonstrate this technique. It assumes the existence of an element range index on the property in the new namespace.

{"search": {
  "options": {
    "constraint": [
      {
        "name": "old-prop",
        "range": {
          "type": "xs:string",
          "element": {
            "name": "my-property",
            "ns": "http://marklogic.com/json"
          },
          "fragment-scope": "properties"
        }
      },
      {
        "name": "new-prop",
        "range": {
          "type": "xs:string",
          "element": {
            "name": "my-property",
            "ns": "http://marklogic.com/xdmp/json/basic"
          },
          "fragment-scope": "properties"
        }
      }
    ]
  },
  "query" : {
    "or-query" : {
      "queries" : [
        {"range-constraint-query": {
          "constraint-name": "old-prop",
          "value": "the value"
        }},
        {"range-constraint-query": {
          "constraint-name": "new-prop",
          "value": "the value"
        }}
    ]}
  }
}}
Create a Field That Spans Both Namespaces

Use this procedure to replace an existing element range index over the property in the old namespace with a field range index that covers both namespaces, and then modify your code and queries to the use new index. To learn more about fields, see Fields Database Settings in Administrator's Guide.

  1. Use the Admin Interface to define namespace prefixes for http://marklogic.com/json and http://marklogic.com/xdmp/json/basic. For details, see Steps 1-8 of Defining Path Range Indexes in Administrator's Guide.
  2. Create a field that includes two paths, one for the property in each namespace. Use the namespace prefixes defined in Step 1. For example, add the paths old-ns:my-property and new-ns:my-property. For details, see Configuring a New Path or Root Field in Administrator's Guide.
  3. Create a field range index over the new field. For details, see Creating a Range Index on a Field in Administrator's Guide.
  4. Modify any extensions, transformations, queries or query options that rely on the old element range index to use the new field range index. The example below shows a combined query modified to use the new field range index instead of the old element range index.

Old New
{ "search": {
 "options": {
   "constraint": {
     "name": "my-prop",
     "range": {
       "type": "xs:string",
       "element": {
         "name": "my-property",
         "ns": "http://marklogic.com/json"
       },
       "fragment-scope": "properties"
  }}},
  "query" : {
    "range-constraint-query": {
      "constraint-name": "my-prop",
      "value": "the value"
  }}
}}
{"search": {
  "options": {
    "constraint": {
      "name": "my-prop",
      "range": {
        "type": "xs:string",
        "field": { 
          "name": "my-new-property" 
        },
        "fragment-scope": "properties"
  }}},
  "query" : {
    "range-constraint-query": {
      "constraint-name": "my-prop",
      "value": "the value"
  }}
}}
mlcp Incompatibilities

MarkLogic Content Pump (mlcp) version 1.1 includes the following changes that potentially affect compatibility for users of mlcp version 1.0-*.

Compressed Input Default URI Includes Input Filename

The following change might affect you if you use mlcp to load documents from compressed files (-input_compressed). Documents created with mlcp v1.0 used the following default URI template:

/path/inside/zip/filename

Starting with version 1.1, the default URI template is:

/compressed-file-path/path/inside/zip/filename 

This change can affect the -output_uri_replace patterns needed to create documents with the same URIs as those created with mlcp 1.0.

For details, see Default Document URI Construction in the mlcp User Guide.

Default Character Encoding Changed to UTF-8

The default content character encoding when importing and exporting documents with mlcp has changed to UTF-8. Previously, mlcp used the platform default encoding for the host which mlcp was running.

You can get the previous behavior by using the following option setting:

-content-encoding system
REST Management API Version Incremented to v2

The REST Management API version has been incremented to v2. The v1 services are no longer available. If you send a request using a v1 URL, MarkLogic Server responds with status code 410 (Gone) and a MANAGE-UNSUPPORTEDVERSION error.

Requests that use LATEST as the version when constructing requests will continue to work. However, you may need to make other changes due to the behavior changes between v1 and v2.

The incompatibilities between v1 and v2 are detailed in the remainder of this section

View Parameter Required Instead of Path Steps

In previous releases, there were two ways to access some views: by path step or using the view request parameter. In MarkLogic 7, the path step form of URL has been removed. You must now use the view parameter. For example, MarkLogic 6 supported the following two ways of requesting database status, where version is v1 or LATEST:

GET /manage/version/databases/{id|name}/status
GET /manage/version/databases/{id|name}?view=status

In MarkLogic 7, only the second form is supported, as shown in the example below, where version is v2 or LATEST.

GET /manage/version/databases/{id|name}?view=status

This change applies to the config, counts, edit, and status views for clusters, databases, forests, groups, hosts, and servers. The table below lists the affected GET methods and the equivalent MarkLogic 7 URL.

Previous Form MarkLogic 7 Form
/manage/v1/clusters/{id|name}/config
/manage/v2/clusters/{id|name}?view=config
/manage/v1/clusters/{id|name}/status
/manage/v2/clusters/{id|name}?view=status
/manage/v1/databases/{id|name}/config
/manage/v2/databases/{id|name}?view=config
/manage/v1/databases/{id|name}/counts
/manage/v2/databases/{id|name}?view=counts
/manage/v1/databases/{id|name}/edit
/manage/v2/databases/{id|name}?view=edit
/manage/v1/databases/{id|name}/status
/manage/v2/databases/{id|name}?view=status
/manage/v1/forests/{id|name}/config
/manage/v2/forests/{id|name}?view=config
/manage/v1/forests/{id|name}/counts
/manage/v2/forests/{id|name}?view=counts
/manage/v1/forests/{id|name}/edit
/manage/v2/forests/{id|name}?view=edit
/manage/v1/forests/{id|name}/status
/manage/v2/forests/{id|name}?view=status
/manage/v1/groups/{id|name}/config
/manage/v2/groups/{id|name}?view=config
/manage/v1/groups/{id|name}/counts
/manage/v2/groups/{id|name}?view=counts
/manage/v1/groups/{id|name}/edit
/manage/v2/groups/{id|name}?view=edit
/manage/v1/groups/{id|name}/status
/manage/v2/groups/{id|name}?view=status
/manage/v1/hosts/{id|name}/config
/manage/v2/hosts/{id|name}?view=config
/manage/v1/hosts/{id|name}/counts
/manage/v2/hosts/{id|name}?view=counts
/manage/v1/hosts/{id|name}/edit
/manage/v2/hosts/{id|name}?view=edit
/manage/v1/hosts/{id|name}/status
/manage/v2/hosts/{id|name}?view=status
/manage/v1/servers/{id|name}/config
/manage/v2/servers/{id|name}?view=config
/manage/v1/servers/{id|name}/edit
/manage/v2/servers/{id|name}?view=edit
/manage/v1/servers/{id|name}/status
/manage/v2/servers/{id|name}?view=status
JSON Output Includes Units

JSON output for the various GET methods now includes units for metrics that were previously flat key-value pairs. The following template summarizes the difference:

For example, in MarkLogic 6, elapsed-time was reported as follows:

"elapsed-time" : "0.023453"

In MarkLogic 7, this field include a unit, as shown in this example:

"elapsed-time" : { "units" : "sec", "value" : "0.023453" }
Element/Key Name Changes in Status Views

The following XML element/JSON key name changes have been made:

For example, in earlier releases, the load-detail section of the report returned by GET /manage/LATEST/hosts/{id|name}?view=status includes a total-query-read-load element/key. The equivalent data is now named query-read-load.

Changes to the Management API Plug-ins

There have been some changes to the Management API plug-ins for MarkLogic 11, so if you have a plugin written against previous versions, depending upon the plugin, you might need to update it.

In previous releases (MarkLogic 5 and MarkLogic 6), plug-ins are installed in the following folder:

Assets/plugins/marklogic/manage/v1

Plug-ins installed to this directory will continue to work with the following exceptions:

For details on extending the Management API with plug-ins, see Extending Management API with Plugins in the Monitoring MarkLogic Guide.

Changes to the Packaging API

The Packaging REST API has changed for MarkLogic 7. Applications written using the MarkLogic 6 Packaging REST API (v1) must be rewritten to work with the MarkLogic 7 Packaging REST API (v2).

The differences between the v1 and v2 versions of the Packaging REST API are summarized in the table below.

MarkLogic 6 MarkLogic 7
/v1/list/package/appserver={name} (GET)
/v2/servers/{name}?view=package (GET)
/v2/packages/{pkgname} (POST)
/v1/list/package/database={name} (GET)
/v2/databases/{name}?view=package (GET)
/v2/packages/{pkgname} (POST)
/v1/package/compare (GET)
/v2/packages/{pkgname}?view=differences (GET)
/v1/package/install (POST)
/v2/packages/{pkgname}/install (POST)
/v1/package-tickets/revert (POST)
/v2/tickets/{ticketnumber}/revert (PUT)
Changes to the Configuration Manager

The Configuration Manager Packaging feature has been more tightly integrated with the Configuration Manager. Rather than exporting configurations to an XML file, as in MarkLogic 6, configurations in MarkLogic 7 are now exported, as XML, to a ZIP file.

You can import a configuration saved by MarkLogic 6 into MarkLogic 7. However, you cannot import a configuration saved in MarkLogic 7 into MarkLogic 6.

xdmp:plan Now Requires a Privilege

In MarkLogic 7, the xdmp:plan function requires a privilege (http://marklogic.com/xdmp/privileges/xdmp-plan). Previously, it did not require a privilege. If you have any code that calls xdmp:plan that is run by non-privileged users, you will need to add the privilege to a role that the users are granted (or to a role that they already have).

fn:analyze-string Now Returns Output in a Different Namespace

In MarkLogic 7, the fn:analyze-string function returns an XML node in the http://www.w3.org/2005/xpath-functions namespace, which is the namespace specified by the working draft of the latest XQuery specification. Previously, this function was not fully specified and it returned XML in the http://www.w3.org/2009/xpath-functions/analyze-string namespace. If you have code that is expecting output in the old namespace, you will need to change that code to expect the new menopause.

« Previous chapter
Next chapter »