MarkLogic 9 Product Documentation

Release Notes — Chapter 3

Support for SSL Connections

The mlcp command line tool can now connect to MarkLogic via an SSL (Secure Socket Layer) connections. For details, see Connecting to MarkLogic Using SSL in the mlcp User Guide.

Greater Control over Host Connections

You can now specify multiple hosts for mlcp to connect to during import, export, and copy jobs. Used by itself, this feature enables mlcp to fall back an alternative host if the initial host is not available.

You can also use this capability in conjunction with the new -restrict_hosts option to prevent mlcp from connecting to any hosts except the ones on the initial host list.

For more details, see Controlling How mlcp Connects to MarkLogic in the mlcp User Guide.

Redaction

MarkLogic 9 introduces the ability to redact sensitive data when extracting documents from the database with the export or copy commands by specifying redaction rule collections in the -redact option.

For more details, see Redacting Content During Export or Copy Operations in the mlcp User Guide and Redacting Document Content in the Application Developer's Guide.

Batch Support for Server-Side Import Transformations

Previously, the batch size was always one when applying a server-side transformation during an mlcp import or copy job. This restriction has been lifted in MarkLogic 9. All documents in a batch are now transformed and inserted into the database as a single statement, greatly improving performance when using a transformation.

Ability to Access and Modify Metadata in a Transformation

Collections, permissions, document quality, and temporal collection specified by the client are now available to a transformation function via the context parameter. In addition, a transformation function can set collections, permissions, quality, temporal collection, and values metadata for its output document(s).

For details, see Transforming Content During Ingestion in the mlcp User Guide.

Bulk Asynchronous Data Movement

The new Data Movement SDK feature of the Java Client API enables you to move large amounts of data into, out of, or within a MarkLogic cluster asynchronously. These interfaces leverage your entire cluster for scale-out performance. The feature supports streaming, asynchronous, long running data movement tasks.

For details, see the com.marklogic.client.datamovement package in the Java Client API Documentation and Asynchronous Multi-Document Operations in the Java Application Developer's Guide.

Enhanced Temporal Document Support

You can now pass a temporal document URI when creating, updating, and patching temporal documents. This is logical document URI in a temporal collection. For more details, see the methods of com.marklogic.client.bitemporal.TemporalDocumentManager that accept a temporalDocumentURI parameter and Working with Temporal Documents in the Developing Applications With the Java Client API.

If you have sufficient privileges, you can now wipe (completely remove) a temporal document using TemporalDocumentManager.wipe.

You can use the Java Client API to protect a temporal document against update, deletion, or wipe for a specified time. See TemporalDocumentManage.protect.

Security and Authentication Improvements

You can now use Kerberos or certificate-based authentication to authenticate with MarkLogic. You can also connect to MarkLogic via a SSL (Secure Socket Layer) connection.

For more details, see Authentication and Connection Security in the Developing Applications With the Java Client API.

Values Metadata Support

MarkLogic 9 adds the ability to associate key-value metadata with a document. You can use the Java Client API to add, update, delete, and search values metadata. For more details, see Values Metadata in the Developing Applications With the Java Client API.

Row-Based Search

You can use the Java Client API to execute a plan produced by the Optic API and receive row-based results in your Java application. For details, see Optic Java API for Relational Operations in the Developing Applications With the Java Client API.

Geospatial Search Enhancements

MarkLogic 9 includes enhancements to geospatial search such as region searches, double precision coordinates, additional coordinate systems. These features are exposed through the Java Client API. For more details, see Geospatial Enhancements, Creating Region Queries Using the Client APIs in the Search Developer's Guide, and StructuredQueryBuilder.geospatial in the Java Client API Documentation.

Authentication and Connection Security

You can now use Kerberos or certificate-based authentication to authenticate with MarkLogic. You can also connect to MarkLogic via a SSL (Secure Socket Layer) connection.

For more details, see Authentication and Connection Security in the Node.js Application Developer's Guide.

Enhanced Temporal Document Support

You can now pass a temporal document URI when creating, updating, and patching temporal documents. This is the logical document URI in a temporal collection. For more details, see the methods in the documents namespace that accept a temporalDocument parameter and Working with Temporal Documents in the Node.js Application Developer's Guide.

If you have sufficient privileges, you can now wipe (completely remove) a temporal document using TemporalDocumentManager.wipe.

Values Metadata Support

MarkLogic 9 adds the ability to associate key-value metadata with a document. You can use the Node.js Client API to add, update, delete, and search values metadata. For more details, see the metadataValues metadata category in Working with Metadata in the Node.js Application Developer's Guide.

Geospatial Search Enhancements

MarkLogic 9 includes enhancements to geospatial search such as region searches, double precision coordinates, additional coordinate systems. These features are exposed through the Node.js Client API. For more details, see Geospatial Enhancements, Creating Region Queries Using the Client APIs in the Search Developer's Guide, and queryBuilder.geospatialRegion in the Node.js Client API Reference.

Minimum Distance on Near Queries

The queryBuilder.near method now accepts a minimum distance for near queries. Previously, you could only specify a maximum distance. For details, see the Node.js Client API Reference and cts:near-query.

Enhanced Temporal Document Support

You can now pass a temporal document URI when creating, updating, and patching temporal documents. This is logical document URI in a temporal collection. For more details see Working with Temporal Documents in the REST Application Developer's Guide and the /v1/documents methods that accept a temporal-document parameter.

If you have sufficient privileges, you can now wipe (completely remove) a temporal document using DELETE /v1/documents?result=wiped. For more details, see DELETE /v1/documents in the MarkLogic REST API Reference.

The new POST /v1/documents/protection method enables you to protect a temporal document against update, deletion, or wipe for a specified time period. For more details, see the MarkLogic REST API Reference.

Row-Based Data Evaluation

The new /rows service enables you to execute a plan produced by the Optic API and receive results in a variety of row-based formats. For more details, see:

• GET /v1/rows in the MarkLogic REST API Reference
• POST /v1/rows in the MarkLogic REST API Reference
• Optic API for Multi-Model Data Access in the Application Developer's Guide

Point-in-Time Operations

Most read and search operations now accept a timestamp request parameter that enables you to make successive requests that will evaluated against the state of the database at a point in time. The timestamp can be obtained from the ML-Effective-Timestamp header returned by the same methods.

For more details, see Performing Point-in-Time Operations in the REST Application Developer's Guide.

cts:query Support

You can use a serialized cts:query in place of a structured query in GET /v1/search and POST /v1/search. For more details, see Searching With cts:query in the REST Application Developer's Guide and the MarkLogic REST API Reference.

Values Metadata Support

MarkLogic 9 adds the ability to associate key-value metadata with a document. You can use the REST Client API to add, update, delete, and search values metadata. You can use the new metadata category metadata-values when working with this type of metadata.

For more details, see Working with Metadata in the REST Application Developer's Guide.

Geospatial Enhancements

MarkLogic 9.0-2 contains the following enhancements related to geospatial region support:

• Extended Units Support in Region Indexes
• Crosses, Equals, and Touches Region Comparison Operators
• New Functions for Probing Region Relationships
• Support for Specifying Tolerance

Additional REST Client API Support for cts:query

You can use a serialized cts:query in place of a structured query in the following methods:

• GET /v1/search and POST /v1/search
• GET /v1/values/{name} and POST /v1/values/{name}

On the GET methods, specify the cts:query as the value of the structuredQuery request parameter. On the POST methods, put the cts:query in the POST body.

In addition, you can include a cts:query serialized as XML or JSON in place of a structured query in a combined query. This is applicable to any method that accepts a combined query input, such as a POST request to /v1/search or /v1/values/{name}.

For more details, see Searching With cts:query in the REST Application Developer's Guide and the MarkLogic REST API Reference.

REST Client API Supports QBE in a Combined Query

You can now include a Query By Example (QBE) in a combined query when using the REST Client API. For more details, see Specifying Dynamic Query Options with Combined Query in the REST Application Developer's Guide.

New Server-Side Transaction Controls

You can now configure the commit mode (auto or explicit) and transaction type (query, update, or auto) independently when configuring a new transaction. This change manifests in the following ways:

• The xdmp:update XQuery prolog option accepts a new value, auto, which specifies that MarkLogic should determine the transaction/statement type (query or update) through static analysis. The pre-existing value false now means the transaction/statement type is query. Use this option plus xdmp:commit instead of the now-deprecated xdmp:transaction-mode prolog option.
• A new XQuery prolog option, xdmp:commit, has been added for specifying whether a main module should run as a single-statement, auto-commit transaction (the default) or a multi-statement explicit-commit transaction. Use this option plus xdmp:update instead of the now-deprecated xdmp:transaction-mode prolog option.
• New commit and update options have been added to the functions listed in the table below. Use these in preference to the transaction-mode option, which has been deprecated.

The following functions support the new commit and update options. For more details, see the function reference documentation for xdmp:eval (XQuery) or xdmp.eval (JavaScript).

For more details on the new capabilities, see Understanding Transactions in MarkLogic Server in the Application Developer's Guide.

For details on transitioning from the old transaction controls to the new ones, see the following topics:

• Changes to xdmp:update XQuery Prolog Option
• xdmp:transaction-mode XQuery Prolog Option Deprecated
• Deprecation of transaction-mode Option to xdmp:eval.

XCC: New Session Methods for Transaction Control

The following methods have been added to the Session class for configuring transactions and querying transaction configuration:

• Session.setAutoCommit and Session.isAutoCommit
• Session.setUpdate and Session.getUpdate

You should use these methods rather Session.setTransactionMode, which has been deprecated. For details, see XCC Session.setTransactionMode is Deprecated.

Session.setAutoCommit controls whether requests submitted during the session run in a transaction with auto-commit semantics (the default) or explicit commit semantics. Executing a request with commit set to explicit starts a multi-statement transaction.

Session.setUpdate controls whether requests submitted during the session run in a query transaction, an update transaction, or if the transaction type should be automatically detected by MarkLogic through analysis of the submitted code. Auto detection is the default behavior.

Note that if you override the Session transaction configuration in an ad hoc query, the behavior differs depending on whether you configure the session using setTransactionMode or setAutoCommit and setUpdate. With setAutoCommit and setUpdate, the transaction configuration reverts to the Session settings once the transaction involving the override completes. With setTransactionMode, the override persists and affects future transactions unless you explicitly change it.

REST Management API: Advancing LSQT

You can use the following new method of the REST Management API to advance LSQT on a temporal collection:

POST /manage/v2/databases/{id|name}/temporal/collections?collection=collname

For more details, see POST /manage/v2/databases/{id|name}/temporal/collections?collection={name} in the MarkLogic REST API Reference.

REST Client API: Advancing LSQT

You can now use the following new method of the REST Client API to advance LSQT on a temporal collection:

POST /v1/temporal/collections/{name}

For more details, see POST /v1/temporal/collections/{name} in the MarkLogic REST API Reference.

Ops Director

Ops Director presents a consolidated view of your MarkLogic infrastructure via dashboards that streamline monitoring and troubleshooting of clusters with alerting, performance, and log data. For details, see the Ops Director Guide.

Amazon Web Services (AWS) 1-Click

MarkLogic is now available for deployment on Amazon Web Services (AWS) by means of AWS 1-Click. For details, see http://developer.marklogic.com/products/cloud/aws.

Entity Services Enhancements

The Entity Services API offers the following additional capabilities as of MarkLogic 9.0-3:

• JSON Envelope Document Support
• Support for Element Range Indexes
• Entity Instance Namespace Support
• Generated Code Refactoring

Search Result Sorting Enhancements

The Search API and REST, Java, and Node.js Client APIs now support additional search result ordering choices through the sort-order query option.

Previously, you could only specify score ordering. You can now sort based on score, fitness, quality, and more. For details, see sort-order in the Search Developer's Guide.

Redaction Built-in for Masking Numbers

A new built-in redaction function, redact-number, is available as of MarkLogic 9.0-3. This function provides fine-grained control over the type, range, and format of masking values for numeric data. You can use redact-number with mlcp's redaction capability, the rdt:redact XQuery function, or the rdt.redact Server-Side JavaScript function.

For details, see redact-number in the Application Developer's Guide.

Client APIs: Performance Improvements for JavaScript Extensions and Transforms

Performance improvements have for Client API server-side transformations and extensions implemented in Server-Side JavaScript. This affects applications using transformations or extensions with the REST, Java, or Node.js Client APIs.

Your applications will only benefit from the improved performance if you reinstall your Server-Side JavaScript transforms and extensions.

Java Client API: Values and Tuples Query Support for cts:query

As of Java Client API version 4.0.3, you can use a serialized cts:query as an additional constraint on a values or tuples query.

Redaction: Support for Salting of Deterministic Masking Values

As of MarkLogic 9.0-4, the mask-deterministic built-in redaction function supports two new options for specifying a salt value for masking value generation. Salting can provide a higher level of security. For more details, see the discussion of the salt and extend-salt options of mask-deterministic in the Application Developer's Guide.

Note that the introduction of these options changes the default behavior of deterministic masking value generation. For details, see Redaction: Deterministic Masking Values Differ.

Redaction: New redact-datetime Built-In Function

As of MarkLogic 9.0-4, you can use the redact-datetime built-in redaction function in your redaction rules. You can mask a dateTime value with a random dateTime value or using a picture string.

For more details, see redact-datetime in the Application Developer's Guide.

Separate Download and Installation of Converters and Filters

As of MarkLogic release 9.0-4, MarkLogic converters/filters are offered as a separate package (called MarkLogic Converters package) from MarkLogic Server package.

This change provides better flexibility and enables you to install/uninstall MarkLogic converters/filters separately from MarkLogic Server.

For more details, see MarkLogic Converters Installation Changes Starting at Release 9.0-4 in the Installation Guide.

Node.js Client API Enhancements

The following new features are available as of Node.js Client API v2.1.1. Unless otherwise noted, MarkLogic 9 is required to take advantage of these changes.

• Support for Optic API queries. The Optic API enables you to query documents, triples, and lexicons to extract row oriented data from MarkLogic. For more details, see Using the Optic API for Relational Operations in the Node.js Application Developer's Guide.
• Point-in-time query support. Many read-only operations, such as document read and query, now support a timestamp parameter or option to enable successive requests to share a consistent view of the database at a given timestamp. For details, see Performing Point-in-Time Operations in the Node.js Application Developer's Guide.
• Partial update (document patch) improvements. Prior to MarkLogic 9.0-4, you could not perform the following operations in a patch. These limitations have been removed. For more details, see Patching Document Content or Metadata in the Node.js Application Developer's Guide.
  • Insert a property immediately under the root node of a JSON object node using last-child position.
  • Address items in nested arrays.
• Support for advancing LSQT for temporal collections. See the function documents.advanceLsqt in the Node.js Client API Reference.

Client API Support for JavaScript Patch Content Constructors

The partial update or patch feature of the REST and Java Client APIs now support replacement content constructor functions implemented in Server-Side JavaScript. Versions of MarkLogic prior to 9.0-4 only support XQuery implementations.

For more details, see Writing an XQuery User-Defined Replacement Constructor in the REST Application Developer's Guide.

Database Restore Enhancement

As of MarkLogic release 9.0-4, you can restore a database from a backup, even if the number of database forests are asymmetrical to the backup forests. As a result, the Admin Interface and related API database restore functions have been changed.

For details, see Restoring a Reconfigured Database in the Administrator's Guide.

1-Click AWS Support

MarkLogic now supports the 1-click launch option in AWS Marketplace. Because of this, the published MarkLogic AMIs will have data volume predefined.

Restricted XPath Changes

The following features impose restrictions on XPath expressions used in their configuration. The existence of the restrictions is unchanged. However, as of MarkLogic 9.0-4, the restrictions have been more formally defined for each feature and somewhat relaxed for some features.

The following list summarizes how the restrictions have changed for affected features. In all cases, the changes enable a larger subset of XPath and do not introduce backward incompatibilities.

• Indexable path expressions: The path expressions usable for defining path-based indexes can include function calls to selected functions in predicates.
• TDE context path expressions: The path expressions usable for defining path-based indexes can include function calls to selected functions in predicates. You can use / as a context path.
• Element Level Security protected paths: The path expressions usable for defining path-based indexes can include function calls to selected functions in predicates.
• Extracting document data from search results (extract-document-data query option), document patch feature of the REST Client APIs, Optic API xpath constructor: The path expressions usable for defining path-based indexes can include function calls to selected functions in predicates. You can use an unnamed node test in the leaf step of a path expression.

For more details, see Restricted XPath in the XQuery and XSLT Reference Guide.

Protected Path Sets Added to Element Level Security

As part of MarkLogic 9.0-4, Element Level Security now includes the protected path set feature. A protected path set is a way to allow multiple protected paths to cover and secure the same element, with both AND and OR relationships between the permissions. The information (the name of the protected path set) is simply a tag on the protected path definition. This enables multiple arbitrary security markings for an element.

For more details, see Protected Path Sets in the Security Guide.

MarkLogic Data Hub Framework

MarkLogic 9.0-5 introduces full support for MarkLogic Data Hub Framework (DHF) version 3.0. DHF is an Open Source data integration framework and set of tools that enable you to quickly integrate data from multiple sources into a single MarkLogic database, and then expose that integrated data through MarkLogic.

DHF includes both client-side libraries and tools for integration and modeling, and server-side framework support. To learn more about DHF, see the following:

• DHF documentation and tutorial: https://marklogic.github.io/marklogic-data-hub/
• DHF Open Source project: https://github.com/marklogic/marklogic-data-hub

To use DHF, you will need DHF version 3.0 or later, MarkLogic 9.0-5 or later, and Oracle Java 8 JRE (client-side). For details, see the DHF documentation.

Entity Enrichment and Extraction Enhancements

MarkLogic 9.0-5 introduces new built-in and library functions for entity enrichment and extraction. The new interfaces use entity dictionaries to identify entities. You can create an entity dictionary in several ways, including from a graph created from a SKOS ontology. For more details, see Entity Extraction and Enrichment in the Search Developer's Guide.

Use the following new functions for entity enrichment and extraction:

Use the following new functions for creating and managing entity dictionaries:

Query Console: Explore Database by URI Pattern

You can now apply a URI filter to the documents listed in the Explorer view. You can specify a single URI (/my/interesting/document.xml) or a wildcard expression (/my/interesting/*.xml). For more details, see Filtering the Explorer View by URI in the Query Console User Guide.

Configuration Management API (CMA) XQuery and JavaScript Libraries

MarkLogic 9.0-5 introduces new library functions for configuration management.

The configuration management functions can be used to:

• retrieve a configuration of an individual resource, a set of resources, or a full cluster;
• generate a configuration from scenarios, such as High Availability (HA) scenario;
• apply a named configuration, overriding parameters and setting options.

Use the following new functions for configuration management:

For more details, see MarkLogic XQuery and XSLT Function Reference and MarkLogic Server-Side JavaScript Function Reference.

Configuration Management API (CMA) REST Endpoints

MarkLogic 9.0-5 introduces a new REST API for configuration management: Configuration Management API (CMA).

The Configuration Management API is a RESTful API that allows retrieving, generating, and applying configurations for MarkLogic clusters, databases, and application servers.

Use the following new endpoints for configuration management:

For more details, see MarkLogic REST API Reference.

Ops Director Enhancements

MarkLogic 9.0-5 includes the new version of the Ops Director application: Ops Director 1.1-1.

The Ops Director 1.1-1 has the following new features and enhancements:

• Enhanced security: default authentication changed from application level to digest.
• Improved support for data integration with other applications. In particular, you may export all tables from the Manage view, the Support view, and the Console Settings view. The tables are exported into CSV files, which might be later on imported into other applications (e.g. Excel) for further data processing and analysis. Also, you may export all metrics from charts in the Analyze view.
• Improved Role Based Access Control (RBAC): added inherited access to resources driven by hierarchical relationships in resource groups.
• Implemented functionality for upgrading Ops Director and updating its configuration.
• Added UX enhancements, such as preserving view state between navigations, and UI performance improvements, such as enhanced rendering speed of the pages.

For more details, see the Ops Director Guide.

Monitoring History Enhancements

The Monitoring History application in MarkLogic 9.0-5 was enhanced with new metrics for XDQP Server Requests performance. For details, see XDQP Server Requests Performance Data in the Monitoring MarkLogic Guide.

Tolerance Support in Geospatial Region Queries

As of MarkLogic 9.0-5, geospatial region queries support a tolerance option.

Tolerance is the largest allowable variation in geometry comparisons. If the distance between two points is less than tolerance, then the two points are considered equal. For more details, see Understanding Tolerance in the Search Developer's Guide.

This change affects the functions cts:geospatial-region-query (XQuery), cts.geospatialRegionQuery (JavaScript), the geo-option portion of a Search API geo-region-path constraint option, the geo-option portion of a geo-region-path-query or geo-region-constraint-query, and the options you can specify in a region query using the Java Client API and Node.js Client API.

Due to the improved tolerance support, when upgrading to MarkLogic 9.0-5 or later from an earlier version of MarkLogic 9, you might not get accurate results for geospatial region queries until you reindex.

Database Replication Enhancements

MarkLogic 9.0-5 and later versions enable you to more easily enable/disable or suspend/resume database replication.

Disabling database replication changes the database configuration, so the disabled state persists across restarts and failovers. Use suspend/resume when lag is high and you need to stop database replication quickly.

Suspending database replication for a forest does not change the configuration, so replication resumes after a restart or failover. Use enable/disable when you want to stop replication for a long period, such as when you move a replica site.

This capability is exposed in the following ways:

• In the Admin UI, navigate to the Database Replication settings for a database, click on the Summary tab, and use the enable, disable, suspend, or resume buttons of the Summary tab. For more details, see Enabling, Disabling, Suspending, and Resuming Database Replication in the Database Replication Guide.
• Use the XQuery functions xdmp:forest-database-replication-suspend and xdmp:forest-database-replication-resume or the Server-Side JavaScript functions xdmp.forestDatabaseReplicationSuspend and xdmp.forestDatabaseReplicationResume.

Fast Fail-over

If one or more host forests are configured for local-disk or shared-disk failover, you now have the option to failover those forests when you shut down the host. A new option, Immediately fail over forests to replica hosts, has been added to the Host Shutdown confirmation page to enable you to fail over the forests to replica hosts.

Switching from Internal to External KMS

In MarkLogic 9.0-6, when switching from an external KMS to an internal KMS, encryption at rest does not limit the files that can be decrypted to only those encrypted with keys that are in the internal KMS. MarkLogic can decrypt any file encrypted with the internal MarkLogic KMS or an external KMS.

It is no longer necessary to decrypt your data before transitioning files encrypted with an external KMS to the internal MarkLogic KMS. In MarkLogic 9.0-5, you needed to decrypt all your data before switching from an external KMS back to internal KMS. This is no longer necessary.

Multiple KMS Hosts for Failover

In MarkLogic 9.0-6, you can now specify multiple hosts, multiple ports, and multiple KMIP credentials to connect to KMIP servers. MarkLogic can store more than one set of external KMIP server credentials, to be used in encrypting and decrypting data. In the case where a KMIP server is unavailable and the first specified KMIP server stops responding, MarkLogic will try to connect to each of the other hosts in the user-specified list until it succeeds.

Geospatial Region Query Tolerance Improvements

MarkLogic 9.0-6 includes refinements to tolerance support in geospatial region queries. Some tolerance related edge cases now produce more accurate results. You can only benefit from this change if you reindex.

If your region queries do not involve polygons that fall into these edge cases, you will not see any change in geospatial region query results even if you do reindex.

Change in Multipart/Form-data Processing

Starting with MarkLogic 9.0-6, forms submitted with blank fields result in field entries with blank string content (for example: ""). Previously, these empty form fields would not result in any field entry.

Database Access with Granular Privileges

In MarkLogic 9.0-7, you can now enable a user to create, reindex, update properties, run operations on, or delete a database if they have the manage role and one of the following granular privileges:

http://marklogic.com/xdmp/privileges/admin/database/database-ID

http://marklogic.com/xdmp/privileges/admin/database/activity/database-ID

where activity is one of the following:

• backup
• merge
• restore
• clear
• index

Forest Access with Granular Privileges

In MarkLogic 9.0-7, you can now enable a user to create, update properties, run operations on, or delete a forest if they have the manage role and one of the following granular privileges:

http://marklogic.com/xdmp/privileges/admin/forest

http://marklogic.com/xdmp/privileges/admin/forest/forest-ID

Query Console: Content Editing

The database Explorer in Query Console now enables you to do the following operations without writing any code:

• Insert a new document into a database.
• Modify an existing document in the database.
• Delete one or selected documents from a database.

For more details, see Editing Database Content in the Query Console User Guide.

Request Monitoring

The Request Monitoring feature enables you to configure logging of information related to requests, including metrics collected during request execution. This feature lets you enable logging of internal preset metrics for requests on specific endpoints. You can also log custom request data by calling the provided Request Logging APIs. This logged information may help you evaluate server performance.

For more details, see Endpoints and Request Monitoring in the Query Performance and Tuning Guide<Default ° Font>.

MarkLogic Services in the Cloud

In MarkLogic 9.0-7 you can now access both Query Service and Data Hub Service as services on AWS. Query Service enables you to respond to varying query workloads for your MarkLogic cluster. This elastic query service eliminates the need to set up and manage the underlying infrastructure required to scale for capacity. MarkLogic Query Service work by attaching MarkLogic e-nodes to to your existing MarkLogic cluster, and auto-scaling as your workload requires. For more information, see https://cloudservices.marklogic.com/and https://www.marklogic.com/product/marklogic-database-overview/query-service/.

MarkLogic Data Hub Service is a cloud service providing the features of the Data Hub Framework in a managed instance available on AWS. The Data Hub Service enables you to create an Operational Data Hub in the cloud. The service is designed to be transparent, scaleable, and easy to deploy. For more details, see https://www.marklogic.com/product/marklogic-database-overview/data-hub-service/. To learn more about Data Hub Framework see the Data Hub Framework website.

Java Client API: Load Balancer Improvements

As of Java Client API 4.1.1, you can specify a connection type when creating a DatabaseClient. The connection type defaults to DIRECT, meaning that direct connections to hosts in your MarkLogic cluster are possible. Specify the GATEWAY connection type when connecting to MarkLogic through a load balancer. A GATEWAY connection is required when using the Data Movement SDK with a load balancer.

For more details, see Connecting Through a Load Balancer in the Java Application Developer's Guide.

Request Monitoring Enhancements

The following features have been added to Request Monitoring:

• Request Monitoring on XDBC Server

  The Request Monitoring feature now enables you to setup and enable request monitoring for an endpoint, a main module, or globally on an XDBC Server.

• Request Cancelling

  The Request Cancelling feature enables you to setup and enable request cancelling for an endpoint, a main module, or globally on an App Server or XDBC Server.

• New Functions

  The following functions have been added to the Request Monitoring API:

  • xdmp:set-request-limit
  • xdmp:request-status

For more details, see Endpoints and Request Monitoring in the Query Performance and Tuning Guide<Default ° Font>.

New Meters Available in Query Meters and Request Logs

The following meters are collected on the server as of MarkLogic Server 9.0-8 and are available in *.api files, request logs, and in the output of xdmp:query-meters():

Synchronize Keystore

Customers who use MarkLogic's Encryption at Rest feature can now synchronize the enveloped keys on MarkLogic with their KMS. For more information, see Synchronizing the KMS Keys in the Security Guide or documentation for the function xdmp:keystore-synchronize.

New Client Issuer Authority Functions

The following XQuery functions have been added to MarkLogic Server:

• admin:appserver-get-ssl-client-issuer-authority-verification
• admin:appserver-set-ssl-client-issuer-authority-verification

The following JavaScript functions have been added to MarkLogic Server:

• admin.appserverGetSslClientIssuerAuthorityVerification
• admin.appserverSetSslClientIssuerAuthorityVerification

See the online documentation provided in the links for details.

Kerberos Authorization and LDAP Authentication Change

MarkLogic 9.0-8 has the parameter ldap-remove-domain, enabling you to choose whether or not to remove the domain name. The default value is false.

Logout Capability Added

MarkLogic 9.0-8 now has a Logout button in the upper right corner of the Admin Interface.

Kerberos Authorization Parameter Change

In MarkLogic 9.0-9, the ldap-remove-domain parameter default value has been changed to true to keep behavior consistent with MarkLogic 9.0-6 and earlier.

Task Server Monitoring at Endpoint Level

MarkLogic 9.0-9 supports endpoint monitoring at the Task Server level. For more information, see the Query Performance and Tuning Guide.

MarkLogic SQL ORDER-BY Keyword

Starting in MarkLogic 9.0-9, ORDER-BY ordering on NULL is set to NULLS LAST. This changes default behavior for nulls in query results and therefore may cause backwards incompatibility for certain queries (unless the Optic Nulls Smallest On trace event is turned on).

Change to ldapSearch Functionality

An ldapSearch call in JavaScript now returns an array if there are multiple values associated with a key. Previously it only returned a single value for every key, even if there were multiple values associated with that key.

Amazon Linux 2 Support

MarkLogic Amazon Web Services (AWS) AMIs are now supported on Amazon Linux 2. This is the recommended base image on which to deploy customized MarkLogic images.

Encryption Support for Thales HSM

With MarkLogic 9.0-9, MarkLogic supports the Thales nCipher nShield Connectd for Encryption at Rest for both Windows and Linux platforms. Thales HSM (Hardware Security Module) implements PKCS #11 (the IETF standard) and when configured, replaces our internal keystore (SoftHSM) for all encryption operations.

MarkLogic Supports ECDH Key Exchange for SSL/TLS

MarkLogic 9.0-9.1, ECDH is a supported cipher for SSL/TLS communication. SSL/TLS works if a ECDH cipher is specified.

OpenSSL Upgrade to Version 1.0.2s

OpenSSL has been upgraded to version 1.0.2s. For more information, please see the list of changes here.

Updates to the BiText Dictionaries

Updates to the BiText Dictionaries:

In most cases, new entries/words come from neologisms or newly-coined words that have a significant number of occurrences in corpus (always including any necessary inflection). In some cases, new entries/words come from previously missing words (normally due to too low occurrences in corpus in previous iterations).

For English we have made a wider change: in our previous version, comparative and superlative forms of adjectives were their own lemmas (so, the lemma for nicest was nicest, and the lemma for nicer was nicer). Now we have linked those forms to the positive form, with that positive form as lemma (so, in that new version, the lemma for nicest and nicer is nice). Anyway, as we weren't sure how this change could affect to your operations, we have prepared two versions of the updated English data:

oldCS.txt, containing the comparatives and superlatives of adjectives with the comparative or superlative as lemma.

newCS.txt, containing the comparatives and superlatives of adjectives with the positive form as lemma.

Below are the details of the changes per language (amount of newly added words):

English (ENG): Around 4,500 new entries added (89,248 ® 93,827).

Arabic (ARB): Around 5,500 new entries added (24,060,893 ® 24,066,479).

Dutch (NLD): Around 25,000 new entries added (376,185 ® 401,741).

French (FRA): Around 700 new entries added to the non-clitics dictionary (375,713 ® 376,449). Around 250 new entries added to the clitics dictionary (985,407 ® 985,676).

German (DEU): Around 26,000 new entries added (1,132,410 ® 1,158,531).

Italian (ITA): Around 900 new entries added (3,320,525 ® 3,321,417).

Korean (KOR): Around 20,000 new entries added (5,013,103 ® 5,032,966).

Norwegian (NOR): Around 400 new entries added to the Bokm•l dictionary (NOB) (378,738 ® 379,112). Around 28 entries added to the Nynorsk dictionary (NNO) (328,872 ® 329,004).

Persian (FAS): Around 1,200 new entries added (403,038 ® 404,283).

Portuguese (POR): Around 2,000 new entries added (10,861,645 ® 10,863,756).

Russian (RUS): Around 1,200 new entries added to the standard dictionary (1,447,917 ® 1,449,175). 16 entries added to the orthographic yo/ye dictionary (61,236 ® 61,252).

Spanish (ESP): Around 7,500 new entries added (3,317,323 ® 3,324,939).

Swedish (SWE): Around 1,000 new entries added (424,824 ® 425,862).

Request Monitoring Enhancements

The logging for each request can be controlled by adding a threshold. The request information is logged only when the threshold conditions are met.

Request monitoring is now enabled by default on some MarkLogic application servers:

• The default App-Services application server, usually port 8000, for all query console-related requests, and for non-query console related requests with a runtime exceeding one second.
• The default Manage application server, usually port 8002, for requests that run longer than one second.

For more information, see the MarkLogic Query Performance and Tuning Guide.

Using MarkLogic Encryption with Microsoft Azure Key Vault

Microsoft Azure Key Vault can encrypt your data in MarkLogic. Azure Key Vault is supported for customers running their cluster on Microsoft Azure. For more information, see the MarkLogic Security Guide.

Rolling Upgrade Status Added

MarkLogic 9.0-12 now has an Upgrade tab in the Admin Interface. During an upgrade, click the Upgrade tab to view the upgrade status of each host in the cluster. For more details, see Rolling Upgrade Status in Admin UI in the Administrator's Guide.

Other Changes

Swap space is automatically configured when running MarkLogic Server on Amazon Web Services (AWS). Swap space is configured during the system startup process with the MARKLOGIC_AWS_SWAP_SIZE configuration variable. For more details, see AWS Configuration Variables and Deployment and Startup in the MarkLogic Server on Amazon Web Services (AWS) Guide.

The Managed Custer feature supports SSL-enabled clusters. For details, see The Managed Cluster Feature in the MarkLogic Server on Amazon Web Services (AWS) Guide.

Packages by Linux Platform Updated

Updated the list of packages required for each supported Linux platform. For more details, see Supported Platforms and Appendix: Packages by Linux Platform in the Installation Guide for All Platforms.

IAM Permissions Updated

Updated the minimum required IAM permissions to create and delete a stack. For more details, see Creating an IAM Role in the MarkLogic Server on Amazon Web Services (AWS) Guide.