Release Notes (PDF)

Release Notes — Chapter 3

« Previous chapter
Next chapter »

New Features in MarkLogic 9

This chapter describes the new features in MarkLogic 9.

Template Driven Extraction (TDE)

MarkLogic 9 enables you to define a relational lens over your document data, so you can query parts of your data using SQL or the new Optic API. Templates let you specify which parts of documents make up rows in a view. You can also use templates to define a semantic lens, specifying which values from a document make up triples in the triple index.

For more details, see Template Driven Extraction (TDE) in the Application Developer's Guide.

SQL Enhancements

MarkLogic is a NoSQL database, where the unit of storage and indexing is a document. The document model makes it possible to express rich, related, varying structures -- anything from a scientific journal article to a complex derivative trade. Many users want to view parts of these rich structures as though they were simple tables -- to see the data in those documents through a relational lens. While it is possible to create and query SQL views with MarkLogic 8, MarkLogic 9 further enhances the SQL capabilities with a number of new features, including Templates, the new Optic API, and an updated ODBC driver.

In MarkLogic 9, you can define a template that specifies which parts of the document make up a row in a view, and then query that view from a server-side program with xdmp:sql() or xdmp.sql() or via ODBC. You can also query that view server-side from the new MarkLogic Optic API -- a fluent JavaScript interface with the ability to perform joins and aggregates on views over documents.

Optic API

The Optic API blends the relational world with rich NoSQL document features by providing the capability to perform joins and aggregates over documents. One of the enabling features, Template Driven Extraction, makes it possible to create a relational lens over documents stored in MarkLogic by using templates to specify the parts of a document that make up a row in a view. You can access that data using the Optic API in XQuery, JavaScript, or Java.

For more details, see Optic API for Multi-Model Data Access in the Application Developer's Guide.

Enhanced Tiered Storage

MarkLogic 9 adds more flexibility to the Tiered Storage feature. First, it allows documents to be matched to a tier based on a query, in addition to the current solution that is based on a range index pair of boundary values. That means that any complex query can be used to create a policy to match data or metadata about documents in order to decide in which tier to save or move the document. Second, it allows tiered storage queries to match dates using age, for instance, you can write a query for high-end tier that matches documents created in the last 30 days, and another for mid-end tier that matches documents created between 1 and 3 years, rather than the fixed pair of boundary values in the current solution. Third, it removes the need of using super databases to create tiers, simplifying the life of developers, which were required to understand the tiering policy in order to write high performance queries or any update.

MarkLogic 9 adds more performance to the Tiered Storage feature. Now queries that use elements which are part of the tiering policy are optimized to run against the nodes that are more likely to have that data. That means that if you have a query that uses the same element as the tiering policy, for instance an element range query on create date > 30 days, the query engine will direct the search only to the data nodes that store that data, in this example high-end tier nodes.

Encryption at Rest

MarkLogic 9 introduces the ability to encrypt 'data at rest' - data that is on media (on disk or in the cloud), as opposed to data that is being used in a process. Encryption can be applied to newly created files, configuration files, or log files. Existing data files can be encrypted by triggering a merge or re-index of the data.

For more information about using Encryption at Rest, see Encryption at Rest in the MarkLogic Security Guide.

Element Level Security

Element Level Security is an addition to the MarkLogic security model that allows you to specify more complex security rules for specific elements within documents. Element Level Security can be applied to either JSON or XML documents. Users without the appropriate permissions cannot view the secured element or JSON property.

Element Level Security can be used in addition to the existing document level security and compartment security. For details about using Element Level Security, see Element Level Security in the Security Guide.

Redaction

MarkLogic 9 introduces a new library module and mlcp command line option that enable you to redact sensitive data when extracting documents from the database. Redaction enables you to obscure or hide portions of a document using a rule-based read transformation.

Redaction is available through the following interfaces:

  • The mlcp -redaction command line option.
  • The rdt:redact XQuery function.
  • The rdt.redact Server-Side JavaScript function.

For more details, see Redacting Document Content in the Application Developer's Guide and Redacting Content During Export or Copy Operations in the mlcp User Guide.

Geospatial Enhancements

MarkLogic 9 includes the following enhancements to geospatial operations:

  • Geospatial region search, including a new geospatial region index type
  • ETRS-89 coordinate system support
  • Double precision coordinate support
  • Support for kilometers, meters, and feet as units of measure

You can take advantage of these enhancements using all of the MarkLogic geospatial and search related interfaces, including cts:search, search:search, JSearch, and the REST, Java, and Node.js Client APIs.

The performance of geospatial region queries for geographic coordinate systems (WGS84 and ETRS89) has not yet been fully optimized. Performance will be improved in a future release.

For details, see Geospatial Search Applications in the Search Developer's Guide.

Entity Services API

The Entity Services API enables you to quickly and easily model your business entities and relationships between them and then generate code and configuration artifacts that provide a framework for an entity based application.

The artifacts you generate with the Entity Services API make it easier to create entities from raw source data, query entities and the relationships between them, and manage your entities and models.

For details, see the Entity Services Developer's Guide.

New Stemming and Tokenization

In MarkLogic 9 the default tokenization and stemming code has been changed for all languages (except English tokenization). Some tokenization and stemming behavior will change between MarkLogic 8 and MarkLogic 9. We expect that in most cases results will be better in MarkLogic 9.

You can now customize stemming and tokenization for a language by selecting one of several built-in stemmer and lexer plug-ins, or by creating your own stemmer and lexer plug-ins in C++.

You can also configure a custom stemming and/or tokenization dictionary for any language. Previously, you could not distinguish between stemming and tokenization dictionaries, and you could not install a dictionary for an unsupported language.

For more details, see the following topics:

Accurate Wildcard Expansion Options

MarkLogic 9 adds three new cts:query constructor options for controlling wildcard expansion.

When evaluating a wildcarded search term, MarkLogic must sometimes make a tradeoff between speed and accuracy. MarkLogic attempts to expand a wildcard term from the lexicon. To prevent this expansion from taking too long, there is a limit on how many words MarkLogic will extract from the lexicon. When the limit is reached, MarkLogic falls back on alternative strategies that accurate for all possible terms containing wildcard characters.

For many searches, these inaccuracies are an acceptable tradeoff for the fast response of an unfiltered search. Where the inaccuracies are not acceptable, you can use the following new options to change the default behavior: -lexicon-expansion-limit=N, -limit-check, -no-limit-check.

For more details, see the function reference documentation for the following functions:

XQuery Server-Side JavaScript
cts:word-query cts.wordQuery
cts:element-word-query cts.elementWordQuery
cts:element-attribute-word-query cts.elementAttributeWordQuery
cts:field-word-query cts.fieldWordQuery
cts:json-property-word-query cts.jsonPropertyWordQuery
cts:element-value-query cts.elementValueQuery
cts:element-attribute-value-query cts.elementAttributeValueQuery
cts:field-value-query cts.fieldValueQuery
cts:json-property-value-query cts.jsonPropertyValueQuery

mlcp Enhancements

The following capabilities have been added to the mlcp command line tool:

In addition the mlcp source code is now available for download through the GitHub marklogic-contentpump project. For details, see Accessing the mlcp Source Code in the mlcp User Guide.

Support for SSL Connections

The mlcp command line tool can now connect to MarkLogic via an SSL (Secure Socket Layer) connections. For details, see Connecting to MarkLogic Using SSL in the mlcp User Guide.

Greater Control over Host Connections

You can now specify multiple hosts for mlcp to connect to during import, export, and copy jobs. Used by itself, this feature enables mlcp to fall back an alternative host if the initial host is not available.

You can also use this capability in conjunction with the new -restrict_hosts option to prevent mlcp from connecting to any hosts except the ones on the initial host list.

For more details, see Controlling How mlcp Connects to MarkLogic in the mlcp User Guide.

Redaction

MarkLogic 9 introduces the ability to redact sensitive data when extracting documents from the database with the export or copy commands by specifying redaction rule collections in the -redact option.

For more details, see Redacting Content During Export or Copy Operations in the mlcp User Guide and Redacting Document Content in the Application Developer's Guide.

Batch Support for Server-Side Import Transformations

Previously, the batch size was always one when applying a server-side transformation during an mlcp import or copy job. This restriction has been lifted in MarkLogic 9. All documents in a batch are now transformed and inserted into the database as a single statement, greatly improving performance when using a transformation.

Ability to Access and Modify Metadata in a Transformation

Collections, permissions, document quality, and temporal collection specified by the client are now available to a transformation function via the context parameter. In addition, a transformation function can set collections, permissions, quality, temporal collection, and values metadata for its output document(s).

For details, see Transforming Content During Ingestion in the mlcp User Guide.

Java Client API Enhancements

The following capabilities have been added to the Java Client API:

Bulk Asynchronous Data Movement

The new Data Movement SDK feature of the Java Client API enables you to move large amounts of data into, out of, or within a MarkLogic cluster asynchronously. These interfaces leverage your entire cluster for scale-out performance. The feature supports streaming, asynchronous, long running data movement tasks.

For details, see the com.marklogic.client.datamovement package in the Java Client API Documentation and Asynchronous Multi-Document Operations in the Java Application Developer's Guide.

Enhanced Temporal Document Support

You can now pass a temporal document URI when creating, updating, and patching temporal documents. This is logical document URI in a temporal collection. For more details, see the methods of com.marklogic.client.bitemporal.TemporalDocumentManager that accept a temporalDocumentURI parameter and Working with Temporal Documents in the Developing Applications With the Java Client API.

If you have sufficient privileges, you can now wipe (completely remove) a temporal document using TemporalDocumentManager.wipe.

You can use the Java Client API to protect a temporal document against update, deletion, or wipe for a specified time. See TemporalDocumentManage.protect.

Security and Authentication Improvements

You can now use Kerberos or certificate-based authentication to authenticate with MarkLogic. You can also connect to MarkLogic via a SSL (Secure Socket Layer) connection.

For more details, see Authentication and Connection Security in the Developing Applications With the Java Client API.

Values Metadata Support

MarkLogic 9 adds the ability to associate key-value metadata with a document. You can use the Java Client API to add, update, delete, and search values metadata. For more details, see Values Metadata in the Developing Applications With the Java Client API.

Row-Based Search

You can use the Java Client API to execute a plan produced by the Optic API and receive row-based results in your Java application. For details, see Optic Java API for Relational Operations in the Developing Applications With the Java Client API.

Geospatial Search Enhancements

MarkLogic 9 includes enhancements to geospatial search such as region searches, double precision coordinates, additional coordinate systems. These features are exposed through the Java Client API. For more details, see Geospatial Enhancements, Creating Region Queries Using the Client APIs in the Search Developer's Guide, and StructuredQueryBuilder.geospatial in the Java Client API Documentation.

Node.js Client API Enhancements

The following capabilities have been added to the Node.js Client API:

Authentication and Connection Security

You can now use Kerberos or certificate-based authentication to authenticate with MarkLogic. You can also connect to MarkLogic via a SSL (Secure Socket Layer) connection.

For more details, see Authentication and Connection Security in the Node.js Application Developer's Guide.

Enhanced Temporal Document Support

You can now pass a temporal document URI when creating, updating, and patching temporal documents. This is the logical document URI in a temporal collection. For more details, see the methods in the documents namespace that accept a temporalDocument parameter and Working with Temporal Documents in the Node.js Application Developer's Guide.

If you have sufficient privileges, you can now wipe (completely remove) a temporal document using TemporalDocumentManager.wipe.

Values Metadata Support

MarkLogic 9 adds the ability to associate key-value metadata with a document. You can use the Node.js Client API to add, update, delete, and search values metadata. For more details, see the metadataValues metadata category in Working with Metadata in the Node.js Application Developer's Guide.

Geospatial Search Enhancements

MarkLogic 9 includes enhancements to geospatial search such as region searches, double precision coordinates, additional coordinate systems. These features are exposed through the Node.js Client API. For more details, see Geospatial Enhancements, Creating Region Queries Using the Client APIs in the Search Developer's Guide, and queryBuilder.geospatialRegion in the Node.js Client API Reference.

Minimum Distance on Near Queries

The queryBuilder.near method now accepts a minimum distance for near queries. Previously, you could only specify a maximum distance. For details, see the Node.js Client API Reference and cts:near-query.

Rest Client API Enhancements

The following capabilities have been added to the REST Client API:

Enhanced Temporal Document Support

You can now pass a temporal document URI when creating, updating, and patching temporal documents. This is logical document URI in a temporal collection. For more details see Working with Temporal Documents in the REST Application Developer's Guide and the /v1/documents methods that accept a temporal-document parameter.

If you have sufficient privileges, you can now wipe (completely remove) a temporal document using DELETE /v1/documents?result=wiped. For more details, see DELETE /v1/documents in the MarkLogic REST API Reference.

The new POST /v1/documents/protection method enables you to protect a temporal document against update, deletion, or wipe for a specified time period. For more details, see the MarkLogic REST API Reference.

Row-Based Data Evaluation

The new /rows service enables you to execute a plan produced by the Optic API and receive results in a variety of row-based formats. For more details, see:

Point-in-Time Operations

Most read and search operations now accept a timestamp request parameter that enables you to make successive requests that will evaluated against the state of the database at a point in time. The timestamp can be obtained from the ML-Effective-Timestamp header returned by the same methods.

For more details, see Performing Point-in-Time Operations in the REST Application Developer's Guide.

cts:query Support

You can use a serialized cts:query in place of a structured query in GET /v1/search and POST /v1/search. For more details, see Searching With cts:query in the REST Application Developer's Guide and the MarkLogic REST API Reference.

Values Metadata Support

MarkLogic 9 adds the ability to associate key-value metadata with a document. You can use the REST Client API to add, update, delete, and search values metadata. You can use the new metadata category metadata-values when working with this type of metadata.

For more details, see Working with Metadata in the REST Application Developer's Guide.

Telemetry

Telemetry is part of our continuous effort to provide better and faster support by automating the data collection process required on most support tickets. When Telemetry is enabled, it collects, encrypts, and sends diagnostic and anonymized system-level information about a MarkLogic cluster to a secure MarkLogic destination.

The Telemetry feature collects only system level information, and sends it to a protected and secure location where it can only be accessed by the MarkLogic technical teams, to be used to facilitate troubleshooting and monitor performance. Telemetry does not collect any personally identifiable information, user data or application logs. See Telemetry in the Monitoring MarkLogic Guide for more information.

XQuery 3.x Features

MarkLogic now supports selected features from the XQuery 3.0 and XQuery 3.1 specifications. These features are only available when using the '1.0-ml' XQuery dialect.

For more details, see XQuery 3.x Features in the XQuery and XSLT Reference Guide.

Query Console Enhancements

MarkLogic 9 includes the following new features and enhancements in the Query Console application. For more details, see the Query Console User Guide.

  • Support for profiling Server-Side JavaScript. The profiling output differs from the XQuery output in being sampling based.
  • Auto-complete suggestions. As you type into a query, Query Console displays a list of functions, keywords, and in-scope variables suggests. For functions, you also see the function reference documentation.
  • The editor now provides auto-closure of parentheses, quotes, braces, and other grouping symbols.
  • Each query has now has its own tab for displaying results. The results of one query are not lost if you switch to another query and run it.
  • Query results description differentiates between a result that is a single item versus a sequence of one item so that you can tell whether or not the query returns a sequence.
  • Query execution time is displayed along with the results. For example, Query Console will report 'Returned sequence of 2 items in 230ms'. The display also includes the time delta between the current and previous run.
  • You can now select a content source and App Server independent of one another.
  • MarkLogic 9 enhances the database Explorer to enable you to view metadata such as collections, permissions, key-value metadata, and document quality, along with the contents of a document. This feature is available when you select a document from the Explorer document list.
  • You can reorder queries in a workspace and reorder tabs in the editor using drag-and-drop.
  • Query results larger than 5M will be truncated.

Application Display Environment Customization

Administrators can now customize the display environment of MarkLogic applications such as Query Console and the Monitoring Dashboard in the following ways:

  • Display a notification dialog when a user navigates to one of the MarkLogic application pages.
  • Display a customized banner across the top of each page of the MarkLogic applications. This enables you to easily differentiate between environments such as production and staging.,

For more details, see Configuring a MarkLogic Application Message and Banner in the Administrator's Guide.

Rolling Upgrades

A rolling upgrade is one way to address the need for highly available clusters under heavy transaction loads to upgrade to a newer version of MarkLogic in a seamless manner. Hosts in a cluster are upgraded one by one, without incurring any downtime in availability or interruption of transactions.

For more details, see Rolling Upgrades in the Administrator's Guide.

Bi-temporal Compliance Enhancements

The MarkLogic bi-temporal data management feature has been enhanced to provide the option to store valid and system axes and archival information outside of temporal documents in metadata, rather than directly in the documents. Storing the axes times in metadata enables MarkLogic to update the axes timestamps without changing the documents and invoking reindexing.

The ability to update nodes in temporal documents has been added.

Temporal documents can be protected from certain temporal operations, such as update, delete or wipe for a specified period of time and then automatically archived to a WORM (Write Once Read Many) device.

MarkLogic also supports uni-temporal documents that have only a system axes.

Secure Credentials

Secure credentials enable a security administrator to manage credentials and to be made available to less privileged users for authentication to other systems without giving them access to the credentials themselves.

Certificate Authentication

Certificate-based authentication requires internal and external users and HTTPS clients to authenticate themselves to MarkLogic via a client certificate, either in addition to, or rather than a password.

New REST Management APIs

In MarkLogic 9, a number of REST Management APIs have been added. A few of the APIs have been updated as well. This table summarizes the changes.

Added Updated
GET /manage/v2/credentials/properties
PUT /manage/v2/credentials/properties
GET:/manage/v2?view=describe
GET /manage/v2/credentials/secure
POST /manage/v2/credentials/secure
GET /manage/v2/databases/{id|name}/partition-queries
POST /manage/v2/databases/{id|name}/partition-queries
DELETE /manage/v2/databases/{id|name}/partition-queries/{partition-number}
GET /manage/v2/databases/{id|name}/partition-queries/{partition-number}
GET /manage/v2/databases/{id|name}/partition-queries/{partition-number}/properties
PUT /manage/v2/databases/{id|name}/partition-queries/{partition-number}/properties
PUT /manage/v2/databases/{id|name}/partition-queries/{partition-number}/properties
POST /manage/v2/hosts
GET /manage/v2/meters
GET /manage/v2/protected-paths
POST /manage/v2/protected-paths
GET /manage/v2/protected-paths/{id}/properties
PUT /manage/v2/protected-paths/{id}/properties
DELETE /manage/v2/protected-paths/{id|name}
GET /manage/v2/protected-paths/{id|name}
GET /manage/v2/query-rolesets
POST /manage/v2/query-rolesets
DELETE /manage/v2/query-rolesets/{id|name}
GET /manage/v2/query-rolesets/{id|name}
GET /manage/v2/query-rolesets/{id|name}/properties
PUT /manage/v2/query-rolesets/{id|name}/properties
GET /manage/v2/security
POST /manage/v2/security
GET /manage/v2/security/properties
PUT /manage/v2/security/properties

New Features in MarkLogic 9.0-2

MarkLogic version 9.0-2 contains the following new features:

Geospatial Enhancements

MarkLogic 9.0-2 contains the following enhancements related to geospatial region support:

Extended Units Support in Region Indexes

You can specify additional units (feet, kilometers, meters) when creating a geospatial region index. Previously, you could only specify miles. The units are only meaningful when using a geodetic coordinate system such as WGS84.

For more details, see admin:database-geospatial-region-path-index in the MarkLogic XQuery and XSLT Function Reference.

Crosses, Equals, and Touches Region Comparison Operators

You can now use the 'crosses', 'equals' and 'touches' operators for geospatial region queries and other region computation functions that previously accepted region comparison operators.

For example, you can uses 'crosses' as an operator name to functions such as cts:geospatial-region-query (XQuery) or cts.geospatialRegionQuery (JavaScript). You can also use DE9IM_CROSSES in query text passed to cts:parse (XQuery) or cts.parse (JavaScript).

New Functions for Probing Region Relationships

The following new functions are available for comparing geospatial region values:

For more details, see the function reference documentation in the MarkLogic XQuery and XSLT Function Reference or the MarkLogic Server-Side JavaScript Function Reference.

Support for Specifying Tolerance

Tolerance is the largest allowable variation in geometry comparisons. If the distance between two points is less than tolerance, then the two points are considered equal. For more details, see Understanding Tolerance in the Search Developer's Guide.

The following functions now support a 'tolerance' option. For more details, see the function reference documentation.

XQuery Server-Side JavaScript
geo:arc-intersection  geo.arcIntersection 
geo:box-intersects  geo.boxIntersects 
geo:circle-intersects  geo.circleIntersects 
geo:polygon-intersects  geo.polygonIntersects 
geo:complex-polygon-intersects  geo.complexPolygonIntersects 
geo:region-intersects  geo.regionIntersects 
geo:polygon-contains  geo.polygonContains 
geo:complex-polygon-contains  geo.complexPolygonContains 
geo:region-contains  geo.regionContains 
geo:geohash-encode  geo.geohashEncode 

Additional REST Client API Support for cts:query

You can use a serialized cts:query in place of a structured query in the following methods:

On the GET methods, specify the cts:query as the value of the structuredQuery request parameter. On the POST methods, put the cts:query in the POST body.

In addition, you can include a cts:query serialized as XML or JSON in place of a structured query in a combined query. This is applicable to any method that accepts a combined query input, such as a POST request to /v1/search or /v1/values/{name}.

For more details, see Searching With cts:query in the REST Application Developer's Guide and the MarkLogic REST API Reference.

REST Client API Supports QBE in a Combined Query

You can now include a Query By Example (QBE) in a combined query when using the REST Client API. For more details, see Specifying Dynamic Query Options with Combined Query in the REST Application Developer's Guide.

New Server-Side Transaction Controls

You can now configure the commit mode (auto or explicit) and transaction type (query, update, or auto) independently when configuring a new transaction. This change manifests in the following ways:

  • The xdmp:update XQuery prolog option accepts a new value, 'auto', which specifies that MarkLogic should determine the transaction/statement type (query or update) through static analysis. The pre-existing value 'false' now means the transaction/statement type is query. Use this option plus xdmp:commit instead of the now-deprecated xdmp:transaction-mode prolog option.
  • A new XQuery prolog option, xdmp:commit, has been added for specifying whether a main module should run as a single-statement, auto-commit transaction (the default) or a multi-statement explicit-commit transaction. Use this option plus xdmp:update instead of the now-deprecated xdmp:transaction-mode prolog option.
  • New commit and update options have been added to the functions listed in the table below. Use these in preference to the transaction-mode option, which has been deprecated.

The following functions support the new commit and update options. For more details, see the function reference documentation for xdmp:eval (XQuery) or xdmp.eval (JavaScript).

XQuery JavaScript
xdmp:eval xdmp.eval
xdmp:javascript-eval xdmp.xqueryEval
xdmp:invoke xdmp.invoke
xdmp:invoke-function xdmp.invokeFunction
xdmp:spawn xdmp.spawn
xdmp:spawn-function

For more details on the new capabilities, see Understanding Transactions in MarkLogic Server in the Application Developer's Guide.

For details on transitioning from the old transaction controls to the new ones, see the following topics:

XCC: New Session Methods for Transaction Control

The following methods have been added to the Session class for configuring transactions and querying transaction configuration:

  • Session.setAutoCommit and Session.isAutoCommit
  • Session.setUpdate and Session.getUpdate

You should use these methods rather Session.setTransactionMode, which has been deprecated. For details, see XCC Session.setTransactionMode is Deprecated.

Session.setAutoCommit controls whether requests submitted during the session run in a transaction with auto-commit semantics (the default) or explicit commit semantics. Executing a request with commit set to explicit starts a multi-statement transaction.

Session.setUpdate controls whether requests submitted during the session run in a query transaction, an update transaction, or if the transaction type should be automatically detected by MarkLogic through analysis of the submitted code. Auto detection is the default behavior.

Note that if you override the Session transaction configuration in an ad hoc query, the behavior differs depending on whether you configure the session using setTransactionMode or setAutoCommit and setUpdate. With setAutoCommit and setUpdate, the transaction configuration reverts to the Session settings once the transaction involving the override completes. With setTransactionMode, the override persists and affects future transactions unless you explicitly change it.

New Features in MarkLogic 9.0-3

MarkLogic version 9.0-3 contains the following new features:

REST Management API: Advancing LSQT

You can use the following new method of the REST Management API to advance LSQT on a temporal collection:

POST /manage/v2/databases/{id|name}/temporal/collections?collection=collname

For more details, see POST /manage/v2/databases/{id|name}/temporal/collections?collection={name} in the MarkLogic REST API Reference.

REST Client API: Advancing LSQT

You can now use the following new method of the REST Client API to advance LSQT on a temporal collection:

POST /v1/temporal/collections/{name}

For more details, see POST /v1/temporal/collections/{name} in the MarkLogic REST API Reference.

Ops Director

Ops Director presents a consolidated view of your MarkLogic infrastructure via dashboards that streamline monitoring and troubleshooting of clusters with alerting, performance, and log data. For details, see the Ops Director Guide.

Amazon Web Services (AWS) 1-Click

MarkLogic is now available for deployment on Amazon Web Services (AWS) by means of AWS 1-Click. For details, see http://developer.marklogic.com/products/cloud/aws.

Entity Services Enhancements

The Entity Services API offers the following additional capabilities as of MarkLogic 9.0-3:

JSON Envelope Document Support

You can now work with either XML or JSON envelope documents. Previously, only XML was supported.

As part of this change, some of the functions generated in instance converter modules have been extended to accept a format parameter. If you omit the format, these functions generate XML instances and envelopes, as they did prior to MarkLogic 9.0-3.

The following generated function interfaces now accept an optional format parameter, for some entity type T.

  • T:instance-to-canonical
  • T:instance-to-envelope

If you want to use JSON envelopes, you should regenerate your instance converter and version translator code.

For more details, see Entity Instance Concepts in the Entity Services Developer's Guide.

Support for Element Range Indexes

An entity type can now include specifications for entity properties that should be backed by an element range index. Previously, you could only specify a path range index.

In an XML model descriptor, use the path-range-index and element-range-index elements to specify entity properties that should be backed by an index. The path-range-index element is identical to the pre-existing range-index element.

In a JSON model descriptor, use the pathRangeIndex and elementRangeIndex properties to specify entity properties that should be backed by an index. The pathRangeIndex property is identical to the pre-existing rangeIndex property.

For more details, see Identifying Entity Properties for Indexing in the Entity Services Developer's Guide.

Entity Instance Namespace Support

An entity type definition can now include a namespace URI and namespace prefix for XML instances of that type. The namespace binding is used when generating entity instances in XML, and when generating index configuration and query option artifacts.

For more details, see Defining a Namespace URI for an Entity Type in the Entity Services Developer's Guide.

Generated Code Refactoring

The instance converter and version translator code you can generate using Entity Services has been refactored to improve readability and ease of customization, and to accommodate both XML and JSON envelopes. Previously generated code will continue to work as-is.

For more details, see Generating Code and Other Artifacts in the Entity Services Developer's Guide.

Search Result Sorting Enhancements

The Search API and REST, Java, and Node.js Client APIs now support additional search result ordering choices through the sort-order query option.

Previously, you could only specify score ordering. You can now sort based on score, fitness, quality, and more. For details, see sort-order in the Search Developer's Guide.

Redaction Built-in for Masking Numbers

A new built-in redaction function, redact-number, is available as of MarkLogic 9.0-3. This function provides fine-grained control over the type, range, and format of masking values for numeric data. You can use redact-number with mlcp's redaction capability, the rdt:redact XQuery function, or the rdt.redact Server-Side JavaScript function.

For details, see redact-number in the Application Developer's Guide.

Client APIs: Performance Improvements for JavaScript Extensions and Transforms

Performance improvements have for Client API server-side transformations and extensions implemented in Server-Side JavaScript. This affects applications using transformations or extensions with the REST, Java, or Node.js Client APIs.

Your applications will only benefit from the improved performance if you reinstall your Server-Side JavaScript transforms and extensions.

Java Client API: Values and Tuples Query Support for cts:query

As of Java Client API version 4.0.3, you can use a serialized cts:query as an additional constraint on a values or tuples query.

New Features in MarkLogic 9.0-4

MarkLogic 9.0-4 introduces the following new features:

Redaction: Support for Salting of Deterministic Masking Values

As of MarkLogic 9.0-4, the mask-deterministic built-in redaction function supports two new options for specifying a salt value for masking value generation. Salting can provide a higher level of security. For more details, see the discussion of the salt and extend-salt options of mask-deterministic in the Application Developer's Guide.

Note that the introduction of these options changes the default behavior of deterministic masking value generation. For details, see Redaction: Deterministic Masking Values Differ.

Redaction: New redact-datetime Built-In Function

As of MarkLogic 9.0-4, you can use the redact-datetime built-in redaction function in your redaction rules. You can mask a dateTime value with a random dateTime value or using a picture string.

For more details, see redact-datetime in the Application Developer's Guide.

Separate Download and Installation of Converters and Filters

As of MarkLogic release 9.0-4, MarkLogic converters/filters are offered as a separate package (called MarkLogic Converters package) from MarkLogic Server package.

This change provides better flexibility and enables you to install/uninstall MarkLogic converters/filters separately from MarkLogic Server.

For more details, see MarkLogic Converters Installation Changes Starting at Release 9.0-4 in the Installation Guide.

Node.js Client API Enhancements

The following new features are available as of Node.js Client API v2.1.1. Unless otherwise noted, MarkLogic 9 is required to take advantage of these changes.

Client API Support for JavaScript Patch Content Constructors

The partial update or 'patch' feature of the REST and Java Client APIs now support replacement content constructor functions implemented in Server-Side JavaScript. Versions of MarkLogic prior to 9.0-4 only support XQuery implementations.

For more details, see Writing an XQuery User-Defined Replacement Constructor in the REST Application Developer's Guide.

Database Restore Enhancement

As of MarkLogic release 9.0-4, you can restore a database from a backup, even if the number of database forests are asymmetrical to the backup forests. As a result, the Admin Interface and related API database restore functions have been changed.

For details, see Restoring a Reconfigured Database in the Administrator's Guide.

1-Click AWS Support

MarkLogic now supports the 1-click launch option in AWS Marketplace. Because of this, the published MarkLogic AMIs will have data volume predefined.

Restricted XPath Changes

The following features impose restrictions on XPath expressions used in their configuration. The existence of the restrictions is unchanged. However, as of MarkLogic 9.0-4, the restrictions have been more formally defined for each feature and somewhat relaxed for some features.

The following list summarizes how the restrictions have changed for affected features. In all cases, the changes enable a larger subset of XPath and do not introduce backward incompatibilities.

  • Indexable path expressions: The path expressions usable for defining path-based indexes can include function calls to selected functions in predicates.
  • TDE context path expressions: The path expressions usable for defining path-based indexes can include function calls to selected functions in predicates. You can use '/' as a context path.
  • Element Level Security protected paths: The path expressions usable for defining path-based indexes can include function calls to selected functions in predicates.
  • Extracting document data from search results (extract-document-data query option), document patch feature of the REST Client APIs, Optic API xpath constructor: The path expressions usable for defining path-based indexes can include function calls to selected functions in predicates. You can use an unnamed node test in the leaf step of a path expression.

For more details, see Restricted XPath in the XQuery and XSLT Reference Guide.

Protected Path Sets Added to Element Level Security

As part of MarkLogic 9.0-4, Element Level Security now includes the protected path set feature. A protected path set is a way to allow multiple protected paths to cover and secure the same element, with both AND and OR relationships between the permissions. The information (the name of the protected path set) is simply a 'tag' on the protected path definition. This enables multiple arbitrary security markings for an element.

For more details, see Protected Path Sets in the Security Guide.

New Features in MarkLogic 9.0-5

MarkLogic 9.0-5 introduces the following new features:

MarkLogic Data Hub Framework

MarkLogic 9.0-5 introduces full support for MarkLogic Data Hub Framework (DHF) version 3.0. DHF is an Open Source data integration framework and set of tools that enable you to quickly integrate data from multiple sources into a single MarkLogic database, and then expose that integrated data through MarkLogic.

DHF includes both client-side libraries and tools for integration and modeling, and server-side framework support. To learn more about DHF, see the following:

To use DHF, you will need DHF version 3.0 or later, MarkLogic 9.0-5 or later, and Oracle Java 8 JRE (client-side). For details, see the DHF documentation.

Entity Enrichment and Extraction Enhancements

MarkLogic 9.0-5 introduces new built-in and library functions for entity enrichment and extraction. The new interfaces use entity dictionaries to identify entities. You can create an entity dictionary in several ways, including from a graph created from a SKOS ontology. For more details, see Entity Extraction and Enrichment in the Search Developer's Guide.

Use the following new functions for entity enrichment and extraction:

XQuery Server-Side JavaScript
entity:enrich entity.enrich
entity:extract entity.extract
cts:entity-highlight cts.entityHighlight
cts:entity-walk cts.entityWalk

Use the following new functions for creating and managing entity dictionaries:

XQuery Server-Side JavaScript
entity:dictionary-insert entity.dictionaryInsert
entity:dictionary-load entity.dictionaryLoad
entity:skos-dictionary entity.skosDictionary
cts:entity-highlight cts.entityHighlight
cts:entity-dictionary cts.entityDictionary
cts:entity-dictionary-get cts.entityDictionaryGet
cts:entity-dictionary-parse cts:entityDictionaryParse

Query Console: Explore Database by URI Pattern

You can now apply a URI filter to the documents listed in the Explorer view. You can specify a single URI ('/my/interesting/document.xml') or a wildcard expression ('/my/interesting/*.xml'). For more details, see Filtering the Explorer View by URI in the Query Console User Guide.

Configuration Management API (CMA) XQuery and JavaScript Libraries

MarkLogic 9.0-5 introduces new library functions for configuration management.

The configuration management functions can be used to:

  • retrieve a configuration of an individual resource, a set of resources, or a full cluster;
  • generate a configuration from scenarios, such as High Availability (HA) scenario;
  • apply a named configuration, overriding parameters and setting options.

Use the following new functions for configuration management:

XQuery Server-Side JavaScript
cma:generate-config cma.generateConfig
cma:apply-config cma.applyConfig

For more details, see MarkLogic XQuery and XSLT Function Reference and MarkLogic Server-Side JavaScript Function Reference.

Configuration Management API (CMA) REST Endpoints

MarkLogic 9.0-5 introduces a new REST API for configuration management: Configuration Management API (CMA).

The Configuration Management API is a RESTful API that allows retrieving, generating, and applying configurations for MarkLogic clusters, databases, and application servers.

Use the following new endpoints for configuration management:

Endpoint Description
GET /manage/v3 This endpoint enables retrieving configuration of an individual resource, a set of resources, or a full cluster. It also enables generating new configurations from scenarios.
POST /manage/v3

This endpoint enables applying named configurations to MarkLogic resources, overriding parameters and setting options.

The configurations may be applied to an individual resource, a set of resources, or a full cluster.

For more details, see MarkLogic REST API Reference.

Ops Director Enhancements

MarkLogic 9.0-5 includes the new version of the Ops Director application: Ops Director 1.1-1.

The Ops Director 1.1-1 has the following new features and enhancements:

  • Enhanced security: default authentication changed from 'application level' to 'digest'.
  • Improved support for data integration with other applications. In particular, you may export all tables from the Manage view, the Support view, and the Console Settings view. The tables are exported into CSV files, which might be later on imported into other applications (e.g. Excel) for further data processing and analysis. Also, you may export all metrics from charts in the Analyze view.
  • Improved Role Based Access Control (RBAC): added 'inherited' access to resources driven by hierarchical relationships in resource groups.
  • Implemented functionality for upgrading Ops Director and updating its configuration.
  • Added UX enhancements, such as preserving view state between navigations, and UI performance improvements, such as enhanced rendering speed of the pages.

For more details, see the Ops Director Guide.

Monitoring History Enhancements

The Monitoring History application in MarkLogic 9.0-5 was enhanced with new metrics for XDQP Server Requests performance. For details, see XDQP Server Requests Performance Data in the Monitoring MarkLogic Guide.

Tolerance Support in Geospatial Region Queries

As of MarkLogic 9.0-5, geospatial region queries support a 'tolerance' option.

Tolerance is the largest allowable variation in geometry comparisons. If the distance between two points is less than tolerance, then the two points are considered equal. For more details, see Understanding Tolerance in the Search Developer's Guide.

This change affects the functions cts:geospatial-region-query (XQuery), cts.geospatialRegionQuery (JavaScript), the geo-option portion of a Search API geo-region-path constraint option, the geo-option portion of a geo-region-path-query or geo-region-constraint-query, and the options you can specify in a region query using the Java Client API and Node.js Client API.

Due to the improved tolerance support, when upgrading to MarkLogic 9.0-5 or later from an earlier version of MarkLogic 9, you might not get accurate results for geospatial region queries until you reindex.

Database Replication Enhancements

MarkLogic 9.0-5 and later versions enable you to more easily enable/disable or suspend/resume database replication.

Disabling database replication changes the database configuration, so the disabled state persists across restarts and failovers. Use suspend/resume when lag is high and you need to stop database replication quickly.

Suspending database replication for a forest does not change the configuration, so replication resumes after a restart or failover. Use enable/disable when you want to stop replication for a long period, such as when you move a replica site.

This capability is exposed in the following ways:

Fast Fail-over

If one or more host forests are configured for local-disk or shared-disk failover, you now have the option to failover those forests when you shut down the host. A new option, 'Immediately fail over forests to replica hosts,' has been added to the Host Shutdown confirmation page to enable you to fail over the forests to replica hosts.

New Features in MarkLogic 9.0-6

MarkLogic 9.0-6 introduces the following new features:

Switching from Internal to External KMS

In MarkLogic 9.0-6, when switching from an external KMS to an internal KMS, encryption at rest does not limit the files that can be decrypted to only those encrypted with keys that are in the internal KMS. MarkLogic can decrypt any file encrypted with the internal MarkLogic KMS or an external KMS.

It is no longer necessary to decrypt your data before transitioning files encrypted with an external KMS to the internal MarkLogic KMS. In MarkLogic 9.0-5, you needed to decrypt all your data before switching from an external KMS back to internal KMS. This is no longer necessary.

Multiple KMS Hosts for Failover

In MarkLogic 9.0-6, you can now specify multiple hosts, multiple ports, and multiple KMIP credentials to connect to KMIP servers. MarkLogic can store more than one set of external KMIP server credentials, to be used in encrypting and decrypting data. In the case where a KMIP server is unavailable and the first specified KMIP server stops responding, MarkLogic will try to connect to each of the other hosts in the user-specified list until it succeeds.

Geospatial Region Query Tolerance Improvements

MarkLogic 9.0-6 includes refinements to tolerance support in geospatial region queries. Some tolerance related edge cases now produce more accurate results. You can only benefit from this change if you reindex.

If your region queries do not involve polygons that fall into these edge cases, you will not see any change in geospatial region query results even if you do reindex.

« Previous chapter
Next chapter »
Powered by MarkLogic Server 7.0-4.1 and rundmc | Terms of Use | Privacy Policy