Loading TOC...
Release Notes (PDF)

Release Notes — Chapter 3

New Features in MarkLogic 10

This chapter describes the new features in MarkLogic 10.

JavaScript Engine Upgrade

In MarkLogic 10, the JavaScript engine has been upgraded to V8 version 6.7. For more details on the new language features, please see Google V8 JavaScript Engine and Converting JavaScript Scripts to Modules.

Element Level Security (ELS) in the Triple Index

In MarkLogic 10, we have extended support for element-level security (ELS) to include the triple index, meaning it can now be leveraged by semantic graphs and SQL. In semantics, individual triples can be protected. In SQL, this allows you to enable column-level security by protecting specific columns in a Template (TDE).

Machine Learning

The Cognitive Toolkit (CNTK) library has the concept of a default device. This sets the default computation device (CPU or GPU) for the API. Some functions have a device parameter that allows you to override the default, but not all. The default device has been set based on the version:

  • The GPU-enabled version of MarkLogic Server has the default device set to GPU (0).
  • The CPU-enabled version of MarkLogic Server has the default device set to CPU.

The default device is enabled during node startup. On GPU enabled instances, it is an exclusive lock. CNTK uses cooperative locking for the device access, whereby only a single process can acquire a device lock. This locking mechanism allows CNTK processes to avoid device oversubscription only if they collectively choose to do so. In other words, the device locked by one CNTK process can still be accessed by another CNTK process without acquiring any locks (the existing device lock can be ignored by other CNTK processes). This cooperative locking mechanism does not guarantee any kind of exclusive access to the device. The proper way to ensure exclusivity is to use the NVIDIA System Management Interface (nvidia-smi) provided by NVIDIA.

Beginning with version 10.0-2 of MarkLogic Server, the CNTK machine learning libraries are loaded dynamically based on the hardware detected at server start time. The GPU-enabled version of MarkLogic Server has the default device set to GPU (0). The CPU-enabled version of MarkLogic Server has the default device set to CPU.

Starting with version 10.0-2 of MarkLogic Server, on Linux, we no longer have separate GPU-enabled and CPU-enabled versions. There is only a single installation RPM file. On Windows, however, we still use separate MSI installation files.

Security Library Upgrades

The following security-related libraries have been upgraded:

  • OpenSSL has been upgraded to version 1.0.2s. For more information, please see the list of changes here.
  • Kerberos has been upgraded to version 1.17.
  • SoftHSM has been upgraded to version 2.5.0.
  • OpenLDAP has been upgraded to version 2.4.50.
  • SASL has been upgraded to version 2.1.27.
  • SoftHSM library has been upgraded from version 2.2.0 to version 2.5.0.

Triggers and Amps Creation

Starting in 9.0-7 for triggers and 10.0-2 for amps, Database names can be used in the trigger and amp creation apis, thus making it easy to support the same functionality on replica clusters for databases with the same names.

Default Assignment Policy for New Databases

Starting in MarkLogic Server version 10.0-2, the default setting for assignment policy for new databases is Segment. Databases created with previous versions of MarkLogic will retain their original assignment policy following an upgrade. After the upgrade to 10.0-2, all new databases will have Segment as the assignment policy.

MarkLogic Supports ECDH Key Exchange for SSL/TLS

In MarkLogic 10.0-1, ECDH is a supported cipher for SSL/TLS communication. SSL/TLS works if an ECDH cipher is specified.

New Features in MarkLogic 10.0-2

Security Improvements

Added support for Azure Key Vault External KMS. For details, see Using MarkLogic Encryption with Microsoft Azure Key Vault in our Security Guide.

Upgraded to version 1.0.2s of the OpenSLL library.

We now use Argon2 for passphrase Key Derivation Function (KDF).

Usability Improvements

Machine Learning using the CNTK API now has support for a single CPU amd GPU on Linux, as well as granular CNTK built-in privileges.

Request Monitoring has been enhanced with: support for triggers; support for a default application server on ports 8000 and 8002. For more details, see Monitoring Requests in our Query Performance and Tuning Guide.

Support for Azure Identity to access storage blob.

Support for Database names for amps.

Performance Improvements

The internal SQL Optimizer has been improved in the following areas:

  • OR operators are now more efficient
  • Support has been added for Power BI inverse filters.
  • SPARQL Query performance has been improved.

New Features in MarkLogic 10.0-3

Usability Improvements

Support for ONNX Runtime API has been added in both JavaScript and XQuery See the Machine Learning with the ONNX API chapter in our Application Developer's Guide.

Language codes are now supported in JSON content. MarkLogic now allows natural language in JSON to be tagged with a language other than the default database language.

The MarkLogic SPARQL engine now supports negated property paths as defined in the W3C 1.1 recommendations, allowing users to query graphs with more flexibility.

The granular privilege create-user-privilege has been added to enable giving users limited privileges. For more information, see Enabling Non-Privileged Users to Create Privileges, Roles, and Users in the Security Guide.

Performance Improvements

The performance has been improved in both our SQL and the SPARQL internal engines.

Other Changes

Swap space is automatically configured when running MarkLogic Server on Amazon Web Services (AWS). Swap space is configured during the system startup process with the MARKLOGIC_AWS_SWAP_SIZE configuration variable. For more details, see AWS Configuration Variables and Deployment and Startup in the MarkLogic Server on Amazon Web Services (AWS) Guide.

The CNTK API is now deprecated and may be removed in a future release. For any new Machine Learning application projects, developers should use the ONNX Runtime API embedded in our server. For more details, please see the Why Using ONNX Runtime in MarkLogic Makes Sense section in our Application Developer's Guide.

The Managed Cluster feature supports SSL-enabled clusters. For details, see The Managed Cluster Feature in the MarkLogic Server on Amazon Web Services (AWS) Guide.

New Features in MarkLogic 10.0-4

Rolling Upgrade Status Added

MarkLogic 10.0-4 now has an Upgrade tab in the Admin Interface. During an upgrade, click the Upgrade tab to view the upgrade status of each host in the cluster. For more details, see Rolling Upgrade Status in Admin UI in the Administrator's Guide.

Permissions Change for Updating Temporal Collections LSQT Properties

The permissions for changing the temporal collection LSQT properties now only requires admin/temporal rights. The scope of this change is within RMA. Previously full admin rights to the database were required.

ODBC Cursor Support

ODBC now supports cursors making it more memory efficient on the client by default. Customers should update to the latest ODBC driver.

New Features in MarkLogic 10.0-5

Packages by Linux Platform Updated

Updated the list of packages required for each supported Linux platform. For more details, see Supported Platforms and Appendix: Packages by Linux Platform in the Installation Guide for All Platforms.

IAM Permissions Updated

Updated the minimum required IAM permissions to create and delete a stack. For more details, see Creating an IAM Role in the MarkLogic Server on Amazon Web Services (AWS) Guide.

FULL OUTER JOIN Updated

FULL OUTER JOIN is now supported in a SQL query.

New Features in MarkLogic 10.0-6

SQL Grouping Sets

In MarkLogic 10.0-6 support for SQL keywords grouping sets, cube, rollup, and the grouping() aggregate has been added. See these APIs for more informaiton:

Optic API for Grouping Sets

In MarkLogic 10.0-6, the Optic API for grouping sets has been added. For more information about Optic, see https://docs.marklogic.com/10.0/guide/app-dev/OpticAPI.

Support for IN Operator in Optic

MarkLogic 10.0-6 now includes support for the IN operator in Optic.

For example:

where(op.in(op.col('columnName'), [1, 2, 3]))

For more information about Optic, see https://docs.marklogic.com/10.0/guide/app-dev/OpticAPI.

Query DSL for Optic API

A human-editable query language representation for the Optic API has been added to the /v1/rows endpoint in MarkLogic_10.0-6. The DSL adds a human-oriented textual representation of an Optic query without limiting the query capabilities. The human-oriented representation can be edited with text editors, displayed in diagnostic views, and so on.

The Optic API supports lossless conversion between the machine-oriented AST and human-oriented DSL representations of an Optic query. Currently, the /v1/rows endpoint is usable only by using MarkLogic client APIs or previously exported ASTs. As a result, the REST API support for Optic queries is currently machine-oriented, but becomes human-oriented with this enhancement.

op.fromSearch() / op:from-search() accessor for Optic API

MarkLogic 10.0-6 now exposes the plan:search function in the Optic API in the form of the new op.fromSearch and op:from-search functions. For more information about Optic, see https://docs.marklogic.com/10.0/guide/app-dev/OpticAPI.

Column binding supported in Optic

In MarkLogic 10.0-6, the op:bind-as operator has been added to bind a new column without affecting existing columns in the row. The bind-as operation is a new, simpler interface to the implementation for the existing op:as or op.as functions. For more information about Optic, see https://docs.marklogic.com/10.0/guide/app-dev/OpticAPI.

Support SQL payloads on /v1/rows

MarkLogic 10.0-6 now supports SQL payloads on /v1/rows. For details, see https://docs.marklogic.com/10.0/REST/POST/v1/rows.

Reactive Auto-Scaling for mlcp Import Jobs

In MarkLogic 10.0-6 mlcp supports reactive auto-scaling for import jobs. This feature maximizes the import process as a Data Hub Service cluster scales to improve performance.

New Features in MarkLogic 10.0-7

Query-Based Access Control

MarkLogic 10.0-7 supports Query-Based Access Control (QBAC) as a way to secure data access at the fundamental level in MarkLogic Server. Query-Based Access Control or QBAC can integrate with all the existing MarkLogic security features, such as Compartment Security, ELS, triples and protected collections. See Query-Based Access Control in the Security Guide for more information.

Query-Based Views

Query-Based Views (QBV) have been added in MarkLogic 10.0-7. A Query-based view is a view created from an Optic query, that can be referenced in subsequent calls to SQL or Optic. The Query-based view feature enables you to create SQL views that reference Template (TDE) views, lexicons, and SPARQL queries. For more information, see Query-Based Views in the Application Developer's Guide.

Hashing Functions Added to TDE

In MarkLogic 10.0-7, these hashing functions have been added to TDE:

  • xdmp:hash32
  • xdmp:hash64
  • xdmp:md5
  • xdmp:sha1
  • xdmp:sha256
  • xdmp:sha384
  • xdmp:sha512
  • xdmp:hmac-md5
  • xdmp:hmac-sha1
  • xdmp:hmac-sha256
  • xdmp:hmac-sha512

See Template Dialect and Data Transformation Functions in the Application Developer's Guide for more information.

Default.sjs and Index.sjs added

In MarkLogic 10.0-7, default.sjs and index.sjs have been added to the list of default modules for an application server to render.

Added Granular Privileges

These execute privileges have been added in MarkLogic 10.0-7:

  • create-data-user
  • create-data-role
  • switch-task-user

See Enabling Non-Privileged Users to Create Privileges, Roles, and Users in the Security Guide for more details.

New Functions Added to Optic API

In MarkLogic 10.0-7, op.existsJoin and op.notExistsJoin have been added to the Optic API. On release, the two functions, op.existsJoin() and op.notExistsJoin() do not perform natural joins between columns with the same identifiers - as other existing Optic join types do. Please use op.on() to specify the join condition.

Optic Redaction on Rows

Redaction on rows using the Optic API has been introduced MarkLogic 10.0-7. An Optic query can redact a column by rebinding a column to an expression. The expression can either transform the column values or generate replacement values in some other way including based on random numbers or UUIDs.

The Optic API now provides helper functions to build column rebindings for common redaction cases including maskDeterministic(), maskRandom(), redactDatetime(), redactEmail(), redactIpv4(), redactNumber(), redactRegex(), redactUsSsn(), and redactUsPhone(). See the Optic APIs at https://docs.marklogic.com/js/ordt (JavaScript) and https://docs.marklogic.com/ordt (XQuery) for more information.

« Previous chapter
Next chapter »