Loading TOC...
Release Notes (PDF)

Release Notes — Chapter 3

New Features in MarkLogic 10

This chapter describes the new features in MarkLogic 10.

JavaScript Engine Upgrade

In MarkLogic 10, the JavaScript engine has been upgraded to V8 version 6.7. For more details on the new language features, please see Google V8 JavaScript Engine and Converting JavaScript Scripts to Modules.

Element Level Security (ELS) in the Triple Index

In MarkLogic 10, we have extended support for element-level security (ELS) to include the triple index, meaning it can now be leveraged by semantic graphs and SQL. In semantics, individual triples can be protected. In SQL, this allows you to enable column-level security by protecting specific columns in a Template (TDE).

Machine Learning

The Cognitive Toolkit (CNTK) library has the concept of a default device. This sets the default computation device (CPU or GPU) for the API. Some functions have a device parameter that allows you to override the default, but not all. The default device has been set based on the version:

  • The GPU-enabled version of MarkLogic Server has the default device set to GPU (0).
  • The CPU-enabled version of MarkLogic Server has the default device set to CPU.

The default device is enabled during node startup. On GPU enabled instances, it is an exclusive lock. CNTK uses cooperative locking for the device access, whereby only a single process can acquire a device lock. This locking mechanism allows CNTK processes to avoid device oversubscription only if they collectively choose to do so. In other words, the device locked by one CNTK process can still be accessed by another CNTK process without acquiring any locks (the existing device lock can be ignored by other CNTK processes). This cooperative locking mechanism does not guarantee any kind of exclusive access to the device. The proper way to ensure exclusivity is to use the NVIDIA System Management Interface (nvidia-smi) provided by NVIDIA.

Beginning with version 10.0-2 of MarkLogic Server, the CNTK machine learning libraries are loaded dynamically based on the hardware detected at server start time. The GPU-enabled version of MarkLogic Server has the default device set to GPU (0). The CPU-enabled version of MarkLogic Server has the default device set to CPU.

Starting with version 10.0-2 of MarkLogic Server, on Linux, we no longer have separate GPU-enabled and CPU-enabled versions. There is only a single installation RPM file. On Windows, however, we still use separate MSI installation files.

Security Library Upgrades

The following security-related libraries have been upgraded:

  • OpenSSL has been upgraded to version 1.0.2s. For more information, please see the list of changes here.
  • Kerberos has been upgraded to version 1.17.
  • SoftHSM has been upgraded to version 2.5.0.
  • OpenLDAP has been upgraded to version 2.4.50.
  • SASL has been upgraded to version 2.1.27.
  • SoftHSM library has been upgraded from version 2.2.0 to version 2.5.0.

Triggers and Amps Creation

Starting in 9.0-7 for triggers and 10.0-2 for amps, Database names can be used in the trigger and amp creation apis, thus making it easy to support the same functionality on replica clusters for databases with the same names.

Default Assignment Policy for New Databases

Starting in MarkLogic Server version 10.0-2, the default setting for assignment policy for new databases is Segment. Databases created with previous versions of MarkLogic will retain their original assignment policy following an upgrade. After the upgrade to 10.0-2, all new databases will have Segment as the assignment policy.

MarkLogic Supports ECDH Key Exchange for SSL/TLS

In MarkLogic 10.0-1, ECDH is a supported cipher for SSL/TLS communication. SSL/TLS works if an ECDH cipher is specified.

New Features in MarkLogic 10.0-2

Security Improvements

Added support for Azure Key Vault External KMS. For details, see Using MarkLogic Encryption with Microsoft Azure Key Vault in our Security Guide.

Upgraded to version 1.0.2s of the OpenSLL library.

We now use Argon2 for passphrase Key Derivation Function (KDF).

Usability Improvements

Machine Learning using the CNTK API now has support for a single CPU and GPU on Linux, as well as granular CNTK built-in privileges.

Request Monitoring has been enhanced with: support for triggers; support for a default application server on ports 8000 and 8002. For more details, see Monitoring Requests in our Query Performance and Tuning Guide.

Support for Azure Identity to access storage blob.

Support for Database names for amps.

Performance Improvements

The internal SQL Optimizer has been improved in the following areas:

  • OR operators are now more efficient
  • Support has been added for Power BI inverse filters.
  • SPARQL Query performance has been improved.

New Features in MarkLogic 10.0-3

Usability Improvements

Support for ONNX Runtime API has been added in both JavaScript and XQuery See the Machine Learning with the ONNX API chapter in our Application Developer's Guide.

Language codes are now supported in JSON content. MarkLogic now allows natural language in JSON to be tagged with a language other than the default database language.

The MarkLogic SPARQL engine now supports negated property paths as defined in the W3C 1.1 recommendations, allowing users to query graphs with more flexibility.

The granular privilege create-user-privilege has been added to enable giving users limited privileges. For more information, see Enabling Non-Privileged Users to Create Privileges, Roles, and Users in the Security Guide.

Performance Improvements

The performance has been improved in both our SQL and the SPARQL internal engines.

Other Changes

Swap space is automatically configured when running MarkLogic Server on Amazon Web Services (AWS). Swap space is configured during the system startup process with the MARKLOGIC_AWS_SWAP_SIZE configuration variable. For more details, see AWS Configuration Variables and Deployment and Startup in the MarkLogic Server on Amazon Web Services (AWS) Guide.

The CNTK API is now deprecated and may be removed in a future release. For any new Machine Learning application projects, developers should use the ONNX Runtime API embedded in our server. For more details, please see the Why Using ONNX Runtime in MarkLogic Makes Sense section in our Application Developer's Guide.

The Managed Cluster feature supports SSL-enabled clusters. For details, see The Managed Cluster Feature in the MarkLogic Server on Amazon Web Services (AWS) Guide.

New Features in MarkLogic 10.0-4

Rolling Upgrade Status Added

MarkLogic 10.0-4 now has an Upgrade tab in the Admin Interface. During an upgrade, click the Upgrade tab to view the upgrade status of each host in the cluster. For more details, see Rolling Upgrade Status in Admin UI in the Administrator's Guide.

Permissions Change for Updating Temporal Collections LSQT Properties

The permissions for changing the temporal collection LSQT properties now only requires admin/temporal rights. The scope of this change is within RMA. Previously full admin rights to the database were required.

ODBC Cursor Support

ODBC now supports cursors making it more memory efficient on the client by default. Customers should update to the latest ODBC driver.

Updates to mlcp in 10.0-4.2

Some features that have been changed in mlcp in the 10.0-4.2 release.

Max Threads

A new command line option called -max_threads refers to the maximum number of threads that run mlcp. This command line option is optional.

Thread Count

This release includes the following behavior changes designed to make mlcp smarter and achieve better concurrency:

  • Instead of using 4 as the default thread count, mlcp now conducts initial polling to identify the available server threads on the port that handles mlcp requests. mlcp then uses half of this value as the default thread count.
  • You can overwrite this calculated value by specifying -thread_count in the command line.
  • If you specify -threads_per_split, each input split will run with the number you have specified. Note, however, that the total thread count is controlled by the newly calculated thread count or, if specified, -thread_count.

New Features in MarkLogic 10.0-5

Packages by Linux Platform Updated

Updated the list of packages required for each supported Linux platform. For more details, see Supported Platforms and Appendix: Packages by Linux Platform in the Installation Guide for All Platforms.

IAM Permissions Updated

Updated the minimum required IAM permissions to create and delete a stack. For more details, see Creating an IAM Role in the MarkLogic Server on Amazon Web Services (AWS) Guide.

FULL OUTER JOIN Updated

FULL OUTER JOIN is now supported in a SQL query.

SQL functions with a null argument now return null

To comply with the SQL specification and better integrate our Tableau connector. Many SQL functions called with a null argument now return null. For instance, the following:

sql:substring()
sql:char()
sql:left()
sql:right()
sql:char-length()
sql:lower()
sql:upper()
sql:concat()

New Features in MarkLogic 10.0-6

SQL Grouping Sets

In MarkLogic 10.0-6 support for SQL keywords grouping sets, cube, rollup, and the grouping() aggregate has been added. See these APIs for more informaiton:

Optic API for Grouping Sets

In MarkLogic 10.0-6, the Optic API for grouping sets has been added. For more information about Optic, see https://docs.marklogic.com/10.0/guide/app-dev/OpticAPI.

Support for IN Operator in Optic

MarkLogic 10.0-6 now includes support for the IN operator in Optic.

For example:

where(op.in(op.col('columnName'), [1, 2, 3]))

For more information about Optic, see https://docs.marklogic.com/10.0/guide/app-dev/OpticAPI.

Query DSL for Optic API

A human-editable query language representation for the Optic API has been added to the /v1/rows endpoint in MarkLogic_10.0-6. The DSL adds a human-oriented textual representation of an Optic query without limiting the query capabilities. The human-oriented representation can be edited with text editors, displayed in diagnostic views, and so on.

The Optic API supports lossless conversion between the machine-oriented AST and human-oriented DSL representations of an Optic query. Currently, the /v1/rows endpoint is usable only by using MarkLogic client APIs or previously exported ASTs. As a result, the REST API support for Optic queries is currently machine-oriented, but becomes human-oriented with this enhancement.

op.fromSearch() / op:from-search() accessor for Optic API

MarkLogic 10.0-6 now exposes the plan:search function in the Optic API in the form of the new op.fromSearch and op:from-search functions. For more information about Optic, see https://docs.marklogic.com/10.0/guide/app-dev/OpticAPI.

Column binding supported in Optic

In MarkLogic 10.0-6, the op:bind-as operator has been added to bind a new column without affecting existing columns in the row. The bind-as operation is a new, simpler interface to the implementation for the existing op:as or op.as functions. For more information about Optic, see https://docs.marklogic.com/10.0/guide/app-dev/OpticAPI.

Support SQL payloads on /v1/rows

MarkLogic 10.0-6 now supports SQL payloads on /v1/rows. For details, see https://docs.marklogic.com/10.0/REST/POST/v1/rows.

Reactive Auto-Scaling for mlcp Import Jobs

In MarkLogic 10.0-6 mlcp supports reactive auto-scaling for import jobs. This feature maximizes the import process as a Data Hub Service cluster scales to improve performance.

New Features in MarkLogic 10.0-7

Query-Based Access Control

MarkLogic 10.0-7 supports Query-Based Access Control (QBAC) as a way to secure data access at the fundamental level in MarkLogic Server. Query-Based Access Control or QBAC can integrate with all the existing MarkLogic security features, such as Compartment Security, ELS, triples and protected collections. See Query-Based Access Control in the Security Guide for more information.

Query-Based Views

Query-Based Views (QBV) have been added in MarkLogic 10.0-7. A Query-based view is a view created from an Optic query, that can be referenced in subsequent calls to SQL or Optic. The Query-based view feature enables you to create SQL views that reference Template (TDE) views, lexicons, and SPARQL queries. For more information, see Query-Based Views in the Application Developer's Guide.

Hashing Functions Added to TDE

In MarkLogic 10.0-7, these hashing functions have been added to TDE:

  • xdmp:hash32
  • xdmp:hash64
  • xdmp:md5
  • xdmp:sha1
  • xdmp:sha256
  • xdmp:sha384
  • xdmp:sha512
  • xdmp:hmac-md5
  • xdmp:hmac-sha1
  • xdmp:hmac-sha256
  • xdmp:hmac-sha512

See Template Dialect and Data Transformation Functions in the Application Developer's Guide for more information.

Default.sjs and Index.sjs added

In MarkLogic 10.0-7, default.sjs and index.sjs have been added to the list of default modules for an application server to render.

Added Granular Privileges

These execute privileges have been added in MarkLogic 10.0-7:

  • create-data-user
  • create-data-role
  • switch-task-user

See Enabling Non-Privileged Users to Create Privileges, Roles, and Users in the Security Guide for more details.

New Functions Added to Optic API

In MarkLogic 10.0-7, op.existsJoin and op.notExistsJoin have been added to the Optic API. On release, the two functions, op.existsJoin() and op.notExistsJoin() do not perform natural joins between columns with the same identifiers - as other existing Optic join types do. Please use op.on() to specify the join condition.

Optic Redaction on Rows

Redaction on rows using the Optic API has been introduced MarkLogic 10.0-7. An Optic query can redact a column by rebinding a column to an expression. The expression can either transform the column values or generate replacement values in some other way including based on random numbers or UUIDs.

The Optic API now provides helper functions to build column rebindings for common redaction cases including maskDeterministic(), maskRandom(), redactDatetime(), redactEmail(), redactIpv4(), redactNumber(), redactRegex(), redactUsSsn(), and redactUsPhone(). See the Optic APIs at https://docs.marklogic.com/js/ordt (JavaScript) and https://docs.marklogic.com/ordt (XQuery) for more information.

Query Console Editor

In MarkLogic 10.0-7, the Query Console includes Editor Options that enable you to configure the auto-close functions for parenthesis using auto complete. You can also control indenting, matching brackets, and closing brackets. A Processing Query window displays the progress of your query as it is running. See the Query Console User Guide for details.

Improved Streaming for XQuery FLWOR Expressions

XQuery FLWOR expressions that only use "let" will now stream the results. Prior to MarkLogic 10.0-7, they would have been buffered in memory. This allows large result sets to be more easily streamed from XQuery modules.

Due to this change, code that relied on the previous behavior of buffered results from FLWOR expression with only a "let", may perform worse if the results are iterated over multiple times. This is due to the fact that once a streaming result has been exhausted, the query has to be rerun to iterate over it again.

Even prior to this change, it is best practice to treat all query calls as lazily-evaluated expressions, and only iterate over them once. If the results need to be iterated multiple times, wrap the search expression in xdmp:eager() or iterate over the results once and assign that to a new variable.

For example, in MarkLogic 10.0-7 and prior versions, the following expression would be lazily-evaluated and run the search multiple times, if the $results variable is iterated over multiple times.

let $_ := xdmp:log("running search")

let $results := cts:search(fn:collection(), cts:word-query("MarkLogic"))

This behavior has not changed in MarkLogic 10.0-7. However, prior to MarkLogic 10.0-7, the following expression would short-circuit the lazy evaluation and buffer all of the results in memory.

let $results :=
    let $_ := xdmp:log("running search")
    return cts:search(fn:collection(), cts:word-query("MarkLogic"))

In MarkLogic 10.0-7, this behavior is now consistent with the other form of the expression above and returns an iterator. The search will be run multiple times if the $results variable is iterated over multiple times.

To achieve the same buffering behavior in MarkLogic 10.0-7, wrap the cts:search() call in xdmp:eager (https://docs.marklogic.com/xdmp:eager) as follows:

let $results :=
    let $_ := xdmp:log("running search")
    return xdmp:eager(cts:search(fn:collection(), cts:word-query("MarkLogic")))

To help understand if a variable will stream or not, the xdmp:streamable function (https://docs.marklogic.com/xdmp:streamable) was also added in MarkLogic 10.0-7.

For more information about lazy evaluation in MarkLogic, see the following resources:

New Features in MarkLogic 10.0-8

Admin UI Access

A new role for accessing the Admin UI has been added in MarkLogic 10.0-8. The admin-ui-user role has been added to enable read-only access to the Admin UI, without providing access to data, to security configuration, or write access to Server configuration. See the Administrator's Guide for more details.

Lightweight Telemetry

MarkLogic 10.0-8 includes a lightweight version of Telemetry, leveraging the existing implementation of Telemetry. It only collects and sends essential information from customers to provide better understand issues and provide useful suggestions. This feature is on by default. See the Telemetry chapter in the Monitoring MarkLogic Guide for more details.

Query Console Improvements

A number of improvements to Query Console have been made in MarkLogic 10.0-8. Editor Options provide auto-complete parameters, along with auto-indent, auto-close and auto-match functions for brackets. The Editor Options allows the configuration of certain conditions (time elapsed, lock count, or read size in bytes), which when met, will lead to auto cancellation of the queries. A separate Processing Query window shows the query plan in a graphical interface. See the Query Console User Guide for more information.

Updated TDE and ELS Behavior

In MarkLogic 10.0-8, the TDE indexing process has been changed so that rows with non-nullable, ELS-protected values are added to the index, rather than skipped. At runtime, a row is skipped if the value for a mandatory (non-nullable) column from that row is missing. ELS-protected triples will display as missing values if the user doesn't have permission to see them. However, rows are only skipped in this way if the column is accessed in the query - otherwise the data isn't read, and the row isn't skipped.

sql:bucket and op:bucket-group Functions Added for the Optic API

MarkLogic 10.0-8 includes a new built-in function that returns the position after which a value would be added to an ordered sequence. This enables efficient bucketed facets in the Optic API for parity with JSearch and the Search API. See sql:bucket and op:bucket-group for more information.

ODBC Supports Querying from a Different Database

The database name is now acknowledged when connecting over ODBC. Be sure to install the latest ODBC driver to allow this capability.

Optic from SPARQL Accessor Now Takes Third Parameter

In MarkLogic 10.0-8, the op.fromSPARQL or op:from-sparql accessor now takes the third parameter, options could be dedup and base.

SPARQL REST API Adds dedup Option

The SPARQL REST APIs GET /v1/graphs/sparql and POST /v1/graphs/sparql in MarkLogic 10.0-8 include a new de-duplication option. The dedup option is dedup=off and dedup=on. The default is dedup=on.

Optic Sample by Function

In MarkLogic 10.0-8, Optic includes a sample by function (AccessPlan.prototype.sampleBy or op:sample-by). This function samples rows from a view, or from a pattern match on the triple index.

Purge Journal Archive Option Added

In MarkLogic 10.0-8, the incremental backup feature in the Admin GUI now includes the option to select Purge Journal Archive. The Configured Backup status will reflect the value. The user is able to create a scheduled backup with purge journal archive by setting this option to true.

Use View-Qualified Wildcards in SQL Select Column Lists

MarkLogic 10.0-8 supports the following uses of SELECT * with SQL:

SELECT *
SELECT <schema>.<view>.*
SELECT <view>.*

This feature supports qualified wildcards in column lists. The asterisk selects visible columns. Hidden columns will still need to be listed explicitly, if they need to be selected as part of the query.

Query Plan Viewer Added to Query Console

In MarkLogic 10.0-8 you can use Query Console to view the query plan for a SQL or SPARQL query. Two types of query plan are available: the estimated plan and the actual plan. Tooltips provide information about the elements of query plan. See Viewing Query Plans in the Query Console User Guide.

In MarkLogic 10.0-8, the Query Plan Viewer does not work with the Windows IE 11 browser.

New cts:column-range-query Function

MarkLogic 10.0-8 includes a new helper function, cts:column-range-query, which constructs a triple range query for a row column. See cts:column-range-query for more information.

New Features in MarkLogic 10.0-9

Namespace Bindings in op.xpath

XPaths on XML elements must be able to specify bindings between namespace prefixes and URIs for namespaced steps. In MarkLogic 10.0-9, op.xpath now supports a namespace map that is added to the in-scope namespace bindings, in the evaluation of the path (and in the AST for the Java and Node.js APIs on the client).

Column Inspection with Optic API

In MarkLogic 10.0-9, the Optic API can be used to inspect names, data types, and the nullability of columns at the Optic level, including on the Java and Node.js clients.

Upgrade OpenSSL to 1.0.2zb

In MarkLogic 10.0-9 we recommend that you upgrade your OpenSSL software to 1.0.2zb to address security vulnerabilities.

Reindexing Triggered if Path Range Indexes Contain Predicates on an Unqualified Axis

In MarkLogic 10.0-9, a bug was fixed where predicates on an unqualified axis are not hashed when setting up path range indexes. An example of these axes is /Node[schema="abc"]. Customers with these types of path range index settings will experience an automatic reindexing on their databases after upgrading to 10.0-9.

Improved Prefix String Lookups

In MarkLogic 10.0-9, a SQL LIKE/GLOB query will run faster if the pattern is a prefix (for example Prefix%) and the left hand side is a column. Optic Queries with where op.fn.startsWith or SQL with FN_STARTS_WITH will improve if the first argument is a column, and the second argument is a prefix. Additionally, you can use op.sql.like and op.sql.glob functions in the Optic API, and strstarts in SPARQL.

HTTP Chunking and Compression

HTTP ChunkingCompression feature is introduced in MarkLogic 10.0-9. The xdmp:set-response-chunked-encoding and xdmp:set-response-compression functions implement parts of the HTTP 1.1. chunk transfer encoding for responses. The compression function uses stream processing. For example when the REST extension sends 1 GB of data back to the client, the 1 GB of data is not compressed all at once, but each network buffer is compressed individually. Each network buffer has a maximum size of a few hundred KBs. With chunking, you get an HTTP Trailer with a content checksum. If an error occurs while streaming the result, the HTTP Trailer provides the error-code/message. This is beneficial for errors that occurs after the HTTP OK code has been sent, enabling you to figure out what went wrong. This feature will improve the overall performance for large responses and maintains the connection (for example, avoiding reconnects).

Request Monitoring for the ODBC Server

Request monitoring is supported for the ODBC App server in MarkLogic 10.0-9. The number of rows and bytes sent over ODBC requests will be recorded. Request cancellation is enabled for the ODBC server as well. See ODBC Request Monitoring and Cancellation in the Administrator's Guide for details.

log4j Updates to Core Server and MLCP

To address the security vulnerability found in log4j 1.2.17, in MarkLogic 10.0-9 both the core MarkLogic Server and mlcp have been upgraded to log4j 2.17.1. MarkLogic 10.0-9 has the update, and mlcp has been updated in the mlcp repo. The log4j.properties file under MLCP_HOME/conf has been replaced by log4j2.xml. For more information, see Enabling Debug Level Messages in the mlcp User Guide.

Update for tde:template-batch-insert

In MarkLogic 10.0-9, the updated tde:template-batch-insert function validates and inserts multiple templates. It can insert templates into the Schemas database, even if the insert is fired from some other database. The tde:template-batch-insert function can also insert templates into the TDE collection, in addition to collections specified for each template before inserting. It validates each new template against all other new and existing templates with same schema/view-name. See the tde:template-batch-insert API for more details.

Query Console Includes Optic Query DSL Dropdown in Query Type

Query Console in MarkLogic 10.0-9 now supports the ability to run Optic Query Domain Specific Language (DSL), and produce estimated and actual query plans. See a plan's result in order to test an Optic query before deploying it to production clusters. Enter an Optic Query DSL in Query Console and see its estimated and actual plan to improve the performance of your Optic query.

QueryPlan Viewer Now Works on IE 11

The QueryPlanViewer now works on IE in MarkLogic 10.0-9. This feature did not work with IE 11 in MarkLogic 10.0-8, but this has been fixed in MarkLogic 10.0-9.

Enhanced Hugepage Allocation for Container Envrionments

In MarkLogic 10.0-9, hugepage allocation for containers has been enhanced for mult-container settings. Previously the first MarkLogic container brought up would consume all available hugepages on the host. Now a detect memory limit is set for each container, to give 3/8 of the memory limit for huge pages. The feature also allows passing in an environment variable to override huge page allocation for a container.

Default Assignment Policy for New Databases

In MarkLogic 10.0-9, the default assignment policy setting for new databases is Bucket. Databases created with previous versions of MarkLogic will retain their original assignment policy following an upgrade.

Cloud Formation Template Changes in 10.0-9.2

AWS Classic Load Balancer Removed from Single Zone Deployments

Since AWS is retiring the Classic Load Balancer (CLB) as of August 15, 2022, the CLB has been removed for single-zone deployments in the MarkLogic CloudFormation templates. The URL in the outputs of the CloudFormation stack is now replaced with a private DNS name, which can be used to access the MarkLogic cluster.

Python Upgrade for Lambda Functions in the MarkLogic CloudFormation Templates

The lambda functions in MarkLogic CloudFormation templates used on AWS are now configured to use Python 3.9. AWS has scheduled the end of support for Python 3.6 by July 2022.

Clear Browser Cache Before Using Query Console

A fix for the JQuery vulnerability issue has been made in MarkLogic 10.0-9.2. Due to this fix, users might have to clear the browser cache before using either the Query Console and/or the Monitoring dashboard. Several JQuery libraries have been removed in MarkLogic 10.0-9.2 to fix the vulnerability.

If the browser cache is not cleared before using the Query Console or the Monitoring dashboard in MarkLogic 10.0-9.2, you might see behaviors like these:

  • Errors reported in browser's console window
  • Not seeing any results in the result pane after running a query

10.0-9.2 Patch Release Not Available on CentOS 8 Azure VM Image

As of December 31 2021, CentOS has ended support for CentOS8. As a result of this, MarkLogic Server versions 10.0-9.2 and later will not be available on CentOS8 Azure VM Images.

Improvements in the 10.0-9.5 Patch Release

XML SVM Classifier can now process an extremely large training set

Prior to 10.0-9.5, when running a cts:classify against an extremely large training dataset, the SVM classifier may have caused a segmentation fault. This is resolved in 10.0-9.5.

Improved logging for merging and stand file management

If obsolete stands are not marked for deletion for an extended period of time or fail to delete, the following log messages will appear in the MarkLogic error log:

XDMP-OBSOLETESTANDNOTDELETED 

Obsolete stand not deleted - As a normal part of the operations in the server, stands are sometimes marked obsolete so they can be deleted later. For example, if stands are merged into a new stand, the old stands are marked obsolete. Typically, these stands will be deleted within seconds or minutes but, if there are long-running transactions or other activities like backups still using obsolete stands, they cannot be deleted until those processes complete. If obsolete stands are not deleted within an hour, the server will log this message for informational purposes.

If the system has long-running transactions that are expected or backups that take more than an hour, these messages can be ignored. If not, these messages could be a reflection of other problems in the system and they can be used to help diagnose when unexpected long-running processes may have started to occur.

XDMP-RECURSIVEREMOVEFAILED 

Recursive remove of a directory failed - An error has occurred when trying to recursively remove a directory.

This is an indication that there is likely a problem with the underlying file system. Inspect the file system on which the error occurred and take action as necessary to address the problem.

MLCP import now works on Windows with OpenJDK 11

Prior to 10.0-9.5, there was an incompatibility between an old Hadoop library and Java 11. The Hadoop libraries have been upgraded in 10.0-9.5 to address this and other issues.

Improved XCC support for Azure Application Gateway, AWS Application Load Balancer and other 3rd party balancers

The XCC client library now properly handles the Connection:close response header. Prior to 10.0-9.5, applications that use XCC, such as MLCP, may have seen ServerConnectionExceptions caused by these responses when running against MarkLogic through an AWS ALB, Azure Application Gateway or other load balancers.

Backup with Journal Archiving to Microsoft Azure Blob Storage is now supported

Prior to 10.0-9.5, in the event that a backup is configured with Journal Archiving and a MarkLogic process restart takes place, forests may remain in a mounted state and are unable to come back up. This has been fixed in 10.0-9.5.

Addressed CodeMirror vulnerability in Query Console

The CodeMirror package used by Query Console was upgraded from version 5.11.0 to 5.65.8 to address CVE-2020-7760.

Addressed security vulnerabilities in MLCP 3rd party libraries

A number of 3rd party libraries that MLCP depends on were updated to address security vulnerabilities. The following vulnerabilities were addressed by these upgrades:

CVE-2009-2625 CVE-2019-0232 CVE-2021-25329 CVE-2021-4104
CVE-2013-4002 CVE-2019-10172 CVE-2021-29425 CVE-2021-42392
CVE-2015-2575 CVE-2019-17563 CVE-2021-29505 CVE-2021-43859
CVE-2015-5262 CVE-2019-17571 CVE-2021-33036 CVE-2022-21363
CVE-2016-3086 CVE-2019-2692 CVE-2021-35515 CVE-2022-22950
CVE-2016-5393 CVE-2020-11979 CVE-2021-35516 CVE-2022-22968
CVE-2016-5725 CVE-2020-13956 CVE-2021-35517 CVE-2022-22971
CVE-2016-6811 CVE-2020-15250 CVE-2021-36090 CVE-2022-23221
CVE-2017-3586 CVE-2020-15522 CVE-2021-36373 CVE-2022-23302
CVE-2017-5637 CVE-2020-1935 CVE-2021-36374 CVE-2022-23305
CVE-2018-1000632 CVE-2020-1945 CVE-2021-37404 CVE-2022-23437
CVE-2018-10237 CVE-2020-26939 CVE-2021-39140 CVE-2022-25168
CVE-2018-11771 CVE-2020-36518 CVE-2021-39144 CVE-2022-25647
CVE-2018-1304 CVE-2020-8908 CVE-2021-39146 CVE-2022-26612
CVE-2018-1336 CVE-2020-9484 CVE-2021-39148 CVE-2022-33980
CVE-2018-8012 CVE-2020-9488 CVE-2021-39150 CVE-2022-38749
CVE-2018-8014 CVE-2020-9492 CVE-2021-39152 CVE-2022-38750
CVE-2018-8088 CVE-2021-22569 CVE-2021-39153 CVE-2022-38751
CVE-2019-0201 CVE-2021-23463 CVE-2021-39154 CVE-2022-38752

New Features in MarkLogic 10.0-10

MarkLogic 10.0-10 is a maintenance release with no new features but there are many bugfixes and a number of performance improvements. The 10.0-10 release containes many fixes that were in the 11.0.0 release as well as some that will be in future releases of MarkLogic 11.

The full list of bugs fixed in 10.0-10 can be found on the Marklogic Support Portal at https://help.marklogic.com/Bugtrack/List. Some highlights include:

AWS Managed Cluster and Cloud Formation Template Improvements

Many new instance types can now be selected when launching the MarkLogic 10.0-10 AMI.

EBS gp3 volumes can now be provisioned via the Cloud Formation Template.

Previously, using special characters in the admin password would cause issues with instances launched via the Cloud Formation Template. This is no longer an issue in 10.0-10.

An improvement was made to prevent the Auto Scaling group from provisioning a new volume for instances when previous instances were still shutting down.

EBS volumes are now mounted with 'nodev' flag and the /var/tmp/marklogic.host file is written with permissions of rw-rw-rw-.

Certificate Handling in the Admin UI Has Been Improved

Prior to MarkLogic 10.0-10, importing a certificate with a passphrase would succeed but MarkLogic could not use the certificate without the passphrase. Starting with MarkLogic 10.0-10, an attempt to import a certificate with a passphrase will throw an exception to make it clear that the certificate cannot be used. Full support for certificates with passphrases will be added in a future version.

Updating a certificate template name or modifying app server settings that use certificate templates with valid non-temp certificates no longer regenerates self-signed certificates.

Query Console Bugfixes

A number of bugs were fixed in Query Console. These are bugs that were fixed in 11.0.0 and are now backported to 10.

xdmp.filesystemFileDelete() is Now a Documented Function

The previously undocumented function xdmp.filesystem.FileDelete is now documented and supported.

MLCP Now Supports -split_input with Multibyte UTF-8 Content

Prior to MarkLogic 10.0-10, MLCP could not be used with the split option with multibyte content. This now works but it requires any multibyte content to be UTF-8 encoded.

Starting Backups No Longer Detects Indexes

Prior to MarkLogic 10.0-10, if a backup is started on any database in the cluster, the index detection process will be run for all databases. This can impact performance of queries in databases that are not being backed up. This no longer occurs in MarkLogic 10.0-10.

Element Level Security Bugfix

Prior to MarkLogic 10.0-10, if multiple protected paths are protecting the same elements but different attribute values, some protected elements fail to be protected while some other protected elements fail to show up to users who do have the permission. This has been fixed in MarkLogic 10.0-10.

Optic API Performance Improvements

MarkLogic 10.0-10 introduces performance improvements for optic queries that use op.fromSearchDocs(), op.joinDocCols(), or op.fromLexicons(). Optic queries that use these functions may be significantly more performant in 10.0-10 but overall performance will depend on the complexity and structure of the complete Optic query.

Install on macOS Running on Apple M1 Processors

MarkLogic 10.0-10 installation now works on macOS running on Apple M1 processors. MarkLogic does not run natively on ARM, but it works well running under Rosetta 2 emulation. Prior to MarkLogic 10.0-10, the install scripts would error out indicating MarkLogic could only be installed on macOS running on an Intel chipset.

Improvements in the 10.0-10.1 Patch Release

MarkLogic connections to AWS KMS and Azure KeyVault now use TLSv1.0

AWS and Azure have discontinued the use of TLS 1.0 and 1.1 and now require TLS 1.2. MarkLogic 10.0-10.1 now supports TLS 1.2 for communication with an external KMS so once AWS and Azure require TLS 1.2, MarkLogic will continue to be able to communicate with AWS KMS or Azure Key Vault.

Improvements in the 10.0-10.2 Patch Release

op.fromSearch() and op.fromSearchDocs() security constraints

Starting with MarkLogic 10.0-6, Optic queries using either op.fromSearch()/op:from-search or op.fromSearchDocs()/op:from-search-docs() operators may return more documents than theoretically accessible. This has been fixed in 11.0.3 as well as 10.0-6.6, 10.0-7.4, 10.0-8.5, 10.0-9.7, and 10.0-10.2.

New Features in MarkLogic 10.0-11

MarkLogic 10.0-11 is a maintenance release with no major new features but there are a number of improvements to the CloudFormation Template, many bug fixes, and a number of performance improvements. The 10.0-11 release contains many fixes that were in the 11.1.0 release as well as some that will be in future releases of MarkLogic 11.

AWS CloudFormation Template (CFT) Improvements

IMDSv2 support

MarkLogic 10.0-11 adds support for version 2 of the EC2 Instance Metadata Service (IMDSv2). This is the latest and most secure option for managing and accessing EC2 instance metadata and is now the default when launching MarkLogic from version 10.0-11 of the CloudFormation Templates. The MarkLogic AMIs still default to IMDSv2 for backward compatibility but the IMDSv2 option is set to "required" by default in the 10.0-11 and later CFTs.

In order to use MarkLogic Server AMIs before 10.0-11 with the new templates, the templates will need to be modified to set IMDSv2 to "optional" as IMDSv2 is not supported in earlier versions of the MarkLogic AMI. See the AWS Security Blog for more details about IMDSv2.

Launch templates

Starting with MarkLogic 10.0-11, the MarkLogic CloudFormation Templates replace the use of Launch Configurations with Launch Templates. This ensures that MarkLogic CFT users can make use of all of new EC2 features now available in AWS Launch Templates. See the AWS Compute Blog for more details about the introduction of AWS Launch Templates.

The use of Launch Templates in the CFTs requires that new privileges be added to the the IAM role used to launch the CloudFormation stacks. Add the following privileges to the IAM role used to launch MarkLogic clusters via the CFTs:

  • ec2:CreateLaunchTemplate
  • ec2:DescribeLaunchTemplates
  • ec2:DeleteLaunchTemplate
  • ec2:ModifyLaunchTemplate
  • Resource:arn:aws:ec2:::launch-template/*

See Creating an IAM Role in Getting Started with MarkLogic Server on AWS for the complete list of additional privileges required.

New Checks and Warnings When Restoring Databases

Prior to MarkLogic 10.0-11, when using the Admin UI to restore a database backup and that includes the Security database, if the backup of the Security database does not include certificate templates or external security configurations that are in use by Admin app server (they may have been added after the backup was taken), no warning that the Admin UI could become inaccessible is provided to the user.

In MarkLogic 10.0-11, if the Security database will be restored with the database, MarkLogic will check if Admin app server is using certificate templates or external security configurations. If they are, a warning is provided to the user along with instructions to reconfigure the Admin app server to ensure that it will be accessible after the pending restore.

Automatic Cache Sizing for Hosts with up to 512 GB of RAM

In versions prior to MarkLogic 10.0-11, automatic cache sizing only supports allocation of caches at the recommended ratios for hosts with up to 256 GB of RAM.

Automatic cache sizing will now allocate caches using the recommended ratios for hosts with up to 512 GB of RAM. Manual cache sizing is required for hosts with more than 512 GB of RAM.

Bugs Fixed in MarkLogic 10.0-11

For versions prior to MarkLogic 10.0-11, the full list of fixed bugs can be found on the Marklogic Support Portal at https://help.marklogic.com/Bugtrack/List. Starting with MarkLogic 10.0-11, fixed bugs will no longer be listed on the support portal as we are in the process of migrating to a new support system. The full list of bugs fixed in 10.0-11 is listed below.

BUG ID Description
BUG-60403 Incorrect document filtering with multiple LIKE conditions

When using SQL queries with multiple LIKE conditions, it is possible that the query will return the wrong results.

BUG-56434

"Automatic" cache sizing only works for hosts with up to 256 GB of RAM

Automatic cache sizing does not allocate caches using the recommended ratios for hosts with more than 256 GB of RAM.

BUG-58187

Journal Archiving on incremental backup does not recover when the disk corruption occurs

When there is a disk failure while archiving journals, journal archiving will not resume until a full backup is performed.

BUG-60269

Data missing after restore from backup and forests failover to replicas

There is a potential for data to be lost if a database with local disk failover configured is restored from a backup and the forests fail over to the replicas when the restore completes.

BUG-60265

BUG-57915

Restoring the security database can break https app servers

When restoring a database backup that includes the Security database, if the backup of the Security database does not include certificate templates or external security configurations that are in use by Admin app server (they may have been added after the backup was taken), no warning that the Admin UI could become inaccessible is provided to the user.

BUG-60014

The V8 engine causes segfaults under heavy load or overloaded infrastructure

Under heavy load, the V8 engine can timeout when instantiating JavaScript modules and cause MarkLogic to segfault.

MLE-1747

CTS queries in JSON payloads are not correctly created by RMA in partition-query and alerting

When doing a PUT of a partition query with an array of CTS queries in JSON format via RMA, only the last query in the array of the queries will be stored in the database.

When doing a POST of an alert rule or a PUT of alert rule properties with an array of CTS queries in JSON format, only the query in the array of the queries will be stored in the database.

BUG-60211

BUG-56402

xdmp:document-filter() does not detect the file type in some cases

When running xdmp:document-filter() against a binary node that is only in memory (has not be retrieved from the database), the file type may not be detected correctly because it does not know the file extension which otherwise comes from the document URI.

MLE-12324

sec:amps-change-modules-database throws an XDMP-COMPARE error

The sec:amps-change-modules-database() function does not correctly cast the database ID and throws an XDMP-COMPARE error.

BUG-60545

Performance regression for cts:triples() with triple range queries

An optimization for certain cts:triples() patterns that was added in 10.0-9 has caused other triple range queries with cts:triples() queries to be slower.

BUG-60342

SSL Secure Client-Initiated Renegotiation is not disabled

SSL Secure Client-Initiated Renegotiation option cannot be disabled.

BUG-60106

Replica databases show size zero in the Admin UI

In the Admin UI, when database replication is used, replica databases where database encryption is enabled report that the encrypted size is zero even though the data is loaded.

BUG-60328

sjs referencing XQuery modules with the same name may use the wrong module

When multiple XQuery modules with the same filename, but different paths exist in the modules database, SJS modules that reference the XQuery modules using the require function may not retrieve the correct module relative to the calling module.

BUG-60393

mlcp has CWE-400 due to 3rd party dependency

mlcp has CWE-400 due to dependency on: com.fasterxml.jackson.core: jackson-core: 2.14.0-rc3

BUG-60394

mlcp has a CWE in 3rd party dependencies

mlcp has CWE-611 due to dependency on: org.apache.hadoop: hadoop-common : 3.3.4, org.apache.hadoop : hadoop-hdfs-client : 3.3.4.

BUG-60363

cts:triple-range-query() can throw "Critical: Bad malloc"

cts:triple-range-query() throws "Critical: Bad malloc" when too many triple values are provided.

BUG-60185

Backup to S3 with object lock and compliance retention does not work

When the object lock and compliance retention is activated on S3 buckets, MarkLogic is unable to backup data to them because of a missing HTTPS header.

BUG-60131

BUG-60480

Select all via ctrl+a is not working in Query Console

In Query Console, select all via ctrl+a is not working correctly when selecting in the query buffer. Once a query is run, ctrl-a stops working in the query buffer. ctrl+a in the results panel only selects one line if a line is clicked on when it should select all of the results.

MLE-508

Top of Form

xdmp:transaction-locks() times out in a cluster when transaction has no locksBottom of Form

When running xdmp:transaction-locks() against a cluster, if there are no transactions, the call will hang and eventually timeout.

BUG-60396

The character "&" is being escaped as "&amp;" in the query console "table" output

When running a SQL or SPARL query that outputs a table in query console, & characters are escaped as &amp;.

BUG-60378

Private keys are not stripped from support dump when requesting all versions of configuration files

The cleansing process that strips out private keys from support dumps does not strip out private keys when Version is set to All. The private key is only stripped from the Latest configuration files.

BUG-57043

sem:query-results-serialize produces invalid JSON when serializing constructed triples

When using sem:query-results-serialize to serialize constructed triples as JSON, the JSON produced may be invalid.

BUG-60178

The XDMP-FORESTTAKEN error reports incorrect host ID when the forest label is corrupt

If the forest label file is corrupt, the XDMP-FORESTTAKEN error may be thrown, incorrectly indicating that a host has taken over a forest. This hides actual error which is that the forest label file has been corrupted.

BUG-60063

POST /manage/v2/databases/{id|name}/partition-queries adds spurious namespace to element name

When posting certain queries via REST, there is an extra namespace added to the element name when using an element-range-query.

CVEs Fixed in MarkLogic 10.0-11

CVE-2023-45853 CVE-2024-26308 CVE-2023-35116 CWE-770
CWE-502 CVE-2023-2976 CVE-2007-1157 CVE-2022-45688
CVE-2023-5072 CVE-2023-6378 CVE-2023-4498 CWE-400
CWE-611
« Previous chapter
Next chapter »