Release Notes — Chapter 3

Compatibility With New Versions of Hadoop

MarkLogic features that leverage Apache Hadoop MapReduce and HDFS can now be used with the following Hadoop distributions:

• Cloudera's Distribution Including Apache Hadoop (CDH) version 5.4
• Hortonworks Data Platform (HDP) version 2.2

CDH version 4.3 is no longer included in the list of compatible distributions.

This change affects mlcp (MarkLogic Content Pump) in distributed mode, the MarkLogic Connector for Hadoop, and the use of HDFS for forest storage.

MarkLogic Connector for Hadoop Supports JavaScript Queries

You can now express input and output queries in either XQuery or Server-Side JavaScript. That is, you can use Server-Side JavaScript in the values of the configuration properties mapreduce.marklogic.input.query and mapreduce.marklogic.output.query. Previously, you could only use XQuery to express input and output queries. Use the properties mapreduce.marklogic.input.queryLanguage and mapreduce.marklogic.output.queryLanguage to indicate which scripting language is used in your input or output query. The default query language is XQuery.

For details, see the MarkLogic Connector for Hadoop Developer's Guide.

XCC Support for JSON Content

You can now use XCC and the Jackson libraries to insert, update, and read JSON content. For example, you can create a ContentFactory specifically for JSON, similar to the following:

JsonNode node = ...;
ContentCreateOptions options = new ContentCreateOptions();
options.setFormat(DocumentFormat.JSON);
Content content = ContentFactory.newContent(uri, node, options);

For details, see Working With JSON Content in the XCC Developer's Guide.

HDFS Client Bundles Available for Download

Previously, using HDFS for forest storage required you to assemble a set of Hadoop HDFS JAR files or install Hadoop on each MarkLogic host containing a forest on HDFS (or to install Hadoop in a well-known location).

You can now download a pre-packaged Hadoop HDFS client bundle from http://developer.marklogic.com/products/hadoop and install this bundle on your MarkLogic hosts. A bundle is available for each supported Hadoop distribution. Use of one of these bundles is required if you use HDFS for forest storage.

The availability of these bundles also changes how and where MarkLogic looks for the JDK and Hadoop libraries.

For details, see HDFS Storage in the Query Performance and Tuning Guide.

Arithmetic and Comparison Methods for dateTime and Duration Objects in Server-Side JavaScript

Beginning with 8.0-3, there are methods that allow you to compare, add, subtract, multiply, and divide duration and date objects in Server-Side JavaScript. These methods allow you to take advantage of richly typed date values available in MarkLogic from Server-Side JavaScript. For details of these new APIs, see JavaScript Duration and Date Arithmetic and Comparison Methods in the JavaScript Reference Guide.

Expanded instanceof Support in Server-Side JavaScript

Beginning in 8.0-3, you can use the instanceof operator in Server-Side JavaScript to test for any of the MarkLogic-typed values, including ValueIterator, cts.query, and so on. For details, see JavaScript instanceof Operator in the JavaScript Reference Guide.

New Features in the Client APIs

The REST Client API, Java Client API, and Node.js Client API now support deleting multiple documents by URI in a single operation.

• REST: Pass multiple uri parameters to the DELETE /v1/documents method.
• Java: Pass multiple URIs to DocumentManager.delete.
• Node.js: Pass multiple URIs to Documents.remove or DatabaseClient.remove.

The Java Client API now includes support for the extract-document-data query option on search operations. Use this option with QueryManager.search to include sparse document projections in your search results. Previously, this capability was only available for multi-document reads. For details, see Extracting a Portion of Matching Documents in the Developing Applications With the Java Client API.

Server-Side JavaScript Features

MarkLogic 8.0-4 includes the following Server-Side JavaScript new features:

• Server-Side JavaScript Search API (jsearch)
• cts:parse
• Alerting API JavaScript Enhancements
• Thesaurus and Spelling API Enhancements

Semantics Enhancements

MarkLogic 8.0-4 includes the following enhancements to Semantics:

• SPARQL Negation
• Java Client API Semantics Enhancements
• MarkLogic Sesame API
• MarkLogic Jena API
• Semantics REST API Additions
• mlcp Enhancements

Management REST API Enhancements

The REST Management API has been expanded to include new alerting, mimetypes, and support endpoints.

Geospatial Search Enhancements

MarkLogic 8.0-4 introduces the following enhancements for working with and searching geospatial data:

• Converting To/From Common Representations
• New Geospatial Utility Functions
• Support for More Recent KML and GML Versions

geo:parse

geo.parse

geo:validate-wkt

geo.validateWkt

geo:parse-wkb

geo.parseWkb

geo:to-wkb

geo.toWkb

geo:validate-wkb

geo.validateWkb

geogml:parse-gml

geogml.parseGml

geogml:to-gml

geogml.toGml

geokml:parse-kml

geokml.parseKml

geokml:to-kml

geogml.toKml

georss:parse-georss

georss.parseGeorss

georss:to-georss

georss.toGeorss

geojson:parse-geojson

geojson.parseGeojson

geojson:to-geojson

geojson.toGeojson

geo:polygon-to-linestring

geo.polygonToLinestring

geo:linestring-reverse

geo.linestringReverse

geo:linestring-concat

geo.linestringConcat

geo:region-affine-transform

geo.regionAffineTransform

geo:region-approximate

geo.regionApproximate

geo:region-clean

geo.regionClean

geo:circle-polygon

geo.circlePolygon

geo:remove-duplicate-vertices

geo.removeDuplicateVertices

geo:count-vertices

geo.countVertices

geo:count-distinct-vertices

geo.countDistinctVertices

geo:interior-point

geo.interiorPoint

geo:distance-convert

geo.distanceConvert

cts:match-regions

cts.matchRegions

Support For Red Hat Enterprise Linux 7

MarkLogic 8.0-4 includes support for RHEL 7. The RHEL 7 package is separate from the RHEL 6 package on developer.marklogic.com. For details on MarkLogic platforms, see Supported Platforms, and for details on installation see the Installation Guide.

Clock Skew Timeouts

XDQP connections between hosts in either a local or foreign cluster will now drop if a host's clock is skewed by more than the host timeout. Attempts to connect will result in a warning message in the log when the first connection attempt is rejected, and every hour after that.

The host timeout is either from the host's group if it is an intra-cluster connection, or from the foreign cluster configuration if it is an inter-cluster connection.

Node.js Client API Supports Searching XML with QBE

You can now search XML documents with using Query By Example when using the Node.js Client API. For details, see Querying XML Content With QBE in the Node.js Application Developer's Guide.

New Features in mlcp

The mlcp command line tool added the following new capabilities in MarkLogic 8.0-4:

• Improved support for loading semantic triples and quads through the new -output_graph and -output_override_graph options. For details, see Loading Triples in the mlcp User Guide.
• The new -query_filter option enables you to select documents to export or copy using a cts query. For details, see Controlling What is Exported, Copied, or Extracted in the mlcp User Guide.
• Create JSON documents from delimited text files, such as CSV files, by using -document_type json with -input_file_type delimited_text. For details, see Creating Documents from Delimited Text Files in the mlcp User Guide.
• Create JSON documents from aggregate JSON files (one JSON object per line). For details, see Creating Documents from Line-Delimited JSON Files in the mlcp User Guide.
• Generate a report of the mlcp version, supported MarkLogic versions, and dependent software versions found in your environment using the new 'version' command. For example:

  $ mlcp.sh version
  ContentPump version: 8.0
  Java version: 1.7.0_45
  Hadoop version: 2.6.0
  Supported MarkLogic versions: 6.0 - 8.0

• Improved error reporting and job output summary. For details, see Understanding mlcp Output in the mlcp User Guide.

MapR Support in mlcp and the Hadoop Connector

You can now use MapR as your Hadoop implementation with mlcp in distributed mode and with the MarkLogic Connector for Hadoop.

Using mlcp with MapR requires special setup. For details, see Required Software and Using mlcp With MapR in the mlcp User Guide or .

Ability to Access and Modify Metadata in a Transformation

Collections, permissions, document quality, and temporal collection metadata specified by the client are now available to a transformation function via the context parameter. In addition, a transformation function can set collections, permissions, quality, and temporal collection metadata for its output document(s).

For details, see Transforming Content During Ingestion in the mlcp User Guide.

New Server-Side Transaction Controls

As of MarkLogic 8.0-7, you can configure the commit mode (auto or explicit) and transaction type (query, update, or auto) independently when configuring a new transaction. This change manifests in the following ways:

• The xdmp:update XQuery prolog option accepts a new value, 'auto', which specifies that MarkLogic should determine the transaction/statement type (query or update) through static analysis. The pre-existing value 'false' now means the transaction/statement type is query. Use this option plus xdmp:commit instead of the now-deprecated xdmp:transaction-mode prolog option.
• A new XQuery prolog option, xdmp:commit, has been added for specifying whether a main module should run as a single-statement, auto-commit transaction (the default) or a multi-statement explicit-commit transaction. Use this option plus xdmp:update instead of the now-deprecated xdmp:transaction-mode prolog option.
• New commit and update options have been added to the functions listed in the table below. Use these in preference to the transaction-mode option, which has been deprecated.

The following functions support the new commit and update options. For more details, see the function reference documentation for xdmp:eval (XQuery) or xdmp.eval (JavaScript).

For more details on the new capabilities, see Understanding Transactions in MarkLogic Server in the Application Developer's Guide, and xdmp:update and xdmp:commit in the XQuery and XSLT Reference Guide.

For details on transitioning from the old transaction controls to the new ones, see the following topics:

• Changes to xdmp:update XQuery Prolog Option
• xdmp:transaction-mode XQuery Prolog Option Deprecated
• Deprecation of transaction-mode Option to xdmp:eval

XCC: New Session Methods for Transaction Control

In MarkLogic 8.0-7, the following methods have been added to the Session class for configuring transactions and querying transaction configuration:

• Session.setAutoCommit and Session.isAutoCommit
• Session.setUpdate and Session.getUpdate

You should use these methods rather Session.setTransactionMode, which has been deprecated. For details, see XCC Session.setTransactionMode is Deprecated.

Session.setAutoCommit controls whether requests submitted during the session run in a transaction with auto-commit semantics (the default) or explicit commit semantics. Executing a request with commit set to explicit starts a multi-statement transaction.

Session.setUpdate controls whether requests submitted during the session run in a query transaction, an update transaction, or if the transaction type should be automatically detected by MarkLogic through analysis of the submitted code. Auto detection is the default behavior.

Note that if you override the Session transaction configuration in an ad hoc query, the behavior differs depending on whether you configure the session using setTransactionMode or setAutoCommit and setUpdate. With setAutoCommit and setUpdate, the transaction configuration reverts to the Session settings once the transaction involving the override completes. With setTransactionMode, the override persists and affects future transactions unless you explicitly change it.

mlcp: Greater Control over Host Connections

You can now specify multiple hosts for mlcp to connect to during import, export, and copy jobs. Used by itself, this feature enables mlcp to fall back an alternative host if the initial host is not available.

You can also use this capability in conjunction with the new -restrict_hosts option to prevent mlcp from connecting to any hosts except the ones on the initial host list.

For more details, see Controlling How mlcp Connects to MarkLogic in the mlcp User Guide.

REST Management API: Advancing LSQT

You can use the following new method of the REST Management API to advance LSQT on a temporal collection:

POST /manage/v2/databases/{id|name}/temporal/collections?collection=collname

For more details, see POST /manage/v2/databases/{id|name}/temporal/collections?collection={name} in the MarkLogic REST API Reference.

REST Client API: Advancing LSQT

You can now use the following new method of the REST Client API to advance LSQT on a temporal collection:

POST /v1/temporal/collections/{name}

For more details, see POST /v1/temporal/collections/{name} in the MarkLogic REST API Reference.

Separate Download and Installation of Converters and Filters

In MarkLogic version 8 releases starting at 8.0-8, MarkLogic converters/filters are offered as a separate package (called MarkLogic Converters package) from MarkLogic Server package.

This change provides better flexibility and enables you to install/uninstall MarkLogic converters/filters separately from MarkLogic Server.

For more details, see MarkLogic Converters Installation Changes in Version 8 Releases Starting at 8.0-8 in the Installation Guide.

SQL Deep Analysis Trace Event

SQL uses cost-based optimization. With the default (shallow) table analysis, the cost estimates are likely to be very wrong, certain optimizations are disabled, and query executions will be longer. However, the cost of performing the deep analysis is expensive.

For long-lived ODBC connections, the cost of analysis is paid once when the connection is created and the better costing numbers will be available for the life time of the connection. The trace event SQL Deep Analysis causes the deep analysis to be run when the connection is initiated automatically. For xdmp:sql, deep analysis is only done if the ANALYZE command is used in the SQL, but the information is only kept until xdmp:sql returns. For example:

xdmp:sql("ANALYZE; SELECT whatever FROM example")

Enabling SQL Deep Analysis will enable full table analysis for each new ODBC connection.

With shallow table analysis, a number of key optimizations are defeated and the costing functions may lead to very bad plans. Customers who use long-running ODBC connections will probably want to enable the SQL Deep Analysist trace event.