Loading TOC...
Release Notes (PDF)

Release Notes — Chapter 3

New Features in MarkLogic 8

This chapter describes the new features in MarkLogic 8.

Server-Side JavaScript

MarkLogic 8 includes JavaScript as a server-side programming language, allowing you to access all of the powerful search and data manipulation capabilities of MarkLogic in a language that is familiar to many developers. Combined with native JSON document support, you can easily build JSON-based data services in JavaScript using MarkLogic.

The Server-Side JavaScript is implemented using the Google V8 JavaScript engine, which is an open-source, high-performance JavaScript engine implemented in C++ that is compiled into MarkLogic Server. There are over 1,000 MarkLogic built-in functions available directly from Server-Side JavaScript, allowing you access to data and search in JavaScript.

The JavaScript support is in addition to the XQuery support, and in fact, you can import XQuery libraries into JavaScript programs, making all of your XQuery code immediately usable in a JavaScript program.

For details on the MarkLogic-specific extensions to JavaScript, see the JavaScript Reference Guide.

Native JSON

MarkLogic 8 includes support for JSON as a native document format. Along with the other document formats (XML, text, and binary), JSON documents are stored in the database with all of the enterprise features you expect in MarkLogic. They are indexed using the universal index (like XML and text documents), and, like XML documents, you can also create range indexes on JSON document properties and path indexes on JSON paths. Similarly, you can create fields on JSON documents. Therefore you can perform fast and complex searches across JSON documents.

JSON has become extremely popular in web applications, and is very popular with JavaScript developers. It is therefore an excellent match for the Server-Side JavaScript and for the Node.js Client API features. MarkLogic 8 also includes a set of constructors in XQuery to make it easy to work with JSON documents in XQuery.

You can choose to model your data in the format that makes sense for your application. Also, you do not have to model everything in one format. Use JSON where it makes sense, use XML where it makes sense, and use RDF triples where they make sense. All MarkLogic document formats are designed to co-exist, working together as your application requirements dictate. For more details on JSON, see Working With JSON in the Application Developer's Guide.

Samplestack Sample Application

Samplestack is a three-tier application that uses MarkLogic as its data layer, and demonstrates a reference architecture for a popular way of building applications. Samplestack is an open source project on GitHub available at the following URL:

https://github.com/marklogic/marklogic-samplestack

Samplestack is a demonstration application that uses data from Stack Overflow to create a question and answer application. For details about Samplestack, see the GitHub project. For an overview of the reference architecture, see Understanding the Reference Architecture in the Reference Application Architecture Guide.

Temporal Documents

MarkLogic 8 includes support for temporal documents, typically used for bitemporal applications. A temporal document stores information about the valid time as well as the system time for a document. This allows you to store date as it was known at various times throughout a document's lifecycle. This is useful in many 'what you knew when you knew it' type of applications, often necessary in compliance applications.

For details about using temporal documents in MarkLogic, see the Temporal Developer's Guide.

REST Management API Improvements

The REST Management API has been significantly expanded in MarkLogic 8. You can now script almost any management task via the REST API, allowing you to create management scripts in whatever scripting language you like, whether that is python, bash, ruby, php, or anything else that allows you to make HTTP calls. For details about the REST Management API, see the MarkLogic REST API Reference and the REST Application Developer's Guide.

More Semantics Features, Including SPARQL 1.1, Inferencing, and SPARQL UPDATE

MarkLogic 8 extends the use of standard SPARQL, enabling you to perform analytics (aggregates) over triples; explore semantics graphs using property paths; and update semantic triples; all using the standard SPARQL 1.1 language over standard protocols. Specifically, MarkLogic 8 includes the following enhancements to semantics:

  • SPARQL 1.1 Support
  • Inferencing
  • SPARQL Update Support
  • Aggregates
  • Property Paths

For details on using the enterprise triple store in MarkLogic, see the Semantics Developer's Guide. For details on inferencing, see Inference, for details on SPARQL Update, see SPARQL Update, for details on aggregates, see SPARQL Aggregates.

Node.js Client API

The MarkLogic Node.js Client API is an open source JavaScript library for Node.js, allowing you to quickly and reliably access MarkLogic from a Node application. Node.js (nodejs.org) is a popular platform for building three-tier applications, where Node.js is typically the middle tier. The MarkLogic Node.js Client API GitHub project is available at the following URL:

https://github.com/marklogic/node-client-api

The Node.js Client API is available on GitHub, and uses Node technologies you would expect such as npm (Node Packaging Manager). For details on getting started with the Node.js Client API, see the GitHub project and Introduction to the Node.js Client API the Node.js Application Developer's Guide.

REST and Java Client API Improvements

The REST Client API and the Java Client API have extensive enhancements to both search and CRUD (create, read, update, delete) operations. The Search enhancements include:

  • extract-document-data query option
  • geo-json-* support in structured query
  • Temporal document support
  • Native JSON query and update support

Additional REST and Java client enhancements include:

  • JavaScript resource service extensions and content transformations.
  • New endpoints for evaluating (eval) and invoking (invoke) code (both JavaScript and XQuery code).
  • Runtime database selection: for the REST Client API, database selection is per request; for the Java Client API, database selection is per DatabaseClient (not per request).
  • Bitemporal support.
  • New config properties for configuring REST instances: forests-per-host, error-format, xdbc-enabled.
  • REST instance available on port 8000 for all installations.
  • Bulk operations for both REST and Java (write, read by URI, read by search).
  • A new POJO data binding interface for the Java Client API, allowing you to persist plain old Java objects (POJOs) into MarkLogic for storage and retrieval.

Additionally, the Java Client API is now an open source project on GitHub available at the following URL:

https://github.com/marklogic/java-client-api

Enhanced HTTP Server Features

The HTTP App Server in MarkLogic 8 includes a declarative XML rewriter, and the default rewriters used by a REST API Instance (as well as the one on port 8000) allow you to use a single App Server for multiple applications, including application that use the REST API, The Java Client API, the Node.js Client API, MLCP, or any XCC application that previously required an XDBC App Server. There is a REST endpoint POST /v1/rest-apis) to create an instance.

For the App Server available in all installations on port 8000, the enhanced features in the HTTP App Server make is very easy for new users to run code without needing to create a separate REST API instance or a separate XDBC App Server; instead just point a REST client or an XCC program (like MLCP) to port 8000 of your MarkLogic installation. For details on REST instances, see Administering MarkLogic REST API Instances in the REST Application Developer's Guide, and for details on the declarative rewriter, see Creating a Declarative XML Rewriter to Support REST Web Services in the Application Developer's Guide.

Flexible Replication Enhancements

Flexible Replication is an existing feature in MarkLogic that makes it easy to copy some or all parts of your data to other MarkLogic clusters, whether they are in the same data center or geographically distributed (and possibly bandwidth or connectivity limited). Flexible Replication is different from Database Replication, as Database Replication is better suited for the purpose of high availability (for example, for failover and disaster recovery). Flexible Replication, on the other hand, is well-suited for applications that need to keep copies of subsets of their data for use by other applications.

In MarkLogic 8, Flexible Replication adds the ability to perform replication based on a saved query (an alert). This query-based flexible replication (QBFR) has the ability to be much faster for highly distributed systems to replicate subsets of their content and makes it efficient to maintain changes in that content. This is especially useful in applications where there are many replicas each replicating different parts of the data, and when some or all of those replicas might have bandwidth or connectivity constraints. For details on configuring QBFR, see Configuring Alerting With Flexible Replication in the Flexible Replication Guide.

Incremental Backup

In addition to the existing backup and journaled backup, MarkLogic 8 adds incremental backup, allowing you to create incremental backups at whatever cadence makes sense for your application. Incremental backups can save backup time and space because it only needs to back up the changes since the last full or incremental backup. You can combine incremental backups with journal archiving, allowing you to restore to the closest incremental backup and then rewind to any time using your journal archive. For details, see Incremental Backup and Restoring from an Incremental Backup with Journal Archiving in the Administrator's Guide.

Document Library Services (DLS) Improvements

Document Library Services (DLS) is an API that allows you to create applications to version documents, perform check-in and check-out operations, and other library services features. In MarkLogic 8, there are improvements to make the system significantly more efficient, especially if you have large DLS repositories.

The improvements require an upgrade operation on any existing DLS repositories, as described in Document Library Services (DLS) Repositories Need To Perform A Bulk Upgrade Operation of these Release Notes.

For details on library services applications, see Library Services Applications in the Application Developer's Guide.

MLCP Enhancements

The MarkLogic Content Pump (MLCP) has many improvements in MarkLogic 8, including:

  • The ability to specify an alternate database
  • Temporal document support for loading
  • JSON document support for loading

Additionally, because of the Enhanced HTTP Server Features, you no longer need to create an XDBC Server to use MLCP; you can target any REST API instance, including the built-in port 8000 instance.

New Features in MarkLogic 8.0-3

MarkLogic version 8.0-3 contains the following new features:

Compatibility With New Versions of Hadoop

MarkLogic features that leverage Apache Hadoop MapReduce and HDFS can now be used with the following Hadoop distributions:

  • Cloudera's Distribution Including Apache Hadoop (CDH) version 5.4
  • Hortonworks Data Platform (HDP) version 2.2

CDH version 4.3 is no longer included in the list of compatible distributions.

This change affects mlcp (MarkLogic Content Pump) in distributed mode, the MarkLogic Connector for Hadoop, and the use of HDFS for forest storage.

MarkLogic Connector for Hadoop Supports JavaScript Queries

You can now express input and output queries in either XQuery or Server-Side JavaScript. That is, you can use Server-Side JavaScript in the values of the configuration properties mapreduce.marklogic.input.query and mapreduce.marklogic.output.query. Previously, you could only use XQuery to express input and output queries. Use the properties mapreduce.marklogic.input.queryLanguage and mapreduce.marklogic.output.queryLanguage to indicate which scripting language is used in your input or output query. The default query language is XQuery.

For details, see the MarkLogic Connector for Hadoop Developer's Guide.

XCC Support for JSON Content

You can now use XCC and the Jackson libraries to insert, update, and read JSON content. For example, you can create a ContentFactory specifically for JSON, similar to the following:

JsonNode node = ...;
ContentCreateOptions options = new ContentCreateOptions();
options.setFormat(DocumentFormat.JSON);
Content content = ContentFactory.newContent(uri, node, options);

For details, see Working With JSON Content in the XCC Developer's Guide.

HDFS Client Bundles Available for Download

Previously, using HDFS for forest storage required you to assemble a set of Hadoop HDFS JAR files or install Hadoop on each MarkLogic host containing a forest on HDFS (or to install Hadoop in a well-known location).

You can now download a pre-packaged Hadoop HDFS client bundle from http://developer.marklogic.com/products/hadoop and install this bundle on your MarkLogic hosts. A bundle is available for each supported Hadoop distribution. Use of one of these bundles is required if you use HDFS for forest storage.

The availability of these bundles also changes how and where MarkLogic looks for the JDK and Hadoop libraries.

For details, see HDFS Storage in the Query Performance and Tuning Guide.

Arithmetic and Comparison Methods for dateTime and Duration Objects in Server-Side JavaScript

Beginning with 8.0-3, there are methods that allow you to compare, add, subtract, multiply, and divide duration and date objects in Server-Side JavaScript. These methods allow you to take advantage of richly typed date values available in MarkLogic from Server-Side JavaScript. For details of these new APIs, see JavaScript Duration and Date Arithmetic and Comparison Methods in the JavaScript Reference Guide.

Expanded instanceof Support in Server-Side JavaScript

Beginning in 8.0-3, you can use the instanceof operator in Server-Side JavaScript to test for any of the MarkLogic-typed values, including ValueIterator, cts.query, and so on. For details, see JavaScript instanceof Operator in the JavaScript Reference Guide.

New Features in the Client APIs

The REST Client API, Java Client API, and Node.js Client API now support deleting multiple documents by URI in a single operation.

  • REST: Pass multiple uri parameters to the DELETE /v1/documents method.
  • Java: Pass multiple URIs to DocumentManager.delete.
  • Node.js: Pass multiple URIs to Documents.remove or DatabaseClient.remove.

The Java Client API now includes support for the extract-document-data query option on search operations. Use this option with QueryManager.search to include sparse document projections in your search results. Previously, this capability was only available for multi-document reads. For details, see Extracting a Portion of Matching Documents in the Java Application Developer's Guide.

New Features in MarkLogic 8.0-4

MarkLogic version 8.0-4 contains the following new features:

Server-Side JavaScript Features

MarkLogic 8.0-4 includes the following Server-Side JavaScript new features:

Server-Side JavaScript Search API (jsearch)

MarkLogic 8.0-4 includes a new Server-Side JavaScript library to help create search applications. This new jsearch API uses common JavaScript design patterns to make it easy to create search applications that include search results with snippets, facets, suggestions, and other search application features. For details on the jsearch API, see Creating JavaScript Search Applications in the Search Developer's Guide and the jsearch API Documentation.

cts:parse

There is a new built-in function called cts:parse in XQuery and cts.parse in JavaScript. The cts.parse function is used by jsearch, but is also available to any XQuery or Server-Side JavaScript code. It returns a cts:query and it useful for parsing a Google-style search grammar that a user might type into a search box in an application, and converting that string into a cts:query to pass into a search. For details, see Creating a Query From Search Text With cts:parse in the Search Developer's Guide.

Alerting API JavaScript Enhancements

In 8.0-4, the alerting API is more convenient to use in JavaScript. The alerting API now allows you to use either XML or JSON format, and it accepts JavaScript objects when called from JavaScript.

It is a best practice to pass XML when the alert action is implemented by an XQuery module and a JavaScript object when the action is implemented by a JavaScript file.

Thesaurus and Spelling API Enhancements

In 8.0-4, the thesaurus and spelling APIs are more convenient to use in JavaScript. The spelling API now allows you to create dictionaries either in XML or JSON format, and both the thesaurus and spelling APIs accept JavaScript objects when called from JavaScript. For details, see Using the Spelling Correction Functions and Using the Thesaurus Functions in the Search Developer's Guide.

Semantics Enhancements

MarkLogic 8.0-4 includes the following enhancements to Semantics:

SPARQL Negation

SPARQL 1.1 Negation (using EXISTS, NOT EXISTS, and MINUS) is part of MarkLogic Semantics in 8.0-4. Used with the FILTER expression, negation operates on matching patterns to refine solution results. See Negation in Filter Expressions in the Semantics Developer's Guide.

Java Client API Semantics Enhancements

In 8.0-4, the Java Client API includes increased support for Semantics. You can use Java for managing graphs and triples, and accessing SPARQL query and SPARQL Update functionality in MarkLogic. MarkLogic now supports graph operations, SPARQL query, and SPARQL Update in the Java Client API. For more information, see Java Client API in the Semantics Developer's Guide, the Java Application Developer's Guide, and GraphManager and SPARQLQueryManager in the Java Client API Documentation. The Java Client project is available on GitHub.

MarkLogic Sesame API

In 8.0-4, MarkLogic Sesame API provides full-featured support for standard Sesame APIs. Java developers familiar with Sesame APIs now have access to MarkLogic Semantics, extensions, and combination queries, simplifying semantic application development. For more information, see MarkLogic Sesame API in the Semantics Developer's Guide and the Sesame project on GitHub.

MarkLogic Jena API

In 8.0-4, MarkLogic Jena API provides full-featured support for standard Jena APIs. Java developers familiar with Jena APIs now have access to MarkLogic Semantics, extensions, and search capabilities, simplifying semantic application development. For more information, see MarkLogic Jena API in the Semantics Developer's Guide and the Jena project on GitHub.

Semantics REST API Additions

The ability to use MarkLogic Semantics with the REST Client API to view, query, and modify triple data and graphs has been enhanced in 8.0-4 with variable bindings, ruleset configuration, and transaction support. For details, see Using Semantics with the REST Client API in the Semantics Developer's Guide.

mlcp Enhancements

When importing triples and quads with the mlcp command line tool, you can now use the new options -output_graph and -output_override_graph to control the graphs into which your semantic data is loaded. For details, see Loading Triples in the mlcp User Guide.

Management REST API Enhancements

The REST Management API has been expanded to include new alerting, mimetypes, and support endpoints.

Geospatial Search Enhancements

MarkLogic 8.0-4 introduces the following enhancements for working with and searching geospatial data:

Converting To/From Common Representations

MarkLogic 8.0-4 adds support for Well Known Binary (WKB) representation of geospatial data, as well as new functions for converting between common geospatial data serializations and the internal MarkLogic representation. For details, see Converting To and From Common Geospatial Representations in the Search Developer's Guide and the XQuery and Server-Side JavaScript API reference documentation.

The following new functions have been added to support this feature:

XQueryJavaScript
geo:parse
geo.parse
geo:validate-wkt
geo.validateWkt
geo:parse-wkb
geo.parseWkb
geo:to-wkb
geo.toWkb
geo:validate-wkb
geo.validateWkb
geogml:parse-gml
geogml.parseGml
geogml:to-gml
geogml.toGml
geokml:parse-kml
geokml.parseKml
geokml:to-kml
geogml.toKml
georss:parse-georss
georss.parseGeorss
georss:to-georss
georss.toGeorss
geojson:parse-geojson
geojson.parseGeojson
geojson:to-geojson
geojson.toGeojson
New Geospatial Utility Functions

The following new geospatial utility functions have been added in 8.0-4. For details, see the XQuery and Server-Side JavaScript API Reference.

XQueryJavaScript
geo:polygon-to-linestring
geo.polygonToLinestring
geo:linestring-reverse
geo.linestringReverse
geo:linestring-concat
geo.linestringConcat
geo:region-affine-transform
geo.regionAffineTransform
geo:region-approximate
geo.regionApproximate
geo:region-clean
geo.regionClean
geo:circle-polygon
geo.circlePolygon
geo:remove-duplicate-vertices
geo.removeDuplicateVertices
geo:count-vertices
geo.countVertices
geo:count-distinct-vertices
geo.countDistinctVertices
geo:interior-point
geo.interiorPoint
geo:distance-convert
geo.distanceConvert
cts:match-regions
cts.matchRegions
Support for More Recent KML and GML Versions

MarkLogic now supports KML 2.2 and GML 3.2. These are now the default versions for KML and GML data. Use the following namespace URIs in your data to identify the version and when converting between a cts point or region and a GML or KML node.

  • KML 2.0: http://earth.google.com/kml/2.0
  • KML 2.2: http://www.opengis.net/kml/2.2
  • GML 3.1.1: http://www.opengis.net/gml
  • GML 3.2: http://www.opengis.net/gml/3.2

When generating GML or KML constructs from a cts point or region, you can use the above namespace URIs in conversion functions like the XQuery geogml:to-gml function or JavaScript geogml.toGml function to request a specific version.

As a side-effect of this feature, the GML and KML geospatial library modules have been moved to a MarkLogic-specific namespace. For details, see Geospatial Namespace and Data Version Changes.

Support For Red Hat Enterprise Linux 7

MarkLogic 8.0-4 includes support for RHEL 7. The RHEL 7 package is separate from the RHEL 6 package on developer.marklogic.com. For details on MarkLogic platforms, see Supported Platforms, and for details on installation see the Installation Guide.

Clock Skew Timeouts

XDQP connections between hosts in either a local or foreign cluster will now drop if a host's clock is skewed by more than the host timeout. Attempts to connect will result in a warning message in the log when the first connection attempt is rejected, and every hour after that.

The host timeout is either from the host's group if it is an intra-cluster connection, or from the foreign cluster configuration if it is an inter-cluster connection.

Node.js Client API Supports Searching XML with QBE

You can now search XML documents with using Query By Example when using the Node.js Client API. For details, see Querying XML Content With QBE in the Node.js Application Developer's Guide.

New Features in mlcp

The mlcp command line tool added the following new capabilities in MarkLogic 8.0-4:

New Features in MarkLogic 8.0-6

MarkLogic 8.0-6 includes the following new features:

MapR Support in mlcp and the Hadoop Connector

You can now use MapR as your Hadoop implementation with mlcp in distributed mode and with the MarkLogic Connector for Hadoop.

Using mlcp with MapR requires special setup. For details, see Required Software and Using mlcp With MapR in the mlcp User Guide or .

« Previous chapter
Next chapter »
Powered by MarkLogic Server 7.0-4.1 and rundmc | Terms of Use | Privacy Policy