MarkLogic 9 Product Documentation

Application Developer's Guide — Chapter 22

What is a Native Plugin?

A native plugin is a dynamically linked library that contains one or more UDF (User Defined Function) implementations. When you package and deploy a native plugin in the expected way, MarkLogic distributes your code across the cluster and makes it available for execution through specific extension points.

The UDF interfaces define the extension points that can take advantage of a native plugin. MarkLogic currently supports the following UDFs:

• AggregateUDF: Use the map-reduce capabilities of MarkLogic to analyze values in lexicons and range indexes. For more details, see Aggregate User-Defined Functions.
• LexerUDF: Define a custom tokenizer for a language. For more details, see User-Defined Lexer Plugins in the Search Developer's Guide.
• StemmerUDF: Define a custom stemmer for a language. For more details, see Using a User-Defined Stemmer Plugin in the Search Developer's Guide.

The implementation requirement for each UDF varies, but they all use the native plugin mechanism for packaging, deployment, and version.

How MarkLogic Server Manages Native Plugins

Native plugins are deployed as dynamically loaded libraries that MarkLogic Server loads on-demand when referenced by an application. The User-Defined Functions (UDFs) implemented by a native plugin are identified by the relative path to the plugin and the name of the UDF. For a list of the supported kinds of UDFs, see What is a Native Plugin?.

When you install a native plugin library, MarkLogic Server stores it in the Extensions database. If the MarkLogic Server instance in which you install the plugin is part of a cluster, your plugin library is automatically propagated to all the nodes in the cluster.

There can be a short delay between installing a plugin and having the new version available. MarkLogic Server only checks for changes in plugin state about once per second. Once a change is detected, the plugin is copied to hosts with an older version.

In addition, each host has a local cache from which to load the native library, and the cache cannot be updated while a plugin is in use. Once the plugin cache starts refreshing, operations that try use a plugin are retried until the cache update completes.

MarkLogic Server loads plugins on-demand. A native plugin library is not dynamically loaded until the first time an application calls a UDF implemented by the plugin. A plugin can only be loaded or unloaded when no plugins are in use on a host.

Building a Native Plugin Library

Native plugins run in the same process context as the MarkLogic Server core, so you must compile and link your library in a manner compatible with the MarkLogic Server executable. Follow these basic steps to build your library:

• Compile your library with a C++ compiler and standard libraries compatible with MarkLogic Server. See the table below. This is necessary because C++ is not guaranteed binary compatible across compiler versions.
• Compile your C++ code with the options your platform requires for creating shared objects. For example, on Linux, compile with the -fPIC option.
• Build a 64-bit library (32-bit on Windows).

The sample plugin in marklogic_dir/Samples/NativePlugins includes a Makefile usable with GNU make on all supported platforms. You should use this makefile as the basis for building your own plugins as it includes all the required compiler options.

The makefile builds a shared library, generates a manifest, and zips up the library and manifest into an install package. The makefile is easily customized for your own plugin by changing a few make variables at the beginning of the file:

PLUGIN_NAME = sampleplugin
PLUGIN_VERSION = 0.1
PLUGIN_PROVIDER = MarkLogic
PLUGIN_DESCRIPTION = Example native plugin

PLUGIN_SRCS = \
  SamplePlugin.cpp

The table below shows the compiler and standard library versions used to build MarkLogic Server. You must build your native plugin with compatible tools.

Packaging a Native Plugin

You must package a native plugin into a zip file to install it. The installation zip file must contain:

• A C++ shared library implementing the plugin interface(s), such as marklogic::AggregateUDF, and the registration function marklogicPlugin.
• A plugin manifest file called manifest.xml. See The Plugin Manifest.
• Optionally, additional shared libraries required by the plugin implementation.

Including dependent libraries in your plugin zip file gives you explicit control over which library versions are used by your plugin and ensures the dependent libraries are available to all nodes in the cluster in which the plugin is installed.

The following example creates the plugin package sampleplugin.zip from the plugin implementation, libsampleplugin.so, a dependent library, libdep.so, and the plugin manifest.

$ zip sampleplugin.zip libsampleplugin.so libdep.so manifest.xml

If the plugin contents are organized into subdirectories, include the subdirectories in the paths in the manifest. For example, if the plugin components are organized as follows in the zip file:

$ unzip -l sampleplugin.zip
Archive:  sampleplugin.zip
  Length     Date   Time    Name
 --------    ----   ----    ----
    28261  06-28-12 12:54   libsampleplugin.so
      334  06-28-12 12:54   manifest.xml
        0  06-28-12 12:54   deps/
    28261  06-28-12 12:54   deps/libdep.so
 --------                   -------
    56856                   4 files

Then manifest.xml for this plugin must include deps/ in the dependent library path:

<?xml version="1.0" encoding="UTF-8"?>
<plugin xmlns="http://marklogic.com/extension/plugin">
  <name>sampleplugin-name</name>
  <id>sampleplugin-id</id>
  ...
  <native>
    <path>libsampleplugin.so</path>
    <dependency>deps/libdep1.so</dependency>
  </native>
</plugin>

Installing a Native Plugin

After packaging your native plugin as described in Packaging a Native Plugin, install or update your plugin using the XQuery function plugin:install-from-zip or the Server-Side JavaScript function plugin:installFromZip.

For example, the following code installs a native plugin contained in the file /space/plugins/sampleplugin.zip. The relative plugin path in the Extensions directory is native.

xquery version "1.0-ml";
import module namespace plugin = "http://marklogic.com/extension/plugin"
  at "MarkLogic/plugin/plugin.xqy";

plugin:install-from-zip("native",
  xdmp:document-get("/space/plugins/sampleplugin.zip")/node())

'use strict';
declareUpdate();
const plugin = require('/MarkLogic/plugin/plugin');

plugin.installFromZip(
  'native', 
  fn.head(
    xdmp.documentGet('/space/plugins/sampleplugin.zip')).root);

If the plugin was already installed on MarkLogic Server, the new version replaces the old.

An installed plugin is identified by its path. The path is of the form scope/plugin-id, where scope is the first parameter to plugin:install-from-zip, and plugin-id is the ID in the <id/> element of the plugin manifest. For example, if the manifest for the above plugin contains <id>sampleplugin-id</id>, then the path is native/sampleplugin-id.

The plugin zip file can be anywhere on the filesystem when you install it, as long as the file is readable by MarkLogic Server. The installation process deploys your plugin to the Extensions database and creates a local on-disk cache inside your MarkLogic Server directory.

Installing or updating a native plugin on any host in a MarkLogic Server cluster updates the plugin for the whole cluster. However, the new or updated plugin may not be available immediately. For details, see How MarkLogic Server Manages Native Plugins.

Uninstalling a Native Plugin

To uninstall a native plugin, call the XQuery function plugin:uninstall or the Server-Side JavaScript function plugin.uninstall. In the first parameter, pass the scope with which you installed the plugin. In the second parameter, pass the plugin ID (the <id/> in the manifest). For example:

xquery version "1.0-ml";
import module namespace plugin = "http://marklogic.com/extension/plugin"
  at "MarkLogic/plugin/plugin.xqy";

plugin:uninstall("native", "sampleplugin-id")

'use strict';
declareUpdate();
const plugin = require('/MarkLogic/plugin/plugin');

// Install a plugin package in the Extensions database under
// the relative path 'my/native/plugin'.
plugin.uninstall('native', 'sampleplugin-id');

The plugin is removed from the Extensions database and unloaded from memory on all nodes in the cluster. There can be a slight delay before the plugin is uninstalled on all hosts. For details, see How MarkLogic Server Manages Native Plugins. There can be a slight delay

Registering a Native Plugin at Runtime

When you install a native plugin, it becomes available for use. The plugin is loaded on demand. When a plugin is loaded, MarkLogic Server uses a registration handshake to cache details about the plugin, such as the version and what UDFs the plugin implements.

Every C++ native plugin library must implement an extern "C" function called marklogicPlugin to perform this load-time registration. The function interface is:

using namespace marklogic;
extern "C" void marklogicPlugin(Registry& r) {...}

When MarkLogic Server loads your plugin library, it calls marklogicPlugin so your plugin can register itself. The exact requirements for registration depend on the interfaces implemented by your plugin, but should include at least the following:

• Register the version of your plugin by calling marklogic::Registry::version.
• Register the interface(s) your plugin implements by calling the appropriate marklogic::Registry registration method. For example, Registry::registerAggregate for implementations of marklogic::AggregateUDF.

Declare marklogicPlugin as required by your platform to make it accessible outside your library. For example, on Microsoft Windows, include the extended attribute dllexport in your declaration:

extern "C" __declspec(dllexport) void marklogicPlugin(Registry& r)...

For example, the following code registers two AggregateUDF implementations. For a complete example, see marklogic_dir/Samples/NativePlugins.

#include MarkLogic.h
using namespace marklogic;

class Variance : public AggregateUDF {...};
class MedianTest : public AggregateUDF {...};

extern "C" void marklogicPlugin(Registry& r)
{
    r.version();
    r.registerAggregate<Variance>("variance");
    r.registerAggregate<MedianTest>("median-test");
}

Versioning a Native Plugin

Your implementation of the registration function marklogicPlugin should include a call to marklogic::Registry::version to register your plugin version. MarkLogic Server uses this information to maintain plugin version consistency across a cluster.

When you deploy a new plugin version, both the old and new versions of the plugin can be present in the cluster for a short time. If MarkLogic Server detects this state when your plugin is used, MarkLogic Server reports XDMP-BADPLUGINVERSION and retries the operation until the plugin versions synchronize.

Calling Registry::version with no arguments uses a default version constructed from the compilation date and time (__DATE__ and __TIME__). This ensures the version number changes every time you compile your plugin. The following example uses the default version number:

extern "C" void marklogicPlugin(Registry& r)
{
    r.version();
    ...
}

You can override this behavior by passing an explicit version to Registry::version. The version must be a numeric value. For example:

extern "C" void marklogicPlugin(Registry& r)
{
    r.version(1);
    ...
}

The MarkLogic Server native plugin API (marklogic_dir/include/MarkLogic.h) is also versioned. You cannot compile your plugin library against one version of the API and deploy it to a MarkLogic Server instance running a different version. If MarkLogic Server detects this mismatch, an XDMP-BADAPIVERSION error occurs.

Checking the Status of Loaded Plugins

Using the Admin Interface or the xdmp:host-status function, you can monitor which native plugin libraries are loaded into MarkLogic Server, as well as their versions and UDF capabilities.

Native plugin libraries are demand loaded when an application uses one of the UDFs implemented by the plugin. Plugins that are installed but not yet loaded will not appear in the host status.

To monitor loaded plugins using the Admin Interface:

1. In your browser, navigate to the Admin Interface: http://yourhost:8001.
2. Click the name of the host you want to monitor, either on the tree menu or the summary page. The host summary page appears.
3. Click the Status tab at the top right. The host status page appears.
4. Scroll down to the native plugin status section.

To examine loaded programatically, open Query Console and run a query similar to the following:

xquery version "1.0-ml";
(: List native plugins loaded on this host :)
xdmp:host-status(xdmp:host())//*:native-plugins

'use strict';
fn.head(xdmp.hostStatus(xdmp.host()))['native-plugins']

You should see output similar to the following if there are plugins loaded. The XQuery code emits XML. The JavaScript code emits a JavaScript object (pretty-printed as JSON by Query Console). This output is the result of installing and loading the sample plugin in MARKLOGIC_DIR/Samples/NativePlugin, which implements several aggregate UDFs (max, min, etc.), a lexer UDF, and a stemmer UDF.

<native-plugins xmlns="http://marklogic.com/xdmp/status/host">
  <native-plugin>
    <path>native/sampleplugin/libsampleplugin.so</path>
    <version>356528850</version>
    <capabilities>
      <aggregate>max</aggregate>
      <aggregate>min_point</aggregate>
      <aggregate>min</aggregate>
      <aggregate>variance</aggregate>
      <aggregate>median-test</aggregate>
      <aggregate>max_dateTime</aggregate>
      <aggregate>max_string</aggregate>
      <lexer>sample_lexer</lexer>
      <stemmer>sample_stemmer</stemmer>
    </capabilities>
  </native-plugin>
</native-plugins>

[{
  "path":"native/sampleplugin/libsampleplugin.so",
  "version":"356528850", 
  "capabilities":[
    "max", "min_point", "min", "variance", 
    "median-test", "max_dateTime", "max_string", 
    "sample_lexer", 
    "sample_stemmer"]
}]

The Plugin Manifest

A native plugin zip file must include a manifest file called manifest.xml. The manifest file must contain the plugin name, plugin id, and a <native> element for each native plugin implementation library in the zip file. The manifest file can also include optional metadata such as provider and plugin description. For full details, see the schema in MARKLOGIC_INSTALL_DIR/Config/plugin.xsd.

Paths to the plugin library and dependent libraries must be relative.

You can use the same manifest on multiple platforms by specifying the native plugin library without a file extension or, on Unix, lib prefix. If this is the case, then MarkLogic Server forms the library name in a platform specific fashion, as shown below:

• Windows: Add a .dll extension
• Linux: Add a lib prefix and a .so extension
• Mac OS X: Add a lib prefix and a .dylib extension

The following example is the manifest for a native plugin with the ID sampleplugin-id, implemented by the shared library libsampleplugin.so.

<?xml version="1.0" encoding="UTF-8"?>
<plugin xmlns="http://marklogic.com/extension/plugin">
  <name>sampleplugin-name</name>
  <id>sampleplugin-id</id>
  <version>1.0</version>
  <provider-name>MarkLogic</provider-name>
  <description>Example native plugin</description>
  <native>
    <path>libsampleplugin.so</path>
  </native>
</plugin>

If the plugin package includes dependent libraries, list them in the <native> element. For example:

<?xml version="1.0" encoding="UTF-8"?>
<plugin xmlns="http://marklogic.com/extension/plugin">
  <name>sampleplugin-name</name>
  ...
  <native>
    <path>libsampleplugin.so</path>
    <dependency>libdep1.so</dependency>
    <dependency>libdep2.so</dependency>
  </native>
</plugin>

Native Plugin Security Considerations

Administering (installing, updating or uninstalling) a native plugin requires the following:

• The http://marklogic.com/xdmp/privileges/plugin-register privilege, or
• The application-plugin-registrar role.

Loading and running a native plugin can be controlled in two ways:

• The native-plugin privilege (http://marklogic.com/xdmp/privileges/native-plugin) enables the use of all native plugins.
• You can define a plugin-specific privelege of the form http://marklogic.com/xdmp/privileges/native-plugin/plugin-path to enable users to use a specific privilege.

The plugin-path is same plugin library path you use when invoking the plugin. For example, if you install the following plugin and its manifest specifies the plugin path as sampleplugin, then the plugin-specific privilege would be http://marklogic.com/xdmp/privileges/native-plugin/native/sampleplugin.

plugin:install-from-zip("native",
  xdmp:document-get("/space/udf/sampleplugin.zip")/node())

The plugin-specific privilege is not pre-defined for you. You must create it. However, MarkLogic Server will honor it if it is present.

Native Plugin Example

You can explore a sample native plugin through the source code and makefile in MARKLOGIC_DIR/Samples/NativePlugins. This example implements several kinds of UDF.

The sample Makefile will lead you through compiling, linking, and packaging the native plugin. The README.txt provides instructions for installing and exercising the plugin library.