MarkLogic Server 11.0 Product Documentation

Search Developer's Guide — Chapter 18

When to Consider a User-Defined Stemmer

MarkLogic provides several built-in stemmers that you can configure for a language if you are not satisfied with the default stemmer. The following are some use cases in which you might consider implementing a your own stemmer:

• You need to stem a language that is not directly supported by MarkLogic.
• You want to use a specific 3rd party library for stemming for a given language.
• You need to use advanced stemming to obtain variants such as normalization and spelling variants not otherwise available.
• You require special format stemming in the context of specific data fields where the requirements are more complicated than simple reclassification.

In some cases, you might also need a custom lexer or custom dictionary. For example, if you're working with a language not supported by MarkLogic, you probably also need a custom lexer. For details, see Custom Tokenization and Custom Dictionaries for Tokenizing and Stemming.

StemmerUDF Interface Summary

You implement a user-defined stemmer as a subclass of the MarkLogic::StemmerUDF base class. StemmerUDF is defined in MARKLOGIC_INSTALL_DIR/include/MarkLogic.h. You can find detailed documentation about the class in the User-Defined Function API reference and in MarkLogic.h. You can find an example implementation in MARKLOGIC_INSTALL_DIR/Samples/NativePlugins.

The following table contains a brief summary of the key methods of StemmerUDF. For a discussions of how the methods are used by MarkLogic, see Understanding User-Defined Stemmer Control Flow.

Understanding User-Defined Stemmer Control Flow

When stemmed searches are enabled, MarkLogic can use either the default stemming plugin for a language, a user-defined stemming plugin, or both (via delegation). This section describes how MarkLogic interacts with a user-defined stemmer. See the following topics:

• When MarkLogic Uses a User-Defined Stemmer
• StemmerUDF Object Creation and Management
• Interaction During Stemming

Implementation Guidelines for User-Defined Stemmers

When implementing a StemmerUDF subclass, keep the following guidelines in mind.

Your implementation runs in the same memory and process space as MarkLogic Server, so errors in your implementation can crash MarkLogic Server. Before deploying a custom lexer, you should read and understand Using Native Plugins in the Application Developer's Guide. See also Testing a User-Defined Stemmer.

• You must implement a subclass of marklogic::StemmerUDF for each stemming algorithm you want to use.
• Stemming is a low-level, inner-loop operation that MarkLogic performs during indexing (including document ingestion) and query evaluation. Your stemmer should introduce as little overhead as possible.
• If the input word has a preferred stem, it should be the first stem returned after calling StemmerUDF::reset or StemmerUDF::start. The preferred stem is the only stem used when stemmed search is configured at the basic level.
• Indicate contractions with the = character and compound words with the # character. For example, de=le for the French word du, or Kind#Platz for the German word Kinderplatz.
• The tokenizer can optionally pass along a Part of Speech (POS) for the word being stemmed. The POS is supplied to StemmerUDF::reset. Stemming only makes use of this data for Japanese by default, but you can choose to use it in your stemmer.
• The stemmer owns the memory allocated for the stem returned by the stem method, and is responsible for releasing it when appropriate. Your stemmer can be called many times per word, so you should choose an efficient allocation strategy.
• Your implementation does not have to be thread safe. MarkLogic will instantiate a new stemmer object in each thread in which it wants to perform stemming.
• Report errors using the Reporter object that is passed to most StemmerUDF methods, rather than by throwing exceptions. For details, see Error Handling and Logging.
• You might also want to support your language with a custom lexer and/or a custom dictionary. To learn more about customization options, see Stemming and Tokenization Customization.

Creating and Deploying a User-Defined Stemmer Plugin

Follow the steps below to create and deploy a stemmer UDF in MarkLogic as a native plugin. A complete example is available in MARKLOGIC_DIR/Samples/NativePlugins.

1. Implement a subclass of the C++ class marklogic::StemmerUDF. See MARKLOGIC_DIR/include/MarkLogic.h for interface details.
2. Implement an extern "C" function called marklogicPlugin to perform plugin registration. For details, see Registering a Native Plugin at Runtime in the Application Developer's Guide.
3. Build a dynamically linked library containing your UDF and registration function. You should use the Makefile in MARKLOGIC_DIR/Samples/NativePlugins as the basis for building your plugin. For more details, see Building a Native Plugin Library in the Application Developer's Guide.
4. Following the directions in Using Native Plugins to package and install your plugin. See the note below about dependent libraries.
5. Configure your stemmer as the stemmer plugin for at least one language. For details, see Configuring Tokenization and Stemming Plugins.

The native plugin interface includes support for bundling dependent libraries in the native plugin zip file. However, many 3rd party natural language processing tools are large, complex, and have strict installation directory requirements. If you are using such a packge, you should install the 3rd party package package independently on each host in the cluster, rather than trying to include it inside your native plugin package.

Registering a User-Defined Stemmer with MarkLogic

A native plugin becomes available for use once you install it, but it will not be loaded until there is a reason to use it. For example, a plugin containing only a stemmer UDF is only loaded if it is associated with at least one language and MarkLogic needs to stem a word in that language.

When MarkLogic loads a native plugin, it performs a registration handshake to obtain details about the plugin such as what capabilities the plugin provides. This handshake is performed through an extern "C" function named marklogicPlugin that must be part of every native plugin.

The following code is an example of a registration function for a plugin that registers only a single stemmer capability. Assume the plugin implements a StemmerUDF subclass named MyStemmerUDF. The stemmer is registered with the plugin id sample_stemmer.

extern "C" PLUGIN_DLL void
marklogicPlugin(Registry& r)
{
  r.version();
  r.registerStemmer<MyStemmerUDF>("sample_stemmer");
}

The plugin id returned by the registerStemmer method, along with the relative path under which the plugin is installed, is used elsewhere to identify your user-defined stemming plugin.

For details, see Registering a Native Plugin at Runtime in the Application Developer's Guide. For a complete example, see the code in MARLOGIC_DIR/Samples/NativePlugins.

Testing a User-Defined Stemmer

You can test your stemmer implementation in the following ways:

• Create standalone test scaffolding.
• Use the cts:stem XQuery function or the cts.stem Server-Side JavaScript function to exercise your plugin after it is installed and configured for at least one language.

Testing your stemmer standalone during development is highly recommended. It is much easier to debug your code in this setup. Also, since it is possible for native plugin code to crash MarkLogic, it is best to test and stabilize your code outside the server environment.

You can find example test scaffolding in MARKLOGIC_DIR/Samples/NativePlugins/TestStemTok.cpp. See the main() function for a starting point.

Error Handling and Logging

Use marklogic::Reporter to log messages and notify MarkLogic Server of fatal errors. Your code should not report errors to MarkLogic Server by throwing exceptions.

Report non-fatal errors and other messages using marklogic::Reporter::log. This method logs a message to the MarkLogic Server error log and returns control to your code. Most methods of LexerUDF accept a marklogic::Reporter input parameter.

Report fatal errors using marklogic::Reporter::error. You should reserve calls to Reporter::error for serious errors from which no recovery is possible. Reporting an error via Reporter::error has the following effects:

• If you report a fatal stemming error during document insertion, the insertion transaction aborts.
• If you report a fatal stemming error during reindexing, reindexing of the document fails.
• Control does not return to your code. Stemming stops.
• MarkLogic Server returns XDMP-UDFERR to the application. Your error message is included in the XDMP-UDFERR error.

The following snippet reports an error and aborts tokenization:

#include "MarkLogic.h"
using namespace marklogic;
...
void ExampleUDF::next(Reporter& r)
{
  ...
  r.log(Reporter::Error, "Bad codepoint.");
}

For more details, see the marklogic::Reporter class in MARKLOGIC_DIR/include/MarkLogic.h.