MarkLogic Server 11.0 Product Documentation

Search Developer's Guide — Chapter 28

Language-Specific Tokenization

When you search for a string (typically a word or a phrase) in MarkLogic Server, or when you load content (which is made up of text strings) into MarkLogic Server, the string is split into parts, each of which is called a token. Each token is classified as a word, punctuation, or whitespace. The process of breaking down strings into tokens is called tokenization.

Tokenization occurs during document loading as well as during query evaluation. The two processes are independent of each other. The tokenization of documents during loading affects indexing. The tokenization of query text affects how search terms are resolved. Though the processes are independent, they use the same tokenizer (for a given language).

Tokenization is language-specific; that is, a given string is tokenized differently depending on the language in which it is tokenized. The language is determined based on the language specified at load or query time (or the database default language if no language is specified) and on any xml:lang attributes in the XML content (for details, see xml:lang Attribute). For JSON content, the language or lang properties determine language-specific tokenization (for details, see Language Support in JSON).

Note the following about the way strings are tokenized in MarkLogic Server:

• You can use cts:tokenize XQuery function or the cts.tokenize Server-Side JavaScript function to see how text is tokenized for a given language.
• If you wrap a call to xdmp:describe around a call to cts:tokenize in XQuery, you can review both the tokens and their classification. Similarly if you wrap xdmp.describe around a call to cts.tokenize in JavaScript. For example:

  xdmp:describe(cts:tokenize("this is, obviously, a phrase", "en"), 100)
  => (cts:word("this"), cts:space(" "), cts:word("is"),
     cts:punctuation(","), cts:space(" "), cts:word("obviously"),
     cts:punctuation(","), cts:space(" "), cts:word("a"), 
     cts:space(" "), cts:word("phrase"))

• Every query has a language associated with it; if the language is not explicitly specified in the cts:query expression, then it takes on the default language of the database.
• MarkLogic Server comes configured such that when an element is in an Asian or Middle Eastern language, the Latin characters tokenize as English. This allows searches to find English words inside Asian or Middle Eastern language elements. For example, a search in English can find Latin characters in a Simplified Chinese element as in the following:

  let $x := <el xml:lang="zh">Chinese-text-here hello</el>
  return
  $x//el[cts:contains(., 
           cts:word-query("hello", ("stemmed", "lang=en")))]
  
  => <el xml:lang="zh">Chinese-text-here hello</el>

  A stemmed search for the Latin characters in a non-English language, however, will not find the non-English word stems (it will only find the non-English word itself, which stems to itself). Similarly, Asian or Middle Eastern characters will tokenize in a language appropriate to the character set, even when they occur in elements that are not in their language. The result is that searches in English sometimes match content that is labeled in an Asian or Middle Eastern character set, and vice-versa. For example, consider the following (zh is the language code for Simplified Chinese):

  let $x := 
  <root>
   <el xml:lang="en">hello</el>
   <el xml:lang="fr">hello</el>
   <el xml:lang="zh">hello</el>
  </root>
  return
  $x//el[cts:contains(., 
           cts:word-query("hello", ("stemmed", "lang=en")))]
  
  => <el xml:lang="en">hello</el>
     <el xml:lang="zh">hello</el>

  This search, even though in English, returns both the element in English and the one in Chinese. It returns the Chinese element because the word hello is in Latin characters and therefore tokenizes as English, and it matches the Chinese query (which also tokenizes hello in English).

• If your application has specialized tokenization requirements, you can use custom tokenizer overrides or a custom lexer to modify how characters are grouped into tokens. For details, see Custom Tokenization.

Stemmed Searches in Different Languages

A stemmed search for a term matches all the terms that have the same stem as the search term (which includes the exact same terms in the language specified in the query). The purpose of stemming is to increase the recall for a search. For details about how stemming works in MarkLogic Server, including the different types of stemming available, see Understanding and Using Stemmed Searches. This section describes how the language settings affect stemmed searches.

Words derived from the same meaning and part of speech have the same stem (for example, mouse and mice). A word can have multiple stems if the word can be used as multiple parts of speech (for example, play can be both a noun and a verb in English), or if there are two words with the same spelling. If you enable advanced stemming, then stemmed searches find all of the words having the same stem as any of the stems. Advanced stemming finds multiple stems for a word.

Stemming is a language-specific operation. For example, the word chat is a different word in French than it is in English. In French, chat is a noun meaning cat, while in English, it is a verb. In French, chatting is not a word, and therefore it does not stem to chat. But in English, chatting does stem to chat. Therefore, stemmed searches in one language might find different results than stemmed searches in another.

When you construct a query, you can specify a language to use for stemmed search. For example, the following cts:query expression specifies a stemmed search in French for the word chat, and it only matches tokens that are stemmed in French.

cts:word-query("chat", ("stemmed", "lang=fr"))

For more details about how languages affect queries, see Querying Documents By Languages.

At load time, the specified language is used to determine in which language to stem the words in the document. For more details about the language aspects of loading documents, see Language Aspects of Loading and Updating Documents.

For details about the syntax of the various cts:query constructors, see the MarkLogic XQuery and XSLT Function Reference.

Tokenization and Stemming

Tokenization and stemming occur when loading documents, just as they do when querying documents (for details, see Language-Specific Tokenization and Stemmed Searches in Different Languages). When loading documents, the stemmed search indexes are created based on the language. The tokenization and stemming at load time is completely independent from the tokenization and stemming at query time.

xml:lang Attribute

You can specify languages in XML documents at the element level by using the xml:lang attribute. MarkLogic Server uses the xml:lang attribute to determine the language with which to tokenize and stem the contents of that element. Note the following about the xml:lang attribute:

• The xml:lang attribute (see https://www.w3.org/TR/xml/#sec-lang-tag) has some special properties such as not needing to declare the namespace bound to the xml prefix, and that it is inherited by all children of the element (unless they explicitly have a different xml:lang value).
• You can explicitly add an xml:lang attribute to the root node of an XML document during loading by specifying the default-language option to xdmp:document-load; without the default-language option, the root node will remain as-is.
• If no xml:lang attribute is present, then the document is processed in the default language of the database into which it is loaded.
• For the purpose of indexing terms, the language specified by the xml:lang attribute only applies to stemmed search terms; the word searches (unstemmed) database configuration setting indexes terms irrespective of language. Tokenization of terms honors the xml:lang value for both stemmed searches and word searches index settings in the database configuration.
• All of the text node children and text node descendants of an element with an xml:lang attribute are treated as the language specified in the xml:lang attribute, unless a child element has an xml:lang attribute with a different value. If so, any text node children and text node descendants are treated as the new language, and so on until no other xml:lang attributes are encountered.
• The value of the xml:lang attribute must conform to the following lexical standard: http://www.ietf.org/rfc/rfc3066.txt. The following are some typical xml:lang attributes (specifying French, Simplified Chinese, and English, respectively):

  xml:lang="fr"
  xml:lang="zh"
  xml:lang="en"

• If an element has an xml:lang attribute with a value of the empty string (xml:lang=""), then any xml:lang value in effect (from some ancestor xml:lang value) is overridden for that element; its value takes on the database language default. Additionally, if a default-language option is specified during loading, any empty string xml:lang values are replaced with the language specified in the default-language option. For example, consider the following XML:

  <rhone xml:lang="fr">
     <wine>vin rouge</wine>
     <wine xml:lang="">red wine</wine>
  </rhone>

  In this sample, the phrase vin rouge is treated as French, and the phrase red wine is treated in the default language for the database (English by default).

  If this sample was loaded with a default-language option specifying Italian (specifying <default-language>it</default-language> for the xdmp:document-load option, for example), then the resulting document would be as follows:

  <rhone xml:lang="fr">
     <wine>vin rouge</wine>
     <wine xml:lang="it">red wine</wine>
  </rhone>

Language-Related Notes About Loading and Updating Documents

When you load content into MarkLogic Server, it determines how to index the content based on several factors, including the language specified during the load operation, the default language of the database, and any languages encoded into the XML content with xml:lang attributes, or into the JSON content with language or lang properties. Note the following about languages with respect to loading content, updating content, and changing language settings on a database:

• Changing the default language starts a reindex operation if reindex enable is set to true.
• XML documents with no xml:lang attribute are indexed upon load or update in the database default language.
• JSON documents with no language or lang properties are indexed upon load or update in the database default language.
• Any XML content within an element having an xml:lang attribute is indexed in that language. Additionally, the xml:lang value is inherited by all of the descendants of that element, until another xml:lang value is encountered.
• Any JSON content within a scope that contains a language or lang property is indexed in that language. Additionally, the language or lang property is inherited by all of the descendants of that element, until another language or lang property is encountered.
• MarkLogic Server comes configured such that when an element is in an Asian or Middle Eastern language, the Latin characters tokenize as English. Therefore, a document with Latin characters in a non-English language will create stemmed index terms in English for those Latin characters. Similarly, Asian or Middle Eastern characters will tokenize in their respective languages, even in elements that are not in their language.

Protecting JSON Files That Should not be Stemmed

The special zxx language code, which means no natural language present, allows users to protect their own configuration files (or other documents, elements, properties that contain no human-readable content) from customized and plugin tokenizers, as well as from stemmers. In the absence of these language codes, text will always be processed using the default database language. These files also process faster, because they are never stemmed and only use a simple tokenizer.

Tokenization, Stemming, and the xml:lang Attribute

Tokenization and stemming are both language-specific; that is, a string can be tokenized and stemmed differently in different languages. By default, a query uses the default language of the database. You can also specify a language when constructing a query. For more details, see Tokenization and Stemming.

For XML nodes constructed in XQuery, any xml:lang attributes are treated the same way as if the document were loaded into a database. For details, see xml:lang Attribute.

Constructed JSON nodes that contain the language or lang properties are indexed in that language. If neither of these properties is present, then they use the default language configured for the database.

Language-Aware Searches

All searches in MarkLogic Server are language-aware. You can specify a language when constructing a query. For example, most cts:query constructors accept a language option. If the language is not explicitly specified, MarkLogic uses the default language configured for the database. For details on the cts:query constructors, see Composing cts:query Expressions.

The language governing a query determines how to tokenize the search terms, whether stemmed search is enabled or not. If stemmed search is enabled, the language is also used to derive stems. Unstemmed searches use the unstemmed (word searches) indexes, which are language independent.

Unstemmed Searches

An unstemmed search matches terms that are exactly like the search term; it does not take into consideration the stem of the word. Unstemmed searches match terms in a language independent way, but tokenize the search according to the specified language. Therefore, when you specify a language in an unstemmed query, the language applies only to tokenization; the unstemmed query will match any text in any language that matches the query.

Note the following characteristics of unstemmed searches:

• Unstemmed searches require word search indexes, otherwise they throw an exception. However, you can perform unstemmed searches without word search indexes using cts:contains. To perform unstemmed searches without the word search indexes enabled, use a let to bind the results of a stemmed search to a variable, and then filter the results using cts:contains with an unstemmed query.

  The following example binds the stemmed search results to a variable, then iterates over the results, filtering out all but the unstemmed results in the where clause (using cts:contains with a cts:query that specifies the unstemmed option).

  let $search := cts:search(doc(), cts:word-query("my words", 
                                    ("stemmed", "lang=en")))
  for $x in $search
  where cts:contains($x, cts:word-query("my words", "unstemmed"))
  return $x

  While it is likely that everything returned by this search will have an English match to the cts:query, it is not guaranteed that everything returned is in English. It is possible for a document to contain words in another language that do not match the language-specific query, but do match the unstemmed query (if the document contains text in multiple languages, and if it has my words in some other language than the one specified in the stemmed cts:query).

• The word search indexes are language-agnostic.
• Unstemmed searches use the lang=<language> query constructor option to determine the language for tokenization.
• Unstemmed searches search all content, regardless of language (and regardless of lang=<language> option). The language only affects how the search terms are tokenized. For example, the following unstemmed search returns true:

  (: returns true :)
  let $x := <el xml:lang="fr">chat</el>
  return
  cts:contains($x, cts:word-query("chat", ("unstemmed", "lang=en")))

  whereas the following stemmed search returns false:

  (: returns false :)
  let $x := <el xml:lang="fr">chat</el>
  return
  cts:contains($x, cts:word-query("chat", ("stemmed", "lang=en")))

Unknown Languages

If the language specified in a search is not one of the languages in which language-specific stemming and tokenization are supported, or if it is a language for which you do not have a license key, then it is treated as a generic language. Typically, generic languages with Latin script are tokenized the same way as English, with token breaks at whitespace and punctuation, and with each word stemming to itself, but this is not always the case (especially for languages supported by MarkLogic Server--see Supported Languages--but for which you are not licensed). For details, see Generic Language Support.

You can implement a custom lexer (for tokenization) and stemmer if the default behavior for an unsupported language does not meet the needs of your application. For details, see User-Defined Lexer Plugins and Using a User-Defined Stemmer Plugin.

Tokenization Customization

With no customizations, each language has a default lexer and default tokenization dictionary associated with it. The default lexer is one of the built-in lexers described in Built-in Lexer Plugin Reference and varies by language.

You can use the following tools to customize tokenization. You can use these features singly or in combination.

• Define tokenizer overrides. Overrides can affect whether a codepoint is classified as a word, punctuation or whitespace character. Overrides are applied independent of the configured lexer. For details, see Custom Tokenizer Overrides.
• Custom tokenization dictionary. You can install a custom dictionary to influence how text is tokenized. You configure custom dictionaries per language. For more details, see Custom Dictionaries for Tokenizing and Stemming.
• Custom lexer. You can use one of the built-in lexer plugins that come with MarkLogic, or create a user-defined lexer plugin using the marklogic::LexerUDF native C++ intefaces. You associate a custom lexer with a specific language. For details, see User-Defined Lexer Plugins and Configuring Tokenization and Stemming Plugins.

You can use tokenization customizations in conjunction with stemming customizations. For details, see Stemming Customization.

Tokenization is a trusted operation. You should be selective about which users can register user-defined lexer plugins and customize language configurations.

Stemming Customization

With no customizations, each language has a base stemmer and stemming dictionary associated with it. The default stemmer is one of the built-in stemmer plugins that come with MarkLogic, and varies by language. For details, see Built-in Stemmer Plugin Reference.

You can use the following tools to customize stemming. You can use these customizations singly or in combination.

• Database text index options. For example, you can enable/disable stemmed searches, and set the level of complexity of the stemmer. For details, see Understanding the Text Index Settings in the Administrator's Guide.
• Custom dictionary. You can install a custom stemming dictionary to influence how words are stemmed. You configure custom dictionaries per language. For more details, see Custom Dictionaries for Tokenizing and Stemming.
• Custom stemmer. You can use one of the built-in stemmer plugins that come with MarkLogic, or create a user-defined stemmer plugin using the marklogic::StemmerUDF native C++ intefaces. You associate a custom stemmer with a specific language. For details, see Using a User-Defined Stemmer Plugin and Configuring Tokenization and Stemming Plugins.

You can use stemming customizations in conjunction with tokenization customizations. For details, see Tokenization Customization.

Stemming is a trusted operation. You should be selective about which users can register user-defined stemming plugins and customize language configurations.

Function Summary for Custom Language Management

Lexer and stemmer plugin configuration is done through the custom language management library module. The module includes the following functions. For more details, see the XQuery/XSLT Function Reference or the MarkLogic Server-Side JavaScript Function Reference.

Customization Using a Built-In Lexer or Stemmer

This sections describes how to construct a custom lexer or stemmer configuration item based on one of the built-in lexers or stemmers, rather than on a user-defined plugin.

Setting the library argument of clang:user-language-plugin or clang.userLanguagePlugin to an empty string tells MarkLogic you are referencing a built-in plugin. For example, the following call constructs a stemmer configuration item based on the built-in Snowball stemmer. Notice that the first parameter (library) is an empty string.

XQuery: clang:user-language-plugin("",(),clang:stemmer("snowball"))

JavaScript: clang.userLanguagePlugin('', null, clang.stemmer('snowball')

The first argument of the stemmer constructor should be one of the built-in stemmer names from Built-in Stemmer Plugin Reference. You can configure a custom lexer at the same time by including a clang.lexer or clang.lexer configuration item as the 3rd parameter.

If you associate a custom lexer dictionary with a language, you must reinstall it if you change the lexer plugin for the language. Similarly, if you associate a custom stemming dictionary with a language, you must reinstall it if you change the stemmer plugin for the language.

The following example creates a configuration item for German. The default lexer for German is ICU. The default stemmer for German is Bitext. The new configuration specifies Snowball as the custom lexer and leaves the default lexer unchanged. In addition, the Snowball stemmer is configured to use the german2 stemming algorithm.

xquery version "1.0-ml";
import module namespace clang =
  "http://marklogic.com/xdmp/custom-language" 
  at "/MarkLogic/custom-language.xqy";

let $stemmer :=
  clang:stemmer("snowball",(),("code=german2"))
let $plugin := clang:user-language-plugin("",(),$stemmer)
let $german := clang:user-language("de",$plugin)

return 
  clang:update-user-language(
    clang:language-config-read(), $german)

'use strict';
const clang = require('/MarkLogic/custom-language');

const stemmer = clang.stemmer(
  'snowball', null, Sequence.from(['code=german2']));
const plugin = clang.userLanguagePlugin('', null, stemmer);
const germanConfig = clang.userLanguage('de', plugin);

clang.updateUserLanguage(
    clang.languageConfigRead(), germanConfig);

Note that this example doesn't actually change the language configuration because it does not call clang:language-config-write (XQuery) or clang.languageConfigWrite (JavaScript).

If you run the example in Query Console, you should see output similar to the following:

<lang:user-languages xml:lang="zxx" xmlns:lang="http://marklogic.com/xdmp/language">
  <lang:user-language>
    <lang:name>de</lang:name>
    <lang:plugin>
      <lang:library/>
      <lang:stemmer>
        <lang:variant>snowball</lang:variant>
        <lang:arg>code=german2</lang:arg>
      </lang:stemmer>
    </lang:plugin>
  </lang:user-language>
</lang:user-languages>

Customization Using a User-Defined Lexer or Stemmer

This describes how to construct a custom lexer or stemmer configuration item based on a user-defined plugin, rather than on one of the built-in plugins. A user-defined lexer or stemmer must be installed as a native plugin before you can use it.

When you construct a lexer (or stemmer) configuration item for a user-defined plugin, you must identify the native plugin and the capability from the plugin library that exposes the LexerUDF or StemmerUDF implementation.

For a lexer, set the variant argument of clang:lexer or clang.lexer to a LexerUDF capability registered by plugin. For a stemmer, set the variant argument of clang:stemmer or clang.stemmer to a StemmerUDF capability registered by plugin. For both, set the library argument to plugin_path/plugin_id.

For example, if you install a plugin with the path native and plugin id sampleplugin, and the lexer UDF capability registered by the plugin is named sample_lexer, then you'd construct a lexer configuration item for it as follows:

XQuery: clang:lexer("sample_lexer", (), (), "native/sampleplugin")

JavaScript: clang.lexer('sample_lexer', null, null, 'native/sampleplugin')

If you configure both a stemmer and lexer from the same native plugin, you can set the plugin library reference (native/sampleplugin) in clang:user-language-plugin or clang.userLanguagePlugin instead. For example:

XQuery: clang:user-language-plugin(
          "native/sampleplugin", 
          clang:lexer("sample_lexer"),
          clang:stemmer("sample_stemmer"))

JavaScript: clang.userLanguagePlugin('
              'native/sampleplugin', 
              clang.lexer('sample_lexer'),
              clang.stemmer('sample_stemmer'));

When a library is specified in both the lexer/stemmer constructor and the language plugin constructor, the library in the lexer/stemmer takes precedence.

If you associate a custom lexer dictionary with a language, you must reinstall it if you change the lexer plugin for the language. Similarly, if you associate a custom stemming dictionary with a language, you must reinstall it if you change the stemmer plugin for the language.

The following example creates a configuration item for German. The default lexer for German is ICU. The default stemmer for German is Bitext. The new configuration specifies a user-defined lexer named sample_lexer as the custom lexer and leaves the default stemmer unchanged. Assume the plugin configuration described above.

xquery version "1.0-ml";
import module namespace clang =
  "http://marklogic.com/xdmp/custom-language" 
  at "/MarkLogic/custom-language.xqy";

let $lexer :=
  clang:lexer("sample_lexer",(),(), "native/sampleplugin")
let $plugin := clang:user-language-plugin("", $lexer, ())
let $german := clang:user-language("de", $plugin)

return 
  clang:update-user-language(
    clang:language-config-read(), $german)

'use strict';
const clang = require('/MarkLogic/custom-language');

const lexer = clang.lexer(
  'sample_lexer', null, null, 'native/sampleplugin');
const plugin = clang.userLanguagePlugin('', lexer, null);
const germanConfig = clang.userLanguage('de', plugin);

clang.updateUserLanguage(
    clang.languageConfigRead(), germanConfig);

Note that this example doesn't actually change the language configuration because it does not call clang:language-config-write (XQuery) or clang.languageConfigWrite (JavaScript).

If you run the example in Query Console, you should see output similar to the following:

<lang:user-languages xml:lang="zxx"
    xmlns:lang="http://marklogic.com/xdmp/language">
  <lang:user-language>
    <lang:name>de</lang:name>
    <lang:plugin>
      <lang:library/>
      <lang:lexer>
        <lang:library>native/sampleplugin</lang:library>
        <lang:variant>sample_lexer</lang:variant>
      </lang:lexer>
    </lang:plugin>
  </lang:user-language>
</lang:user-languages>

Example: Adding Configuration for a Language

Use the clang:user-language-plugin XQuery function or the clang.userLanguagePlugin Server-Side JavaScript function to define a binding between a language and custom tokenization and stemming plugins. For more details, see Customization Using a Built-In Lexer or Stemmer and Customization Using a User-Defined Lexer or Stemmer.

To put the configuration change into effect, use the following pattern. A complete example follow.

clang:language-config-write(
  clang:update-user-language(
    clang:language-config-read(), $changed-lang)
  )

clang.languageConfigWrite(
  clang.updateUserLanguage(
    clang.languageConfigRead(), changedLang));

This operation is an overwrite: Any previous configuration for the language will be replaced. Thus, if you are going to configure both a lexer and a stemmer for a language, do it in a single call to clang:update-user-language or clang.updateUserLanguage.

Calling clang:language-config-write or clang.languageConfigWrite causes MarkLogic to restart.

The following example configures a custom stemmer and lexer for the Catalan language using a user-defined plugin. Assume the plugin registers a lexer named sample_lexer and a stemmer named sample_stemmer.

xquery version "1.0-ml";
import module namespace clang="http://marklogic.com/xdmp/custom-language"
  at "/MarkLogic/custom-language.xqy";

(: Construct custom lexer and stemmer bindinf for Catalan :)
let $catalan :=
  clang:user-language("ca",
    clang:user-language-plugin("native/sampleplugin",
      clang:lexer("sample_lexer"),
      clang:stemmer("sample_stemmer")
    )
  )
(: Get the existing config so we update it :)
let $existing :=  clang:language-config-read()
return
  (: Update the current config and commit the changes :)
  (: NOTE: Causes a restart :)
  clang:language-config-write(
    clang:update-user-language($existing, $catalan)
  )

'use strict';
const clang = require('/MarkLogic/custom-language.xqy');

// Construct a custom lexer and stemmer binding for Catalan
const catalan = clang.userLanguage(
  'ca', 
  clang.userLanguagePlugin(
    'native/sampleplugin', 
    clang.lexer('sample_lexer'), 
    clang.stemmer('sample_stemmer'))
  );
// Get the existing config so we can update it
const existing = clang.languageConfigRead();

// Update the current config and commit the changes.
// NOTE: Causes a restart
clang.languageConfigWrite(
  clang.updateUserLanguage(existing, catalan));

You can configure just a lexer or just a stemmer for a language by including just that reference when calling clang:user-language-plugin or clang.userLanguagePlugin. For example, the following code only configures a custom stemmer.

xquery version "1.0-ml";
import module namespace clang="http://marklogic.com/xdmp/custom-language"
  at "/MarkLogic/custom-language.xqy";

let $catalan :=
  clang:user-language("ca",
    clang:user-language-plugin("native/sampleplugin",
      (), clang:stemmer("sample_stemmer")
    )
  )
let $existing :=  clang:language-config-read()
return
  clang:language-config-write(
    clang:update-user-language($existing, $catalan)
  )

'use strict';
const clang = require('/MarkLogic/custom-language.xqy');

const catalan = clang.userLanguage(
  'ca', 
  clang.userLanguagePlugin(
    'native/sampleplugin', null,
    clang.stemmer('sample_stemmer'))
  );
const existing = clang.languageConfigRead();

clang.languageConfigWrite(
  clang.updateUserLanguage(existing, catalan));

To configure a lexer and stemmer from different plugin libraries for the same language, specify the plugin path to the lexer and stemmer reference constructors. For example, the following code configures a lexer and a stemmer from two different plugins:

xquery version "1.0-ml";
import module namespace clang="http://marklogic.com/xdmp/custom-language"
  at "/MarkLogic/custom-language.xqy";

let $catalan :=
  clang:user-language("ca",
    clang:user-language-plugin("",
      clang:lexer("my_lexer",(),(), "plugin1/lexers"),
      clang:stemmer("my_stemmer", (), (), "plugin2/stemmers")
    )
  )
let $existing :=  clang:language-config-read()
return
  clang:language-config-write(
    clang:update-user-language($existing, $catalan)
  )

'use strict';
const clang = require('/MarkLogic/custom-language.xqy');

const catalan = clang.userLanguage(
  'ca', 
  clang.userLanguagePlugin(
    '', 
    clang.lexer('my_lexer', null, null, 'plugin1/lexers'), 
    clang.stemmer('my_stemmer', null, null, 'plugin2/stemmers'))
  );
const existing = clang.languageConfigRead();

clang.languageConfigWrite(
  clang.updateUserLanguage(existing, catalan));

Example: Removing Configuration for a Language

Use the clang:delete-user-language XQuery function or the clang.deleteUserLanguage JavaScript function to remove the custom configuration for a specific language. You must call clang:language-config-write (XQuery) or clang.languageConfigWrite (JavaScript) for your change to take effect, and doing so will cause MarkLogic to restart.

The following example removes the configuration for the Catalan language (language code ca).

xquery version "1.0-ml";
import module namespace clang="http://marklogic.com/xdmp/custom-language"
  at "/MarkLogic/custom-language.xqy";

let $language := "ca"
let $existing :=  clang:language-config-read()
return
  clang:language-config-write(
    clang:delete-user-language($existing, $language)
  )

'use strict';
const clang = require('/MarkLogic/custom-language.xqy');

const language = 'ca';
const existing = clang.languageConfigRead();

clang.languageConfigWrite(
  clang.deleteUserLanguage(existing, language));

Example: Resetting Configuration for All Languages

To remove custom stemmer and lexer bindings for all languages, use the clang:language-config-delete XQuery function or the clang.languageConfigDelete Server-Side JavaScript function.

Calling these functions restarts MarkLogic.

The following example code removes all language customizations and restarts the server.

xquery version "1.0-ml";
import module namespace clang="http://marklogic.com/xdmp/custom-language"
  at "/MarkLogic/custom-language.xqy";

clang:language-config-delete()

'use strict';
const clang = require('/MarkLogic/custom-language.xqy');

clang.languageConfigDelete();

Understanding Stemming Delegation

You can use delegation to control whether stemming consults the default stemmer in addition to the custom plugin for a language. Delegation can be controlled at two levels:

• User-defined stemming plugins and some built-in plugins include a delegation control on their interface. For example, the built-in Bitext plugin accepts a delegation option, and the StemmerUDF interface for user-defined plugins has a delegate method.
• The clang:user-language-plugin XQuery function and the clang.userLanguagePlugin JavaScript function accept a boolean delegate parameter. When set to true (the default), the stemming process asks the plugin whether or not to delegate.

If delegation is enabled at the language plugin configuration level, then the stemming process will consult the custom plugin about whether or not to delegate. For example, it will call the delegate method on StemmerUDF. If delegation is disabled at the language plugin configuration level, then the stemming process will not consult the custom plugin and will never delegate to the default stemmer.

Delegation has the following effect on the stemming results. The first column indicates whether or not the delegate parameter of clang:user-language-plugin or clang.userLanguagePlugin is set to true. The second column indicates whether or not the custom plugin agrees to delegate; for example, whether StemmerUDF::delegate returns true.

The following table contains examples of the stemming result with various delegation and stem count combinations. The with Delegation column signifies the plugin was consulted and agreed to delegation. The without Delegation column signifies either the plugin was not consulted or the plugin did not agree to delegation.

A custom plugin determines its own delegation policy. For example, a plugin might choose among policies such as always (delegate regardless of the number of stems found), never (never delegate), or on empty (delegate only if the plugin found no stems).

Custom Dictionary Security Considerations

When you configure a language to use a custom user-defined stemmer or lexer, and also associate a custom dictionary with the language, then you can create special security privileges to enable finer control over who can administer the dictionary.

A custom dictionary is associated with both a language and a specific stemmer or lexer plugin. The lexer or stemmer is implicit in the configuration of the language. Usually, any user with the custom-dictionary-admin role or equivalent privileges can add, update, or delete a custom dictionary for any language-stemmer or language-lexer configuration.

You can create a privileges of the following form to make it possible to control dictionary management on a per stemmer/lexer basis.

http://marklogic.com/xdmp/privileges/custom-dictionary-admin/library
http://marklogic.com/xdmp/privileges/xdmp-write-cluster-config-file/library

Where library is of the form plugin_path/plugin_id and identifies a user-defined lexer or stemmer plugin.

For example, if you install a user-defined lexer plugin with the plugin path native and the plugin id sampleplugin, then you would create a privileges of the following form:

http://marklogic.com/xdmp/privileges/custom-dictionary-admin/native/sampleplugin
http://marklogic.com/xdmp/privileges/xdmp-write-cluster-config-file/native/sampleplugin

MarkLogic will not create these privilege for you, but it will check for and enforce them if the privileges exist.

Built-in Lexer Plugin Reference

The table below lists the built-in lexers (tokenizers), which languages each one is configured for by default, and what configuration options (if any) are available for customization. Use the configuration lexer name and configuration options when calling the clang:lexer XQuery function or the clang.lexer JavaScript function; for details, see Customization Using a Built-In Lexer or Stemmer.

model_filename

search-mode
normal-mode

Built-in Stemmer Plugin Reference

The tables below list the built-in stemmers. Use the stemmer name and options when constructing a stemmer configuration item using the clang:stemmer XQuery function or the clang.stemmer JavaScript function.

MarkLogic uses the following built-in stemmers by default:

See the following topics for configuration options.

• Bitext Stemmer Options
• Snowball Stemmer Options
• Atilika Stemmer Options

decompounding
no-decompounding

use-algorithm
no-use-algorithm

use-dictionary
no-use-dictionary

lowercase
no-lowercase

nfkd
no-nfkd

code=value

add-reading
no-add-reading

delegation

Overview

Beginning with version 10.0-1 of our server, MarkLogic allows natural language in JSON to be tagged with a language other than the default database language. When a language or lang tag is present in a JSON object, all textual content in that object will be interpreted as being processed under the language referred to by the ISO code in that tag. JSON language processing is very similar to XML language processing (see the ksjdf section, for details) with the following differences:

• Because JSON has no attributes, the language applies to sibling properties, as well as children of these properties.
• The xml:lang attribute has a schema in scope, you if it isn't a valid ISO code you may get errors.
• If the language or langtag does not contain a valid language code (per RFC4646), then it will be treated analogously to such a value forxml:lang today: It will be treated as an unknown language for tokenization and stemming purposes.

A JSON document is allowed to have multiple language or lang tags in its content. A JSON node containing the key language will be processed according to that language. All descendant nodes will be processed according to that language. Language tags may be placed at any level in the JSON and are applied in a simple hierarchical way.

{
    language: "en-US", description: "This is US English text",
    components: [
        Still US english,
        {
            language: nl,
            data: Dutch stuff
        }
        {
            language: es,
            data: Spanish stuff
        }
        {
            data: More US English
        }
    ]
}

In the above example, content indexed with a particular language will have the key for that language added to the re-indexer keys stored with the document, as is now the case with XML content.

When JSON is being parsed (for example from a file), making the language tag apply to preceding siblings would be expensive and require us to parse the whole object before doing any node construction on it. Serialization will put the language child first.

API Changes

The fn:lang function and the underlying datamodel functions that support it now handle JSON nodes as well as XML nodes.

fn:lang($testlang as xs:string?, [$node as node()]) as xs:boolean

The function fn:lang already exists with the above signature. In previous versions, it always returns false for a JSON node. Starting with version 10.0-1 of MarkLogic, it will return true if the JSON node or one of its ancestors has a lang or language key that matches the $testlang per the rules defined for the xml:lang attribute on XML nodes.

For example, the following will return true.

fn:lang("en", object-node{ "language" : "en-US", "item" : "example" } )

JSON Serialization

Serialization of JSON objects will put the language tag first. (with the limitations noted above).

For example:

xdmp:to-json( object-node{ "item" : "example", "language" : "en-US" } )

will return

{"language":"en-US", "item":"example"}

Upgrade Considerations

It is possible that you may already have JSON documents may already have language or lang properties used for some other purpose. In that case, normal language processing attempts to look up a given language, and will treat all unknown tags as equivalent. The content of the language property itself will still be indexed normally: the issue is that content will be indexed as "unknown language" instead of "default database language". That is a potential incompatibility, and a potential risk. This risk is attenuated by the fact that some JSON formats already use language or lang for precisely the purpose we want gives us some comfort that this will not be an issue in practice. In addition, MarkLogic will only attempt to apply a language or lang property if the node is a text node.