MarkLogic Server 11.0 Product Documentation

Search Developer's Guide — Chapter 24

The Spelling Built-In Functions

The spelling correction functions are built-in functions and do not require the import module statement in the XQuery prolog. The following are the spelling correction functions:

The spell:double-metaphone / spell.doubleMetaphone and spell:levenshtein-distance / spell.levenshteinDistance functions return the raw values from which spell:suggest / spell.suggest, spell:suggest-detailed / spell.levenshteinDistance, and spell:is-correct / spell.isCorrect calculate their values.

The difference between spell:suggest (JavaScript spell.suggest) and spell:suggest-detailed (JavaScript spell.suggestDetailed) is that spell:suggest-detailed provides some of the information used in calculating the suggestions, and it returns a report (an XML representaiton in XQuery and an array of objects in JavaScript), whereas spell:suggest returns a sequence of suggested words. For most spelling applications, spell:suggest is sufficient, but if you want finer control of the suggestions you provide (for example, if you want to calculate your own order of returning the suggestions), you can use spell:suggest-detailed and then filter on some of the criteria returned in its XML or JSON output.

The Spelling Dictionary Management Module Functions

There is an XQuery module to perform management of dictionary documents. You can use this module in either XQuery or in Server-Side JavaScript. The spelling correction dictionary management functions are installed into the following XQuery module file:

• install_dir/Modules/MarkLogic/spell.xqy

where install_dir is the directory in which MarkLogic Server is installed. The functions in the spelling module use the spell: namespace prefix, which is predefined in the server.

To use the functions in this module in an XQuery program, include the module declaration in the prolog of your XQuery program as follows:

import module namespace spell = "http://marklogic.com/xdmp/spell" 
   at "/MarkLogic/spell.xqy";

To use the functions in this module in a JavaScript program, include a line similar to the following in your Server-Side JavaScript program:

const spell = require("/MarkLogic/spell");

XML Dictionary Document

Any XML dictionary documents loaded into MarkLogic must have the following basic structure:

<dictionary xmlns="http://marklogic.com/xdmp/spell">
    <metadata>
    </metadata>
    <word></word>
    <word></word>
    ......
</dictionary>

The <metadata> element is optional. Use spell:make-dictionary / spell.makeDictionary and spell:load / spell.load to create your own dictionary documents.

JSON Dictionary Document

Any JSON dictionary documents loaded into MarkLogic must have the following basic structure:

{
 "metadata": { ... },
 "words": ["word1", "word2", ... ]
}

The metadata property is optional. Use spell:make-dictionary / spell.makeDictionary and spell:load / spell.load to create your own dictionary documents.

Loading Dictionary Documents in XQuery

To use a dictionary in a query, it must be in the database. To load a dictionary document using XQuery, use the spell:load function or the spell:insert function. For example, to load a dictionary document with a URI /mySpell/spell.xml, execute a query similar to the following:

xquery version "1.0-ml";
import module "http://marklogic.com/xdmp/spell" 
  at "/MarkLogic/spell.xqy";

spell:load("c:\dictionaries\spell.xml", "/mySpell/spell.xml")

This XQuery adds all of the <word> elements from the c:\dictionaries\spell.xml file to a dictionary with the URI /mySpell/spell.xml. If the document already exists, then it is overwritten with the new content from the specified file.

Loading Dictionary Documents in JavaScript

To use a dictionary in a query, it must be in the database. To load a dictionary document using JavaScript, use the spell.load function or the spell.insert function. For example, to load a dictionary document with a URI /mySpell/spell.json, execute a program similar to the following:

const spell = require("/MarkLogic/spell");
declareUpdate();
spell.load("c:/dictionaries/spell.json", "/mySpell/spell.json");

This loads the file at the specified path into the dictionary JSON document at the specified URI.

Loading one of the Sample XML Dictionaries

You can download a sample dictionary from the MarkLogic Community site (). The community site links to github, which has small, medium, and large versions of the dictionary. Once you download a dictionary XML file, you can load it as a dictionary document using the spell:load function.

Perform the following steps to download and load a sample dictionary:

1. Go to the Code page of developer.marklogic.com:

   http://developer.marklogic.com/code/#dictionaries

2. Navigate to the dictionary document section, then click the github link:

   https://github.com/marklogic/dictionaries

3. In the dictionaries folder, choose the small-dictionary.xml, medium-dictionary.xml, or large-dictionary.xml file (or any other dictionary docments that might be available). The large dictionary has approximately 100,000 words and is about 3 MB to download. ALternately, you can choose small-dictionary.json, medium-dictionary.json, or large-dictionary.json file to load a JSON dictionary.
4. Save <size>-dictionary.xml (or the corresponding JSON document) to a file (for example, c:\dictionaries\spell.xml).
5. Load the dictionary with a command similar to the following:

   xquery version "1.0-ml";
   import module "http://marklogic.com/xdmp/spell" at
                 "/MarkLogic/spell.xqy";
   
   spell:load("c:\dictionaries\spell.xml", "/mySpell/spell.xml")

Updating a Dictionary Document

Use the following dictionary functions to modify existing dictionary documents:

• spell:add-word / spell.addWord
• spell:remove-word / spell.removeWord

The spell:insert XQuery function or the spell.insert JavaScript function will overwrite an existing dictionary if you specify an existing dictionary document (as well as creates a new one if one does not exist at the specified URI).

The transactional unit in MarkLogic Server is a query; therefore, if you are performing multiple updates to the same dictionary document, be sure to perform those updates as part of separate queries. In XQuery, you can place a semi-colon between the update statements to start a new query (and therefore a new transaction). If you use a semicolon to start any new queries that uses spelling correction functions in XQuery, each query must include the import statement in the prolog to resolve the spelling module.

The following topics are about updating dictionary documents:

• Example: Adding a New Word to a Dictionary in XQuery
• Example: Adding a New Word to a Dictionary in JavaScript
• Example: Removing a Word From a Dictionary in XQuery
• Example: Removing a Word From a Dictionary in JavaScript

xquery version "1.0-ml";
import module "http://marklogic.com/xdmp/spell" at
              "/MarkLogic/spell.xqy";

spell:add-word("/mySpell/spell.xml", "albatross")

const spell = require("/MarkLogic/spell.xqy");
declareUpdate();

spell.addWord("/mySpell/spell.json", "albatross");

xquery version "1.0-ml";
import module "http://marklogic.com/xdmp/spell" at
              "/MarkLogic/spell.xqy";

spell:remove-word("/mySpell/spell.xml", "albatross")

const spell = require("/MarkLogic/spell.xqy");
declareUpdate();

spell.removeWord("/mySpell/spell.json", "albatross")

Security Considerations With Dictionary Documents

Dictionary documents are stored in XML or JSON format in the database. Therefore, they can be queried just like any other document. Note the following about security and dictionary documents:

• By default, dictionary documents are loaded into the following collections:
  • http://marklogic.com/xdmp/documents
  • http://marklogic.com/xdmp/spell
• Dictionary documents are loaded with the default permissions of the user who loads them. Make sure users who load dictionary documents have appropriate privileges, otherwise the documents might not have the needed permissions for reading and updating. For more information, see Setting Document Permissions in the Loading Content Into MarkLogic Server Guide.
• If you want to control access (read and/or write) to dictionary documents beyond the default permissions with which the documents are loaded, perform an xdmp:document-set-permissions (JavaScript xdmp.documentSetPermissions) after a spell:load / spell.load operation.