Search Developer's Guide — Chapter 30

Overview of Collations

A collation specifies the order for sorting strings. The collation settings determine the order for operations where the order is specified (either implicitly or explicitly) and for operations that use Range Indexes. Examples of operations that specify the order are XQuery statements with an order by clause, XQuery standard functions that compare order (for example, fn:compare, fn:substring-after, fn:substring-before, and so on), and lexicon functions (for example, cts:words, cts:element-word-match, cts:element-values, and so on). Additionally, collations determine uniqueness in string comparisons, so two strings that are equal according to one collation might be not be equal according to another.

The codepoint-order collation sorts according to the Unicode codepoint order, which does not take into account any language-specific information. There are other collations that are often used to specify language-specific sorting differences. For example, a code point sort puts all uppercase letters before lower-case letters, so the word Zounds sorts before the word abracadabra. If you use a collation that sorts upper and lower-case letters together (for example, the order A a B b C c, and so on), then abracadabra sorts before Zounds.

Collations are specified with a URI (for example, http://marklogic.com/collation/). The collation URIs are specific to MarkLogic Server, but they specify collations according to the Unicode collation standards. There are many variations to collations, and many sort orders that are based on preferences and traditions in various languages. The following section describes the syntax of collation URIs. Although there are a huge number of collation URIs possible, most applications will use only a small number of collations. For more information about collations, see http://icu.sourceforge.net/userguide/Collate_Concepts.html.

Two Common Collation URIs

The following are two very common collation URIs used in MarkLogic Server:

• http://marklogic.com/collation/
• http://marklogic.com/collation/codepoint

The first one is the UCA Root Collation (see UCA Root Collation), and is the system default. The second is the codepoint order collation, and was the default in pre-3.2 releases of MarkLogic Server.

Collation URI Syntax

Collations in MarkLogic Server are specified by a URI. All collations begin with the string http://marklogic.com/collation/. The syntax for collations is as follows:

http://marklogic.com/collation/<locale>[/<attribute>]*

This section describes the following parts of the syntax:

• Locale Portion of the Collation URI
• Attribute Portion of the Collation URI

<locale> ::= <language>[-<script>][_<region>][@(collation=<value>;)+]

http://www.loc.gov/standards/iso639-2/php/code_list.php

http://www.unicode.org/iso15924/iso15924-codes.html

http://www.iso.org/iso/en/prods-services/iso3166ma/02iso-3166-code-lists/list-en1.html

en

http://marklogic.com/collation/en

en_US

http://marklogic.com/collation/en_US

zh

http://marklogic.com/collation/zh

de@collation=phonebook

http://marklogic.com/collation/de@collation=phonebook

<attribute> ::= <strength> | <case-level> | <case-first> | 
                <alternate> | <numeric-collation> | 
                <variable-top> | <normalization-checking> | 
                <french> | <hiragana>

<strength>

S1

S2

S3

S4

SI

<case-level>

EO

EX

<case-first>

CU

CL

CX

<alternate>

AN

AS

<numeric-collation>

MO

MX

<variable-top>

T0000

T0020

A B = AB and AB < A.B.

T00BB

<normalization-checking>

NO

NX

<french>

FO

FX

<hiragana>

HO

HX

Backward Compatibility with 3.1 Range Indexes and Lexicons

Range Indexes and lexicons that were created in MarkLogic Server 3.1 use the Unicode codepoint collation order. If you want them to use a different collation in any of these indexes and/or lexicons, you must change the collation and re-create the index, and then reindex the database (if reindex enable is set to true, it will automatically begin reindexing).

UCA Root Collation

The Unicode collation algorithm (UCA) root collation in MarkLogic Server is used when no default exists. It uses the Unicode codepoint collation with S3 (case and diacritic sensitive) strength, and it has the following URI:

http://marklogic.com/collation/

The UCA root collation adds more useful case and diacritic sensitivity to the Unicode codepoint order, so it will make more sensible sort orders when you take case sensitivity and diacritic sensitivity into consideration. For more details about the UCA, see http://www.unicode.org/unicode/reports/tr10/.

How Collation Defaults are Determined

The collation used for requests in MarkLogic Server is based on the settings of various parameters in the Admin Interface and on what is specified in your XQuery code. Each App Server has a default collation specified, and that is used in the absence of anything else that overrides it. Note the following about collations and their defaults.

• Collations are specified at the App Server level, on Range Indexes, and on lexicons.
• App Servers, Range Indexes, and lexicons upgraded from 3.1 remain in codepoint order (http://marklogic.com/collation/codepoint).
• New App Servers default to the UCA Root Collation (http://marklogic.com/collation/).
• New Range Indexes and lexicons default to UCA Root Collation (http://marklogic.com/collation/).
• You can specify a default collation in an XQuery prolog, which overrides the App Server default. For example, the following query will use the French collation:

  xquery version "1.0-ml";
  declare default collation "http://marklogic.com/collation/fr";
  
  for $x in ("côte", "cote", "coté", "côté", "cpte" )
  order by $x 
  return $x

• The codepoint collation URI is as follows:

  http://marklogic.com/collation/codepoint

  The following is an alias to the codepoint collation URI (used with the 1.0 strict XQuery dialect):

  http://www.w3.org/2005/xpath-functions/collation/codepoint

• Collation URIs displayed in the Admin Interface are stored and displayed as the canonical representation of the URI entered. The canonical representation is equivalent to the URI entered, but changes the order and simplifies portions of the collation URI string to a predetermined order. The xdmp:collation-canonical-uri built-in XQuery function returns the canonical URI of any valid collation URI.
• The empty string URI becomes codepoint collation. Therefore, the following returns as shown:

  xdmp:collation-canonical-uri("")
  => http://marklogic.com/collation/codepoint

• The collation used in an XQuery module is determined on a per-module basis. Therefore, a module might call another module that uses a different collation, as each module determines its collation independent of the module that called it (based on the App Server defaults, collation prolog declaration, and so on).
• When a module is invoked or spawned from another module, or when a request is submitted via an xdmp:eval call from another module, the new request inherits the collation context of the calling module. That context can be overridden in the query (for example, with a declare default collation expression in the prolog), but it will default to the context from the calling module.
• If no other collations are in effect (for example, for scheduled tasks), the codepoint collation is used.

Specifying Collations

You can specify collations in many places. Some common places to specify collations are:

• In the order by clause of a FLWOR expression.
• In an App Server configuration in the Admin Interface.
• In a lexicon or Range Index specification in the Admin Interface.
• In many W3C standard XQuery functions (for example, fn:compare, fn:contains, fn:starts-with, fn:ends-with, fn:substring-after, fn:substring-before, fn:deep-equals, fn:distinct-values, fn:index-of, fn:max, fn:min).
• In the lexicon APIs (cts:words, cts:word-match, cts:element-words, cts:element-values, and so on).
• In the range query constructors (cts:element-range-query, cts:element-attribute-range-query).