MarkLogic 10 Product Documentation

Administrator's Guide — Chapter 25

Understanding the Text Index Settings

The following table describes the different types of indexes available. The indexes are not mutually independent. If both the word search and stemmed search indexes are disabled, the configuration of the remaining indexes is irrelevant, as they all depend on the existence of the word and/or stemmed-search index.

Viewing Text Index Configuration

To view text index configuration for a particular database, complete the following procedure:

1. Click on the Databases icon on the left tree menu.
2. Locate the database for which you want to view text index configuration settings, either in the tree menu or in the Database Summary table.
3. Click the name of the database for which you want to view the settings.
4. Scroll down until the text index settings are visible. The following screen shots show the default configuration of text indexing for a database:

   

   

   

Configuring Text Indexes

To configure text indexes for a particular database, complete the following procedure:

1. Click on the Databases icon on the left tree menu.
2. Locate the database for which you want to view text index configuration settings, either in the tree menu or in the Database Summary table.
3. Click the name of the database for which you want to view the settings.
4. Scroll down until the text indexing controls are visible.
5. Configure the text indexes for this database by selecting the appropriate radio buttons for each index type.

   Click on the true radio button for a particular text index type if you want that index to be maintained. Click on the false radio button for a particular text index type if you do not want that index to be maintained.

   If word searches and stemmed searches are disabled (that is, the false radio button is selected for word searches and off is selected for stemmed searches), the settings for the other text indexes are ignored, as explained above.

6. Leave the rest of the parameters unchanged.
7. Scroll to the top or bottom of the right frame and click OK.

The database now has the new text indexing configurations.

Phrasing Control

By default, MarkLogic Server assumes that any XML element constructor acts as a phrase boundary. This means that phrase searches (for example, searches for sequences of terms) will not match a sequence of terms that contains one or more XML element constructors. Phrasing control lets you specify which XML elements should be transparent to phrase boundaries (for example, a bold or italic element), and which XML elements should be ignored for phrase purposes (for example, footnotes or graphic captions).

For example, consider the following sample XML fragment:

<paragraph>
  These two words <italic>are italicized</italic>. The italic element
  <footnote>Elements are defined in the W3C XML standard.</footnote>
  is a standard part of this document's schema.
</paragraph>

By default, MarkLogic Server would extract the following five sequences of text for phrase matching purposes (ignoring punctuation and case for simplicity):

• these two words
• are italicized
• the italic element
• elements are defined in the w3c xml standard
• is a standard part of this document's schema

If you then attempted to match the phrases words are italicized or element is a standard part against this XML fragment, no matches would be found, because of the embedded XML element constructors.

In fact, a human looking at this XML fragment would realize that the italic element should be transparent for phrasing purposes, and that the footnote element is a completely independent text container. Seen from this viewpoint, the XML fragment shown above contains only two text sequences (again, ignoring punctuation and case for simplicity):

• these two words are italicized the italic element is a standard part of this document's schema
• elements are defined in the w3c xml standard

In this case, words are italicized and element is a standard part would each properly generate a match. But a search for the w3c xml standard is a standard would not result in a match.

MarkLogic Server lets you achieve this type of phrasing control by specifying particular XML element names as phrase-through, phrase-around, and element-word-query-through elements:

Phrase controls are configured on a per-database basis. You should complete this configuration before loading any documents into the specified database; otherwise, in order for the changes to take effect with your existing content, you must either reload the content or reindex the database after changing the configuration.

Element Word Query Throughs

Element-word-query-throughs allow you to specify elements that should be included in text searches that use cts:element-word-query on a parent element. For example, consider the following XML fragment:

<a>
  <b>hello</b>
  <c>goodbye</c>
</a>

If you perform a cts:element-word-query on <a> searching for the word hello, the search does not find any matches in this fragment. The following query shows this pattern:

cts:search(fn:doc(), cts:element-word-query(xs:QName("a"), "hello"))

This query does not find any matches because cts:element-word-query only searches for text nodes that are immediate children of the element <a>, not text nodes that are children of any child nodes of <a>. Because hello is in a text node that is a child of <b>, it does not satisfy the cts:element-word-query.

If you add an element-word-query-through for the element <b>, however, then the cts:element-word-query on <a> searching for the word hello returns a match. The element-word-query-through on <b> causes the text node children of <b> behave like the text node children of its parent (in this case, <a>).

If an element is specified as a phrase-through, then it also behaves as an element-word-query-through, and therefore you do not need to specify it as an element-word-query-through.

Procedures

Use the following procedures to configure phrase controls for a particular database:

• Viewing Phrasing and Element-Word-Query Settings
• Configuring Phrasing and Element-Word-Query Settings
• Deleting a Phrasing or Element-Word-Query Setting

Understanding the Reindexer Enable Settings

At the database level, you can enable or disable automatic reindexing by setting the reindexer enable setting to true or false for that database. When the reindexer is enabled, any index or fragment changes to the database settings will cause all documents in the database that are not indexed/fragmented according to the settings to initiate a reindex operation. Note the following about the database settings and the reindex operation:

• When reindexing is enabled, the reindex operation runs as a background task. You can set a higher or lower priority on the reindexing task by increasing or decreasing the setting of the reindexer throttle.
• Any new documents added to or updated in the database will get the new database settings. This is true both with reindexing enabled and with reindexing disabled.
• After changing index or fragmentation settings in a database, because new or modified documents get the new settings, the database can get into a state where some documents are indexed/fragmented differently from other documents in the database.
• After changing index or fragmentation settings in a database in which reindexing is enabled, the old documents are reindexed according to the new settings, but the new settings do not take effect for queries until the reindex operation has completed and all documents are indexed to the state matching the database settings.
• After changing index or fragmentation settings in a database in which reindexing is disabled, new and changed documents get the current settings, but queries will not take advantage of the new settings until all documents in the database match the database settings.
• Even if reindexing is disabled, when you add tokenizer overrides to a field, those tokenization changes take effect immediately, so all new queries against the field will use the new tokenization (even if it is indexed with the previous tokenization).

Query Evaluation According to the Lowest Common Denominator

When queries are evaluated, they use the index settings that are calculated for the database at a given time. The current index settings for a query are determined at the time of query evaluation, and are based on the lowest common denominator of (that is, the index/fragmentation settings that are the least of) the following:

• The index/fragmentation settings defined in the database configuration.
• The actual index/fragmentation of documents/fragments in the database.

At any given time, the current lowest common denominator is invalidated upon the following events:

• system startup
• a change to the database configuration settings
• when a reindexing operation completes

If the lowest common denominator is invalidated, it is recalculated the next time a query is issued against the database.

The net impact is that, when index/fragmentation settings have changed on a database after any data is loaded, queries cannot take advantage of the new settings until the new settings meet the lowest common denominator criteria. Depending on the types of index setting changes you make, this can cause queries that behaved one way before index settings were changed to behave differently after the changes. The next section provides a sample scenario to help illustrate this behavior.

Reindexing Does Not Apply to Point-In-Time Versions of Fragments

If you have set a merge timestamp on the database to retain older versions of fragments for point-in-time queries, the older versions of the fragments will retain the indexing properties of the database at the time when they were updated. Because of this, reindexing a database that uses point-in-time queries can cause unpredictable query results. MarkLogic recommends that you do not reindex a database that has the merge timestamp parameter set to anything but 0. For details on point-in-time queries, see the Point-In-Time Queries chapter in the Application Developer's Guide. For details on setting the merge timestamp parameter, see Merges and Point-In-Time Queries.

Example Scenario

This section describes a simple scenario showing the effect of changing index settings on query behavior over time.

The following figure shows how changing the index settings can effect queries that initiate after index setting changes occur.

In this scenario, the query issued at time T3 sees the doc1 document with stemming and 3-character wildcard indexes enabled. Wildcard queries such as abc* will be successful. The same wildcard query at time T5, however, will not be successful, because the 3-character index (which is required for the abc* query) was disabled at time T4. Note that the document doc1 is actually indexed with 3-character and stemming, but the query at time T5 only is able to use the stemming index. At time T7, the database has doc1 indexed with both stemming and 3-character indexes, but doc2 only has the stemming index. With reindexing disabled, the query at T7 will use the lowest common denominator, which is in this case stemming.