Administrator's Guide — Chapter 19

Terms Used in this Chapter

• A Partition is a set of forests sharing the same name prefix and same partition range definition. Typically forests in a partition share the same type of storage and configuration such as updates allowed, availability, and enabled status. Partitions are based on forest naming conventions. A forest's partition name prefix and the rest of the forest name are separated by a dash (-). For example, a forest named 2011-0001 belongs to the 2011 partition.
• A Partition Key defines an element or attribute on which a range index, collection lexicon, or field is set and defines the context for the range set on the partitions in the database. The partition key is a database-level setting.
• A Partition Range defines a range of values for a partition. Documents with a partition key value that fall within the range specified for a partition are stored in that partition.
• A Default Partition is a partition with no defined range. Documents that have no partition key or a partition key value that does not fall into any of the partition ranges are stored in the default partition.
• A Super-database is a database containing other databases (sub-databases) so that they can be queried as if they were a single logical database.
• A Sub-database is a database contained in a super-database.
• Active Data is data that requires low-latency queries and updates. The 'activeness' of a particular document is typically determined by its recency and thus changes over time.
• Historical Data is less critical for the lowest-latency queries than 'active' data, but still requires online access for queries. Historical data is not typically updated.
• Archived Data is data that has aged beyond its useful life in the online storage tiers and is typically taken offline.
• An Online partition or forest is available for queries and updates.
• An Offline partition or forest is not available for queries, but is tracked by the cluster. The benefit of taking data offline is to spare the RAM, CPU, and network resources for the online data.
• The Availability of a partition or forest refers to its online/offline status.

Overview of Tiered Storage

The MarkLogic tiered storage APIs enable you to actively and easily move your data between different tiers of storage. For example, visualize how data might be tiered in different storage devices in a pyramid-like manner, as illustrated below.

As data ages and becomes less updated and queried, it can be migrated to less expensive and more densely-packed storage devices to make room for newer, more frequently accessed and updated data, as illustrated in the graph below.

The illustration below shows the basic tiered storage operations:

• Migrate a partition to a different database, host, and/or directory, which may be mounted on another storage device.
• Resize the partition to expand or contract the number of forests it contains.
• Combine a number of forests into a single forest.
• Reset the update-allowed state of a partition. For example, make the partition read-only, so it can be stored more compactly on a device that is not required to reserve space for forest merges.
• Take a partition offline to archive the partition. The partition data is unavailable to query, update, backup, restore and replicate operations.
• Take a partition online to make the partition data available again.
• Delete a partition when its data has outlived its useful life.

Forest migrate, forest combine, partition migrate and partition resize may result in potential data loss when used during XA transactions.

Partitions, Partition Keys, and Partition Ranges

MarkLogic Server tiered storage manages data in partitions. Each partition consists of a group of database forests that share the same name prefix and the same partition range.

When deploying forests in a cluster, you should align forests and forest replicas across hosts for parellelization and high availability, as described in the Scalability, Availability, and Failover Guide.

The range of a partition defines the scope of element or attribute values for the documents to be stored in the partition. This element or attribute is called the partition key. The partition key is based on a range index, collection lexicon, or field set on the database. The partition key is set on the database and the partition range is set on the partition, so you can have several partitions in a database with different ranges.

For example, you have a database, named WorkingVolumes, that contains nine forests that are grouped into three partitions. Among the range indexes in the WorkingVolumes database is an element range index for the update-date element with a type of date. The WorkingVolumes database has its partition key set on the update-date range index. Each forest in the WorkingVolumes database contains a lower bound and upper bound range value of type date that defines which documents are to be stored in which forests, as shown in the following table:

When Lower Bound Included is set to false on a database, the lower bound of the partition ranges are ignored. With this setting, documents with a partition key value that match the lower bound value are excluded from the partition and documents that match the upper bound value are included.

In this example, a document with an update-date element value of 2011-05-22 would be stored in one of the forests in the Vol2 partition. Should the update-date element value in the document get updated to 2012-01-02 or later, the document will be automatically moved to the Vol3 partition. How the documents are redistributed among the partitions is handled by the database rebalancer, as described in Range Assignment Policy.

Below is an illustration of the WorkingVolumes database, showing its range indexes, partition key, and its partitions and forests.

In a few months, the volumes of documents grow to 5 and there is no longer enough space on the fast SSD device to hold all of them. Instead, the oldest and least queried volumes (Vol1-Vol3) are migrated to a local disk drive, which represents a slower storage tier.

After years of data growth, the volumes of documents grow to 50. After migrating between storage tiers, the partitions are eventually distributed among the storage tiers, as shown below.

Multiple databases, even those that serve on different storage tiers, can be grouped into a super-database in order to allow a single query to be done across multiple tiers of data. Databases that belong to a super-database are referred to as sub-databases. A single sub-database can belong to multiple super-databases.

Configuring a Database to Participate in Tiered Storage

If a database is to participate in a tiered storage scheme, it must have the following set:

• Rebalancer enable set to true
• Rebalancer Assignment Policy set to range
• Locking set to strict
• A range index established for the partition key, as described in Range Indexes and Lexicons
• A partition key, as described in Defining a Partition Key
• Partitions, as described in Creating Partitions

  All of the forests in a database configured for tiered storage must be part of a partition.

For details on how to configure the database rebalancer with the range assignment policy, see the sections Range Assignment Policy, Configuring the Rebalancer on a Database, and Configuring the Rebalancer on a Forest.

Defining a Partition Key

The partition key describes a common element or attribute in the stored documents. The value of this element or attribute in the document determines the partition in which the document is stored. A partition key is based on a range index, collection lexicon, or field of the same name set for the database. The range index, collection lexicon, or field used by the partition key must be created before the partition key is created.

For example, assume your documents all have an update-date element with a date value. The following procedure describes how to create a partition key for the update-date element:

1. Create an element range index, named update-date, on the database of type date. The details on how to create an element range index are described in Defining Element Range Indexes.
2. In the Admin UI, open the configuration page for the database, set the assignment policy to range. Additional settings appear under the assignment policy.

   

3. Set the Lower Bound Included to true if you want to include documents with a partition key value that matches the lower bound value and exclude documents that match the upper bound value. Set the Lower Bound Included to false, if you want to exclude documents with a partition key value that matches the lower bound value and include documents that match the upper bound value. For example, if the range is 2011-01-01 (lower) to 2012-01-01 (upper) and Lower Bound Included is set to false, documents with an update-date value of 2011-01-01 will not be included in the partition, but documents with an update-date value of 2011-01-02 and 2012-01-01 will be included.
4. Note the type and scalar type of the range index, field, or collection lexicon you want to use as your partition key. In this example, we use an Element range index with a scalar type of date. Set the index and scalar types in the drop down menus to list the matching range indexes, fields, or collection lexicons set for the database.

   

5. Select the range index, field, or collection lexicon you want to use as your partition key, which is update-date in this example.

   

Creating Partitions

Partitions are based on forest naming conventions. A forest's partition name prefix and the rest of the forest name are separated by a dash (-). For example, a forest named June-0001 belongs to the June partition.

It is a best practice to create a default partition (a partition without a range) before creating partitions with ranges. Doing this will allow you to load documents into the default partition before you have finished creating the other partitions. As new partitions with ranges are created, the documents will be automatically moved from the default partition to the partitions with matching ranges.

All of the forests in a database configured for tiered storage must be part of a partition.

There are two ways to create a partition:

• Creating a Partition with New Forests
• Creating a Partition from Existing Forests

$ cat create-partition.xml
<partition xmlns="http://marklogic.com/manage">
  <partition-name>2011</partition-name>
  <upper-bound>2012-01-01</upper-bound>
  <lower-bound>2011-01-01</lower-bound>
  <forests-per-host>4</forests-per-host>
  <hosts>
    <host>MyHost1</host>
    <host>MyHost2</host>
  </hosts>
</partition>
$ curl --anyauth --user user:password -X POST \
-d@"./create-partition.xml" -H 'Content-type: application/xml' \
'http://MyHost:8002/manage/v2/databases/Documents/partitions'

Overview of the Tiered Storage REST API

Tiered storage is supported by the XQuery, JavaScript, and REST APIs. All of the operations you will want to integrate into your storage-management scripts to automate repetitive storage management operations are available through the REST API. However, some of the initial, one-time set-up operations, such as those related to setting the range policy and partition key on the database, are only supported by the Admin Interface and the XQuery API.

The Tiered Storage REST API supports both JSON and XML formats. The XML format is used for all of the examples in this chapter.

The topics in this section are:

• Asynchronous Operations
• Privileges
• /manage/v2/databases/{id|name}/partitions
• /manage/v2/databases/{id|name}/partitions/{name}
• /manage/v2/databases/{id|name}/partitions/{name}/properties
• /manage/v2/databases/{id|$name}/sub-databases
• /manage/v2/databases/{id|name}/sub-databases/{id|name}
• /manage/v2/databases/{id|name}/super-databases
• /manage/v2/databases/{id|name}/super-databases/{id|name}
• /manage/v2/forests
• /manage/v2/forests/{id|name}
• /manage/v2/forests/{id|name}/properties

/manage/v2/tickets/{id}?view=process-status.

/manage/v2/tickets/8681809991198462214?view=process-status

http://MyHost:8002/manage/v2/tickets/8681809991198462214?view=process-status

Common Forest and Partition Operations

This section describes the following partition operations:

• Viewing Partitions
• Migrating Forests and Partitions
• Resizing Partitions
• Transferring Partitions between Databases
• Combining Forests
• Retiring Forests
• Taking Forests and Partitions Online and Offline
• Setting the Updates-allowed State on Partitions
• Deleting Partitions

Some of these operations operate asynchronously and immediately return a ticket number that you can use to check the status of the operation. For example, if the following is returned:

<link><kindref>process-status</kindref><uriref>/manage/v2/tickets/4678516920057381194?view=process-status</uriref></link>

You can check the status of the operation by entering a resource address like the following:

http://MyHost:8002/manage/v2/tickets/4678516920057381194?view=process-status

For details on asynchronous processes, see Asynchronous Operations.

curl -X GET --anyauth --user admin:admin --header \
"Content-Type:application/xml" \
http://MyHost:8002/manage/v2/databases/Documents/partitions/2011

$ cat migrate-partition.xml
<migrate xmlns="http://marklogic.com/manage">
  <hosts>
    <host>OurHost</host>
  </hosts>
  <data-directory>/warm-storage</data-directory>
  <options>
    <option>failover=none</option>
    <option>local-to-shared</option>
  </options>
</migrate>
$ curl --anyauth --user user:password -X PUT \
-d@"./migrate-partition.xml" -H 'Content-type: application/xml' \
'http://MyHost:8002/manage/v2/databases/Documents/partitions/2011'

$ cat migrate-forests.xml
<forest-migrate xmlns="http://marklogic.com/manage">
  <forests>
    <forest>2011-0001</forest>
    <forest>2011-0002</forest>
  </forests>
  <host>MyHost</host>
  <data-directory>/warm-storage</data-directory>
  <options>
    <option>local-to-shared</option>
  </options>
</forest-migrate>
$ curl --anyauth --user user:password -X PUT \
-d@"./migrate-forests.xml" -H 'Content-type: application/xml' \
'http://MyHost:8002/manage/v2/forests'

$ cat resize-partition.xml
<resize xmlns="http://marklogic.com/manage">
  <forests-per-host>5</forests-per-host>
  <hosts>
    <host>MyHost</host>
  </hosts>
</resize>
$ curl --anyauth --user user:password -X PUT \
-d@"./resize-partition.xml" -H 'Content-type: application/xml' \
'http://MyHost:8002/manage/v2/databases/Documents/partitions/2011'

$ cat transfer-partition.xml
<transfer xmlns="http://marklogic.com/manage">
  <destination-database>Documents2</destination-database>
</transfer>
$ curl --anyauth --user user:password -X PUT \
-d@"./transfer-partition.xml" -H 'Content-type: application/xml' \
'http://MyHost:8002/manage/v2/databases/Documents1/partitions/2011'

$ cat combine-forests.xml
<forest-combine xmlns="http://marklogic.com/manage">
  <forests>
    <forest>2011-0001</forest>
    <forest>2011-0002</forest>
  </forests>
  <forest-name>2011</forest-name>
  <hosts>
    <host>MyHost</host>
  </hosts>
</forest-combine>
$ curl --anyauth --user user:password -X PUT \
-d@"./combine-forests.xml" -H 'Content-type: application/xml' \
'http://MyHost:8002/manage/v2/forests'

curl -i -X POST --digest --user user:password -H \
"Content-Type:application/x-www-form-urlencoded" \
--data "state=retire&database=Documents" \
http://MyHost:8002/manage/v2/forests/2011

$ cat partition-offline.xml
<partition-properties xmlns="http://marklogic.com/manage">
  <availability>offline</availability>
</partition-properties>
$ curl --anyauth --user user:password -X PUT \
-d@"./partition-offline.xml" -H 'Content-type: application/xml' \
'http://MyHost:8002/manage/v2/databases/Documents/partitions/2011/properties'

$ cat read-only-partition.xml
<partition-properties xmlns="http://marklogic.com/manage">
  <updates-allowed>read-only</updates-allowed>
</partition-properties>
$ curl --anyauth --user user:password -X PUT \
-d@"./read-only-partition.xml" -H 'Content-type: application/xml' \
'http://MyHost:8002/manage/v2/databases/Documents/partitions/2011/properties'

$ curl --anyauth --user user:password -X DELETE \
-H 'Content-type: application/xml' \
'http://MyHost:8002/manage/v2/databases/Documents/partitions/2011'

Sub-databases and Super-databases

Multiple databases can be grouped into a super-database in order to allow a single query to be done across multiple databases distributed on different storage tiers. Databases contained in a super-database are called sub-databases. A sub-database can be either active (online) or archive (offline), as specified by the kind element.

Only one level of sub-databases is supported for a super-database, which means that a sub-database cannot also be configured as a super-database with sub-databases of its own.

Sub-databases and their super-databases must have the same index settings. Otherwise, queries will not work.

Because super-databases and their sub-databases are effectively a single database, you cannot have documents with the same URI in super-databases and their sub-databases. It is a best practice to use directories to ensure that your document URIs are unique.

The operations described in this section are:

• Creating Sub-databases
• Creating Super-databases
• Viewing Super-databases and Sub-databases

$ cat create-subdatabase.xml
<sub-database xmlns="http://marklogic.com/manage">
  <database-name>subdb1</database-name>
  <kind>active</kind>
</sub-database>
$ curl --anyauth --user user:password -X POST \
-d@"./create-subdatabase.xml" -H 'Content-type: application/xml' \
'http://MyHost:8002/manage/v2/databases/superdb1/sub-databases'

$ cat create-superdatabase.xml
<super-database xmlns="http://marklogic.com/manage">
  <database-name>superdb1</database-name>
</super-database>
$ curl --anyauth --user user:password -X POST \
-d@"./create-superdatabase.xml" -H 'Content-type: application/xml' \
'http://MyHost:8002/manage/v2/databases/subdb1/super-databases'

$ curl --anyauth --user user:password -X GET \
-H 'Content-type: application/xml' \
'http://MyHost:8002/manage/v2/databases/subdb1/super-databases'

$ curl --anyauth --user user:password -X GET \
-H 'Content-type: application/xml' \
'http://MyHost:8002/manage/v2/databases/superdb1/sub-databases'

Partitions with Forest-Level Failover

The partition create, migrate and resize operations allow you to specify an options element to create replica forests for shared-disk or local-disk failover, as described in the Configuring Local-Disk Failover for a Forest and Configuring Shared-Disk Failover for a Forest chapters in the Scalability, Availability, and Failover Guide.

To create replica forests for forest-level failover, you must create the partition on at least two hosts. For each master forest created on one host a replica forest will be created on another host. For example, to create a single replica forest for each forest in the partition and configure the forests for local-disk failover between MyHost1, MyHost2, and MyHost3, do the following.

$ cat create-partition.xml
<partition xmlns="http://marklogic.com/manage">
  <partition-name>2011</partition-name>
  <upper-bound>2012-01-01</upper-bound>
  <lower-bound>2011-01-01</lower-bound>
  <forests-per-host>4</forests-per-host>
  <data-directory>/forests</data-directory>
  <hosts>
    <host>MyHost1</host>
    <host>MyHost2</host>
    <host>MyHost3</host>
  </hosts>
  <data-directory></data-directory>
  <large-data-directory></large-data-directory>
  <fast-data-directory></fast-data-directory>
  <options>
    <option>replicas=1</option>
    <option>failover=local</option>
  </options>
</partition>
$ curl --anyauth --user user:password -X POST \
-d@"./create-partition.xml" -H 'Content-type: application/xml' \
'http://MyHost:8002/manage/v2/databases/Documents/partitions'

Keep in mind the following when configuring partitions or forests with forest-level failover:

• If failover is configured on your forests, do a full backup of database after doing a partition or forest migrate or a forest combine to ensure that you can recover your data should something go wrong. You may also need to increase the timeout setting on the migrate or combine operation, as these operations will take longer when failover is configured.
• It is not recommended to configure local-disk failover for forests attached to a database with journaling set to off.
• You cannot configure a partition with shared-disk or local-disk failover on Amazon Simple Storage Service (S3), unless its fast data directory, as designated by <fast-data-directory>, is not on S3.
• If your deployment of MarkLogic is on Amazon Elastic Compute Cloud (EC2) or is distributed across multiple data centers, be sure to specify an equal number of hosts on different zones when creating, migrating, or resizing your partition with forest-level failover. For example, two hosts on us-east-1a, two hosts on us-east-1b, and two hosts on us-east-1c. In this example, tiered storage will ensure that master and their replica forests are created on hosts in different zones. This ensures that the partition will remain accessible should a forest, host, or entire zone go down.