MarkLogic 10 Product Documentation

Loading Content Into MarkLogic Server — Chapter 4

Consider If You Really Want to Specify a Forest

For most applications, you should not specify the forest in which you want to load a document. MarkLogic Server has efficient ways of determining which forest to load a document, and those ways are almost always better than explicitly specifying the forest. The default way MarkLogic spreads documents across forests is optimized for both query and loading efficiency. If you are using Tiered Storage (for details, see Tiered Storage), it has its own way of partitioning documents that you should follow.

One of the pitfalls of specifying a forest, is that the URI you are loading may already exist in another forest within the same database. This is a form of content corruption and will cause searches that select that URI to return with an XDMP-DBDUPURI error. If you run into this error, this Knowledge Base article contains a solution as well as some strategies for preventing duplicate URIs.

If you really want to specify the forest to which you load a document, the following describes some details about forest-specific loading.

Some Potential Advantages of Specifying a Forest

Because backup operations are performed at either the database or the forest level, loading a set of documents into specific forests allows you to effectively perform backup operations on that set of documents (by backing up the database or forest, for example).

Specifying a forest also allows you to have more control over the filesystems in which the documents reside. Each forest configuration includes a directory where the files are stored. By specifying the forest in which a document resides, you can control the directories (and in turn, the filesystems) in which the documents are stored. For example, you might want to place large, frequently accessed documents in a forest which resides on a RAID filesystem with complete failover and redundancy, whereas you might want to place documents which are small and rarely accessed in a forest which resides in a slower (and less expensive) filesystem.

Once a document is loaded into a forest, you cannot move it to another forest. If you want to change the forest in which a document resides, you must reload the document and specify another forest.

Example: Examining a Document to Decide Which Forest to Specify

You can use the xdmp:document-get function to a load a document into memory. One use for loading a document into memory is the ability to perform some processing or logic on the document before you load the document onto disk.

For example, if you want to make a decision about which forest to load a document into based on the document contents, you can put some simple logic in your load script as follows:

let $memoryDoc := xdmp:document-get("c:\myFiles\newDocument.xml")
let $forest :=
     if( $memoryDoc//ID gt "1000000" )
     then xdmp:forest("LargeID")
     else xdmp:forest("SmallID")
return
     xdmp:document-insert("/myCompany/newDocument.xml", 
                          $memoryDoc,
          xdmp:default-permissions(), 
          xdmp:default-collections(), 
                          0,
                          $forest)

This code loads the document newDocument.xml into memory, finds the ID element in the in-memory document, and then inserts the node into the forest named LargeID if the ID is greater than 1,000,000, or inserts the node into the forest named SmallID if the ID is less than 1,000,000.

More Examples

The following command loads the document into the forest named myForest:

xdmp:document-load("c:\myFile.xml",
       <options xmlns="xdmp:document-load">
          <uri>/myDocs/myDocument.xml</uri>
          <permissions>{xdmp:default-permissions()}</permissions> 
          <collections>{xdmp:default-collections()}</collections> 
          <repair>full</repair>
          <forests>
            <forest>{xdmp:forest("myForest")}</forest>
          </forests>
        </options> )

The following command loads the document into either the forest named redwood or the forest named aspen:

xdmp:document-load("c:\myFile.xml",
       <options xmlns="xdmp:document-load">
          <uri>/myDocs/myDocument.xml</uri>
          <permissions>{xdmp:default-permissions()}</permissions> 
          <collections>{xdmp:default-collections()}</collections> 
          <repair>full</repair>
          <forests>
            <forest>{xdmp:forest("redwood")}</forest>
            <forest>{xdmp:forest("aspen")}</forest>
          </forests>
        </options> )