MarkLogic Server 11.0 Product Documentation

Temporal Developer's Guide — Chapter 3

Creating Temporal Collections

There are a number of ways to create a temporal collection, depending on how you want to structure your temporal documents. The fundamental types of temporal collections are:

• Bi-temporal Collections
• Uni-temporal Collections

const temporal = require("/MarkLogic/temporal.xqy");
const collectionResult = temporal.collectionCreate(
"uniTemp1", "sys-axes");

declareUpdate();
const temporal = require("/MarkLogic/temporal.xqy");
const root =
    {"tempdoc": 
       {"content": "content here"}
    };
temporal.documentInsert("uniTemp1", "doc.json", root);

const temporal = require("/MarkLogic/temporal.xqy");
const collectionResult = temporal.collectionCreate(
"uniTemp2", "systemAxes");

declareUpdate();
const temporal = require("/MarkLogic/temporal.xqy");
const root =
    {"tempdoc": 
       {"content": "content here"}
    };
const options =
    {metadata:
       {systemStart: "1601-01-01T13:59:00Z",
        systemEnd: "9999-12-31T11:59:59Z"}
    };
temporal.documentInsert("uniTemp2", "doc.json", root, options);

Inserting and Loading Temporal Documents

There are a number of ways to insert and update temporal documents in MarkLogic Server. These include:

• The XQuery functions, temporal:document-insert and temporal:document-load.
• The REST Client API resource addresses, POST /v1/documents and PUT /v1/documents with the temporal-collection parameter.
• The Java Client API, as described in Working with Temporal Documents in the Java Application Developer's Guide.
• The Node.js Client API, through the documents interface. For details, see the Node.js Client API Reference.
• The MarkLogic Content Pump (mlcp), as described in Using MarkLogic Content Pump (MLCP) to Load Temporal Documents.
• XCC ContentCreateOptions class functions: getTemporalCollection and setTemporalCollection to get and set the temporal collection object associated with the inserted documents.
• Hadoop Connector mapreduce.marklogic.output.temporalcollection property to specify the temporal collection that is used to ingest the documents.

Though MarkLogic manages temporal documents in the same manner regardless of the tool you use, this section describes the use of the XQuery functions, temporal:document-insert and temporal:document-load, to insert and update temporal documents into MarkLogic Server.

Calling either temporal:document-insert or temporal:document-load on an existing URI updates the existing temporal document. An update on a temporal document results in a new document, rather than an overwrite of the original document.

You can use xdmp:document-set-properties to set properties on temporal documents, but you cannot use the xdmp:document-set-quality, xdmp:document-set-permissions, or xdmp:document-set-collections functions to set their respective attributes on temporal documents.

As described in Bi-temporal Collections. the bi-temporal document being inserted or updated must specify valid start and end times either in the document or in metadata. When the system start and end times are stored in the document, they also must be included. These times must be dateTime values identified by elements that map to the range indexes that represent the valid start and end time period. On insert, MarkLogic sets the system start time to the current time system time and the system end time to the farthest possible time (infinity).

If you have enabled LSQT, as described in Last Stable Query Time (LSQT) and Application-controlled System Time, you can alternatively have your application call the temporal:statement-set-system-time function to specify a system start time along with your insert. The dateTime given must be later than the LSQT returned by the temporal:get-lsqt function.

JavaScript Example:

1. Set LSQT for the temporal collection.

   declareUpdate();
   const temporal = require("/MarkLogic/temporal.xqy");
   temporal.setUseLsqt("kool", true);

2. Get the LSQT.

   const temporal = require("/MarkLogic/temporal.xqy");
   temporal.getLsqt("kool");

3. On document insert, set the system start time to some time after the LSQT.

   declareUpdate();
   const temporal = require("/MarkLogic/temporal.xqy");
   const root =
       {"tempdoc": 
          {"content": "content here"}
       };
   const options =
       {metadata:
          {validStart: "2016-06-01T13:59:00Z",
           validEnd: "9999-12-31T11:59:59Z"}
       };
   temporal.documentInsert("kool", "doc.json", root, options);
   temporal.statementSetSystemTime(xs.dateTime("2016-06-01T14:00:00Z"));

   XQuery Example:

4. Set LSQT for the temporal collection.

   xquery version "1.0-ml";
   import module namespace temporal = "http://marklogic.com/xdmp/temporal" 
       at "/MarkLogic/temporal.xqy";   
   temporal:set-use-lsqt("kool", fn:true())

5. Get the LSQT.

   xquery version "1.0-ml";
   import module namespace temporal = "http://marklogic.com/xdmp/temporal" 
       at "/MarkLogic/temporal.xqy";  
   temporal:get-lsqt("kool")

6. On document insert, set the system start time to some time after the LSQT.

   xquery version "1.0-ml";
   import module namespace temporal = "http://marklogic.com/xdmp/temporal" 
       at "/MarkLogic/temporal.xqy";   
   let $root :=   
   <tempdoc>
       <content>v1-content here</content>
   </tempdoc>
   let $options :=  
   <options xmlns="xdmp:document-insert">
       <metadata>
          <map:map xmlns:map="http://marklogic.com/xdmp/map">
            <map:entry key="validStart">
              <map:value>2016-06-01T13:59:00Z</map:value>
            </map:entry>
            <map:entry key="validEnd">
              <map:value>9999-12-31T11:59:59Z</map:value>
            </map:entry> 
          </map:map>
       </metadata> 
   </options> 
   return 
   temporal:document-insert("kool", "koolorder.xml", $root, $options),
   temporal:statement-set-system-time(xs:dateTime("2016-06-01T14:00:00Z"))

   Document properties are not updated on temporal documents, so do not use Content Processing Framework (CPF) or Library Services (DLS) on temporal data.

Last Stable Query Time (LSQT) and Application-controlled System Time

You can manually set the system start time when inserting or updating a document in a collection. This feature is useful when you need to maintain a master system time across multiple clients that are concurrently inserting and updating bi-temporal documents, without the need for the clients to communicate with one another in order to coordinate their system times.

The system start times for document versions with the same URI must progress along the system time axis, so that an update to a document cannot have a system start time that is earlier than that of the document that chronicles its last update. However, when managing documents with different URIs in a temporal collection, it is necessary to ensure that the system time progresses at the same rate for every document insert and update.

A special timestamp, called the LSQT (Last Stable Query Time), can be enabled on a temporal collection to manage system start times across documents with different URIs. A temporal document with a system start time before the LSQT can only be queried and a document with a system start time after the LSQT can be updated / ingested, but not queried. You can advance the LSQT, either manually or automatically, to manage which documents are available to be queried and which documents can be updated.

You can use the temporal:set-use-lsqt function to enable or disable LSQT on a temporal collection. When LSQT is enabled, the LSQT is stored in a document in the database, with a name of the form collection-name.lsqt. You can call the temporal:advance-lsqt function to manually advance the LSQT or use the temporal:set-lsqt-automation function to direct MarkLogic to automatically advance the LSQT at set periods.

When LSQT is enabled on a temporal collection, the LSQT value starts at 0 (lowest timestamp) When advanced, document reads and writes are quiesced until the LSQT is reset to the maximum system start time in the database. You must have LSQT enabled in order to use the temporal:statement-set-system-time function to set the system start time.

You can only call the temporal:statement-set-system-time function once per statement

For example, the following query first checks to make sure the application time (simulated by the current time) is greater than the LSQT:

JavaScript Example:

declareUpdate();
const temporal = require("/MarkLogic/temporal.xqy");

const curtime = fn.currentDateTime();
const lsqt = temporal.getLsqt("kool");
const root =
    {"tempdoc": 
       {"content": "12"}
    };
const options =
    {metadata:
       {validStart: "2016-06-20T14:06:13Z",
        validEnd: "9999-12-31T11:59:59Z"}
    };

temporal.documentInsert("kool", "test.json", root, options);
if(curtime > lsqt) {
      temporal.statementSetSystemTime(xs.dateTime(curtime));}
else {fn.concat("Can't update document with specified system ",
                "time because it is earlier than the set LSQT")};

XQuery Example:

xquery version "1.0-ml";
import module namespace temporal = "http://marklogic.com/xdmp/temporal" 
    at "/MarkLogic/temporal.xqy";   
let $curtime := fn:current-dateTime()
let $LSQT := temporal:get-lsqt("kool")
let $root := 
<tempdoc>
    <content>v1-content here</content>  
</tempdoc>
let $options :=  
<options xmlns="xdmp:document-insert">
    <metadata>
      <map:map xmlns:map="http://marklogic.com/xdmp/map">
         <map:entry key="validStart">
           <map:value>2016-06-20T14:06:13Z</map:value>
         </map:entry>
         <map:entry key="validEnd">
           <map:value>9999-12-31T11:59:59Z</map:value>
         </map:entry> 
      </map:map>
    </metadata> 
</options> 
let $systemTime := 
      if ($curtime > $LSQT)
      then (temporal:statement-set-system-time(xs:dateTime($curtime)))
      else (fn:concat("Can't update document with specified system",
                      "time because it is earlier than the set LSQT")) 
return (
   temporal:document-insert( 
     "kool", "doc.xml", $root, $options), 
     $systemTime )

Using MarkLogic Content Pump (MLCP) to Load Temporal Documents

You can use the MarkLogic Content Pump (MLCP) to import documents into MarkLogic as part of a temporal collection, copy non-temporal documents in one database into a temporal collection in another database, or copy temporal documents from one database to another. See the following topics for details:

• Importing Documents Into a Temporal Collection
• Copying Non-Temporal Documents Into a Temporal Collection
• Copying Temporal Documents Between Databases

mlcp.sh import -temporal_collection kool -input_file_path /etc/orders \
-host desthost -port 8006 -username user1

mlcp.sh copy -mode local -input_host srchost -input_port 8006 \
-input_username user1 -input_password password1 \
-output_host desthost -output_port 8010 -temporal_collection kool \
-output_username user2 -output_password password2

mlcp.sh copy -mode local -input_host srchost -input_port 8006 \
-input_username inuser -input_password inpassword \
-output_host desthost -output_port 8010 -copy_collections true \
-output_username outuser -output_password outpassword

mlcp.sh copy -mode local -input_host srchost -input_port 8006 \
  -input_username inuser -input_password inpassword \
  -output_host desthost -output_port 8010 -copy_collections true \
  -output_username my_override_user -output_password outpassword \
  -copy_collections true

(: XQuery :)
xquery version "1.0-ml"; 
import module namespace temporal = "http://marklogic.com/xdmp/temporal"
    at "/MarkLogic/temporal.xqy";
temporal:collection-create("temporalCollection", "system", "valid"
                           ("allow-nonempty"))

// Server-Side JavaScript
var temporal = require('/MarkLogic/temporal.xqy');
temporal.collectionCreate('kool', 'system', 'valid',
                          ['allow-nonempty']);

Setting the URI for a New Version of a Temporal Document

By default, MarkLogic creates a new URI of the form filename[number].extension for each new version of a temporal document. You can optionally set the URI when updating a temporal document by calling the temporal:statement-set-document-version-uri or temporal.statementSetDocumentVersionUri function.

For example, the initial version (the URI provided during the initial insert) of a temporal document has the following URI:

important.json

And you want the new version of the temporal document to have the following URI:

/version2/important.json

You could call the temporal.statementSetDocumentVersionUri function as follows:

declareUpdate();
const temporal = require("/MarkLogic/temporal.xqy");

const root =
    {"tempdoc":
       {"content": "content here"}
    };

const options =
    {metadata:
       {validStart: "1601-01-01T13:59:00Z",
        validEnd: "9999-12-31T11:59:59Z"}
    };

temporal.statementSetDocumentVersionUri(
     "important.json", 
     "/version2/important.json")

temporal.documentInsert("kool", "important.json", root, options);

Protecting and Archiving Temporal Documents

Temporal documents can be protected from certain temporal operations, such as update, delete or wipe for a specified period of time. You can use the temporal:document-protect function to set the duration of protection, when that protection will expire (overrides duration, if different), and where to archive the temporal document.

You should not use document-protect to protect a document with an expectation that you will remove the protection in the future. Protections are meant to be firm, and based on absolute dates and times.

There are three levels of protection, listed below in the descending order of restrictive level:

• no-update -- do not allow any modification of the archived document. This includes no-delete and no-wipe.
• no-delete -- do not allow a temporal delete document operation to delete any version of the document. This includes no-wipe. This is the default, if you do not specify a protection type.
• no-wipe -- do not allow a wipe operation to delete all versions of the document.

When setting protection on a temporal document, you can specify a path to an archive, such as a WORM device. As described in WORM (Write Once Read Many) Support, once written to a WORM device, the temporal document can be read, but cannot be modified or deleted until its expiration date.

The archive settings are set by an XQuery, Javascript, or REST temporal document protect operation. For example, to set the protection level to no-update with an expire time to 2:00pm on 7/20/2016 on the doc.xml document in the temporal collection, kool, do the following:

declareUpdate();
const temporal = require("/MarkLogic/temporal.xqy");

temporal.documentProtect ("kool", "doc.xml", 
    {level: "noUpdate", expireTime: "2016-07-20T14:00:00Z"})

Deleting and Wiping Temporal Documents

You can use the temporal:document-delete function to delete temporal documents. Deleting a temporal document maintains the document and all of its versions in the URI collection and updates the deleted document and all of its versions that have a system end time of infinity to the time of the delete.

Deleting a temporal document removes the document from the latest collection. So the latest collection is the source of all of the documents that are currrently valid and the URI collections are the source of the history of each document.

Should you insert a document using the same URI as a deleted document, the deleted document, and all of its previous versions remain in the same URI collection as the newly inserted document. The newly inserted document is then added to the latest collection.

You can also use the temporal:document-wipe function to remove all versions of a temporal document that has expired. The expiration date for a document is set by the temporal:document-protect function, as described in Protecting and Archiving Temporal Documents.

Before all of the versions of a temporal document can be wiped, the current version of the temporal document must first have an expire time set by the temporal.documentProtect function.

For example, to wipe the doc.xml document when its protection time has expired, do the following:

declareUpdate();
const temporal = require("/MarkLogic/temporal.xqy");

const curtime = fn.currentDateTime()
const exptime = xs.dateTime(
        xdmp.documentGetMetadataValue("doc.xml","temporalProtectExTime")) 
if (exptime.lt(curtime)) {temporal.documentWipe("kool", "doc.xml")}
else {"This document is protected or does not exist"}

Example: The Lifecycle of a Temporal Document

The example in this section builds on the example described in Quick Start. The purpose of this example is to show how new versions of a temporal document are generated and updated from a series of changes to a stock purchase order.

1. The stock of KoolCo is trading around $12.65. John places a limit order to buy 100 shares for $12 at 11:00:00 on 3-Apr-2014 (this is the valid start time). A temporal document for the order is recorded in the broker's database at 11:00:01 on 3-Apr-2014 (this is the system start time).
2. At 11:30:00, John changes his order to buy the stock at $13. The change is recorded as another version in the broker's database at 11:30:01.
3. At 12:10:00, John changes his mind again and decides to change his order to a limit order to buy at $12.50. This transaction is recorded as another version with a valid time of 12:10:00, but due to heavy trading, the change is not recorded in the broker's database until 12:10:12.
4. At 13:00:00, the purchase order has not been filled and John decides he no longer wants to buy the stock, so he cancels his order. This cancellation is recorded as another version with a valid time of 13:00:00 and recorded in the broker's database at 13:00:02.
5. However, at 13:00:01, the stock hits $12.50 and John's order is filled.
6. The broker's policy is to honor the valid times for all orders. At 13:00:03, the order fulfillment application reviews the valid and system times recorded at the time of the cancellation, determines that John in fact cancelled his order before it was filled, and does not debit his account for the stock purchase.

The valid and system times are each dateTime ranges that define a start and end time. The start time represents the time at which the information is known (as both valid and system times) and the end time represents the time at which the information is no longer true.

The above stock purchase example was kept simple for clarity. The following shows the insert query and the resulting documents with the actual valid and system times for the example, along with their respective start and end times. The graphic at the end displays the relationships between the documents in terms of valid and system times.

11:00:00 -- Initial order:

Insert Query (JavaScript):

declareUpdate();
const temporal = require("/MarkLogic/temporal.xqy");
const root =
    {"tempdoc": 
       {"order 1": "12"}
    };
const options =
    {metadata:
       {validStart: "2014-04-03T11:00:00",
        validEnd: "9999-12-31T11:59:59Z"}
    };
temporal.documentInsert("kool", "koolorder.json", root, options);

Insert Query (XQuery):

xquery version "1.0-ml";
import module namespace temporal = "http://marklogic.com/xdmp/temporal" 
    at "/MarkLogic/temporal.xqy";   
let $root :=   
<tempdoc>
    <order1>12</order1>
</tempdoc>
let $options :=  
<options xmlns="xdmp:document-insert">
    <metadata>
       <map:map xmlns:map="http://marklogic.com/xdmp/map">
         <map:entry key="validStart">
           <map:value>2014-04-03T11:00:00</map:value>
         </map:entry>
         <map:entry key="validEnd">
           <map:value>9999-12-31T11:59:59Z</map:value>
         </map:entry> 
       </map:map>
    </metadata> 
</options> 
return 
temporal:document-insert("kool", "koolorder.xml", $root, $options)

Results:

Original Document:

Order Price: $12
System Start: 2014-04-03T11:00:01
System End: 9999-12-31T11:59:59Z <-- infinity
Valid Start: 2014-04-03T11:00:00
Valid End: 9999-12-31T11:59:59Z <-- infinity

The date-time 9999-12-31T11:59:59Z represents infinity and is used when there is no valid date-time yet for that start or end time.

11:30:00 -- Changed order from $12 to $13.

Update Query (JavaScript):

declareUpdate();
const temporal = require("/MarkLogic/temporal.xqy");
const root =
    {"tempdoc": 
       {"order 2": "13"}
    };
const options =
    {metadata:
       {validStart: "2014-04-03T11:30:00",
        validEnd: "9999-12-31T11:59:59Z"}
    };
temporal.documentInsert("kool", "koolorder.json", root, options);

Update Query (XQuery):

xquery version "1.0-ml";
import module namespace temporal = "http://marklogic.com/xdmp/temporal" 
    at "/MarkLogic/temporal.xqy";   
let $root := 
<tempdoc>
    <order2>13</order2>
</tempdoc>
let $options := 
<options xmlns="xdmp:document-insert">
    <metadata>
       <map:map xmlns:map="http://marklogic.com/xdmp/map">
         <map:entry key="validStart">
           <map:value>2014-04-03T11:30:00</map:value>
         </map:entry>
         <map:entry key="validEnd">
           <map:value>9999-12-31T11:59:59Z</map:value>
         </map:entry> 
       </map:map>
    </metadata>
</options>
return
temporal:document-insert("kool", "koolorder.xml", $root, $options)

Results:

Original Document (updated):

Order Price: $12
System Start: 2014-04-03T11:00:01
System End: 2014-04-03T11:30:01 <-- changed
Valid Start: 2014-04-03T11:00:00
Valid End: 9999-12-31T11:59:59Z <-- infinity

Split 1 from Original Document:

Order Price: $12
System Start: 2014-04-03T11:30:01
System End: 9999-12-31T11:59:59Z <-- infinity
Valid Start: 2014-04-03T11:00:00
Valid End: 2014-04-03T11:30:00

Version 2:

Order Price: $13
System Start: 2014-04-03T11:30:01
System End: 9999-12-31T11:59:59Z <-- infinity
Valid Start: 2014-04-03T11:30:00
Valid End: 9999-12-31T11:59:59Z <-- infinity

12:10:00 -- Changed order from $13 to $12:50:

Update Query (JavaScript):

declareUpdate();
const temporal = require("/MarkLogic/temporal.xqy");
const root =
    {"tempdoc": 
       {"order 3": "12.50"}
    };
const options =
    {metadata:
       {validStart: "2014-04-03T12:10:00",
        validEnd: "9999-12-31T11:59:59Z"}
    };
temporal.documentInsert("kool", "koolorder.json", root, options);

Update Query (XQuery):

xquery version "1.0-ml";
import module namespace temporal = "http://marklogic.com/xdmp/temporal" 
    at "/MarkLogic/temporal.xqy";   
let $root :=   
<tempdoc>
    <order3>12.50</order3>
</tempdoc>
let $options :=  
<options xmlns="xdmp:document-insert">
    <metadata>
       <map:map xmlns:map="http://marklogic.com/xdmp/map">
         <map:entry key="validStart">
           <map:value>2014-04-03T12:10:00</map:value>
         </map:entry>
         <map:entry key="validEnd">
           <map:value>9999-12-31T11:59:59Z</map:value>
         </map:entry> 
       </map:map>
    </metadata> 
</options> 
return 
temporal:document-insert("kool", "koolorder.xml", $root, $options)

Results:

Original Document (no change)

Split 1 (no change)

Version 2 (update):

Order Price: Closing price
System Start: 2014-04-03T11:30:01
System End: 2014-04-03T12:10:12 <-- changed
Valid Start: 2014-04-03T11:30:00
Valid End: 9999-12-31T11:59:59Z <-- infinity

Split 2 (new)

Order Price: $12
System Start: 2014-04-03T12:10:12
System End: 9999-12-31T11:59:59Z <-- infinity
Valid Start: 2014-04-03T11:00:00
Valid End: 2014-04-03T12:10:00 

Version 3 (new):

Order Price: $12.5
System Start: 2014-04-03T12:10:12
System End: 9999-12-31T11:59:59Z <-- infinity
Valid Start: 2014-04-03T12:10:00
Valid End: 9999-12-31T11:59:59Z <-- infinity

13:00:00 -- Cancelled Order:

Update Query (JavaScript):

declareUpdate();
const temporal = require("/MarkLogic/temporal.xqy");
const root =
    {"tempdoc": 
       {"cancel": "0"}
    };
const options =
    {metadata:
       {validStart: "2014-04-03T13:00:00",
        validEnd: "9999-12-31T11:59:59Z"}
    };
temporal.documentInsert("kool", "koolorder.json", root, options);

Update Query (XQuery):

xquery version "1.0-ml";
import module namespace temporal = "http://marklogic.com/xdmp/temporal" 
    at "/MarkLogic/temporal.xqy";   
let $root :=   
<tempdoc>
    <cancel>0</cancel>
</tempdoc>
let $options :=  
<options xmlns="xdmp:document-insert">
    <metadata>
       <map:map xmlns:map="http://marklogic.com/xdmp/map">
         <map:entry key="validStart">
           <map:value>2014-04-03T13:00:00</map:value>
         </map:entry>
         <map:entry key="validEnd">
           <map:value>9999-12-31T11:59:59Z</map:value>
         </map:entry> 
       </map:map>
    </metadata> 
</options> 
return 
temporal:document-insert("kool", "koolorder.xml", $root, $options)

Results:

Original Document (no change)

Split 1 (no change)

Version 2 (no change)

Split 2 (no change)

Version 3 (updated)

Order Price: $12.5
System Start: 2014-04-03T12:10:12
System End: 2014-04-03T13:00:02 <-- changed
Valid Start: 2014-04-03T12:10:00
Valid End: 9999-12-31T11:59:59Z <-- infinity

Split 3 (new)

Order Price: $12.5
System Start: 2014-04-03T13:00:02
System End: 9999-12-31T11:59:59Z <-- infinity
Valid Start: 2014-04-03T11:00:00
Valid End: 2014-04-03T13:00:00

Version 4 (new):

Order Price: $0
System Start: 2014-04-03T13:00:02
System End: 9999-12-31T11:59:59Z <-- infinity
Valid Start: 2014-04-03T13:00:00
Valid End: 9999-12-31T11:59:59Z <-- infinity