Loading TOC...
Temporal Developer's Guide (PDF)

Temporal Developer's Guide — Chapter 3

Managing Temporal Documents

This chapter describes how to insert and update temporal documents, and includes the following sections:

Inserting and Loading Temporal Documents

There are a number of ways to insert and update temporal documents in MarkLogic Server. These include:

Though MarkLogic manages temporal documents in the same manner regardless of the tool you use, this section describes the use of the XQuery functions, temporal:document-insert and temporal:document-load, to insert and update temporal documents into MarkLogic Server.

Calling either temporal:document-insert or temporal:document-load on an existing URI 'updates' the existing temporal document. An update on a temporal document results in a new document, rather than an overwrite of the original document.

You can use xdmp:document-set-properties to set properties on temporal documents, but you cannot use the xdmp:document-set-quality, xdmp:document-set-permissions, or xdmp:document-set-collections functions to set their respective attributes on temporal documents.

The document being inserted or updated must include a valid start and end times. These times must be dateTime values identified by elements that map to the range indexes that represent the valid start and end time period. On insert, MarkLogic sets the system start time to the current time system time and the end time to the farthest possible time (infinity).

JavaScript Example:

var temporal = require("/MarkLogic/temporal.xqy");
var root =
    { "tempdoc": {
          "systemStart": null,
          "systemEnd": null,
          "validStart": "2014-06-03T14:13:05",
          "validEnd": "9999-12-31T11:59:59Z",
          "content": "v1-content here"
          }
    };
declareUpdate();
temporal.documentInsert("temporalCollection", "doc.json", root);

XQuery Example:

xquery version "1.0-ml";
import module namespace temporal = "http://marklogic.com/xdmp/temporal" 
    at "/MarkLogic/temporal.xqy";   
let $root := 
<tempdoc>
    <systemStart/>
    <systemEnd/>
    <validStart>2014-06-03T14:13:05</validStart> 
    <validEnd>9999-12-31T11:59:59Z</validEnd> 
    <content>v1-content here</content>  
</tempdoc>
return 
    temporal:document-insert("temporalCollection", "doc.xml", $root)

If you don't specify a system time, MarkLogic Server will fill in the system start and end times. If you have enabled LSQT, as described in Last Stable Query Time (LSQT) and Application-controlled System Time, you can alternately have your application call the temporal:statement-set-system-time function to specify a system time along with your insert. The dateTime given must be later than the LSQT (Last Stable Query Time) returned by the temporal:get-lsqt function.

JavaScript Example:

var temporal = require("/MarkLogic/temporal.xqy");
var root =
    { "tempdoc": {
          "systemStart": null,
          "systemEnd": null,
          "validStart": "2014-06-03T14:13:05",
          "validEnd": "9999-12-31T11:59:59Z",
          "content": "v1-content here"
          }
    };
declareUpdate();
temporal.documentInsert("temporalCollection", "doc.json", root), 
temporal.statementSetSystemTime(xs.dateTime("2014-06-03T14:15:00"));

XQuery Example:

xquery version "1.0-ml";
import module namespace temporal = "http://marklogic.com/xdmp/temporal" 
    at "/MarkLogic/temporal.xqy";
let $root := 
<tempdoc>
    <systemStart/>
    <systemEnd/>
    <validStart>2014-06-03T14:13:05</validStart> 
    <validEnd>9999-12-31T11:59:59Z</validEnd> 
    <content>v1-content here</content>  
</tempdoc>
return 
    temporal:document-insert("temporalCollection","doc.xml",$root),
    temporal:statement-set-system-time(
        xs:dateTime("2014-06-03T14:15:00"))

Document properties are not updated on temporal documents, so do not use Content Processing Framework (CPF) or Library Services (DLS) on temporal data.

Last Stable Query Time (LSQT) and Application-controlled System Time

You can manually set the system start time when inserting or updating a document in a collection. This feature is useful when you need to maintain a 'master' system time across multiple clients that are concurrently inserting and updating bitemporal documents, without the need for the clients to communicate with one another in order to coordinate their system times.

The system start times for document versions with the same URI must progress along the system time axis, so that an update to a document cannot have a system start time that is earlier than that of the document that chronicles its last update. However, when managing documents with different URIs in a temporal collection, it is necessary to ensure that the system time progresses at the same rate for every document insert and update.

A special timestamp, called the LSQT (Last Stable Query Time), can be enabled on a temporal collection to manage system start times across documents with different URIs. A temporal document with a system start time before the LSQT can only be queried and a document with a system start time after the LSQT can be updated / ingested, but not queried. You can advance the LSQT, either manually or automatically, to manage which documents are available to be queried and which documents can be updated.

You can use the temporal:set-use-lsqt function to enable or disable LSQT on a temporal collection. When LSQT is enabled, the LSQT is stored in a document in the database, with a name of the form collection-name.lsqt. You can call the temporal:advance-lsqt function to manually advance the LSQT or use the temporal:set-lsqt-automation function to direct MarkLogic to automatically advance the LSQT at set periods.

When LSQT is enabled on a temporal collection, the LSQT value starts at 0 (lowest timestamp) When advanced, document reads and writes are quiesced until the LSQT is reset to the maximum system start time in the database. You must have LSQT enabled in order to use the temporal:statement-set-system-time function to set the system start time.

You can only call the temporal:statement-set-system-time function once per statement

For example, the following query first checks to make sure the application time (simulated by the current time) is greater than the LSQT:

xquery version "1.0-ml";
import module namespace temporal = "http://marklogic.com/xdmp/temporal" 
    at "/MarkLogic/temporal.xqy";   
let $appTime := fn:current-dateTime()
let $LSQT := temporal:get-lsqt("temporalcollection")
let $root := 
<tempdoc>
    <systemStart/> 
    <systemEnd/> 
    <validStart>2014-06-03T14:13:05</validStart> 
    <validEnd>9999-12-31T11:59:59Z</validEnd> 
    <content>v1-content here</content>  
</tempdoc>
let $systemTime := 
      if ($appTime > $LSQT)
      then (temporal:statement-set-system-time(xs:dateTime($appTime)))
      else ()  
return (
   temporal:document-insert( 
     "temporalcollection",
     "doc.xml",
     $root), 
     $systemTime )

Using MarkLogic Content Pump (MLCP) to Load Temporal Documents

You can use the MarkLogic Content Pump (MLCP) with the -temporal_collection option to load temporal documents into a specific temporal collection in the MarkLogic Server database. The two MLCP commands that are supported for loading temporal documents are import and copy.

For example, to import the temporal documents in the /etc/orders directory on the filesystem into the temporal collection, named 'kool,' into the temporal database on the host, desthost, you would use the following MLCP command:

mlcp.sh import -temporal_collection kool -input_file_path /etc/orders \
-host desthost -port 8006 -username user1

If you omit -port, MLCP will use port 8000.

You can use the copy command to copy non-temporal documents from a database and ingest them as temporal documents into the temporal collection in another database. For example to migrate the temporal documents from the database used by the host, srchost, to the temporal collection named 'kool' in the database used by the host, desthost, you would use a command like the following:

mlcp.sh copy -mode local -input_host srchost -input_port 8006 \
-input_username user1 -input_password password1 \
-output_host desthost -output_port 8010 -temporal_collection kool \
-output_username user2 -output_password password2

You cannot import or copy binary files as temporal documents or specify an -input_file_type of rdf or forest for temporal documents.

For details on using MLCP to load documents, see Loading Content Using MarkLogic Content Pump in the Loading Content Into MarkLogic Server Guide.

Deleting Temporal Documents

You can use the temporal:document-delete function to delete temporal documents. Deleting a temporal document maintains the document and all of its versions in the URI collection and updates the deleted document and all of its versions that have a system end time of infinity to the time of the delete.

Deleting a temporal document removes the document from the latest collection. So the latest collection is the source of all of the documents that are currrently valid and the URI collections are the source of the history of each document.

Should you insert a document using the same URI as a deleted document, the deleted document, and all of its previous versions remain in the same URI collection as the 'newly' inserted document. The newly inserted document is then added to the latest collection.

Example: The Lifecycle of a Temporal Document

The example in this section builds on the example described in Quick Start. The purpose of this example is to show how temporal documents are generated and updated from a series of changes to a transaction.

  1. The stock of KoolCo is trading around $12.65. John places a limit order to buy 100 shares for $12 at 11:00:00 on 3-Apr-2014 (this is the valid start time). The document for the transaction is recorded in the broker's database at 11:00:01 on 3-Apr-2014 (this is the system start time).
  2. John looks at the trading pattern of the stock over the last week and notices that it always dips duing the last second of the trading day. At 11:30:00, John changes his order to buy the stock at the closing price (15:59:59). The change is recorded as another document in the broker's database at 11:30:01.
  3. At 12:10:00, John changes his mind again and decides to change his order to a limit order to buy at $12.50. This transaction is recorded as another document with a valid time of 12:10:00, but due to heavy trading, the change is not recorded in the broker's database until 12:10:12.
  4. At 13:00:00, the purchase order has not been filled and John decides he no longer wants to buy the stock, so he cancels his order. This cancellation is recorded as another document with a valid time of 13:00:00 and recorded in the broker's database at 13:00:02.
  5. However, at 13:00:01, the stock hits $12.50 and John's order is filled.
  6. The broker's policy is to honor the valid times for all orders. At 13:00:03, the order fulfillment application reviews the valid and system times recorded in the cancellation document, determines that John in fact cancelled his order before it was filled, and does not debit his account for the stock purchase.
  7. At 16:00:00, the broker deletes the order.

The valid and system times are each dateTime ranges that define a start and end time. The start time represents the time at which the information is known (as both valid and system times) and the end time represents the time at which the information is no longer true.

The above stock purchase example was kept simple for clarity. The following shows the insert query and the resulting documents with the actual valid and system times for the example, along with their respective start and end times. The graphic at the end displays the relationships between the documents in terms of valid and system times.

11:00:01 -- Initial order. 1 Document:

Insert Query (JavaScript):

var temporal = require("/MarkLogic/temporal.xqy");
var root =
    { "tempdoc": {
          "systemStart": null,
          "systemEnd": null,
          "validStart": "2014-04-03T11:00:00",
          "validEnd": "2014-04-03T16:00:00",
          "content": "12"
          }
    };
declareUpdate();
temporal.documentInsert("kool", "koolorder.json", root);

Insert Query (XQuery):

xquery version "1.0-ml";
import module namespace temporal = "http://marklogic.com/xdmp/temporal" 
      at "/MarkLogic/temporal.xqy";
let $root :=   
<tempdoc>
    <systemStart/> 
    <systemEnd/> 
    <validStart>2014-04-03T11:00:00</validStart> 
    <validEnd>2014-04-03T16:00:00</validEnd> 
    <content>12</content>  
</tempdoc>
return temporal:document-insert("kool", "koolorder.xml", root)

Results:

Document 1:

Order Price: $12
System Start: 2014-04-03T11:00:01
System End: 9999-12-31T11:59:59Z <-- infinity
Valid Start: 2014-04-03T11:00:00
Valid End: 2014-04-03T16:00:00

The date-time 9999-12-31T11:59:59Z represents infinity and is used when there is no valid date-time yet for that start or end time.

11:30:00 -- Changed order from $12 to buy at end of the day (16:00:00).

Update Query (JavaScript):

var temporal = require("/MarkLogic/temporal.xqy");
var root =
    { "tempdoc": {
          "systemStart": null,
          "systemEnd": null,
          "validStart": "2014-04-03T15:59:59",
          "validEnd": "2014-04-03T16:00:00",
          "content": "Closing Price"
          }
    };
declareUpdate();
temporal.documentInsert("kool", "koolorder.json", root);

Update Query (XQuery):

xquery version "1.0-ml";
import module namespace temporal = "http://marklogic.com/xdmp/temporal" 
      at "/MarkLogic/temporal.xqy";   
let $root :=   
<tempdoc>
    <systemStart/> 
    <systemEnd/> 
    <validStart>2014-04-03T15:59:59</validStart>
    <validEnd>2014-04-03T16:00:00</validEnd> 
    <content>Closing Price</content>  
</tempdoc>
return temporal:document-insert("kool", "koolorder.xml", $root)

Results:

Document 1 (updated):

Order Price: $12
System Start: 2014-04-03T11:00:01
System End: 2014-04-03T11:30:01 <-- changed
Valid Start: 2014-04-03T11:00:00
Valid End: 2014-04-03T16:00:00

Document 2 (new -- split from Document 1):

Order Price: $12
System Start: 2014-04-03T11:30:01
System End: 9999-12-31T11:59:59Z <-- infinity
Valid Start: 2014-04-03T11:00:00
Valid End: 2014-04-03T15:59:59

Document 3 (new):

Order Price: Closing price
System Start: 2014-04-03T11:30:01
System End: 9999-12-31T11:59:59Z <-- infinity
Valid Start: 2014-04-03T15:59:59
Valid End: 2014-04-03T16:00:00

12:10:00 -- Changed order from closing price to $12:50. 5 Documents:

Update Query (JavaScript):

var temporal = require("/MarkLogic/temporal.xqy");
var root =
    { "tempdoc": {
          "systemStart": null,
          "systemEnd": null,
          "validStart": "2014-04-03T12:10:00",
          "validEnd": "2014-04-03T16:00:00",
          "content": "12.50"
          }
    };
declareUpdate();
temporal.documentInsert("kool", "koolorder.json", root);

Update Query (XQuery):

xquery version "1.0-ml";
import module namespace temporal = "http://marklogic.com/xdmp/temporal" 
      at "/MarkLogic/temporal.xqy";   
let $root :=   
<tempdoc>
   <systemStart/> 
   <systemEnd/> 
   <validStart>2014-04-03T12:10:00</validStart> 
   <validEnd>2014-04-03T16:00:00</validEnd> 
   <content>12.50</content> 
</tempdoc>
return temporal:document-insert("kool", "koolorder.xml", $root)

Results:

Document 1 (no change)

Document 2 (update)

Order Price: $12
System Start: 2014-04-03T11:30:01
System End: 2014-04-03T12:10:12 <-- changed
Valid Start: 2014-04-03T11:00:00
Valid End: 2014-04-03T15:59:59

Document 3 (update):

Order Price: Closing price
System Start: 2014-04-03T11:30:01
System End: 2014-04-03T12:10:12 <-- changed
Valid Start: 2014-04-03T15:59:59
Valid End: 2014-04-03T16:00:00

Document 4 (new -- split from Document 2)

Order Price: $12
System Start: 2014-04-03T12:10:12
System End: 9999-12-31T11:59:59Z <-- infinity
Valid Start: 2014-04-03T11:00:00
Valid End: 2014-04-03T12:10:00 

Document 5 (new):

Order Price: $12.5
System Start: 2014-04-03T12:10:12
System End: 9999-12-31T11:59:59Z <-- infinity
Valid Start: 2014-04-03T12:10:00
Valid End: 2014-04-03T16:00:00

Doc 3 was not split because the new Doc 5 contains the same period as Doc 3.

13:00:00 -- Cancelled Order. 6 Documents:

Update Query (JavaScript):

var temporal = require("/MarkLogic/temporal.xqy");
var root =
    { "tempdoc": {
          "systemStart": null,
          "systemEnd": null,
          "validStart": "2014-04-03T13:00:00",
          "validEnd": "2014-04-03T16:00:00",
          "content": "0"
          }
    };
declareUpdate();
temporal.documentInsert("kool", "koolorder.json", root);

Update Query (XQuery):

xquery version "1.0-ml";
import module namespace temporal = "http://marklogic.com/xdmp/temporal" 
      at "/MarkLogic/temporal.xqy";   
let $root :=   
<tempdoc>
   <systemStart/> 
   <systemEnd/> 
   <validStart>2014-04-03T13:00:00</validStart> 
   <validEnd>2014-04-03T16:00:00</validEnd> 
   <content>0</content> 
</tempdoc>
return temporal:document-insert("kool", "koolorder.xml", $root)

Results:

Document 1 (no change)

Document 2 (no change)

Document 3 (no change)

Document 4 (no change)

Document 5 (updated):

Order Price: $12.5
System Start: 2014-04-03T13:00:02 <-- changed
System End: 9999-12-31T11:59:59Z <-- infinity
Valid Start: 2014-04-03T12:10:00
Valid End: 2014-04-03T13:00:00 <-- changed

Document 6 (new -- split from Document 5):

Order Price: $12.5
System Start: 2014-04-03T12:10:12
System End: 2014-04-03T13:00:02
Valid Start: 2014-04-03T12:10:00
Valid End: 2014-04-03T16:00:00

Document 7 (new):

Order Price: $0
System Start: 2014-04-03T13:00:02
System End: 9999-12-31T11:59:59Z <-- infinity
Valid Start: 2014-04-03T13:00:00
Valid End: 2014-04-03T16:00:00

16:00:00 -- Order Deleted. 5 Documents:

Delete Query (JavaScript):

declareUpdate();
var temporal = require("/MarkLogic/temporal.xqy");
temporal.documentDelete("kool", "koolorder.json")

Delete Query (XQuery):

xquery version "1.0-ml";
import module namespace temporal = "http://marklogic.com/xdmp/temporal" 
      at "/MarkLogic/temporal.xqy";   
temporal:document-delete("kool", "koolorder.xml")

Results:

Document 1 (no change)

Document 2 (no change)

Document 3 (no change)

Document 4 (updated)

Order Price: $12
System Start: 2014-04-03T12:10:12
System End: 2014-04-03T16:00:00 <-- changed
Valid Start: 2014-04-03T11:00:00
Valid End: 2014-04-03T12:10:00 

Document 5 (updated):

Order Price: $12.5
System Start: 2014-04-03T13:00:02
System End: 2014-04-03T16:00:00 <-- changed
Valid Start: 2014-04-03T12:10:00
Valid End: 2014-04-03T13:00:00

Document 6 (no change)

Document 7 (updated):

Order Price: $0
System Start: 2014-04-03T13:00:02
System End: 2014-04-03T16:00:00 <-- changed
Valid Start: 2014-04-03T13:00:00
Valid End: 2014-04-03T16:00:00

When a temporal document is deleted, it is removed from the latest collection.

« Previous chapter
Next chapter »
Powered by MarkLogic Server 7.0-4.1 and rundmc | Terms of Use | Privacy Policy