Loading TOC...
Application Developer's Guide (PDF)

Application Developer's Guide — Chapter 3

Understanding Transactions in MarkLogic Server

MarkLogic Server is a transactional system that ensures data integrity. This chapter describes the transaction model of MarkLogic Server, and includes the following sections:

For additional information about using multi-statement and XA/JTA transactions from XCC Java applications, see the XCC Developer's Guide.

Terms and Definitions

Although transactions are a core feature of most database systems, various systems support subtly different transactional semantics. Clearly defined terminology is key to a proper and comprehensive understanding of these semantics. To avoid confusion over the meaning of any of these terms, this section provides definitions for several terms used throughout this chapter and throughout the MarkLogic Server documentation. The definitions of the terms also provide a useful starting point for describing transactions in MarkLogic Server.

Term Definition
statement

An XQuery main module, as defined by the W3C XQuery standard, to be evaluated by MarkLogic Server. A main module consists of an optional prolog and a complete XQuery expression.

A Server-Side JavaScript program (or script) is considered a single statement for transaction purposes.

Statements are either query statements or update statements, determined statically through static analysis prior to beginning the statement evaluation.

query statement

A statement that contains no update calls.

Query statements have a read-consistent view of the database. Since a query statement does not change the state of the database, the server optimizes it to hold no locks or lightweight locks, depending on the type of the containing transaction.

update statement

A statement with the potential to perform updates (that is, it contains one or more update calls).

A statement can be categorized as an update statement whether or not the statement performs an update at runtime. Update statements run with readers/writers locks, obtaining locks as needed for documents accessed in the statement.

transaction

A set of one or more statements which either all fail or all succeed.

A transaction is either an update transaction or a query (read-only) transaction, depending on the transaction type and the kind of statements in the transaction. A transaction is either a single-statement transaction or a multi-statement transaction, depending on the commit mode at the time it is created.

transaction mode Controls the transaction type and the commit semantics of newly created transactions. If you need to control the transaction type and/or commit semantics of a transaction, set them individually, rather than setting transaction mode. For details, see Transaction Mode.
single-statement transaction Any transaction created in auto commit mode. Single-statement transactions always contain only one statement and are automatically committed on successful completion or rolled back on error.
multi-statement transaction A transaction created in explicit commit mode, consisting of one or more statements which commit or rollback together. Changes made by one statement in the transaction are visible to subsequent statements in the same transaction prior to commit. Multi-statement transactions must be explicitly committed by calling xdmp:commit or xdmp.commit.
query transaction

A transaction which cannot perform any updates; a read-only transaction. A transaction consisting of a single query statement in auto update mode, or any transaction created with query transaction type. Attempting to perform an update in a query transaction raises XDMP-UPDATEFUNCTIONFROMQUERY.

Instead of acquiring locks, query transactions run at a particular system timestamp and have a read-consistent view of the database.

update transaction

A transaction that can perform updates (make changes to the database). A transaction consisting of a single update statement in auto commit mode, or any transaction created with update transaction type.

Update transactions run with readers/writers locks, obtaining locks as needed for documents accessed in the transaction.

commit End a transaction and make the changes made by the transaction visible in the database. Single-statement transactions are automatically committed upon successful completion of the statement. Multi-statement transactions are explicitly committed using xdmp:commit, but the commit only occurs if and when the calling statement successfully completes.
rollback Immediately terminate a transaction and discard all updates made by the transaction. All transactions are automatically rolled back on error. Multi-statement transactions can also be explicitly rolled back using xdmp:rollback, or implicitly rolled back due to timeout or reaching the end of the session without calling xdmp:commit.
system timestamp A number maintained by MarkLogic Server that increases every time a change or a set of changes occurs in any of the databases in a system (including configuration changes from any host in a cluster). Each fragment stored in a database has system timestamps associated with it to determine the range of timestamps during which the fragment is valid.
readers/writers locks

A set of read and write locks that lock documents for reading and update at the time the documents are accessed.

MarkLogic Server uses readers/writers locks during update statements. Because update transactions only obtain locks as needed, update statements always see the latest version of a document. The view is still consistent for any given document from the time the document is locked. Once a document is locked, any update statements in other transactions wait for the lock to be released before updating the document. For more details, see Update Transactions: Readers/Writers Locks.

program The expanded version of some XQuery code that is submitted to MarkLogic Server for evaluation, such as a query expression in a .xqy file or XQuery code submitted in an xdmp:eval statement. The program consists not only of the code in the calling module, but also any imported modules that are called from the calling module, and any modules they might call, and so on.
session A conversation with a database on a MarkLogic Server instance. The session encapsulates state information such as connection information, credentials, and transaction settings. The precise nature of a session depends on the context of the conversation. For details, see Sessions.
request Any invocation of a program, whether through an App Server, through a task server, through xdmp:eval, or through any other means. In addition, certain client calls to App Servers (for example, loading an XML document through XCC, downloading an image through HTTP, or locking a document through WebDAV) are also requests.

Overview of MarkLogic Server Transactions

This section summarizes the following key transaction concepts in MarkLogic Server for quick reference.

The remainder of the chapter covers these concepts in detail.

Key Transaction Attributes

MarkLogic supports the following transaction models:

  • Single-statement transactions, which are automatically committed at the end of a statement. This is the default transaction model in MarkLogic.
  • Multi-statement transactions, which can span multiple requests or statements and must be explicitly committed.

Updates made by a statement in a multi-statement transaction are visible to subsequent statements in the same transaction, but not to code running outside the transaction.

An application can use either or both transaction models. Single statement transactions are suitable for most applications. Multi-statement transactions are powerful, but introduce more complexity to your application. Focus on the concepts that match your chosen transactional programming model.

In addition to being single or multi-statement, transactions are typed as either update or query. The transaction type determines what operations are permitted and if, when, and how locks are acquired. By default, MarkLogic automatically detects the transaction type, but you can also explicitly specify the type.

The transactional model (single or multi-statement), commit mode (auto or explicit), and the transaction type (auto, query, or update) are fixed at the time a transaction is created. For example, if a block of code is evaluated by an xdmp:eval (XQuery) or xdmp.eval (JavaScript) call using same-statement isolation, then it runs in the caller's transaction context, so the transaction configuration is fixed by the caller, even if the called code attempts to change the settings.

The default transaction semantics vary slightly between XQuery and Server-Side JavaScript. The default behavior for each language is shown in the following table, along with information about changing the behavior. For details, see Transaction Type.

Langauge Default Transaction Behavior Alternative
XQuery single-statement, auto-commit, with auto-detection of transaction type Use the update prolog option to explicitly set the transaction type to auto, update or query. Use the commit option to set the commit mode to auto (single-statement) or explicit (multi-statement). Similar controls are available through options on functions such as xdmp:eval and xdmp:invoke.
Server-Side JavaScript single-statement, auto-commit, with query transaction type Use the declareUpdate function to explicitly set the transaction type to update and control whether the commit mode is auto (single-statement) or explicit (multi-statement). Auto detection of transaction type is not available. Similar controls are available through options on functions such as xdmp.eval and xdmp.invoke.

A statement can be either a query statement (read only) or an update statement. In XQuery, the first (or only) statement type determines the transaction type unless you explicitly set the transaction type. The statement type is determined through static analysis. In JavaScript, query statement type is assumed unless you explicitly set the transaction to update.

In the context of transactions, a statement has different meanings for XQuery and JavaScript. For details, see Understanding Statement Boundaries.

Understanding Statement Boundaries

Since transactions are often described in terms of statements, you must understand what constitutes a statement in your server-side programming language:

Transactional Statements in XQuery

In XQuery, a statement for transaction purposes is one complete XQuery statement that can be executed as a main module. You can use the semi-colon separator to include multiple statements in a single block of code.

For example, the following code block contains two statements:

xquery version "1.0-ml";
xdmp:document-insert('/some/uri/doc.xml', <data/>);
(: end of statement 1 :)

xquery version "1.0-ml";
fn:doc('/some/uri/doc.xml'); 
(: end of statement 2 :)

By default, the above code executes as two auto-detect, auto-commit transactions.

If you evaluate this code as a multi-statement transaction, both statements would execute in the same transaction; depending on the evaluation context, the transaction might remain open or be rolled back at the end of the code since there is no explicit commit.

For more details, see Semi-Colon as a Statement Separator.

Transactional Statements in Server-Side JavaScript

In JavaScript, an entire script or main module is considered a statement for transaction purposes, no matter how many JavaScript statements it contains. For example, the following code is one transactional statement, even though it contains multiple JavaScript statements:

'use strict';
declareUpdate();
xdmp.documentInsert('/some/uri/doc.json', {property: 'value'});
console.log('I did something!');
// end of module

By default, the above code executes in a single transaction that completes at the end of the script. If you evaluate this code in the context of a multi-statement transaction, the transaction remains open after completion of the script.

Single-Statement Transaction Concept Summary

If you use the default model (single-statement, auto-commit), it is important to understand the following concepts:

Single-Statement Transaction Concept Where to Learn More
Statements run in a transaction. Single vs. Multi-statement Transactions
A transaction contains exactly one statement. Single-Statement, Automatically Committed Transactions
Transactions are automatically committed at the end of every statement. Single-Statement, Automatically Committed Transactions
Transactions have either update or query type.
  • Query transactions use a system timestamp instead of locks.
  • Update transactions acquire locks.
Transaction Type

In XQuery, transaction type can be detected by MarkLogic, or explicitly set.

In JavaScript, transaction type is assumed to be query unless you explicitly set it to update. Auto detection is not available.

Transaction Type
Updates made by a statement are not visible until the statement (transaction) completes. Update Transactions: Readers/Writers Locks

In XQuery, semi-colon can be used as a statement/transaction separator to include multiple statements in a main module.

Each JavaScript program is considered a statement for transaction purposes, regardless how many JavaScript statements it contains.

Understanding Statement Boundaries

Single-Statement, Automatically Committed Transactions

Semi-Colon as a Statement Separator

Multi-Statement Transaction Concept Summary

If you use multi-statement transactions, it is important to understand the following concepts:

Multi-Statement Transaction Concept Where to Learn More
Statements run in a transaction. Single vs. Multi-statement Transactions
A transaction contains one or more statements that either all succeed or all fail. Multi-Statement, Explicitly Committed Transactions
Multi-statement transactions must be explicitly committed using xdmp:commit (XQuery) or xdmp.commit (JavaScript).

Multi-Statement, Explicitly Committed Transactions

Committing Multi-Statement Transactions

Rollback can be implicit or explicit. For explicit rollback, use xdmp:rollback (XQuery) or xdmp.rollback (JavaScript).

Multi-Statement, Explicitly Committed Transactions

Rolling Back Multi-Statement Transactions

Transactions have either update or query type.
  • Query transactions use a system timestamp instead of acquiring locks
  • Update transactions acquire locks.
Transaction Type

In XQuery, transaction type can be detected by MarkLogic, or explicitly set.

In JavaScript, transaction type is assumed to be query unless you explicitly set it to update. Auto detection is not available.

Transaction Type
Transactions run in a session. Sessions
Sessions have a transaction mode that affects the following:
  • transaction type
  • commit semantics
  • how many statements a transaction can contain
Transaction Mode
Setting the commit mode to explicit always creates a multi-statement transaction, explicit-commit transaction.

Single vs. Multi-statement Transactions

Multi-Statement, Explicitly Committed Transactions

Updates made by a statement are not visible until the statement completes. Update Transactions: Readers/Writers Locks
Updates made by a statement are visible to subsequent statements in the same transaction while the transaction is still open. Multi-Statement, Explicitly Committed Transactions
In XQuery, semi-colon can be used as a statement separator to include multiple statements in a transaction.

Understanding Statement Boundaries

Semi-Colon as a Statement Separator

Commit Mode

A transaction can run in either auto or explicit commit mode.

The default behavior for a single-statement transaction is auto commit, which means MarkLogic commits the transaction at the end of a statement, as defined in Understanding Statement Boundaries.

Explicit commit mode is intended for multi-statement transactions. In this mode, you must explicitly commit the transaction by calling xdmp:commit (XQuery) or xdmp.commit (JavaScript), or explicitly roll back the transaction by calling xdmp:rollback (XQuery) or xdmp.rollback (JavaScript). This enables you to leave a transaction open across multiple statements or requests.

You can control the commit mode in the following ways:

The following functions support commit and update options that enable you to control the commit mode (explicit or auto) and transaction type (update, query, or auto). For details, see the function reference for xdmp:eval or xdmp.eval.

XQuery JavaScript
xdmp:eval xdmp.eval
xdmp:javascript-eval xdmp.xqueryEval
xdmp:invoke xdmp.invoke
xdmp:invoke-function xdmp.invokeFunction
xdmp:spawn xdmp.spawn
xdmp:spawn-function

Transaction Type

This section covers the following information related to transaction type. This information applies to both single-statement and multi-statement transactions.

Transaction Type Overview

Transaction type determines the type of operations permitted by a transaction and whether or not the transaction uses locks. Transactions have either update or query type. Statements also have query or update type, depending on the type of operations they perform.

Update transactions and statements can perform both query and update operations. Query transactions and statements are read-only and may not attempt update operations. A query transaction can contain an update statement, but an error is raised if that statement attempts an update operation at runtime; for an example, see Query Transaction Mode.

MarkLogic Server determines transaction type in the following ways:

  • Auto: (XQuery only) MarkLogic determines the transaction type through static analysis of the first (or only) statement in the transaction. Auto is the default behavior in XQuery.
  • Explicit: Your code explicitly specifies the transaction type as update or query through an option, a call to xdmp:set-transaction-mode (XQuery) or xdmp.setTransactionMode (JavaScript), or by calling declareUpdate (JavaScript only).

For more details, see Controlling Transaction Type in XQuery or Controlling Transaction Type in JavaScript.

Query transactions use a system timestamp to access a consistent snapshot of the database at a particular point in time, rather than using locks. Update transactions use readers/writers locks. See Query Transactions: Point-in-Time Evaluation and Update Transactions: Readers/Writers Locks.

The following table summarizes the interactions between transaction types, statements, and locking behavior. These interactions apply to both single-statement and multi-statement transactions.

Transaction Type Statement Behavior
query query Point-in-time view of documents. No locking required.
update Runtime error.
update query Read locks acquired, as needed.
update Readers/writers locks acquired, as needed.

Controlling Transaction Type in XQuery

You do not need to explicitly set transaction type unless the default auto-detection is not suitable for your application. When the transaction type is auto (the default), MarkLogic determines the transaction type through static analysis of your code. In a multi-statement transaction, MarkLogic examines only the first statement when auto-detecting transaction type.

To explicitly set the transaction type:

Use the xdmp:update prolog option when you need to set the transaction type before the first transaction is created, such as at the beginning of a main module. For example, the following code runs as a multi-statement update transaction because of the prolog options:

declare option xdmp:commit "explicit";
declare option xdmp:update "true";

let $txn-name := "ExampleTransaction-1"
return (
  xdmp:set-transaction-name($txn-name),
  fn:concat($txn-name, ": ",
    xdmp:host-status(xdmp:host())
      //hs:transaction[hs:transaction-name eq $txn-name]
        /hs:transaction-mode)
  );
  xdmp:commit();

For more details, see xdmp:update and xdmp:commit in the XQuery and XSLT Reference Guide.

Setting transaction mode with xdmp:set-transaction-mode affects both the commit semantics (auto or explicit) and the transaction type (auto, query, or update). Setting the transaction mode in the middle of a transaction does not affect the current transaction. Setting the transaction mode affects the transaction creation semantics for the entire session.

The following example uses xdmp:set-transaction-mode to demonstrate that the currently running transaction is unaffected by setting the transaction mode to a different value. The example uses xdmp:host-status to examine the mode of the current transaction. (The example only uses xdmp:set-transaction-name to easily pick out the relevant transaction in the xdmp:host-statusresults.)

xquery version "1.0-ml";

declare namespace hs="http://marklogic.com/xdmp/status/host";

(: The first transaction created will run in update mode :)
declare option xdmp:commit "explicit";
declare option xdmp:update "true";

let $txn-name := "ExampleTransaction-1"
return (
  xdmp:set-transaction-name($txn-name),
  xdmp:set-transaction-mode("query"),  (: no effect on current txn :)
  fn:concat($txn-name, ": ",
    xdmp:host-status(xdmp:host())
      //hs:transaction[hs:transaction-name eq $txn-name]
        /hs:transaction-mode)
  );

(: complete the current transaction :)
xdmp:commit();

(: a new transaction is created, inheriting query mode from above :)
declare namespace hs="http://marklogic.com/xdmp/status/host";
let $txn-name := "ExampleTransaction-2"
return (
  xdmp:set-transaction-name($txn-name),
    fn:concat($txn-name, ": ",
      xdmp:host-status(xdmp:host())
        //hs:transaction[hs:transaction-name eq $txn-name]
          /hs:transaction-mode)
  );

If you paste the above example into Query Console, and run it with results displayed as text, you see the first transaction runs in update mode, as specified by xdmp:transaction-mode, and the second transaction runs in query mode, as specified by xdmp:set-transaction-mode:

ExampleTransaction-1: update
ExampleTransaction-2: query

You can include multiple option declarations and calls to xdmp:set-transaction-mode in your program, but the settings are only considered at transaction creation. A transaction is implicitly created just before evaluating the first statement. For example:

xquery version "1.0-ml";
declare option xdmp:commit "explicit";
declare option xdmp:update "true";

(: begin transaction :)
"this is an update transaction";
xdmp:commit();
(: end transaction :)

xquery version "1.0-ml";
declare option xdmp:commit "explicit";
declare option xdmp:update "false";

(: begin transaction :)
"this is a query transaction";
xdmp:commit();
(: end transaction :)

The following functions support commit and update options that enable you to control the commit mode (explicit or auto) and transaction type (update, query, or auto). For details, see the function reference for xdmp:eval or xdmp.eval.

XQuery JavaScript
xdmp:eval xdmp.eval
xdmp:javascript-eval xdmp.xqueryEval
xdmp:invoke xdmp.invoke
xdmp:invoke-function xdmp.invokeFunction
xdmp:spawn xdmp.spawn
xdmp:spawn-function

Controlling Transaction Type in JavaScript

By default, Server-Side JavaScripts runs in a single-statement, auto-commit, query transaction. You can control transaction type in the following ways:

  • Use the declareUpdate function to set the transaction type to update and/or specify the commit semantics, or
  • Set the update option in the options node passed to functions such as xdmp.eval, xdmp.invoke, or xdmp.spawn; or
  • Call xdmp.setTransactionModeprior to creating transactions that will run in that mode.
Configuring a Transaction Using declareUpdate

By default, JavaScript runs in auto commit mode with query transaction type. You can use the declareUpdate function to change the transaction type to update and/or the commit mode from auto to explicit.

MarkLogic cannot use static analysis to determine whether or not JavaScript code performs updates. If your JavaScript code makes updates, one of the following requirements must be met:

  • You call the declareUpdate function to indicate your code will make updates.
  • The caller of your code sets the transaction type to one that permits updates.

Calling declareUpdate with no arguments is equivalent to auto commit mode and update transaction type. This means the code can make updates and runs as a single-statement transaction. The updates are automatically committed when the JavaScript code completes.

You can also pass an explicitCommit option to declareUpdate, as shown below. The default value of explicitCommit is false.

declareUpdate({explicitCommit: boolean});

If you set explicitCommit to true, then your code starts a new multi-statement update transaction. You must explicitly commit or rollback the transaction, either before returning from your JavaScript code or in another context, such as the caller of your JavaScript code or another request executing in the same transaction.

For example, you might use explicitCommit to start a multi-statement transaction in an ad-hoc query request through XCC, and then subsequently commit the transaction through another request.

If the caller sets the transaction type to update, then your code is not required to call declareUpdate in order to perform updates. If you do call declareUpdate in this situation, then the resulting mode must not conflict with the mode set by the caller.

For more details, see declareUpdate Function in the JavaScript Reference Guide.

Configuring Transactions in the Caller

The following are examples of cases in which the transaction type and commit mode might be set before your code is called:

  • Your code is called via an eval/invoke function such as the XQuery function xdmp:javascript-eval or the JavaScript functions xdmp.eval, and the caller specifies the commit, update, or transaction-mode option.
  • Your code is a server-side import transformation for use with the mlcp command line tool.
  • Your code is a server-side transformation, extension, or other customization called by the Java, Node.js, or REST Client APIs. The pre-set mode depends on the operation which causes your code to run.
  • Your code runs in the context of an XCC session where the client sets the commit mode and/or transaction type.

The following functions support commit and update options that enable you to control the commit mode (explicit or auto) and transaction type (update, query, or auto). For details, see the function reference for xdmp:eval (XQuery) or xdmp.eval (JavaScript).

XQuery JavaScript
xdmp:eval xdmp.eval
xdmp:javascript-eval xdmp.xqueryEval
xdmp:invoke xdmp.invoke
xdmp:invoke-function xdmp.invokeFunction
xdmp:spawn xdmp.spawn
xdmp:spawn-function

Query Transactions: Point-in-Time Evaluation

Query transactions are read-only and never obtain locks on documents. This section explores the following concepts related to query transactions:

System Timestamps and Fragment Versioning

To understand how transactions work in MarkLogic Server, it is important to understand how documents are stored. Documents are made up of one or more fragments. After a document is created, each of its fragments are stored in one or more stands. The stands are part of a forest, and the forest is part of a database. A database contains one or more forests.

Each fragment in a stand has system timestamps associated with it, which correspond to the range of system timestamps in which that version of the fragment is valid. When a document is updated, the update process creates new versions of any fragments that are changed. The new versions of the fragments are stored in a new stand and have a new set of valid system timestamps associated with them. Eventually, the system merges the old and new stands together and creates a new stand with only the latest versions of the fragments. Point-in-time queries also affect which versions of fragments are stored and preserved during a merge. After the merge, the old stands are deleted.

The range of valid system timestamps associated with fragments are used when a statement determines which version of a document to use during a transaction. For more details about merges, see Understanding and Controlling Database Merges in the Administrator's Guide. For more details on how point-in-time queries affect which versions of documents are stored, see Point-In-Time Queries.

Query Transactions Run at a Timestamp (No Locks)

Query transactions run at the system timestamp corresponding to transaction creation time. Calls to xdmp:request-timestamp return the same system timestamp at any point during a query transaction; they never return the empty sequence. Query transactions do not obtain locks on any documents, so other transactions can read or update the document while the transaction is executing.

Query Transactions See Latest Version of Documents Up To Timestamp of Transaction

When a query transaction is created, MarkLogic Server gets the current system timestamp (the number returned when calling the xdmp:request-timestamp function) and uses only the latest versions of documents whose timestamp is less than or equal to that number. Even if any of the documents that the transaction accesses are updated or deleted outside the transaction while the transaction is open, the use of timestamps ensures that all statements in the transaction always see a consistent view of the documents the transaction accesses.

Update Transactions: Readers/Writers Locks

Update transactions have the potential to change the database, so they obtain locks on documents to ensure transactional integrity. Update transactions run with readers/writers locks, not at a timestamp like query transactions. This section covers the following topics:

Identifying Update Transactions

When MarkLogic creates a transaction in auto-detect mode, the transaction type is determined through static analysis of the first (or only) statement in the transaction. If MarkLogic detects the potential for updates during static analysis, then the transaction is considered an update transaction.

Depending on the specific logic of the transaction, it might not actually update anything, but a transaction that MarkLogic determines to be an update transaction always runs as an update transaction, not a query transaction.

For example, the following transaction runs as an update transaction even though the xdmp:document-insert can never occur:

if ( 1 = 2 ) 
then ( xdmp:document-insert("fake.xml", <a/>) ) 
else ()

In a multi-statement transaction, the transaction type always corresponds to the transaction type settings in effect when the transaction is created. If the transaction type is explicitly set to update, then the transaction is an update transaction, even if none of the contained statements perform updates. Locks are acquired for all statements in an update transaction, whether or not they perform updates.

Similarly, if you use auto-detect mode and MarkLogic determines the first statement in a multi-statement transaction is a query statement, then the transaction is created as a query transaction. If a subsequent statement in the transaction attempts an update operation, MarkLogic throws an exception.

Calls to xdmp:request-timestamp always return the empty sequence during an update transaction; that is, if xdmp:request-timestamp returns a value, the transaction is a query transaction, not an update transaction.

Locks Are Acquired on Demand and Held Throughout a Transaction

Because update transactions do not run at a set timestamp, they see the latest view of any given document at the time it is first accessed by any statement in the transaction. Because an update transaction must successfully obtain locks on all documents it reads or writes in order to complete evaluation, there is no chance that a given update transaction will see half or some of the updates made by some other transactions; the statement is indeed transactional.

Once a lock is acquired, it is held until the transaction ends. This prevents other transactions from updating the read locked document and ensures a read-consistent view of the document. Query (read) operations require read locks. Update operations require readers/writers locks.

When a statement in an update transaction wants to perform an update operation, a readers/writers lock is acquired (or an existing read lock is converted into a readers/writers lock) on the document. A readers/writers lock is an exclusive lock. The readers/writers lock cannot be acquired until any locks held by other transactions are released.

Lock lifetime is an especially important consideration in multi-statement transactions. Consider the following single-statement example, in which a readers/writers lock is acquired only around the call to xdmp:node-replace:

  (: query statement, no locks needed :)
fn:doc("/docs/test.xml");
  (: update statement, readers/writers lock acquired :)
xdmp:node-replace(fn:doc("/docs/test.xml")/node(), <a>hello</a>);
  (: readers/writers lock released :)
  (: query statement, no locks needed :)
fn:doc("/docs/test.xml");

If the same example is rewritten as a multi-statement transaction, locks are held across all three statements:

declare option xdmp:transaction-mode update;

  (: read lock acquired :)
fn:doc("/docs/test.xml");
  (: the following converts the lock to a readers/writers lock :)
xdmp:node-replace(fn:doc("/docs/test.xml")/node(), <a>hello</a>);
  (: readers/writers lock still held :)
fn:doc("/docs/test.xml");

  (: after the following statement, txn ends and locks released :)
xdmp:commit()
Visibility of Updates

Updates are only visible within a transaction after the updating statement completes; updates are not visible within the updating statement. Updates are only visible to other transactions after the updating transaction commits. Pre-commit triggers run as part of the updating transaction, so they see updates prior to commit. Transaction model affects the visibility of updates, indirectly, because it affects when commit occurs.

In the default single-statement transaction model, the commit occurs automatically when the statement completes. To use a newly updated document, you must separate the update and the access into two single-statement transactions or use multi-statement transactions.

In a multi-statement transaction, changes made by one statement in the transaction are visible to subsequent statements in the same transaction as soon as the updating statement completes. Changes are not visible outside the transaction until you call xdmp:commit.

An update statement cannot perform an update to a document that will conflict with other updates occurring in the same statement. For example, you cannot update a node and add a child element to that node in the same statement. An attempt to perform such conflicting updates to the same document in a single statement will fail with an XDMP-CONFLICTINGUPDATES exception.

Example: Query and Update Transaction Interaction

The following figure shows three different transactions, T1, T2, and T3, and how the transactional semantics work for each one:

Assume T1 is a long-running update transaction which starts when the system is at timestamp 10 and ends up committing at timestamp 40 (meaning there were 30 updates or other changes to the system while this update statement runs).

When T2 reads the document being updated by T1 (doc.xml), it sees the latest version that has a system timestamp of 20 or less, which turns out to be the same version T1 uses before its update.

When T3 tries to update the document, it finds that T1 has readers/writers locks on it, so it waits for them to be released. After T1 commits and releases the locks, then T3 sees the newly updated version of the document, and performs its update which is committed at a new timestamp of 41.

Single vs. Multi-statement Transactions

This section discusses the details of and differences between the two transaction programming models supported by MarkLogic Server, single-statement and multi-statement transactions. The following topics are covered:

Single-Statement, Automatically Committed Transactions

By default, all transactions in MarkLogic Server are single-statement, auto-commit transactions. In this default model, MarkLogic creates a transaction to evaluate each statement. When the statement completes, MarkLogic automatically commits (or rolls back, in case of error) the transaction, and then the transaction ends.

In Server-Side JavaScript, a JavaScript program (or script) is considered a single statement in the transactional sense. For details, see Understanding Statement Boundaries.

In a single statement transaction, updates made by a statement are not visible outside the statement until the statement completes and the transaction is committed.

The single-statement model is suitable for most applications. This model requires less familiarity with transaction details and introduces less complexity into your application:

  • Statement and transaction are nearly synonymous.
  • The server determines the transaction type through static analysis.
  • If the statement completes successfully, the server automatically commits the transaction.
  • If an error occurs, the server automatically rolls back any updates made by the statement.

Updates made by a single-statement transaction are not visible outside the statement until the statement completes. For details, see Visibility of Updates.

Use the semi-colon separator extension in XQuery to include multiple single-statement transactions in your program. For details, see Semi-Colon as a Statement Separator.

In Server-Side JavaScript, you need to use the declareUpdate() function to run an update. For details, see Controlling Transaction Type in JavaScript.

Multi-Statement, Explicitly Committed Transactions

When a transaction is created in a context in which the commit mode is set to explicit, the transaction will be a multi-statement transaction. This section covers the following related topics:

For details on setting the transaction type and commit mode, see Transaction Type.

For additional information about using multi-statement transactions in Java, see Multi-Statement Transactions in the XCC Developer's Guide.

Characteristics of Multi-Statement Transactions

Using multi-statement transactions introduces more complexity into your application and requires a deeper understanding of transaction handling in MarkLogic Server. In a multi-statement transaction:

  • In XQuery, semi-colon acts as a separator between statements in the same transaction.
  • In Server-Side JavaScript, the entire program (script) is considered a single transactional statement, regardless how many JavaScript statements it contains.
  • Each statement in the transaction sees changes made by previously evaluated statements in the same transaction.
  • The statements in the transaction either all commit or all fail.
  • You must use xdmp:commit (XQuery) or xdmp.commit (JavaScript) to commit the transaction.
  • You can use xdmp:rollback (XQuery) or xdmp.rollback (JavaScript) to abort the transaction.

A multi-statement transaction is bound to the database in which it is created. You cannot use a transaction id created in one database context to perform an operation in the same transaction on another database.

The statements in a multi-statement transaction are serialized, even if they run in different requests. That is, one statement in the transaction completes before another one starts, even if the statements execute in different requests.

A multi-statement transaction ends only when it is explicitly committed using xdmp:commit or xdmp.commit, when it is explicitly rolled back using xdmp:rollback or xdmp.rollback, or when it is implicitly rolled back through timeout, error, or session completion. Failure to explicitly commit or roll back a multi-statement transaction might retain locks and keep resources tied up until the transaction times out or the containing session ends. At that time, the transaction rolls back. Best practice is to always explicitly commit or rollback a multi-statement transaction.

The following example contains 3 multi-statement transactions (because of the use of the commit prolog option). The first transaction is explicitly committed, the second is explicitly rolled back, and the third is implicitly rolled back when the session ends without a commit or rollback call. Running the example in Query Console is equivalent to evaluating it using xdmp:eval with different transaction isolation, so the final transaction rolls back when the end of the query is reached because the session ends. For details about multi-statement transaction interaction with sessions, see Sessions.

xquery version "1.0-ml";

declare option xdmp:commit "explicit";
(: Begin transaction 1 :)
xdmp:document-insert('/docs/mst1.xml', <data/>);
(: This statement runs in the same txn, so sees /docs/mst1.xml :)
xdmp:document-insert('/docs/mst2.xml', fn:doc('/docs/mst1.xml'));
xdmp:commit();
(: Transaction ends, updates visible in database :)

declare option xdmp:commit "explicit";
(: Begin transaction 2 :)
xdmp:document-delete('/docs/mst1.xml');
xdmp:rollback();
(: Transaction ends, updates discarded :)

declare option xdmp:commit "explicit";
(: Begin transaction 3 :)
xdmp:document-delete('/docs/mst1.xml');
(: Transaction implicitly ends and rolls back due to
 :   reaching end of program without a commit :)

As discussed in Update Transactions: Readers/Writers Locks, multi-statement update transactions use locks. A multi-statement update transaction can contain both query and update operations. Query operations in a multi-statement update transaction acquire read locks as needed. Update operations in the transaction will upgrade such locks to read/write locks or acquire new read/write locks if needed.

Instead of acquiring locks, a multi-statement query transaction uses a system timestamp to give all statements in the transaction a read consistent view of the database, as discussed in Query Transactions: Point-in-Time Evaluation. The system timestamp is determined when the query transaction is created, so all statements in the transaction see the same version of accessed documents.

Committing Multi-Statement Transactions

Multi-statement transactions are explicitly committed by calling xdmp:commit. If a multi-statement update transaction does not call xdmp:commit, all its updates are lost when the transaction ends. Leaving a transaction open by not committing updates ties up locks and other resources.

Once updates are committed, the transaction ends and evaluation of the next statement continues in a new transaction. For example:

xquery version "1.0-ml";

declare option xdmp:commit "explicit";

(: Begin transaction 1 :)
xdmp:document-insert('/docs/mst1.xml', <data/>);
(: This statement runs in the same txn, so sees /docs/mst1.xml :)
xdmp:document-insert('/docs/mst2.xml', fn:doc('/docs/mst1.xml'));
xdmp:commit();
(: Transaction ends, updates visible in database :)

Calling xdmp:commit commits updates and ends the transaction only after the calling statement successfully completes. This means updates can be lost even after calling xdmp:commit, if an error occurs before the committing statement completes. For this reason, it is best practice to call xdmp:commit at the end of a statement.

The following example preserves updates even in the face of error because the statement calling xdmp:commit always completes.:

xquery version "1.0-ml";

declare option xdmp:commit "explicit";

(: transaction created :)
xdmp:document-insert("not-lost.xml", <data/>)
, xdmp:commit();
fn:error(xs:QName("EXAMPLE-ERROR"), "An error occurs here");
(: end of session or program :)

(: ==> Insert is retained because the statement
       calling commit completes successfully. :)

By contrast, the update in this example is lost because the error occurring in the same statement as the xdmp:commit call prevents successful completion of the committing statement:

xquery version "1.0-ml";

declare option xdmp:commit "explicit";

(: transaction created :)
xdmp:document-insert("lost.xml", <data/>)
, xdmp:commit()
, fn:error(xs:QName("EXAMPLE-ERROR"), "An error occurs here");
(: end of session or program :)

(: ==> Insert is lost because the statement
       terminates with an error before commit can occur. :)

Uncaught exceptions cause a transaction rollback. If code in a multi-statement transaction might raise an exception that must not abort the transaction, wrap the code in a try-catch block and take appropriate action in the catch handler. For example:

xquery version "1.0-ml";

declare option xdmp:commit "explicit";

xdmp:document-insert("/docs/test.xml", <a>hello</a>);
try {
  xdmp:document-delete("/docs/nonexistent.xml")
} catch ($ex) {
  (: handle error or rethrow :)
  if ($ex/error:code eq 'XDMP-DOCNOTFOUND') then ()
  else xdmp:rethrow()
}, xdmp:commit();
(: start of a new txn :)
fn:doc("/docs/test.xml")//a/text()
Rolling Back Multi-Statement Transactions

Multi-statement transactions are rolled back either implicitly (on error or when the containing session terminates), or explicitly (using xdmp:rollback or xdmp.rollback). Calling xdmp:rollback immediately terminates the current transaction. Evaluation of the next statement continues in a new transaction. For example:

xquery version "1.0-ml";

declare option xdmp:commit "explicit";
                                            (: begin transaction :)
xdmp:document-insert("/docs/mst.xml", <data/>);
xdmp:commit()
, "this expr is evaluated and committed";
                                            (: end transaction :)
                                            (:begin transaction :)
declare option xdmp:commit "explicit";
xdmp:document-insert("/docs/mst.xml", <data/>);
xdmp:rollback()                             (: end transaction :)
, "this expr is never evaluated";
                                            (:begin transaction :)
"execution continues here, in a new transaction"
                                            (: end transaction :)

The result of a statement terminated with xdmp:rollback is always the empty sequence.

Best practice is to explicitly rollback when necessary. Waiting on implicit rollback at session end leaves the transaction open and ties up locks and other resources until the session times out. This can be a relatively long time. For example, an HTTP session can span multiple HTTP requests. For details, see Sessions.

Sessions

A session is a conversation with a database in a MarkLogic Server instance. A session encapsulates state about the conversation, such as connection information, credentials, and transaction settings. When using multi-statement transactions, you must understand when evaluation might occur in a different session because:

  • transaction mode is an attribute of a session.
  • uncommitted transactions automatically roll back when the containing session ends.

For example, since a query evaluated by xdmp:eval (XQuery) or xdmp.eval (JavaScript) with different-transaction isolation runs in its own session, it does not inherit the transaction mode setting from the caller. Also, if the transaction is still open (uncommitted) when evaluation reaches the end of the eval'd query, the transaction automatically rolls back.

By contrast, in an HTTP session, the transaction settings might apply to queries run in response to multiple HTTP requests. Uncommitted transactions remain open until the HTTP session times out, which can be a relatively long time.

The exact nature of a session depends on the conversation context. The following table summarizes the most common types of sessions encountered by a MarkLogic Server application and their lifetimes:

Session Type Session Lifetime

HTTP

An HTTP client talking to an HTTP App Server.

A session is created when the first HTTP request is received from a client for which no session already exists. The session persists across requests until the session times out.

XCC

An XCC Java application talking to an XDBC App Server

A session is created when a Session object is instantiated and persists until the Session object is finalized, you call Session.close(), or the session times out.
Standalone query evaluated:
  • by xdmp:eval or xdmp:invoke with different-transaction isolation
  • by xdmp:spawn
  • as task on the Task Server)
A session is created to evaluate the eval/invoke/spawn'd query or task and ends when the query or task completes.

Session timeout is an App Server configuration setting. For details, see admin:appserver-set-session-timeout in XQuery and XSLT Reference Guide or the Session Timeout configuration setting in the Admin Interface for the App Server.

Semi-Colon as a Statement Separator

MarkLogic Server extends the XQuery language to include the semi-colon ( ; ) in the XQuery body as a separator between statements. Statements are evaluated in the order in which they appear. Each semi-colon separated statement in a transaction is fully evaluated before the next statement begins.

In a single-statement transaction, the statement separator is also a transaction separator. Each statement separated by a semi-colon is evaluated as its own transaction. It is possible to have a program where some semi-colon separated parts are evaluated as query statements and some are evaluated as update statements. The statements are evaluated in the order in which they appear, and in the case of update statements, one statement commits before the next one begins.

Semi-colon separated statements in auto commit mode (the default) are not multi-statement transactions. Each statement is a single-statement transaction. If one update statement commits and the next one throws a runtime error, the first transaction is not rolled back. If you have logic that requires a rollback if subsequent transactions fail, you must add that logic to your XQuery code, use multi-statement transactions, or use a pre-commit trigger. For information about triggers, see Using Triggers to Spawn Actions.

In a multi-statement transaction, the semi-colon separator does not act as a transaction separator. The semi-colon separated statements in a multi-statement transaction see updates made by previous statements in the same transaction, but the updates are not committed until the transaction is explicitly committed. If the transaction is rolled back, updates made by previously evaluated statements in the transaction are discarded.

The following diagram contrasts the relationship between statements and transactions in single and multi-statement transactions:

Transaction Mode

This section covers the following topics related to transaction mode:

Transaction Mode Overview

Transaction mode combines the concepts of commit mode (auto or explicit) and transaction type (auto, update, or query). The transaction mode setting is session wide. You can control transaction mode in the following ways:

  • Call the xdmp:set-transaction-mode XQuery function or the xdmp.setTransactionMode JavaScript function.
  • Deprecated: Use the transaction-mode option of xdmp:eval (XQuery) or xdmp.eval (JavaScript) or related functions eval/invoke/spawn functions. Use the commit and update options instead.
  • Deprecated: Use the XQuery prolog option xdmp:transaction-mode. Use the xdmp:commit and xdmp:update XQuery prolog options instead.

Be aware that the xdmp:commit and xdmp:update XQuery prolog options affect only the next transaction created after their declaration; they do not affect an entire session. Use xdmp:set-transaction-mode or xdmp.setTransactionMode if you need to change the settings at the session level.

Use the more specific commit mode and transaction type controls instead of setting transaction mode. These controls provide finer grained control over transaction configuration.

For example, use the following table to map the xdmp:transaction-mode XQuery prolog options to the xdmp:commit and xdmp:update prolog options. For more details, see Controlling Transaction Type in XQuery.

xdmp:transaction-mode Value Equivalent xdmp:commit and xdmp:update Option Settings
"auto"
declare option xdmp:commit "auto";
declare option xdmp:update "auto";
"update-auto-commit"
declare option xdmp:commit "auto";
declare option xdmp:update "true";
"query-single-statement"
declare option xdmp:commit "auto";
declare option xdmp:update "false";
"multi-auto"
declare option xdmp:commit "explicit";
declare option xdmp:update "auto";
"update"
declare option xdmp:commit "explicit";
declare option xdmp:update "true";
"query"
declare option xdmp:commit "explicit";
declare option xdmp:update "false";

Use the following table to map between the transaction-mode option and the commit and update options for xdmp:eval and related eval/invoke/spawn functions.

transaction-mode Option Value Equivalent commit and update Option Values
auto
commit: "auto"
update: "auto"
update-auto-commit
commit: "auto"
update: "true"
query-single-statement
commit: "auto"
update: "false"
multi-auto
commit: "explicit"
update: "auto"
update
commit: "explicit"
update: "true"
query
commit "explicit"
update "false"

Server-Side JavaScript modules use the declareUpdate function to indicate when the transaction mode is update-auto-commit or update. For more details, see Controlling Transaction Type in JavaScript.

To use multi-statement transactions in XQuery, you must explicitly set the transaction mode to multi-auto, query, or update. This sets the commit mode to explicit and specifies the transaction type. For details, see Transaction Type.

Selecting the appropriate transaction mode enables the server to properly optimize your queries. For more information, see Multi-Statement, Explicitly Committed Transactions.

The transaction mode is only considered during transaction creation. Changing the mode has no effect on the current transaction.

Explicitly setting the transaction mode affects only the current session. Queries run under xdmp:eval or xdmp.eval or a similar function with different-transaction isolation, or under xdmp:spawn do not inherit the transaction mode from the calling context. See Interactions with xdmp:eval/invoke.

Auto Transaction Mode

The default transaction mode is auto. This is equivalent to auto commit mode and auto transaction type. In this mode, all transactions are single-statement transactions. See Single-Statement, Automatically Committed Transactions.

Most XQuery applications use auto transaction mode. Using auto transaction mode allows the server to optimize each statement independently and minimizes locking on your files. This leads to better performance and decreases the chances of deadlock, in most cases.

Most Server-Side JavaScript applications use auto mode for code that does not perform updates, and update-auto-commit mode for code that performs updates. Calling declareUpdate with no arguments activates update-auto-commit mode; for more details, see Controlling Transaction Type in JavaScript..

In auto transaction mode:

  • All transactions are single-statement transactions, so a new transaction is created for each statement.
  • Static analysis of the statement prior to evaluation determines whether the created transaction runs in update or query mode.
  • The transaction associated with a statement is automatically committed when statement execution completes, or automatically rolled back if an error occurs.

The update-auto-commit differs only in that the transaction is always an update transaction.

In XQuery, you can set the mode to auto explicitly with xdmp:set-transaction-mode or the xdmp:transaction-mode prolog option, but this is not required unless you've previously explicitly set the mode to update or query.

Query Transaction Mode

Query transaction mode is equivalent to explicit commit mode plus query transaction type.

In XQuery, query transaction mode is only in effect when you explicitly set the mode using xdmp:set-transaction-mode or the xdmp:transaction-mode prolog option. Transactions created in this mode are always multi-statement transactions, as described in Multi-Statement, Explicitly Committed Transactions.

You cannot create a multi-statement query transaction from Server-Side JavaScript.

In query transaction mode:

  • Transactions can span multiple statements.
  • The transaction is assumed to be read-only, so no locks are acquired. MarkLogic executes all statements in the transaction as point-in-time queries, using the system timestamp at the start of the transaction.
  • All statements in the transaction must functionally be query (read-only) statements. MarkLogic raises an error at runtime if an update operation is attempted.
  • Transactions which are not explicitly committed using xdmp:commit roll back when the session times out. However, since there are no updates to commit, rollback is only distinguishable if an explicit rollback occurs before statement completion.

An update statement can appear in a multi-statement query transaction, but it must not actually make any update calls at runtime. If a transaction running in query mode attempts an update operation, XDMP-UPDATEFUNCTIONFROMQUERY is raised. For example, no exception is raised by the following code because the program logic causes the update operation not to run:

xquery version "1.0-ml";
declare option xdmp:transaction-mode "query";

if (fn:false())then
  (: XDMP-UPDATEFUNCTIONFROMQUERY only if this executes :)
  xdmp:document-insert("/docs/test.xml", <a/>)
else ();
xdmp:commit();

Update Transaction Mode

Update transaction mode is equivalent to explicit commit mode plus update transaction type.

In XQuery, update transaction mode is only in effect when you explicitly set the mode using xdmp:set-transaction-mode or the xdmp:transaction-mode prolog option. Transactions created in update mode are always multi-statement transactions, as described in Multi-Statement, Explicitly Committed Transactions.

In Server-Side JavaScript, setting explicitCommit to true when calling declareUpdate puts the transaction into update mode.

In update transaction mode:

  • Transactions can span multiple statements.
  • The transaction is assumed to change the database, so readers/writers locks are acquired as needed.
  • Statements in an update transaction can be either update or query statements.
  • Transactions which are not explicitly committed using xdmp:commit roll back when the session times out.

Update transactions can contain both query and update statements, but query statements in update transactions still acquire read locks rather than using a system timestamp. For more information, see Update Transactions: Readers/Writers Locks.

Query-Single-Statement Transaction Mode

The query-single-statement transaction mode is equivalent to auto commit mode plus query transaction type.

In XQuery, this transaction mode is only in effect when you explicitly set the mode using xdmp:set-transaction-mode or the xdmp:transaction-mode prolog option. Transactions created in this mode are always single-statement transactions, as described in Single-Statement Transaction Concept Summary.

You cannot explicitly create a single-statement query-only transaction from Server-Side JavaScript, but this is the default transaction mode when declareUpdate is not present.

In this transaction mode:

  • All transactions are single-statement transactions, so a new transaction is created for each statement.
  • The transaction is assumed to be read-only, so no locks are acquired. The statement is evaluated as a point-in-time query, using the system timestamp at the start of the transaction.
  • An error is raised at runtime if an update operation is attempted by the transaction.
  • The transaction is automatically committed when statement execution completes, or automatically rolled back if an error occurs.

An update operation can appear in a this type of transaction, but it must not actually make any updates at runtime. If a transaction running in query mode attempts an update operation, XDMP-UPDATEFUNCTIONFROMQUERY is raised.

Multi-Auto Transaction Mode

Setting transaction mode to multi-auto is equivalent to explicit commit mode plus auto transaction type. In this mode, all transactions are multi-statement transactions, and MarkLogic determines for you whether the transaction type is query or update.

In multi-auto transaction mode:

  • Transactions can span multiple statements.
  • Static analysis of the first statement in the transaction determines whether the transaction type is query or update.
  • Transactions which are not explicitly committed using xdmp:commit or xdmp.commit roll back when the session times out.

In XQuery, multi-auto transaction mode is only in effect when you explicitly set the mode using xdmp:set-transaction-modeor the xdmp:transaction-mode prolog option.

There is no equivalent to multi-auto transaction mode for Server-Side JavaScript.

Interactions with xdmp:eval/invoke

The xdmp:eval and xdmp:invoke family of functions enable you to start one transaction from the context of another. The xdmp:eval XQuery function and the xdmp.eval JavaScript function submit a string to be evaluated. The xdmp:invoke XQuery function and the xdmp.invoke JavaScript function evaluate a stored module. You can control the semantics of eval and invoke with options to the functions, and this can subtly change the transactional semantics of your program. This section describes some of those subtleties and includes the following parts:

Isolation Option to xdmp:eval/invoke

The xdmp:eval and xdmp:invoke XQuery functions and their JavaScript counterparts accept a set of options as an optional third parameter. The isolation option determines the behavior of the transaction that results from the eval/invoke operation, and it must be one of the following values:

  • same-statement
  • different-transaction

In same-statement isolation, the code executed by eval or invoke runs as part of the same statement and in the same transaction as the calling statement. Any updates done in the eval/invoke operation with same-statement isolation are not visible to subsequent parts of the calling statement. However, when using multi-statement transactions, those updates are visible to subsequent statements in the same transaction.

You may not perform update operations in code run under eval/invoke in same-statement isolation called from a query transaction. Since query transactions run at a timestamp, performing an update would require a switch between timestamp mode and readers/writers locks in the middle of a transaction, and that is not allowed. Statements or transactions that do so will throw XDMP-UPDATEFUNCTIONFROMQUERY.

You may not use same-statement isolation when using the database option of eval or invoke to specify a different database than the database in the calling statement's context. If your eval/invoke code needs to use a different database, use different-transaction isolation.

When you set the isolation to different-transaction, the code that is run by eval/invoke runs in a separate session and a separate transaction from the calling statement. The eval/invoke session and transaction will complete before continuing the rest of the caller's transaction. If the calling transaction is an update transaction, any committed updates done in the eval/invoke operation with different-transaction isolation are visible to subsequent parts of the calling statement and to subsequent statements in the calling transaction. However, if you use different-transaction isolation (which is the default isolation level), you need to ensure that you do not get into a deadlock situation (see Preventing Deadlocks).

The following table shows which isolation options are allowed from query statements and update statements.

Calling Statement Called Statement (xdmp:eval, xdmp:invoke)
same-statement isolation different-transaction isolation
query statement update statement query statement update statement
query statement (timestamp mode) Yes Yes, if no update takes place. If an update takes place, throws exception. Yes Yes
update statement (readers/writers locks mode) Yes (see Note) Yes Yes Yes (possible deadlock if updating a document with any lock)

This table is slightly simplified. For example, if an update statement calls a query statement with same-statement isolation, the query statement is actually run as part of the update statement (because it is run as part of the same transaction as the calling update statement), and it therefore runs with readers/writers locks, not in a timestamp.

Preventing Deadlocks

A deadlock is where two processes or threads are each waiting for the other to release a lock, and neither process can continue until the lock is released. Deadlocks are a normal part of database operations, and when the server detects them, it can deal with them (for example, by retrying one or the other transaction, by killing one or the other or both requests, and so on).

There are, however, some deadlock situations that MarkLogic Server cannot do anything about except wait for the transaction to time out. When you run an update statement that calls an xdmp:eval or xdmp:invoke statement, and the eval/invoke in turn is an update statement, you run the risk of creating a deadlock condition. These deadlocks can only occur in update statements; query statements will never cause a deadlock.

A deadlock condition occurs when a transaction acquires a lock of any kind on a document and then an eval/invoke statement called from that transaction attempts to get a write lock on the same document. These deadlock conditions can only be resolved by cancelling the query or letting the query time out.

To be completely safe, you can prevent these deadlocks from occurring by setting the prevent-deadlocks option to true, as in the following example:

xquery version "1.0-ml";
(: the next line ensures this runs as an update statement :)
declare option xdmp:update "true";
xdmp:eval("xdmp:node-replace(doc('/docs/test.xml')/a, <b>goodbye</b>)",
          (),
          <options xmlns="xdmp:eval">
            <isolation>different-transaction</isolation>
            <prevent-deadlocks>true</prevent-deadlocks>
          </options>) ,
doc("/docs/test.xml")

This statement will then throw the following exception:

XDMP-PREVENTDEADLOCKS: Processing an update from an update with
  different-transaction isolation could deadlock

In this case, it will indeed prevent a deadlock from occurring because this statement runs as an update statement, due to the xdmp:document-insert call, and therefore uses readers/writers locks. In line 2, a read lock is placed on the document with URI /docs/test.xml. Then, the xdmp:eval statement attempts to get a write lock on the same document, but it cannot get the write lock until the read lock is released. This creates a deadlock condition. Therefore the prevent-deadlocks option stopped the deadlock from occurring.

If you remove the prevent-deadlocks option, then it defaults to false (that is, it will allow deadlocks). Therefore, the following statement results in a deadlock:

This code is for demonstration purposes; if you run this code, it will cause a deadlock and you will have to cancel the query or wait for it to time out to clear the deadlock.

(: the next line ensures this runs as an update statement :)
if ( 1 = 2) then ( xdmp:document-insert("foobar", <a/>) ) else (),
doc("/docs/test.xml"),
xdmp:eval("xdmp:node-replace(doc('/docs/test.xml')/a, <b>goodbye</b>)",
          (),
          <options xmlns="xdmp:eval">
            <isolation>different-transaction</isolation>
          </options>) ,
doc("/docs/test.xml")

This is a deadlock condition, and the deadlock will remain until the transaction either times out, is manually cancelled, or MarkLogic is restarted. Note that if you take out the first call to doc("/docs/test.xml") in line 2 of the above example, the statement will not deadlock because the read lock on /docs/test.xml is not called until after the xdmp:eval statement completes.

Seeing Updates From eval/invoke Later in the Transaction

If you are sure that your update statement in an eval/invoke operation does not try to update any documents that are referenced earlier in the calling statement (and therefore does not result in a deadlock condition, as described in Preventing Deadlocks), then you can set up your statement so updates from an eval/invoke are visible from the calling transaction. This is most useful in transactions that have the eval/invoke statement before the code that accesses the newly updated documents.

If you want to see the updates from an eval/invoke operation later in your statement, the transaction must be an update transaction. If the transaction is a query transaction, it runs in timestamp mode and will always see the version of the document that existing before the eval/invoke operation committed.

For example, consider the following example, where doc("/docs/test.xml") returns <a>hello</a> before the transaction begins:

(: doc("/docs/test.xml") returns <a>hello</a> before running this :)
(: the next line ensures this runs as an update statement :)
if ( 1 = 2 ) then ( xdmp:document-insert("fake.xml", <a/>) ) else (),
xdmp:eval("xdmp:node-replace(doc('/docs/test.xml')/node(), <b>goodbye</b>)", (),
          <options xmlns="xdmp:eval">
            <isolation>different-transaction</isolation>
            <prevent-deadlocks>false</prevent-deadlocks>
          </options>) ,
doc("/docs/test.xml")

The call to doc("/docs/test.xml") in the last line of the example returns <a>goodbye</a>, which is the new version that was updated by the xdmp:eval operation.

You can often solve the same problem by using multi-statement transactions. In a multi-statement transaction, updates made by one statement are visible to subsequent statements in the same transaction. Consider the above example, rewritten as a multi-statement transaction. Setting the transaction mode to update removes the need for fake code to force classification of statements as updates, but adds a requirement to call xdmp:commit to make the updates visible in the database.

declare option xdmp:transaction-mode "update";

(: doc("/docs/test.xml") returns <a>hello</a> before running this :)
xdmp:eval("xdmp:node-replace(doc('/docs/test.xml')/node(), <b>goodbye</b>)", (),
          <options xmlns="xdmp:eval">
            <isolation>different-transaction</isolation>
            <prevent-deadlocks>false</prevent-deadlocks>
          </options>);
(: returns <a>goodbye</b> within this transaction :)
doc("/docs/test.xml"),
(: make updates visible in the database :)
xdmp:commit()

Running Multi-Statement Transactions under xdmp:eval/invoke

When you run a query using xdmp:eval or xdmp:invoke or their JavaScript counterparts with different-transaction isolation, or via xdmp:spawn or xdmp.spawn, a new transaction is created to execute the query, and that transaction runs in a newly created session. This has two important implications for multi-statement transactions evaluated with xdmp:eval or xdmp:invoke:

  • Transaction mode is not inherited from the caller.
  • Uncommitted updates are automatically rolled back when an eval/invoke'd or spawned query completes.

Therefore, when using multi-statement transactions in code evaluated under eval/invoke with different-transaction isolation or under xdmp:spawn or xdmp.spawn:

  • Set the commit option to explicit in the options node if the transaction must run as a multi-statement transaction or use the XQuery xdmp:commit prolog option or JavaScript declareUpdate function to specify explicit commit mode.
  • Always call xdmp:commit or xdmp.commit inside an eval/invoke'd multi-statement query if updates must be preserved.

Setting the commit mode in the XQuery prolog of the eval/invoke'd query is equivalent to setting it by passing an options node to xdmp:eval/invoke with commit set to explicit. Setting the mode through the options node enables you to set the commit mode without modifying the eval/invoke'd query.

For an example of using multi-statement transactions with different-transaction isolation, see Example: Multi-Statement Transactions and Different-transaction Isolation.

The same considerations apply to multi-statement queries evaluated using xdmp:spawn or xdmp.spawn.

Transactions run under same-statement isolation run in the caller's context, and so use the same transaction mode and benefit from committing the caller's transaction. For a detailed example, see Example: Multi-statement Transactions and Same-statement Isolation.

Functions With Non-Transactional Side Effects

Update transactions use various update built-in functions which, at the time the transaction commits, update documents in a database. These updates are technically known as side effects, because they cause a change to happen outside of what the statements in the transaction return. The side effects from the update built-in functions (xdmp:node-replace, xdmp:document-insert, and so on) are transactional in nature; that is, they either complete fully or are rolled back to the state at the beginning of the update statement.

Some functionsevaluate asynchronously as soon as they are called, whether called from an update transaction or a query transaction. These functions have side effects outside the scope of the calling statement or the containing transaction (non-transactional side effects). The following are some examples of functions that can have non-transactional side effects:

When evaluating a module that performs an update transaction, it is possible for the update to either fail or retry. That is the normal, transactional behavior, and the database will always be left in a consistent state if a transaction fails or retries. However, if your update transaction calls a function with non-transactional side effects, that function evaluates even if the calling update transaction fails and rolls back.

Use care or avoid calling any of these functions from an update transaction, as they are not guaranteed to only evaluate once (or to not evaluate if the transaction rolls back). If you are logging some information with xdmp:log or xdmp.log in your transaction, it might or might not be appropriate for that logging to occur on retries (for example, if the transaction is retried because a deadlock is detected). Even if it is not what you intended, it might not do any harm.

Other side effects, however, can cause problems in updates. For example, if you use xdmp:spawn or xdmp.spawn in this context, the action might be spawned multiple times if the calling transaction retries, or the action might be spawned even if the transaction fails; the spawn call evaluates asynchronously as soon as it is called. Similarly, if you are calling a web service with xdmp:http-get or xdmp.httpGet from an update transaction, it might evaluate when you did not mean for it to evaluate.

If you do use these functions in updates, your application logic must handle the side effects appropriately. These types of use cases are usually better suited to triggers and the Content Processing Framework. For details, see Using Triggers to Spawn Actions and the Content Processing Framework Guide manual.

Reducing Blocking with Multi-Version Concurrency Control

You can set the multi-version concurrency control App Server configuration parameter to nonblocking to minimize transaction blocking, at the cost of queries potentially seeing a less timely view of the database. This option controls how the timestamp is chosen for lock-free queries. For details on how timestamps affect queries, see Query Transactions: Point-in-Time Evaluation.

Nonblocking mode can be useful for your application if:

  • Low query latency is more important than update latency.
  • Your application participates in XA transactions. XA transactions can involve multiple participants and non-MarkLogic Server resources, so they can take longer than usual.
  • Your application accesses a replica database which is expected to significantly lag the master. For example, if the master becomes unreachable for some time.

The default multi-version concurrency control is contemporaneous. In this mode, MarkLogic Server chooses the most recent timestamp for which any transaction is known to have committed, even if other transactions have not yet fully committed for that timestamp. Queries can block waiting for the contemporaneous transactions to fully commit, but the queries will see the most timely results. The block time is determined by the slowest contemporaneous transaction.

In nonblocking mode, the server chooses the latest timestamp for which all transactions are known to have committed, even if there is a slightly later timestamp for which another transaction has committed. In this mode, queries do not block waiting for contemporaneous transactions, but they might not see the most up to date results.

You can run App Servers with different multi-version concurrency control settings against the same database.

Administering Transactions

The MarkLogic Server XQuery API include built-in functions helpful for debugging, monitoring, and administering transactions.

Use xdmp:host-status to get information about running transactions. The status information includes a <transactions> element containing detailed information about every running transaction on the host. For example:

<transactions xmlns="http://marklogic.com/xdmp/status/host">
  <transaction>
    <transaction-id>10030469206159559155</transaction-id>
    <host-id>8714831694278508064</host-id>
    <server-id>4212772872039365946</server-id>
    <name/>
    <mode>query</mode>
    <timestamp>11104</timestamp>
    <state>active</state>
    <database>10828608339032734479</database>
    <canceled>false</canceled>
    <start-time>2011-05-03T09:14:11-07:00</start-time>
    <time-limit>600</time-limit>
    <max-time-limit>3600</max-time-limit>
    <user>15301418647844759556</user>
    <admin>true</admin>
  </transaction>
  ...
</transactions>

In a clustered installation, transactions might run on remote hosts. If a remote transaction does not terminate normally, it can be committed or rolled back remotely using xdmp:transaction-commit or xdmp:transaction-rollback. These functions are equivalent to calling xdmp:commit and xdmp:rollback when xdmp:hostis passed as the host id parameter. You can also rollback a transaction through the Host Status page of the Admin Interface. For details, see Rolling Back a Transaction in the Administrator's Guide.

Though a call to xdmp:transaction-commit returns immediately, the commit only occurs after the currently executing statement in the target transaction successfully completes. Calling xdmp:transaction-rollback immediately interrupts the currently executing statement in the target transaction and terminates the transaction.

For an example of using these features, see Example: Generating a Transaction Report With xdmp:host-status. For details on the built-ins, see the XQuery & XSLT API Reference.

Transaction Examples

This section includes the following examples:

Example: Multi-statement Transactions and Same-statement Isolation

The following example demonstrates the interactions between multi-statement transactions and same-statement isolation, discussed in Interactions with xdmp:eval/invoke.

The goal of the sample is to insert a document in the database using xdmp:eval, and then examine and modify the results in the calling module. The inserted document will be visible to the calling module immediately, but not visible outside the module until transaction completion.

xquery version "1.0-ml";
declare option xdmp:transaction-mode "update";

(: insert a document in the database :)
let $query :=
  'xquery version "1.0-ml";
   xdmp:document-insert("/examples/mst.xml", <myData/>)
  '
return xdmp:eval(
  $query, (),
  <options xmlns="xdmp:eval">
    <isolation>same-statement</isolation>
  </options>);

(: demonstrate that it is visible to this transaction :)
if (fn:empty(fn:doc("/examples/mst.xml")//myData))
then ("NOT VISIBLE")
else ("VISIBLE");

(: modify the contents before making it visible in the database :)
xdmp:node-insert-child(doc('/examples/mst.xml')/myData, <child/>),
xdmp:commit()

(: result: VISIBLE :)

The same operation (inserting and then modifying a document before making it visible in the database) cannot be performed as readily using the default transaction model. If the module attempts the document insert and child insert in the same single-statement transaction, an XDMP-CONFLICTINGUPDATES error occurs. Performing these two operations in different single-statement transactions makes the inserted document immediately visible in the database, prior to inserting the child node. Attempting to perform the child insert using a pre-commit trigger creates a trigger storm, as described in Avoiding Infinite Trigger Loops (Trigger Storms).

The eval'd query runs as part of the calling module's multi-statement update transaction since the eval uses same-statement isolation. Since transaction mode is not inherited by transactions created in a different context, using different-transaction isolation would evaluate the eval'd query as a single-statement transaction, causing the document to be immediately visible to other transactions.

The call to xdmp:commit is required to preserve the updates performed by the module. If xdmp:commit is omitted, all updates are lost when evaluation reaches the end of the module. In this example, the commit must happen in the calling module, not in the eval'd query. If the xdmp:commit occurs in the eval'd query, the transaction completes when the statement containing the xdmp:eval call completes, making the document visible in the database prior to inserting the child node.

Example: Multi-Statement Transactions and Different-transaction Isolation

The following example demonstrates how different-transaction isolation interacts with transaction mode for multi-statement transactions. The same interactions apply to queries executed with xdmp:spawn. For more background, see Transaction Mode and Interactions with xdmp:eval/invoke.

In this example, xdmp:eval is used to create a new transaction that inserts a document whose content includes the current transaction id using xdmp:transaction. The calling query prints its own transaction id and the transaction id from the eval'd query.

xquery version "1.0-ml";

(: init to clean state; runs as single-statement txn :)
xdmp:document-delete("/docs/mst.xml");

(: switch to multi-statement transactions :)
declare option xdmp:transaction-mode "query";

let $sub-query :=
  'xquery version "1.0-ml";
   declare option xdmp:transaction-mode "update";       (: 1 :)
   xdmp:document-insert("/docs/mst.xml", <myData/>);
   
   xdmp:node-insert-child(
     fn:doc("/docs/mst.xml")/myData,
     <child>{xdmp:transaction()}</child>
   );
   xdmp:commit()                                        (: 2 :)
  '
return xdmp:eval($sub-query, (),
  <options xmlns="xdmp:eval">
    <isolation>different-transaction</isolation>
  </options>);
  
(: commit to end this transaction and get a new system
 : timestamp so the updates by the eval'd query are visible. :)
xdmp:commit();                                                (: 3 :)
(: print out my transaction id and the eval'd query transaction id :)
fn:concat("My txn id: ", xdmp:transaction()),           (: 4 :)
fn:concat("Subquery txn id: ", fn:doc("/docs/mst.xml")//child)

Setting the transaction mode in statement (: 1 :) is required because different-transaction isolation makes the eval'd query a new transaction, running in its own session, so it does not inherit the transaction mode of the calling context. Omitting xdmp:transaction-mode in the eval'd query causes the eval'd query to run in the default, auto, transaction mode.

The call to xdmp:commit at statement (: 2 :)is similarly required due to different-transaction isolation. The new transaction and the containing session end when the end of the eval'd query is reached. Changes are implicitly rolled back if a transaction or the containing session end without committing.

The xdmp:commit call at statement (: 3 :) ends the multi-statement query transaction that called xdmp:eval and starts a new transaction for printing out the results. This causes the final transaction at statement (: 4 :) to run at a new timestamp, so it sees the document inserted by xdmp:eval. Since the system timestamp is fixed at the beginning of the transaction, omitting this commit means the inserted document is not visible. For more details, see Query Transactions: Point-in-Time Evaluation.

If the query calling xdmp:eval is an update transaction instead of a query transaction, the xdmp:commit at statement (: 3 :) can be omitted. An update transaction sees the latest version of a document at the time the document is first accessed by the transaction. Since the example document is not accessed until after the xdmp:eval call, running the example as an update transaction sees the updates from the eval'd query. For more details, see Update Transactions: Readers/Writers Locks.

Example: Generating a Transaction Report With xdmp:host-status

Use the built-in xdmp:host-status to generate a list of the transactions running on a host, similar to the information available through the Host Status page of the Admin Interface.

This example generates a simple HTML report of the duration of all transactions on the local host:

xquery version "1.0-ml";

declare namespace html = "http://www.w3.org/1999/xhtml";
declare namespace hs="http://marklogic.com/xdmp/status/host";

<html>
  <body>
    <h2>Running Transaction Report for {xdmp:host-name()}</h2>
    <table border="1" cellpadding="5">
    <tr>
      <th>Transaction Id</th>
      <th>Database</th><th>State</th>
      <th>Duration</th>
    </tr>
    {
      let $txns:= xdmp:host-status(xdmp:host())//hs:transaction
      let $now := fn:current-dateTime()
      for $t in $txns
      return
        <tr>
          <td>{$t/hs:transaction-id}</td>
          <td>{xdmp:database-name($t/hs:database-id)}</td>
          <td>{$t/hs:transaction-state}</td>
          <td>{$now - $t/hs:start-time}</td>
        </tr>
    }
    </table>
  </body>
</html>

If you paste the above query into Query Console and run it with HTML output, the query generates a report similar to the following:

Many details about each transaction are available in the xdmp:host-status report. For more information, see xdmp:host-status in the XQuery & XSLT API Reference.

If we assume the first transaction in the report represents a deadlock, we can manually cancel it by calling xdmp:transaction-rollback and supplying the transaction id. For example:

xquery version "1.0-ml";
xdmp:transaction-rollback(xdmp:host(), 6335215186646946533)

You can also rollback transactions from the Host Status page of the Admin Interface.

« Previous chapter
Next chapter »