MarkLogic Server 11.0 Product Documentation

XCC Developer's Guide — Chapter 2

Required Libraries

The XCC interfaces include an integration with Jackson for manipulating JSON data in Java. To use XCC methods such as JsonItem.asJsonNode or the ContentFactory.newJsonContent overload that accepts a JsonNode, you must have an installation of Jackson and put the Jackson jar files on your classpath.

Exactly which libraries you need to add to your classpath depends on the Jackson features you use, but you will probably need at least the Jackson core libraries, available from http://github.com/FasterXML/jackson.

For example, you might need to add the following libraries to your classpath:

• jackson-core-version.jar
• jackson-annotations-version.jar
• jackson-databind-version.jar

For information on version restrictions, see XML Contentbase Connector for Java (XCC/J) Requirements.

Inserting JSON Documents

You can JSON data into the database the same way you insert other data. If the document URI extension is mapped to the JSON document format in the MarkLogic Server MIME type mappings, then a JSON document is automatically created.

For example, the following code snippet reads JSON data from a file and inserts it into the database as a JSON document.

ContentSource cs = ContentSourceFactory.newContentSource(SERVER_URI);
Session session = cs.newSession();

File inputFile = new File("data.json");
String uri = "/xcc/fromFile.json";
Content content = ContentFactory.newContent(uri, inputFile, null);
session.insertContent(content);

If there is no URI extension or you use an extension that is not mapped to JSON, you can explicitly specify JSON using ContentCreateOptions.setFormat. For example:

ContentCreateOptions options = new ContentCreateOptions();
options.setFormat(DocumentFormat.JSON);
Content content = ContentFactory.newContent(uri, inputFile, options);

The following code snippet inserts a JSON document into the database using an in-memory String representation of the contents:

ContentSource cs = ContentSourceFactory.newContentSource(SERVER_URI);
Session session = cs.newSession();

String uri = "/xcc/fromString.json";
String data = "{\"num\":1, \"arr\":[0, 1], \"str\":\"value\"}";

ContentCreateOptions options = ContentCreateOptions.newJsonInstance();
Content content = ContentFactory.newContent(uri, data, options);

session.insertContent(content);

You can also use Jackson to build up JSON content, and then pass a Jackson JsonNode in to ContentFactory.newJsonContent. To learn more about Jackson, see http://github.com/FasterXML/jackson-docs.

Processing JSON Query Results

If you run an ad hoc query that returns JSON (or a JavaScript object or array), you can use Jackson to traverse and manipulate the data in your Java application.

For example, the following code snippet evaluates an ad hoc Server-Side JavaScript query that retrieves a JSON document from the database, and then accesses the value in the document's num property as an integer:

Session session = cs.newSession();
AdhocQuery request = 
    session.newAdhocQuery("cts.doc('/xcc/fromString.json')");
RequestOptions options = new RequestOptions();
options.setQueryLanguage("javascript");
request.setOptions(options);

ResultSequence rs = session.submitRequest(request);
while (rs.hasNext()) {
    XdmItem item = rs.next().getItem();
    if (item instanceof JsonItem) {
        JsonItem jsonItem = (JsonItem) item;
        JsonNode node = jsonItem.asJsonNode();
        // process the value...
    }
}

You can use JsonItem.asJsonNode to convert a JSON result item into a Jackson JsonNode (com.fasterxml.jackson.databind.JsonNode). For example:

JsonItem jsonItem = (JsonItem) item;
JsonNode node = jsonItem.asJsonNode();

You can also use the Jackson interfaces to manipulate native JavaScript objects and arrays returned by ad hoc Server-Side JavaScript queries. That is, the above conversion to a JsonNode works whether the item is a JSON node or an atomic JS_OBJECT result.

Then you can use any of the Jackson interfaces to manipulate the contents. For example, the following code snippet accesses the value of the num JSON property as an integer:

node.get("num").asInt()

To learn more about Jackson, see http://github.com/FasterXML/jackson-docs.

Creating a Trust Manager

This section describes how to use a simple Trust Manager for X.509-based authentication. The Trust Manager shown here does not validate certificate chains and is therefore unsafe and should not be used for production code. See your Java documentation for details on how to create a more robust Trust Manager for your specific application or how to obtain a Certificate Authority from a keystore.

To enable SSL access using a trust manager, import the following classes in addition to those described in Coding Basics:

import javax.net.ssl.SSLContext;
import com.marklogic.xcc.SecurityOptions;
import javax.net.ssl.TrustManager;
import javax.net.ssl.X509TrustManager;
import java.security.KeyManagementException;
import java.security.cert.X509Certificate;
import java.security.cert.CertificateException;

Create a trust manager and pass it to the SSLContext.init() method:

protected SecurityOptions newTrustOptions()
  throws Exception
{
   TrustManager[] trust = new TrustManager[] { new X509TrustManager() {
    public void checkClientTrusted(
       X509Certificate[] x509Certificates, 
       String s)
          throws CertificateException {
     // nothing to do
    }
    public void checkServerTrusted(
       X509Certificate[] x509Certificates, 
       String s)
          throws CertificateException {
     // nothing to do
    }
    public X509Certificate[] getAcceptedIssuers() {
     return null;
    }
  }
};
  SSLContext sslContext = SSLContext.getInstance("SSLv3");
  sslContext.init(null, trust, null);
  return new SecurityOptions(sslContext);
}

Call ContentSourceFactory.newContentSource() with a host name, port, user name, password, and SSL security options defined by newTrustOptions():

ContentSource cs = 
    ContentSourceFactory.newContentSource (host,
                                           port,
                                           username,
                                           password,
                                           null,
                                           newTrustOptions());

If you are passing a URI to ContentSourceFactory.newContentSource(), specify a connection scheme of xccs, rather than xcc., as shown in Accessing a Keystore.

Accessing a Keystore

You can use the Java keytool utility to import a MarkLogic certificate into a keystore. See the Java JSSE documentation for details on the use of the keytool and your keystore options.

You can explicitly specify a keystore, as shown in this example, or you can specify a null keystore. Specifying a null keystore causes the TrustManagerFactory to locate your default keystore, as described in the Java Secure Socket Extension (JSSE) Reference Guide.

To enable SSL by accessing certificates in a keystore, import the following classes in addition to those described in Coding Basics:

import com.marklogic.xcc.SecurityOptions;
import com.marklogic.xcc.ContentSource;
import com.marklogic.xcc.ContentSourceFactory;
import java.io.FileInputStream;
import java.net.URI;
import javax.net.ssl.KeyManager;
import javax.net.ssl.KeyManagerFactory;
import javax.net.ssl.TrustManager;
import javax.net.ssl.TrustManagerFactory;
import javax.net.ssl.X509TrustManager;
import javax.net.ssl.SSLContext;
import java.security.KeyStore;
import java.security.cert.X509Certificate;

Get the signed certificate from a keystore and pass it to the SSLContext.init() method:

protected SecurityOptions newTrustOptions()
  throws Exception
{
// Load key store with trusted signing authorities.
  KeyStore trustedKeyStore = KeyStore.getInstance("JKS");
  trustedKeyStore.load(
    new FileInputStream("C:/users/myname/.keystore"), 
    null);
// Build trust manager to validate server certificates using the
   specified key store.
  TrustManagerFactory trustManagerFactory =
    TrustManagerFactory.getInstance("SunX509");
  trustManagerFactory.init(trustedKeyStore);
  TrustManager[] trust = trustManagerFactory.getTrustManagers();
  SSLContext sslContext = SSLContext.getInstance("SSLv3");
  sslContext.init(null, trust, null);
  return new SecurityOptions(sslContext);
}

Call ContentSourceFactory.newContentSource() with a URI:

ContentSource cs = 
    ContentSourceFactory.newContentSource (uri,
                                           newTrustOptions());

The URI is passed from the command line in the form of:

xccs://username:password@hostname:port

Managing Client Side Authentication

You can define a KeyManager, if your client application is required to send authentication credentials to the server. The following example adds client authentication to the newTrustOptions method shown in Accessing a Keystore:

protected SecurityOptions newTrustOptions()
  throws Exception
{
// Load key store with trusted signing authorities.
  KeyStore trustedKeyStore = KeyStore.getInstance("JKS");
  trustedKeyStore.load(
    new FileInputStream("C:/users/myname/.keystore"), 
    null);
// Build trust manager to validate server certificates using the
   specified key store.
  TrustManagerFactory trustManagerFactory =
    TrustManagerFactory.getInstance("SunX509");
  trustManagerFactory.init(trustedKeyStore);
  TrustManager[] trust = trustManagerFactory.getTrustManagers();
// Load key store with client certificates.
  KeyStore clientKeyStore = KeyStore.getInstance("JKS");
  clientKeyStore.load(
    new FileInputStream("C:/users/myname/.keystore"),
    null);
// Get key manager to provide client credentials.
  KeyManagerFactory keyManagerFactory =
    KeyManagerFactory.getInstance("SunX509");
  keyManagerFactory.init(clientKeyStore, passphrase);
  KeyManager[] key = keyManagerFactory.getKeyManagers();
// Initialize the SSL context with key and trust managers.
  SSLContext sslContext = SSLContext.getInstance("SSLv3");
  sslContext.init(key, trust, null);
  return new SecurityOptions(sslContext);
}

Overview

Use the following procedure to use multi-statement, explicitly committed transactions with XCC:

1. Create a Session object in the usual way.
2. Call Session.setAutoCommit with a value of false. The next transaction created in the session will run as a multi-statement, explicit commit transaction.
3. Optionally, call Session.setUpdate to specify an explicit transaction type. By default, MarkLogic determines the transaction type through static analysis of the first statement in a request, but you can explicitly set the transaction type to update or query using Session.setUpdate.
4. Call Session.submitRequest as usual to operate on your data. All requests run in the same transaction until the transaction is committed or rolled back.
5. Call Session.commit or Session.rollback to commit or rollback the transaction. If the session ends or times out without explicitly commit or rolling back, the transaction is rolled back.
6. To restore a session to the default, single-statement transaction model, call Session.setAutoCommit with a value of true and Session.setUpdate with a value of AUTO.

Note that the transaction configuration defined by setAutoCommit and setUpdate remain in effect for all transactions created by a session until explicitly changed. If you override the transaction configuration in an ad hoc query, the override applies only to the current transaction.

Multi-statement query transactions allow all the statements in a transaction to share the same point-in-time view of the database, as discussed in Point-In-Time Queries.

In a multi-statement update transaction, updates performed by one statement (or request) are visible to subsequent statements in the same transaction, without being visible to other transactions.

A multi-statement transaction remains open until it is committed or rolled back. Use Session.commit. to commit a multi-statement transaction and make the changes visible in the database. Use Session.rollback to roll back a multi-statement transaction, discarding any updates. Multi-statement transactions are implicitly rolled back when the containing session ends or the transaction times out. Failure to explicitly commit or rollback a multi-statement update transaction can tie up resources, hold locks unnecessarily, and increase the chances of deadlock.

You may receive a java.lang.IllegalStateException if you call Session.commit from an exception handler when there are no pending updates in the current transaction. Committing from a handler is not recommended.

For a detailed discussion of multi-statement transactions, see the Understanding Transactions chapter of the Application Developer's Guide<Default ° Font>.

Multi-statement transactions impose special re-try semantics on XCC applications. For details, see Retrying Multi-statement Transactions.

Example: Using Multi-Statement Transactions in Java

The following example demonstrates using multi-statement transactions in Java. The first multi-statement transaction in the session inserts two documents into the database, calling Session.commit to complete the transaction and commit the updates. The second transaction demonstrates the use of Session.rollback. The third transaction demonstrates implicitly rolling back updates by closing the session.

import java.net.URI;

import com.marklogic.xcc.ContentSource;
import com.marklogic.xcc.ContentSourceFactory;
import com.marklogic.xcc.Session;

public class SimpleMST {
  public static void main(String[] args) throws Exception {
    if (args.length != 1) {
      System.err.println("usage: xcc://user:password@host:port/contentbase");
      return;
    }

    // Obtain a ContentSource object for the server at the URI.
    URI uri = new URI(args[0]);
    ContentSource contentSource =
        ContentSourceFactory.newContentSource(uri);

    // Create a Session and set the transaction mode to trigger
    // multi-statement transaction use.
    Session updateSession = contentSource.newSession();
    updateSession.setAutoCommit(false);
    updateSession.setUpdate(Session.Update.TRUE);

    // The request starts a new, multi-statement transaction. 
    updateSession.submitRequest(updateSession.newAdhocQuery(
    "xdmp:document-insert('/docs/mst1.xml', <data/>)"));

    // This request executes in the same transaction as the previous
    // request and sees the results of the previous update.
    updateSession.submitRequest(updateSession.newAdhocQuery(
    "xdmp:document-insert('/docs/mst2.xml', fn:doc('/docs/mst1.xml'));"));

    // After commit, updates are visible to other transactions.
    // Commit ends the transaction after current stmt completes.
    updateSession.commit();                    // txn ends, updates kept

    // Rollback discards changes and ends the transaction. 
    updateSession.submitRequest(updateSession.newAdhocQuery(
    "xdmp:document-delete('/docs/mst1.xml')"));
    updateSession.rollback();                // txn ends, updates lost

    // Closing session without calling commit causes a rollback.
    updateSession.submitRequest(updateSession.newAdhocQuery(
    "xdmp:document-delete('/docs/mst1.xml')"));
    updateSession.close();                    // txn ends, updates lost
  }
}

Terminating a Transaction in an Exception Handler

Calling Session.commit from an exception handler that wraps a request participating in a multi-statement transaction may raise java.lang.IllegalStateException. You may always safely call Session.rollback from such a handler.

Usually, an exception raised during multi-statement transaction processing leaves the Session open, allowing you to continue working in the transaction after handling the exception. However, in order to preserve consistency, exceptions occurring under the following circumstances always roll back the transaction:

• After an XQuery statement has finished but before the XCC request is completed
• In the middle of an explicit commit or rollback

If such a rollback occurs, the current transaction is terminated before control reaches your exception handler. Calling Session.commit when there is no active transaction raises a java.lang.IllegalStateException. Calling Session.rollback when there is no active transaction does not raise an exception, so rollback from a handler is always safe.

Therefore, it is usually only safe to call Session.commit from an exception handler for specific errors you expect to receive and for which you can predict the state of the transaction.

Retrying Multi-statement Transactions

MarkLogic Server sometimes detects the need to retry a transaction. For example, if the server detects a deadlock, it may cancel one of the deadlocked transactions, allowing the other to complete; the cancelled transaction should be re-tried.

With single-statement, auto-commit transactions, the server can usually retry automatically because it has the entire transaction available at the point of detection. However, the statements in a multi-statement transactions from XCC clients may be interleaved with arbitrary application-specific code of which the server has no knowledge.

In such cases, instead of automatically retrying, the server throws a RetryableXQueryException. The calling application is then responsible for re-trying all the requests in the transaction up to that point. This exception is more likely to occur when using multi-statement transactions.

The following example demonstrates logic for re-trying a multi-statement transaction. The multi-statement transaction code is wrapped in a retry loop with an exception handler that waits between retry attempts. The number of retries and the time between attempts is up to the application.

import java.net.URI;

import com.marklogic.xcc.ContentSource;
import com.marklogic.xcc.ContentSourceFactory;
import com.marklogic.xcc.Session;
import com.marklogic.xcc.exceptions.RetryableXQueryException;

public class TransactionRetry {
  public static final int MAX_RETRY_ATTEMPTS = 5;
  public static final int RETRY_WAIT_TIME = 1;
  
  public static void main(String[] args) throws Exception {
    if (args.length != 1) {
      System.err.println("usage: xcc://user:password@host:port/contentbase");
      return;
    }

    // Obtain a ContentSource object for the server at the URI.
    URI uri = new URI(args[0]);
    ContentSource contentSource =
        ContentSourceFactory.newContentSource(uri);

    // Create a Session and set the transaction mode to trigger
    // multi-statement transaction use.
    Session session = contentSource.newSession();
    Session.setAutoCommit(false);
    Session.setUpdate(Session.Update.TRUE);

    // Re-try logic for a multi-statement transaction
    for (int i = 0; i < MAX_RETRY_ATTEMPTS; i++) {
      try {
        session.submitRequest(session.newAdhocQuery(
          "xdmp:document-insert('/docs/mst1.xml', <data/>)"));
        session.submitRequest(session.newAdhocQuery(
          "xdmp:document-insert('/docs/mst2.xml', fn:doc('/docs/mst1.xml'));"));
        session.commit();
        break;
      } catch (RetryableXQueryException e) {
        Thread.sleep(RETRY_WAIT_TIME);
      }
    }
    session.close(); 
  }
}

Using Multi-Statement Transactions With Older MarkLogic Versions

If you use multi-statement transactions or set the transaction time limit when using XCC version 8.0-2 or later with versions of MarkLogic Server older than 8.0-2, you should set the system property xcc.txn.compatible to true. If you do not set this, then you will get an exception when trying to set the transaction mode or transaction time limit.

You can set the property on the java command line with an argument of the following form:

java -Dxcc.txn.compatible=true

You can also set the property programmatically by calling System.setProperty.

You do not need to set the property if your XCC application does not use multi-statement transactions or if your application communicates with MarkLogic Server version 8.0-2 or later.

Overview

XA is a standard for distributed transaction processing defined by The Open Group. For details about XA, see:

https://www2.opengroup.org/ogsys/jsp/publications/PublicationDetails.jsp?catalogno=c193

JTA is the Java Transaction API, a Java standard which supports distributed transactions across XA resources. For details about JTA, see http://java.sun.com/products/jta/.

The XCC API includes support for registering MarkLogic Server as a resource with an XA Transaction Manager. For details, see Enlisting MarkLogic Server in an XA Transaction.

The basic XA architecture as it applies to MarkLogic Server is shown in the following diagram:

Predefined Security Roles for XA Participation

The following security roles are predefined for participating in and administering XA transactions:

• The xa user role allows creation and management of one's own XA transaction branches in MarkLogic Server.
• The xa-admin role allows creation and management of any user's XA transaction branches in MarkLogic Server.

The xa role is required to participate in XA transactions. The xa-admin role is intended primarily for Administrators who need to complete or forget XA transactions; see Heuristically Completing a Stalled Transaction.

Enlisting MarkLogic Server in an XA Transaction

To use MarkLogic Server in an XA transaction, use the Session.getXAResource method to register your XCC session as a javax.transaction.xa.XAResource with the XA Transaction Manager.

The following code snippet shows how to enlist an XCC session in a global XA transaction. Once you enlist the Session, any work performed in the session is part of the global transaction. For complete code, see the sample code in com.marklogic.xcc.examples.XA.

javax.transaction.TransactionManager tm = ...;
Session session = ...;

try {
    // Begin a distributed transaction
    tm.begin();

    // Add the MarkLogic Session to the distributed transaction
    javax.transaction.xa.XAResource xaRes = session.getXAResource();
    tm.getTransaction().enlistResource(xaRes);

    // Perform MarkLogic Server updates under the global transaction
    session.submitRequest(session.newAdhodquery(
        "xdmp:document-insert('a', <a/>)"));

    // Update other databases here

    //Commit all updates together
    tm.commit();
} catch (Exception e) {
    e.printStackTrace();
    if (tm. getTransaction != null) tm.rollback();
} finally {
    session.close();
}

When MarkLogic Server acts as an XA transaction Resource Manager, requests submitted to the server are always part of a multi-statement update transaction, with the following important differences:

• The Session.setAutoCommit and Session.setTransactionMode settings are ignored. The transaction is always a multi-statement update transaction, even if only a single request is submitted to MarkLogic Server during the global transaction.
• The application should not call Session.commit or xdmp:commit. The transaction is committed (or rolled back) as part of the global XA transaction. To commit the global transaction, use the Transaction Manager with which the MarkLogic Server XAResource is registered.
• The application may call Session.rollback or xdmp:rollback. Doing so eventually causes rollback of the global XA transaction. Rolling back via the Transaction Manager is usually preferable.

To learn more about multi-statement transactions, see Multi-Statement Transactions.

Heuristically Completing a Stalled Transaction

Under extreme circumstances, a MarkLogic Server administrator may need to heuristically complete (intervene to manually commit or rollback) the MarkLogic Server portion of a prepared XA transaction. This section covers the following topics related to heuristic completion:

• Understanding Heuristic Completion
• Heuristically Completing a MarkLogic Server Transaction
• Cleaning Up After Heuristic Completion

(: commit and remember the transaction outcome:)
xdmp:xa-complete($forest-id, $txn-id, fn:true(), fn:true())

(: roll back and remember the transaction outcome:)
xdmp:xa-complete($forest-id, $txn-id, fn:false(), fn:true())

(: commit and forget the transaction outcome :)
xdmp:xa-complete($forest-id, $txn-id, fn:true(), fn:false())

xquery version "1.0-ml";
for $f in xdmp:database-forests(xdmp:database())
return xdmp:forest-status($f)//*:transaction-participants

Reducing Blocking Caused by Slow XA Transactions

Since XA transactions may involve multiple participants and non-MarkLogic Server resources, they may take longer than usual. A slow XA transaction may cause other queries on the same App Server to block for an unacceptably long time.

You may set the multi-version concurrency control App Server configuration parameter to nonblocking to minimize blocking, at the cost of less timely results. For details, see Reducing Blocking with Multi-Version Concurrency Control in the Application Developer's Guide.

Enable HTTP 1.1 Compliance

Enabling HTTP compliant mode guarantees the traffic between your XDBC App Server and your XCC client is compliant with the HTTP 1.1 protocol. This enables properly configured load balancers to detect the SessionID cookie generated by MarkLogic Server and use it to enforce session affinity.

To enable this mode for a Java application, set the xcc.httpcompliant system property to true on the Java command line. For example:

java -Dxcc.httpcompliant=true ...

Setting xcc.httpcompliant to true is incompatible with enabling content entity resolution using ContentCreateOptions.setResolveEntities.

If xcc.httpcompliant is not set explicitly, then xcc.httpcompliant is false.

You must also configure your load balancer to use the value in the SessionID cookie for session affinity. Some routers or load balancers may need to have xcc.httpcompliant enabled to allow any traffic through, regardless of session affinity issues.

Configure the Load Balancer

In addition to setting xcc.httpcompliant to true, you must configure your load balancer to use the SessionID cookie generated by MarkLogic Server for session affinity. You might also need to enable session affinity or sticky sessions in your load balancer. The exact configuration steps depend on the load balancer; see your load balancer documentation for details.

Your load balancer must be HTTP 1.1 compliant and support cookie-based session affinity to use this feature of XCC.

A SessionID cookie looks similar to the following:

SessionID=25b877c32807aa9f