MarkLogic Server 11.0 Product Documentation

Node.js Application Developer's Guide — Chapter 2

Overview

Use DatabaseClient.documents.write to insert or update whole documents and/or metadata. To update only a portion of a document or its metadata, use DatabaseClient.documents.patch; for details, see Patching Document Content or Metadata.

The primary input to the write function is one or more document descriptors. Each descriptor encapsulates a document URI with the content and/or metadata to be written. For details, see Input Document Descriptors.

For example, the following call writes a single document with the URI /doc/example.json:

const db = marklogic.createDatabaseClient(...);
db.documents.write(
  { uri: '/doc/example.json',
    contentType: 'application/json',
    content: { some: 'data' }
  }
);

Write multiple documents by passing in multiple descriptors. For example, the following call writes 2 documents:

db.documents.write(
    { uri: '/doc/example1.json',
      contentType: 'application/json',
      content: { data: 'one' }
    },
    { uri: '/doc/example2.json',
      contentType: 'application/json',
      content: { data: 'two' }
    },
);

Descriptors can be passed as individual parameters, in an array, or in an encapsulating call object. For details, see Calling Convention.

You can take action based on the success or failure of a write operation by calling the result() function on the return value. For details, see Supported Result Handling Techniques.

For example, the following code snippet prints an error message to the console if the write fails:

db.documents.write(
  { uri: '/doc/example.json',
    contentType: 'application/json',
    content: { some: 'data' }
  }
).result(null, function(error) {
    console.log(
})

Input Document Descriptors

Each document to be written is described by a document descriptor. The document descriptor must include a URI and either content, metadata, or both content and metadata. For details, see Document Descriptor.

For example, the following is a document descriptor for a document with URI /doc/example.json. The document contents are expressed as a JavaScript object containing a single property.

{ uri: '/doc/example.json', content: {'key': 'value'} }

The content property in a document descriptor can be an object, a string, a Buffer, or a ReadableStream.

Metadata is expressed as properties of a document descriptor when it applies to a specific document. Metadata is expressed as properties of a call object when it applies to multiple documents; for details, see Inserting or Updating Metadata for One Document.

For example, the following document descriptor includes collection and document quality metadata for the document with URI /doc/example.json:

{ uri: '/doc/example.json', 
  content: {'key': 'value'},
  collections: [ 'collection1', 'collection2' ],
  quality: 2
}

Calling Convention

You must pass at least one document descriptor to DatabaseClient.documents.write. You can also include additional properties such as a transform name or a transaction id. The parameters passed to documents.write can take one of the following forms:

• One or more document descriptors: db.documents.write(desc1, desc2,...).
• An array of one or more document descriptors: db.documents.write([desc1, desc2, ...]).
• A call object that encapsulates a document descriptor array and additional optional properties: db.documents.write({documents: [desc1, desc2, ...], txid: ..., ...}).

The following calls are equivalent:

// passing document descriptors as parameters
db.documents.write(
  {uri: '/doc/example1.json', content: {...}},
  {uri: '/doc/example2.json', content: {...}}
);
// passing document descriptors in an array
db.documents.write([
  {uri: '/doc/example1.json', content: {...}},
  {uri: '/doc/example1.json', content: {...}}
]);
// passing document descriptors in a call object
db.documents.write({
  documents: [
    {uri: '/doc/example1.json', content: {...}},
    {uri: '/doc/example2.json', content: {...}}
  ],
  additional optional properties
});

The additional optional properties can include a transform specification, transaction id, or temporal collection name; for details, see the Node.js API Reference. You can always specify such properties as properties of a call object.

For example, the following call includes a transaction id (txid) as an additional property of the call object:

// passing a transaction id as a call object property
db.documents.write({
  documents: [
    {uri: '/doc/example1.json', content: {...}},
    {uri: '/doc/example2.json', content: {...}}
  ],
  txid: '1234567890'
});

For convenience, if and only if there is a single document descriptor, the additional optional properties can be passed as properties of the document descriptors, as an alternative to using a call object. For example, the following call includes a transaction id inside the single document descriptor:

// passing a transaction id as a document descriptor property
db.documents.write(
  { uri: '/doc/example1.json',
    content: {...},
    txid: '1234567890'
  }
);

Example: Loading A Single Document

This example inserts a single document into the database using DatabaseClient.documents.write.

The document to load is identified by a document descriptor. The following document descriptor describes a JSON document with the URI /doc/example.json. The document content is expressed as a JavaScript object here, but it can also be a string, Buffer, or ReadableStream.

{ uri: '/doc/example.json',
  contentType: 'application/json',
  content: { some: 'data' }
})

The code below creates a database client and calls DatabaseClient.documents.write to load the document. The example checks for a write failure by calling the result function and passing in an error handler. In this example, no action is taken if the write succeeds, so null is passed as the first parameter to result().

const marklogic = require('marklogic');
const my = require('./my-connection.js');
const db = marklogic.createDatabaseClient(my.connInfo);

db.documents.write(
    { uri: '/doc/example.json',
      contentType: 'application/json',
      content: { some: 'data' }
    })
  .result(null, function(error) {
      console.log(JSON.stringify(error));
    });

For additional examples, see examples/before-load.js and examples/write-remove.js in the node-client-api source directory.

To include metadata, add metadata properties to the document descriptor. For example, to add the document to a collection, you can add a collections property to the descriptor:

db.documents.write(
    { uri: '/doc/example.json',
      contentType: 'application/json',
      content: { some: 'data' },
      collections: [ 'collection1', 'collection2' ]
    })

You can include optional additional parameters such as a transaction id or a write transform by using a call object. For details, see Calling Convention.

Example: Loading Multiple Documents

This example builds on Example: Loading A Single Document to insert multiple documents into the database using DatabaseClient.documents.write.

To insert or update multiple documents in a single request to MarkLogic Server, pass multiple document descriptors to DatabaseClient.documents.write.

The following code inserts 2 documents into the database with URIs /doc/example1.json and /doc/example2.json:

const marklogic = require('marklogic');
const my = require('./my-connection.js');
const db = marklogic.createDatabaseClient(my.connInfo);

db.documents.write(
    { uri: '/doc/example1.json',
      contentType: 'application/json',
      content: { data: 'one' }
    },
    { uri: '/doc/example2.json',
      contentType: 'application/json',
      content: { data: 'two' }
    }
).result(null, function(error) {
    console.log(JSON.stringify(error));
});

A multi-document write returns an object that contains a descriptor for each document written. The descriptor includes the URI, the MIME type the contents were interpreted as, and whether the write updated content, metadata, or both.

For example, the return value of the above call is as follows:

{ documents: [
  { uri: '/doc/example1.json',
    mime-type: 'application/json',
    category: ['metadata','content']
  },{
    uri: '/doc/example2.json',
    mime-type: 'application/json',
    category: ['metadata','content']
  }
]}

Note that the category property indicates both content and metadata were updated even though no metadata was explicitly specified. This is because system default metadata values were implicitly assigned to the documents.

To include metadata for a document when you load multiple documents, include document-specific metadata in the descriptor for that document. To specify metadata that applies to multiple documents include a metadata descriptor in the parameter list or documents property.

For example, to add the two documents to the collection examples, add a metadata descriptor before the document descriptors, as shown below. The order of the descriptors matters as the set of descriptors is processed in the order it appears. A metadata descriptor only affects document descriptors that appear after it in the parameter list or documents array.

const marklogic = require('marklogic');
const my = require('./my-connection.js');
const db = marklogic.createDatabaseClient(my.connInfo);

db.documents.write({
  documents: [
    { contentType: 'application/json',
      collections: [ 'examples' ]
    },
    { uri: '/doc/example1.json',
      contentType: 'application/json',
      content: { data: 'one' }
    },
    { uri: '/doc/example2.json',
      contentType: 'application/json',
      content: { data: 'two' }
    }
  ]
}).result(null, function(error) {
      console.log(JSON.stringify(error));
});

Inserting or Updating Metadata for One Document

To insert or update metadata for a specific document, include one or more metadata properties in the document descriptor passed to DatabaseClient.documents.write. To insert or update the same metadata for multiple documents, you can include a metadata descriptor in a multi-document write; for details, see Working with Metadata.

When setting permissions, at least one update permission must be included.

Metadata is replaced on update, not merged. For example, if your document descriptor includes a collections property, then calling DatabaseClient.documents.write replaces all existing collection associations for the document.

The following example inserts a document with URI /doc/example.json and adds it to the collections examples and metadata-examples. If the document already exists and is part of other collections, it is removed from those collections.

const marklogic = require('marklogic');
const my = require('./my-connection.js');
const db = marklogic.createDatabaseClient(my.connInfo);

db.documents.write(
    { uri: '/doc/example.json',
      collections: ['examples', 'metadata-examples'],
      contentType: 'application/json',
      content: { some: 'data' }
    })
  .result(null, function(error) {
      console.log(JSON.stringify(error));
    });

To insert or update just metadata for a document, omit the content property. For example, the following code sets the quality to 2 and the collections to some-collection, without changing the document contents:

const marklogic = require('marklogic');
const my = require('./my-connection.js');
const db = marklogic.createDatabaseClient(my.connInfo);

db.documents.write(
    { uri: '/doc/example.json',
      collections: ['some-collection'],
      quality: 2,
    })
  .result(null, function(error) {
      console.log(JSON.stringify(error));
    });

Automatically Generating Document URIs

You can have document URIs automatically generated during insertion by replacing the uri property in your document descriptor with an extension property, as described below.

You can only use this feature to create new documents. To update an existing document, you must know its URI.

To use this feature, construct a document descriptor with the following characteristics:

• Omit the uri property.
• Include an extension property that specifies the generated URI extension, such as xml or json. Do not include a dot (.) prefix. That is, specify json, not .json.
• Optionally, include a directory property that specifies a database directory prefix for the generated URI. The directory prefix must end in a forward slash (/).

The following example inserts a document into the database with a URI of the form /my/directory/auto-generated.json.

const marklogic = require('marklogic');
const my = require('./my-connection.js');
const db = marklogic.createDatabaseClient(my.connInfo);

db.documents.write(
    { extension: 'json',
      directory: '/my/directory/',
      content: { some: 'data' },
      contentType: 'application/json'
    }
).result(
    function(response) {
      console.log('Loaded ' + response.documents[0].uri);
    }, 
    function(error) {
      console.log(JSON.stringify(error));
    }
);

Running the above script results in output similar to the following upon success:

Loaded /my/directory/16764526972136717799.json

Transforming Content During Ingestion

You can transform content during ingestion by applying a custom write transform. A transform is server-side XQuery, JavaScript, or XSLT that you install in the modules database associated with your REST API instance. You can install transforms using the config.transforms functions. This topic describes how to apply a transform during ingestion. For more details and examples, see Working with Content Transformations.

To apply a transform when creating or updating documents, call documents.write with a call object that includes the transform property. The transform property encapsulates the transform name and any parameters expected by the transform. The transform property has the following form:

transform: [transformName, {param1: value, param2: value, ...}]

For example, the following code snippet applies a transform installed under the name my-transform and passes in values for 2 parameters:

db.documents.write({
  documents: [
    {uri: '/doc/example1.json', content: {...}},
    {uri: '/doc/example2.json', content: {...}}
  ],
  transform: [
    'my-transform',
    { my-first-param: 'value',
      my-second-param: 42
    }
  ]
});

As a convenience, you can embed the transform property in the document descriptor when inserting or updating just one document. For example:

db.documents.write({
  uri: '/doc/example1.json', 
  content: {...}},
  transform: [
    'my-transform',
    { my-first-param: 'value',
      my-second-param: 42
    }
  ]
});

Retrieving the Contents of a Document By URI

To retrieve the contents of a document from the database using its URI, use DatabaseClient.documents.read or DatabaseClient.read. You can also retrieve the contents and metadata of documents that match a query; for details, see Querying Documents and Metadata.

DatabaseClient.read and DatabaseClient.Documents.read both enable you to read documents by URI, but they differ in power and complexity. If you need to read metadata, use a multi-statement transaction, apply a read transformation, or access information such as the content-type, use DatabaseClient.Documents.read. DatabaseClient.read accepts only a list of URIs as input and returns just the contents of the requested documents.

The two functions return different output. DatabaseClient.Documents.read returns an array of document descriptors, instead of just the content. DatabaseClient.read returns an array containing only the content of each document, in the same order as the input URI list. You can process the output of both functions using a callback, Promise, or Stream; for details, see Supported Result Handling Techniques.

For example, the following code uses DatabaseClient.Documents.read to read the document with URI /doc/example1.json and processes the output using the Promise returned by the result method.

const marklogic = require('marklogic');
const my = require('./my-connection.js');
const db = marklogic.createDatabaseClient(my.connInfo);

db.documents.read('/doc/example1.json')
   .result( function(documents) {
      documents.forEach(function(document) {
        console.log(JSON.stringify(document));
      });
    }, function(error) {
      console.log(JSON.stringify(error, null, 2));
    });

The complete descriptor returned by result looks like the following. The returned array contains a document descriptor item for each document returned. In this case, there is only a single document.

{
  "partType":"attachment",
  "uri":"/doc/example1.json",
  "category":"content",
  "format":"json",
  "contentType":"application/json",
  "contentLength":"14",
  "content":{"data":"one"}
}

If you read the same document using DatabaseClient.read, you get the output shown below. Notice that it is just the content, not a descriptor.

db.read('/doc/example1.json').result(...);
==>
{"data":"one"}

You can read multiple documents by passing in multiple URIs. The following example reads two documents and uses the Stream pattern to process the results. The data handler receives a document descriptor as input.

const marklogic = require('marklogic');
const my = require('./my-connection.js');
const db = marklogic.createDatabaseClient(my.connInfo);

db.documents.read('/doc/example1.json', '/doc/example2.json')
  .stream().on('data', function(document) {
    console.log('Read ' + document.uri);
  }).
  on('end', function() {
    console.log('finished');
  }).
  on('error', function(error) {
    console.log(JSON.stringify(error));
    done();
  });

You can request metadata by passing a call object parameter to Documents.read and including a categories property that specifies which document parts to return. For details, see Retrieving Metadata About a Document.

When calling Documents.read, you can use a read transform to apply server-side transformations to the content before the response is constructed. For details, see Transforming Content During Retrieval.

To perform reads across multiple operations with a consistent view of the database state, pass a Timestamp object to Documents.read. For more details, see Performing Point-in-Time Operations.

Retrieving Metadata About a Document

To retrieve metadata when reading documents by URI, pass a call object to DatabaseClient.documents.read that includes a categories property. You can retrieve all metadata (category value 'metadata') or a subset of metadata (category values 'collections', 'permissions', 'properties', 'metadataValues', and 'quality'). To retrieve both content and metadata, include the category value 'content'.

The metadataValues metadata category represents simple key-value metadata, sometimes called metadata fields. For more details, see Metadata Fields in the Administrator's Guide.

For example, the following code retrieves all metadata about the document /doc/example.json, but not the contents.

const marklogic = require('marklogic');
const my = require('./my-connection.js');
const db = marklogic.createDatabaseClient(my.connInfo);

db.documents.read({
  uris: ['/doc/example.json'], 
  categories: ['metadata']
}).result(
    function(documents) {
      for (const i in documents)
        console.log('Read metadata for ' + documents[i].uri);
    }, 
    function(error) {
      console.log(JSON.stringify(error));
    }
);

The result is a document descriptor that includes all metadata properties for the requested document, as shown below. For details on metadata categories and formats, see Working with Metadata.

[{
  "partType":"attachment",
  "uri":"/doc/example.json",
  "category":"metadata",
  "format":"json",
  "contentType":"application/json",
  "contentLength":"168",
  "collections":[],
  "permissions":[
    {"role-name":"rest-writer","capabilities":["update"]},
    {"role-name":"rest-reader","capabilities":["read"]}
  ],
  "properties":{},
  "quality":0
}]

The following example retrieves content and collections metadata about 2 JSON documents:

const marklogic = require('marklogic');
const my = require('./my-connection.js');
const db = marklogic.createDatabaseClient(my.connInfo);

db.documents.read({
  uris: ['/doc/example1.json', '/doc/example2.json'], 
  categories: ['collections', 'content']
}).stream().
  on('data', function(document) {
    console.log('Collections for ' + document.uri + ': ' 
      + JSON.stringify(document.collections));
  }).
  on('end', function() {
    console.log('finished');
  }).
  on('error', function(error) {
    console.log(JSON.stringify(error));
  });

The result is a document descriptor for each document that includes a collections and a content property:

{
  "partType" : "attachment",
  "uri" : "/doc/example2.json",
  "category" : "content",
  "format" : "json",
  "contentType" : "application/json",
  "contentLength" : "14",
  "collections" : ["collection1", "collection2"],
  "content" : {"data":"two"}
}

Example: Retrieving Content and Metadata

The example demonstrates reading content and metadata for a document.

The script below first writes an example document to the database that is in the examples collection and has document quality 2. Then, the document and metadata are read back in a single operation by including both 'content' and 'metadata' in the categories property of the read document descriptor. You can also specify specific metadata properties, such as 'collections' or 'permissions'.

To run the example, copy the script below to a file and run it using the node command.

const marklogic = require('marklogic');
const my = require('./my-connection.js');

const db = marklogic.createDatabaseClient(my.connInfo);

// (1) Seed the database with an example document that has custom metadata
db.documents.write({
  uri: '/read/example.json',
  contentType: 'application/json',
  collections: ['examples'],
  metadataValues: {key1: 'val1', key2: 2},
  quality: 2,
  content: { some: 'data' }
}).result().then(function(response) {
  // (2) Read back the content and metadata
  return db.documents.read({
    uris: [response.documents[0].uri], 
    categories: ['content', 'metadata']
  }).result();
}).then(function(documents) {
  // Emit the read results
  console.log('CONTENT: ' + 
              JSON.stringify(documents[0].content));
  console.log('COLLECTIONS: ' + 
              JSON.stringify(documents[0].collections));
  console.log('PERMISSIONS: ' + 
              JSON.stringify(documents[0].permissions, null, 2));
  console.log('PROPERTIES: ' + 
              JSON.stringify(documents[0].properties, null, 2));
  console.log('QUALITY: ' + 
              JSON.stringify(documents[0].quality, null, 2));
  console.log("METADATAVALUES: " + 
              JSON.stringify(documents[0].metadataValues, null, 2));
});

The script produces output similar to the following:

CONTENT: {"some": "data"}
COLLECTIONS: ["examples"]
PERMISSIONS: [
  {
    "role-name": "rest-writer",
    "capabilities": [
      "update"
    ]
  },
  {
    "role-name": "rest-reader",
    "capabilities": [
      "read"
    ]
  }
]
PROPERTIES: {}
QUALITY: 2
METADATAVALUES: {
  "key2": 2,
  "key1": "val1"
}

Transforming Content During Retrieval

You can apply custom server-side transforms to a document before returning the content to the requestor. A transform is a JavaScript module, XQuery module, or XSLT stylesheet that you install in your REST API instance using the DatabaseClient.config.transforms.write function. For details, see Working with Content Transformations.

You can configure a default read transform that is automatically applied whenever a document is retrieved. You can also specify a per-request transform by including a transform property in the call object passed to most read operations. If there is both a default transform and a per-request transform, the transforms are chained together, with the default transform running first. Thus, the output of the default transform is the input to the per-request transform, as shown in the following diagram:

To configure a default transform, set the document-transform-out configuration parameter of the REST API instance. Instance-wide configuration parameters are set using the DatabaseClient.config.serverprops.write function. For details, see Configuring Instance Properties.

To specify a per-request transform, use the call object form of a read operation such as DatabaseClient.documents.read and include a transform property. The value of the transform property is an array where the first item is the transform name and the optional additional items specify the name and value of parameters expected by the transform. That is, the transform property has the following form:

transform: [transformName, {param1: value, param2: value, ...}]

The following example applies a transform installed under the name example that expects a single parameter named reviewer. For a complete example, see Example: Read, Write, and Query Transforms.

const marklogic = require('marklogic');
const my = require('./my-connection.js');
const db = marklogic.createDatabaseClient(my.connInfo);

db.documents.read({
    uris: ['/doc/example.json'],
    transform: ['example', {reviewer: 'me'}]
  }).result(
    function(documents) {
      for (const i in documents)
        console.log('Document ' + documents[i].uri + ': ');
        console.log(JSON.stringify(documents[i].content));
    } 
);

Removing Documents By URI

To remove one or more documents by URI, use DatabaseClient.remove or DatabaseClient.documents.remove. Both functions enable you to remove documents by URI, but DatabaseClient.remove offers a simpler, more limited, interface. You can also remove multiple documents by collection or directory; for details, see Removing Sets of Documents.

Removing a document also removes the associated metadata. Removing a binary document with extracted metadata stored in a separate XHTML document also deletes the properties document; for details, see Working with Binary Documents.

The following example removes the documents /doc/example1.json and /docs/example2.json.

const marklogic = require('marklogic');
const my = require('./my-connection.js');
const db = marklogic.createDatabaseClient(my.connInfo);

db.documents.remove('/doc/example1.json','/doc/example2.json').result(
    function(response) {
      console.log(JSON.stringify(response));
    } 
);

The response contains the document URIs and a success indicator. For example, the above program produces the following output:

{ "uris":["/doc/example1.json", "/doc/example2.json"],
  "removed":true }

When you remove documents with DatabaseClient.remove, the response only includes the URIs. For example:

db.remove('/doc/example1.json')
==> ['/doc/example1.json']

db.remove('/doc/example1.json','/doc/example2.json')
==> ['/doc/example1.json','/doc/example2.json']

For additional examples, see the examples and tests in the node-client-api sources available on GitHub at http://github.com/marklogic/node-client-api.

Attempting to remove a document that does not exist produces the same output as successfully removing a document that exists.

When removing multiple documents, you can specify the URIs as individual parameters (if the call has no other parameters), an array of URIs, or an object with a uris property whose value is the URIs. When removing multiple documents, the whole batch fails if there is an error removing any one of the documents.

You can supply additional parameters to DatabaseClient.documents.remove, such as a transaction id and temporal collection information. For details, see the Node.js API Reference.

Removing Sets of Documents

You can use DatabaseClient.documents.removeAll to remove all documents in a collection or all documents in a database directory.

To remove documents by collection, use the following form:

db.documents.removeAll({collection:..., other-properties...}

To remove documents by directory, use the following form:

db.documents.removeAll({directory:..., other-properties...}

The optional other-properties can include a transaction id. For details, see the Node.js API Reference.

Removing all documents in a collection or directory requires the rest-writer role or equivalent privileges.

When removing documents by directory, the directory name must include a trailing slash (/).

The following example removes all documents in the collection /countries:

const marklogic = require('marklogic');
const my = require('./my-connection.js');
const db = marklogic.createDatabaseClient(my.connInfo);

db.documents.removeAll({collection: '/countries'}).result(
    function(response) {
      console.log(JSON.stringify(response));
    } 
);
==> {"exists":false,"collection":"/countries"}

The following example removes all documents in the directory /doc/:

const marklogic = require('marklogic');
const my = require('./my-connection.js');
const db = marklogic.createDatabaseClient(my.connInfo);

db.documents.removeAll({directory: '/doc/'}).result(
    function(response) {
      console.log(JSON.stringify(response));
    } 
);
==> {"exists":false,"directory":"/doc/"}

You can also include a txid property in the call object passed to DatabaseClient.documents.removeAll to specify a transaction in which to perform the deletion. For example:

db.documents.removeAll({directory: '/doc/', txid: '1234567890'})

For additional examples, see the examples and tests in the node-client-api sources available on GitHub at http://github.com/marklogic/node-client-api.

Removing All Documents

To remove all documents in the database, call DatabaseClient.documents.removeAll and include an all property with value true in the call object. For example:

db.documents.removeAll({all: true})

Removing all documents in the database requires the rest-admin or equivalent privileges.

There is no confirmation or other safety net when you clear the database in this way. Creating a backup is recommended.

The following example removes all documents in the database:

const marklogic = require('marklogic');
const my = require('./my-connection.js');
const db = marklogic.createDatabaseClient(my.connInfo);

db.documents.removeAll({all: true}).result(
    function(response) {
      console.log(JSON.stringify(response));
    } 
);
==> {"exists":false,"allDocuments":true}

You can also include a txid property in the call object passed to DatabaseClient.documents.removeAll to specify a transaction in which to perform the deletion. For example:

db.documents.removeAll({all: true, txid: '1234567890'})

Understanding Optimistic Locking

Consider an application that reads a document, makes modifications, and then updates the document in the database with the changes. The traditional approach to ensuring document integrity is to perform the read, modification, and update in a multi-statement transaction. This holds a lock on the document from the point when the document is read until the update is committed. However, this pessimistic locking blocks access to the document and incurs more overhead on the App Server.

With optimistic locking, the application does not hold a lock on a document between read and update. Instead, the application saves the document state on read, and then checks for changes at the time of update. The update fails if the document has changed between read and update. This is a conditional update.

Optimistic locking is useful in environments where integrity is important, but contention is rare enough that it is useful to minimize server load by avoiding unnecessary multi-statement transactions.

The Node.js Client API uses content versioning to implement optimistic locking. When content versioning is enabled on your REST API instance, MarkLogic Server associates an opaque version id with each document when it is created or updated. The version id changes each time you update the document. The version id is returned when you read a document, and you can pass it back in an update or delete operation to test for changes prior to commit.

Enabling content versioning does not implement document versioning. MarkLogic Server does not keep multiple versions of a document or track what changes occur. The version id can only be used to detect that a change occurred.

Using optimistic locking in your application requires the following steps:

1. Enable Optimistic Locking in the REST API instance.
2. Obtain a Version Id for documents you wish to conditionally update.
3. Apply a Conditional Update by including the version in your update operations.

You enable optimistic locking by setting the update-policy REST API instance property; for details. You send and receive version ids via the versionId property in a document descriptor.

Enable Optimistic Locking

To enable optimistic locking, call DatabaseClient.config.serverprops.write and set the update-policy property to version-required or version-optional. For example, the following code sets update-policy to version-optional:

const marklogic = require('marklogic');
const my = require('./my-connection.js');
const db = marklogic.createDatabaseClient(my.connInfo);

db.config.serverprops.write({'update-policy': 'version-optional'})
  .result(function(response) {
      console.log(JSON.stringify(response));
  });

Set the property to version-required if you want every document update or delete operation to use optimistic locking. Set the property to version-optional to allow selective use of optimistic locking.

The table below describes how each setting for this property affects document operations.

Obtain a Version Id

When optimistic locking is enabled, a version id is included in the response when you read documents. You can only obtain a version id if optimistic locking is enabled by setting update-policy; for details, see Enable Optimistic Locking.

You can obtain a version id for use in conditional updates in the following ways:

• Call DatabaseClient.documents.read. The version id is available through the versionId property of the returned document descriptor(s).
• Call DatabaseClient.documents.probe. No documents are fetched, but the version id is available through the versionId property of the returned document descriptor(s).

You can mix and match these methods of obtaining a version id.

The following example returns the version for multiple documents:

const marklogic = require('marklogic');
const my = require('./my-connection.js');
const db = marklogic.createDatabaseClient(my.connInfo);

db.documents.read('/doc/example1.json', '/doc/example2.json').
  .stream().on('data', function(document) {
    console.log('Read ' + document.uri + 
       ' with version ' + document.versionId);
  }).on('end', function() {
    console.log('finished');
  });

Apply a Conditional Update

To apply a conditional update, include a versionId property in the document descriptor passed to an update or delete operation. The version id is ignored if optimistic locking is not enabled; for details, see Enable Optimistic Locking.

When a document descriptor passed to an update or delete operation includes a version id, MarkLogic Server checks for a version id match before committing the update or delete. If the input version id does not match a document's current version id, the operation fails. In a multi-document update, the entire batch of updates is rejected if the conditional update of any document in the batch fails.

The following example performs a conditional update of two documents by including the versionId property in each input document descriptor.

const marklogic = require('marklogic');
const my = require('./my-connection.js');
const db = marklogic.createDatabaseClient(my.connInfo);

db.documents.write({
  documents: [
    { uri: '/doc/example1.json',
      contentType: 'application/json',
      content: { data: 1 },
      versionId: 14115098125553360
    },
    { uri: '/doc/example2.json',
      contentType: 'application/json',
      content: { data: 2 },
      versionId: 14115098125553350
    }
  ]
}).result(
  function(success) {
    console.log('Loaded the following documents:');
    for (const i in success.documents)
      console.log(success.documents[i].uri);
  }
);

Similarly, the following example removes the document /doc/example.json only if the current version id of the document in the database matches the version id in the input document descriptor:

const marklogic = require('marklogic');
const my = require('./my-connection.js');
const db = marklogic.createDatabaseClient(my.connInfo);

db.documents.remove({
  uris: ['/doc/example.json'], 
  versionId: 14115105931044000}).
result(
  function(response) {
    console.log(JSON.stringify(response));
});

You cannot use conditional delete when removing multiple documents in a single operation.

If a conditional update or delete fails due to a version id mismatch, MarkLogic Server responds with an error similar to the following. (The object contents have been reformatted for readability.)

{message: "remove document: response with invalid 412 
           status (on /doc/example1.json)",
  statusCode:412,
  body:{
    error: {
      status-code: "412",
      status: "Precondition Failed",
      message-code: "RESTAPI-CONTENTWRONGVERSION",
      message: "RESTAPI-CONTENTWRONGVERSION: (err:FOER0000) 
                Content version mismatch: uri /doc/example.json 
                doesn't match if-match: 14115105931044000"
}}}

Type of Binary Documents

This section provides a brief summary of binary document types. For details, see Working With Binary Documents in the Application Developer's Guide.

MarkLogic Server can store binary documents in three representations:

• Small binary documents are stored entirely in the database.
• Large binary documents are stored on disk with a small reference fragment in the database. The on-disk content is managed by MarkLogic Server.
• External binary documents are stored on disk with a small reference fragment in the database. However, the on-disk content is not managed by MarkLogic Server.

MarkLogic automatically determines whether a binary document is a small or large binary document when you insert or update the document, based on the document size and the database configuration.

Though external binary documents cannot be created using the Node.js Client API, you can retrieve them, just like any other document.

Streaming Binary Content

By using streaming techniques to access binary content, you can avoid loading potentially large documents into memory on MarkLogic Server and in your client application.

You can stream binary and other data into MarkLogic using by using a stream as the input source. For details, see Streaming Into the Database.

When you retrieve a large or external binary document from a database, MarkLogic Server automatically streams the content out of the database under the following conditions:

• Your request is for a single document, rather than being a bulk read.
• The size of the binary content returned is over the large binary size threshold. For details, see Working With Binary Documents in the Application Developer's Guide.
• The request is for content only. That is, no metadata is requested.
• The MIME type of the content is determinable from the Accept header or the document URI file extension.
• No content transformation is applied.

You can also use range requests to incrementally retrieve pieces of a binary document that meets the above constraints. For details, see Retrieving Binary Content with Range Requests.

You can avoid loading the entire document into memory in your client application by using a streaming result handler, such as the chunked stream pattern described in Stream Result Handling Pattern.

Retrieving Binary Content with Range Requests

When you just use db.documents.read to retrieve a binary document, the goal is to eventually retrieve the entire document, even if your application code processes it in chunks. By contrast, using range requests to retrieve parts of a binary document enables retryable, random access to parts of a binary document.

To use range requests, your retrieval operation must meet the conditions described in Streaming Binary Content. Specify the range of bytes to retrieve by including a range property in the call object passed to DatabaseClient.documents.read.

The following example requests the first 500K of the binary document with URI /binary/song.m4a:

db.documents.read({
    uris: '/binary/song.m4a',
    range: [0,511999]
  })

The document descriptor returned by such a call is similar to the following. The contentLength property indicates how many bytes were returned. This value will be smaller than the size of the requested range if you request a range that extends past the end of the source document.

result: [{
  content: {
    type': 'Buffer',
    data': [ theData ]
  },
  uri: '/binary/song.m4a',
  category: [ 'content' ],
  format: 'binary',
  contentLength: '10',
  contentType: 'audio/mp4'
}]

Metadata Categories

When working with documents and their metadata, metadata is usually expressed as properties of a document descriptor. For example, the following descriptor includes collections metadata for the document /doc/example.json, whether the descriptor is input output:

{ uri: '/doc/example.json',
  collections: ['collection1','collection2']

Some operations support a categories property for indicating what parts of a document and its metadata you want to read or write. For example, you can read just the collections and quality associated with /doc/example.json by calling DatabaseClient.documents.read similar to the following:

db.documents.read({
  uris: ['/doc/example.json'], 
  categories: ['collections', 'quality']
})

The following categories are supported:

• collections 
• permissions 
• properties 
• quality 
• metadataValues 
• metadata 
• content 

The metadataValues category represents simple key-value metadata property, sometimes called metadata fields. This category can contain both system-managed metadata, such as certain properties of a temporal document, and user-defined metadata. The value of a property in metadataValues is always stored as a string in MarkLogic. For more details, see Metadata Fields in the Administrator's Guide.

The metadata category is shorthand for collections, permissions, properties, and quality. Some operations, such as DatabaseClient.documents.read, also support the content category as a convenience for retrieving or updating content and metadata together.

Metadata Format

Metadata takes the following form in a document descriptor. You do not need to specify all categories of metadata when including it in a write or query operation. This structure applies to both input and output metadata.

{
  "collections" : [ string ],
  "permissions" : [
    { 
      "role-name" : string,
      "capabilities" : [ string ]
    }
  ],
  "properties" : {
    property-name : property-value
  },
  "quality" : integer,
  "metadataValues": { key: value, key: value, ... }
}

The following example is an output document descriptor from reading the metadata for the document with URI /doc/example.json. The document is in two collections and has two permissions, two document properties, and two metadataValues key-value pairs:

{
  "partType": "attachment",
  "uri": "/doc/example.json",
  "category": "metadata",
  "format": "json",
  "contentType": "application/json",
  "collections": [ "collection1", "collection2" ],
  "permissions": [
    {
      "role-name": "rest-writer",
      "capabilities": [ "update" ]
    },
    {
      "role-name": "rest-reader",
      "capabilities": [ "read" ]
    }
  ],
  "properties": {
    "prop1": "this is my prop",
    "prop2": "this is my other prop"
  },
  "metadataValues": {
    "key1": "value1",
    "key2": 2
  },
  "quality": 0
}

The following example shows metadata as XML. All elements are in the namespace http://marklogic.com/rest-api. You can have 0 or more <collection/>, <permission/> or property elements. There can be only one <quality/> element. The element name and contents of each property element depends on the property.

<metadata xmlns="http://marklogic.com/rest-api">
  <collections>
    <collection>collection-name</collection>
  </collections>
  <permissions>
    <permission>
      <role-name>name</role-name>
      <capability>capability</capability>
    </permission>
  </permissions>
  <properties>
    <property-element/>
  </properties>
  <quality>integer</quality>
  <metadata-values>
    <metadata-value key="key1">value1</metadata-value>
    <metadata-value key="key2">2</metadata-value>
  <metadata-values>
</metadata>

Working with Document Properties

Document properties are a kind of metadata. You can use document properties to add queryable user-defined data to a document without changing the document contents. For details, see Properties Documents and Directories in the Application Developer's Guide.

In most cases, your application should work with document properties in a consistent format. That is, if you insert or update properties as JSON, then you should retrieve and query them as JSON. If you insert or update properties as XML, you should retrieve them as XML. If you mix and match XML and JSON formats, you can encounter namespace inconsistencies.

Document properties are always stored in the database as XML. A document property inserted as JSON is converted internally into an XML element in the namespace http://marklogic.com/xdmp/json/basic, with an XML local name that matches the JSON property name. For example, a document property expressed in JSON as { myProperty : 'value' } has the following XML representation:

<rapi:metadata uri="/doc/example.json" ...>
  <prop:properties xmlns:prop="http://marklogic.com/xdmp/property">
    <myProperty type="string"
        xmlns="http://marklogic.com/xdmp/json/basic">
      value
    </myProperty>
  </prop:properties>
</rapi:metadata>

As long as you consistently use a JSON representation for document properties on input and output, this internal representation is transparent to you. However, if you query or read document properties using XML, you must be aware of the namespace and the internal representation. Similarly, you can use XML to insert document properties in no namespace or in your own namespace, but the namespace cannot be reflected in the JSON representation of the property. Therefore, it is best to be consistent in how you work with properties.

In order to support for passing properties as XML in a JSON string, users need to specify the namespace such as in     {'$ml.xml':     '<prop:properties xmlns:prop="http://marklogic.com/xdmp/property">         <myProps>Property 1</myProps></prop:properties>'     } Failing to mention namespace in a property with XML format in a JSON string can trigger Status 500: XDMP-DOCNONSBIND error.

The one exception is in code that works with properties on the server, such as resource service extensions and transformations. Such code always accesses document properties as XML.

If you configure an index based on a user-defined document property inserted using JSON, you should use the http://marklogic.com/xdmp/json/basic namespace in your configuration.

Protected system properties such as last-updated cannot be modified by your application. The JSON representation of such properties wraps them in an object with the key $ml.prop. For example:

{ "properties": {
    "$ml.prop": {
      "last-updated": "2013-11-06T10:01:11-08:00"
    }
} }

Disabling Metadata Merging

If you use the REST Client API to ingest a large number of documents at one time and you find the performance unacceptable, you might see a small performance improvement by disabling metadata merging. This topic explains the tradeoff and how to disable metadata merging.

• When to Consider Disabling Metadata Merging
• How to Disable Metadata Merging

const marklogic = require('marklogic');
const my = require('./my-connection.js');
const db = marklogic.createDatabaseClient(my.connInfo);

db.config.serverprops.write({'update-policy': 'overwrite-metadata'})
  .result(function(response) {
      console.log(JSON.stringify(response));
  });