MarkLogic Server 11.0 Product Documentation

Node.js Application Developer's Guide — Chapter 5

Introduction to the Optic Interfaces

The Optic API enables you to perform relational operations on indexes and documents. For example, you can extract data in row format, perform joins, and perform relational queries on XML and JSON documents. You can also extract a row view of data from other sources, such as lexicons and semantic triples.

The Optic capabilities of the Node.js Client API closely mirror the server-side Optic API described in Optic API for Multi-Model Data Access in the Application Developer's Guide. Refer to that guide for conceptual details.

The usage model for an optic query in Node.js is as follows:

1. Build an Optic execution plan on the client using the planBuilder interface. For details, see Generating a Plan.
2. Execute the plan on MarkLogic, resulting in generation of a row set. For details, see Invoking a Plan.
3. Process the results returned by MarkLogic on the client. For details on result formats, see Configuring Row Set Format.

Execution of a plan can yield a row that satisfies any of several common use cases:

• A traditional flat list of atomic values with names and XML Schema atomic datatypes.
• A dynamic JSON or XML document with substructure and leaf atomic values or mixed text.
• An envelope with out-of-band metadata properties and relations for a list of documents.

In addition to executing a query plan, you can also perform the following Optic related operations:

• Generate an execution plan explanation that reflects the logical flow of the plan as a sequence of atomic operations.
• Export a serializable version of the plan for later use.

Interface Summary

The following are the key Optic interfaces in the Node.js Client API:

Preparing to Run the Examples

The examples in this chapter use the data, templates, and a plan from SQL on MarkLogic Server Quick Start in the SQL Data Modeling Guide. If you want to run the examples, you should use the SQL quick start to set up your environment. See the following topics in the SQL Data Modeling Guide:

• Setup MarkLogic Server
• Load the Data
• Create Template Views

If you set up a database named SQLdata, as directed in the Quick Start, then add a database property to your DatabaseClient connection information. For example, if your my-connection.js module should be similar to the following when running the Optic examples:

module.exports = {
  connInfo: {
    host: 'localhost',
    port: 8000,
    user: your-ml-username,
    password: your-ml-user-password,
    database: 'SQLdata'
  }
};

For more details on configuring a DatabaseClient for running the examples, see Using the Examples in This Guide.

Generating a Plan

Use marklogic.planBuilder to construct an Optic query plan. Usually, you then use the plan to extract row data from MarkLogic. You can also generate a plan explanation or serialize a built plan.

An Optic plan defines a pipeline of relational operations to be applied to a row set.

1. Select a data source using one of the data accessor methods, planBuilder.from*. For example, planBuilder.fromLiterals, planBuilder.fromView, planBuilder.fromTriples, or planBuilder.fromLexicons.
2. Refine your plan using modifier and composer operations, such as select, joinInner, where, and orderBy.
3. Optionally, specify a mapper or reducer to be applied to each row. See planBuilder.PreparePlan.map or planBuilder.PreparePlan.reduce. The mapper or reducer function runs on MarkLogic.

For example, the following code snippet builds a plan to list the employee ID, first name, and last name of all employees, in order of employee ID, using the configuration and data from the SQL on MarkLogic Server Quick Start chapter in the SQL Data Modeling Guide.

const marklogic = require('marklogic');
const my = require('./my-connection.js');

const db = marklogic.createDatabaseClient(my.connInfo);
const pb = marklogic.planBuilder;

pb.fromView('main', 'employees')
  .select(['EmployeeID', 'FirstName', 'LastName'])
  .orderBy('EmployeeID')

The plan is not executed until you process it with rows.query or rows.queryAsStream. For details, see Invoking a Plan.

For details on the logical structure of a plan and the available operators, see Objects in an Optic Pipeline in the Application Developer's Guide. The Node.js planBuilder interface exposes methods with the same names and purpose as the Server-Side JavaScript Optic API.

The planBuilder interface includes namespaces that expose proxies for many server-side operations. For example, you can call cts query constructors or a server-side function in the fn, xdmp, or map namespaces. The placeholder in the plan translates into an equivalent invocation of a server-side function during plan execution.

For example, the following code constructs a plan that includes cts query constructor proxy calls:

const marklogic = require('marklogic');
const my = require('./my-connection.js');

const db = marklogic.createDatabaseClient(my.connInfo);
const pb = marklogic.planBuilder;

db.rows.query(
  pb.fromView('main', 'employees')
    .where(pb.cts.andQuery([
             pb.cts.wordQuery('Senior'), 
             pb.cts.wordQuery('Researcher')]))
    .select(['FirstName', 'LastName', 'Position']),
  {columnTypes: 'header'}
).then( function(rows) {
    console.log(JSON.stringify(rows, null, 2));
}).catch(
  function(error) {
    console.log(JSON.stringify(error, null, 2));
});

For more information on the proxy functions, see the planBuilder API Reference and the following topics in the Application Developer's Guide:

• Expression Functions For Processing Column Values
• Functions Equivalent to Boolean, Numeric, and String Operators
• Node Constructor Functions

For details on the behavior of a particular server-side function, see the MarkLogic Server-Side JavaScript Function Reference.

Invoking a Plan

Use rows.query or rows.queryAsStream to execute a plan and generate a row set. You can also pass in an options object to control the output format, specify parameter bindings, or pin the query to a specific point-in-time.

Unlike most other Node.js Client API operations, query and queryAsStream do not support a result method. Instead, the plan is sent to MarkLogic for evaluation when you invoke the then or on methods. See the examples below.

To perform a query at a specific point in time, pass a Timestamp object as the value of the timestamp property of the options parameter. For more details, see Performing Point-in-Time Operations.

For example, the following code constructs a plan, executes it on MarkLogic, and then processes the resulting row set. The plan and results are based on the configuration and data from the SQL on MarkLogic Server Quick Start chapter in the SQL Data Modeling Guide.

const marklogic = require('marklogic');
const my = require('./my-connection.js');

const db = marklogic.createDatabaseClient(my.connInfo);
const pb = marklogic.planBuilder;

db.rows.query(
  pb.fromView('main', 'employees')
    .select(['EmployeeID', 'FirstName', 'LastName'])
    .orderBy('EmployeeID')
).then( function(rows) {
  console.log(JSON.stringify(rows, null, 2));
}).catch( function(error) {
  console.log(JSON.stringify(error, null, 2));
});

The following example is the equivalent code using queryAsStream to stream the results as JavaScript objects instead of returning them all at once. Several stream modes are available; for details, see Streaming Row Data.

const marklogic = require('marklogic');
const my = require('./my-connection.js');

const db = marklogic.createDatabaseClient(my.connInfo);
const pb = marklogic.planBuilder;

db.rows.queryAsStream(
  pb.fromView('main', 'employees')
    .select(['EmployeeID', 'FirstName', 'LastName'])
    .orderBy('EmployeeID'),
  'object'
).on( 'data', function(rows) {
    console.log(JSON.stringify(rows, null, 2));
}).on ('end', function() {
  console.log('done');
});

The plan returns results similar to the following. You can use the options parameter of the query and queryAsStream methods to customize the output format; for details, see Configuring Row Set Format.

{ "columns": [
    { "name": "main.employees.EmployeeID" },
    { "name": "main.employees.FirstName" },
    { "name": "main.employees.LastName" }
  ],
  "rows": [
    { "main.employees.EmployeeID": {
        "type": "xs:integer",
        "value": 1
      },
      "main.employees.FirstName": {
        "type": "xs:string",
        "value": "John"
      },
      "main.employees.LastName": {
        "type": "xs:string",
        "value": "Widget"
      }
    },
    { "main.employees.EmployeeID": {
        "type": "xs:integer",
        "value": 2
      },
      "main.employees.FirstName": {
        "type": "xs:string",
        "value": "Jane"
      },
      "main.employees.LastName": {
        "type": "xs:string",
        "value": "Lead"
      }
    },
    { "main.employees.EmployeeID": {
        "type": "xs:integer",
        "value": 3
      },
      "main.employees.FirstName": {
        "type": "xs:string",
        "value": "Steve"
      },
      "main.employees.LastName": {
        "type": "xs:string",
        "value": "Manager"
      }
    }, ...
  ]
}

Configuring Row Set Format

When you invoke a query plan as described in Invoking a Plan, the result is a row set. You can use the options parameter of rows.query and rows.queryAsStream to configuring the layout of the row set.

This section covers the following topics related to how configuration options affect the layout of a row set.

• Configuration Options
• Layout Examples

{format:'json',
 structure: 'object',
 columnTypes: 'rows'}

{ columns: [
    {name: 'main.employees.Employee'},
    {name: 'main.employees.FirstName'},
    {name: 'main.employees.LastName'}
  ],
  rows: [
    { 'main.employees.EmployeeID': {
        type: 'xs:integer', value: 1},
      'main.employees.FirstName': {
        type: 'xs:string', value: 'John'},
      'main.employees.LastName': {
        type: 'xs:string', value: 'Widget'}
    },
    {...}, ...
  ]
}

{format:'json',
 structure: 'object',
 columnTypes: 'header'}

{ columns: [
    { name: 'main.employees.EmployeeID',
      type: 'xs:integer' },
    { name: 'main.employees.FirstName',
      type: 'xs:string' },
    { name: 'main.employees.LastName',
      type: 'xs:string' }
  ],
  rows: [
    { 'main.employees.EmployeeID': 1,
      'main.employees.FirstName': 'John',
      'main.employees.LastName': 'Widget'
    },
    {...}, ...
  ]
}

{format:'json',
 structure: 'array',
 columnTypes: 'rows'}

[
  [ { name: 'main.employees.EmployeeID' },
    { name: 'main.employees.FirstName' },
    { name: 'main.employees.LastName' }
  ],
  [ { type: 'xs:integer', value: 1 },
    { type: 'xs:string', value: 'John' },
    { type: 'xs:string', value: 'Widget' }
  ],
  [...], ...
]

{format:'json',
 structure: 'array',
 columnTypes: 'header'}

[
  [ { name: 'main.employees.EmployeeID',
      type: 'xs:integer' },
    { name: 'main.employees.FirstName',
      type: 'xs:string' },
    { name: 'main.employees.LastName',
      type: 'xs:string' }
  ],
  [ 1, 'John', 'Widget' ],
  [...], ...
]

{format: 'csv'
 structure: 'object'}

main.employees.EmployeeID,main.employees.FirstName,main.employees.LastName
1,John,Widget
...

{format: 'csv'
 structure: 'array'}

["main.employees.EmployeeID", "main.employees.FirstName", "main.employees.LastName"]
[1, "John", "Widget"]
...

{format:'xml',
 columnValues: 'rows'}

<t:table xmlns:t="http://marklogic.com/table">
  <t:columns>
    <t:column name="main.employees.EmployeeID"/>
    <t:column name="main.employees.FirstName"/>
    <t:column name="main.employees.LastName"/>
  </t:columns>
  <t:rows>
    <t:row>
      <t:cell name="main.employees.EmployeeID"
              type="xs:integer">1</t:cell>
      <t:cell name="main.employees.FirstName"
              type="xs:string">John</t:cell>
      <t:cell name="main.employees.LastName"
              type="xs:string">Widget</t:cell>
    </t:row>
    ...
  </t:rows>
</t:table

{format:'xml',
 columnValues: 'header'}

<t:table xmlns:t="http://marklogic.com/table">
  <t:columns>
    <t:column name="main.employees.EmployeeID"
              type="xs:integer"/>
    <t:column name="main.employees.FirstName"
              type="xs:string"/>
    <t:column name="main.employees.LastName"
              type="xs:string"/>
  </t:columns>
  <t:rows>
    <t:row>
      <t:cell name="main.employees.EmployeeID">1</t:cell>
      <t:cell name="main.employees.FirstName">John</t:cell>
      <t:cell name="main.employees.LastName">Widget</t:cell>
    </t:row>
  <t:row>
  ...
  </t:rows>
</t:table>

Streaming Row Data

You can use rows.queryAsStream to stream a row set from MarkLogic. Depending on the streaming mode, data is returned to your handler in chunks of bytes, JavaScript objects, or JSON text sequence records.

• Object Mode Streaming
• Chunked Mode Streaming
• Sequence Mode Streaming

Row sets formatted as XML or CSV can only be streamed in chunked mode. Rows sets formatted as JSON can be streamed in any mode.

db.rows.queryAsStream(plan, 'object', options)

{format:'json',
 structure: 'object',
 columnTypes: 'rows'}

{ columns: [
    { name: 'main.employees.EmployeeID' },
    { name: 'main.employees.FirstName' },
    { name: 'main.employees.LastName' }
]}

{ 'main.employees.EmployeeID': {
    type: 'xs:integer', value: 1 },
  'main.employees.FirstName': {
    type: 'xs:string', value: "John' },
  'main.employees.LastName': {
    type: 'xs:string', value: 'Widget' }
}

{format:'json',
 structure: 'object',
 columnTypes: 'header'}

{ columns: [
    { name: 'main.employees.EmployeeID',
      type: 'xs:integer'
    }, {
      name: 'main.employees.FirstName',
      type: 'xs:string'
    }, {
      name: 'main.employees.LastName',
      type: 'xs:string'
    }
]}

{ 'main.employees.EmployeeID': 1,
  'main.employees.FirstName': 'John',
  'main.employees.LastName': 'Widget'
}

{format:'json',
 structure: 'array',
 columnTypes: 'rows'}

[ { name: 'main.employees.EmployeeID' },
  { name: 'main.employees.FirstName' },
  { name: 'main.employees.LastName' } ]

[ { type: 'xs:integer',
    value: 1 },
  { type: 'xs:string',
    value: 'John' },
  { type: 'xs:string',
    value: 'Widget' } ]

{format:'json',
 structure: 'array',
 columnTypes: 'header'}

[ { name: 'main.employees.EmployeeID',
    type: 'xs:integer'
  }, {
    name: 'main.employees.FirstName',
    type: 'xs:string'
  }, {
    name: 'main.employees.LastName',
    type: 'xs:string'
  } ]

[ 1, 'John', 'Widget' ]

db.rows.queryAsStream(plan, 'chunked', options)

db.rows.queryAsStream(plan, 'sequence', options)

const marklogic = require('marklogic');
const my = require('./my-connection.js');
const fs = require('fs');

const db = marklogic.createDatabaseClient(my.connInfo);
const plan = fs.readFileSync('./plan.json', 'utf8');
const options = {
  format: 'json',
  structure: 'object',
  columnTypes: 'header'
};

db.rows.queryAsStream(plan, 'sequence', options)
  .on( 'data', function(record) {
    const asString = record.toString();
    console.log(asString.substring(1,asString.length-2));
  }).on ('end', function() {
    console.log('done');
  });

{"columns":[{"name":"main.employees.EmployeeID","type":"xs:integer"},{"name":"main.employees.FirstName","type":"xs:string"},{"name":"main.employees.LastName","type":"xs:string"}]}
{"main.employees.EmployeeID":1,"main.employees.FirstName":"John","main.employees.LastName":"Widget"}
...

[{"name":"main.employees.EmployeeID","type":"xs:integer"},{"name":"main.employees.FirstName","type":"xs:string"},{"name" :"main.employees.LastName","type":"xs:string"}]
[1,"John","Widget"]
...

Passing Parameters into a Plan

If your plan uses placeholder parameters, use the bindings option to specify values for the placeholders when you invoke the plan.

The bindings option value is a JavaScript object where the property names correspond to parameters, and the values are either a parameter value or an object that specifies a type or language key and a value. That is, each property must take one of forms shown below:

{bindings : {
  paramName1 : value1,
  paramName2 : {value: value2, type: typeNameString},
  paramName3 : {value: value3, lang: languageCode}
}}

The type name can be any derivation of xs:atomicType other than xs:QName. For example, you can use type names such as 'string', 'integer', and 'decimal'. If you do not specify a type, the value is interpreted as a string. Use a language code to bind language-tagged strings.

For example, if you defined a placeholder variable named start in your plan definition, then you could specify a value for start in the bindings option in any of the following ways:

{bindings:{
  start: 'apple'
}

{bindings:{
  start: {value: 'apple', type: 'string'}
}

{bindings:{
  start: {value: 'apple', lang: 'en'}
}

Handling Complex Column Values

If your row set includes column values with non-atomic type, such as XML elements, JSON arrays, JSON objects, binary content, or text nodes, they are serialized inline by default. For example, the following row contains a column named complex whose value is a serialized XML element:

{ "id":{
    "type":"xs:integer",
    "value":1
  },
  "complex":{
    "type":"element",
    "value":"<root><A>Detail 1</A><B>2015-12-01</B></root>"
}}

You can use the complexValues option of rows.queryAsStream to specify whether such complex values should be included inline (as shown above) or by reference. You cannot configure the handling of complex values when executing a plan using rows.query.

For example, the following option setting specifies complex values should be returned by reference.

{complexValues: 'reference'}

For example, suppose your row set includes a column named node whose value is an XML element. The default behavior (inline) yields a serialized string of the following form for the node value:

"node": {
  "type": "element",
  "value": "<alpha><a>true</a></alpha>"
}

If you set the complexValues option to reference, then the same column value is rendered as follows:

"node": {
  "contentType": "application/xml",
  "format": "xml",
  "content": "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<beta><b>false</b></beta>"
}

The node value is actually returned as a reference to a complex value by MarkLogic, but the reference is resolved and expanded for you in the results returned from the Node.js Client API.

The details of the format depend on the output formatting options you choose and the type of complex value. For example, if the complex value is a binary document, then it would be returned as a base64 encoded value inline, but as a Node.js Buffer when fetched by reference.

Generating an Execution Plan

Use rows.explain to generate an execution plan that expresses the logical flow of an Optic plan as a sequence of atomic operations. For more details, see Optic Execution Plan in the Application Developer's Guide.

For example, the following code generates an execution plan based on the data from SQL on MarkLogic Server Quick Start in the SQL Data Modeling Guide.

const marklogic = require('marklogic');
const my = require('./my-connection.js');

const db = marklogic.createDatabaseClient(my.connInfo);
const pb = marklogic.planBuilder;

db.rows.explain(
  pb.fromView('main', 'employees')
    .select(['EmployeeID', 'FirstName', 'LastName'])
    .orderBy('EmployeeID')
    .limit(3)
).then( function(rows) {
    console.log(JSON.stringify(rows, null, 2));
}).catch(
  function(error) {
    console.log(JSON.stringify(error, null, 2));
});

Serializing a Plan

Use the export method of a plan object to generate a serializable version of a plan that can be saved to persistent storage for later use. The export method produces a JavaScript object representation of a plan, which can be serialized to JSON using, for example, JSON.stringfy.

For example, the following code generates a JSON serialization of a plan and logs it to the console:

const marklogic = require('marklogic');
const my = require('./my-connection.js');

const db = marklogic.createDatabaseClient(my.connInfo);
const pb = marklogic.planBuilder;

console.log( JSON.stringify(
  pb.fromView('main', 'employees')
    .select(['EmployeeID', 'FirstName', 'LastName'])
    .orderBy('EmployeeID')
    .limit(3)
    .export(), 
  null,2));

To subsequently use a serialized plan, convert it back into a JavaScript object and pass the object to rows.query or rows.queryAsStream. For example:

const marklogic = require('marklogic');
const my = require('./my-connection.js');

const db = marklogic.createDatabaseClient(my.connInfo);
const pb = marklogic.planBuilder;

// Read serialized plan from a file or other storage, then...
const serializedPlan = ...;

const thePlan = JSON.parse(serializedPlan);
db.rows.query(thePlan)
  .then( function(rows) {
    console.log(JSON.stringify(rows, null, 2));
}).catch(
  function(error) {
    console.log(JSON.stringify(error, null, 2));
});