MarkLogic 9 Product Documentation

Administrator's Guide — Chapter 21

When Cluster Has Nodes at Different Software Version Levels

The rolling upgrade feature is designed is to enable business continuity while a cluster is being upgraded. The window of time when a cluster has nodes of varying versions should be small. During this time, do not make application code changes and/or configuration changes.

Configuration changes involve the following:

• Changes to index, forest, database, application server, host, group, and cluster setting
• Changes to security settings such adding/changing/deleting roles, users, privileges, credentials, certificates, etc.
• Adding/removing/updating TDE templates
• Adding/removing/updating redaction rules

In addition, do not perform any manual merges and disable reindexing while the cluster has nodes that are at different software version levels. Changing error log settings and adding trace events to debug issues should be fine.

Rolling Upgrade Process

You can upgrade your cluster with a minimal amount of transactional downtime (less than 5-10 minutes) without using the rolling upgrade feature. Consider whether the tradeoff in added complexity warrants using rolling upgrades instead of the regular upgrade process. See Upgrading from Previous Releases in the Installation Guide for information about the regular upgrade process.

Here are the steps in the rolling upgrade process:

• Backup - back up any hosts that you are going to upgrade. See Backing Up and Restoring a Database for details.
• Preparation - prepare any code or application that you may need to prior to the upgrade (See Interaction with Other MarkLogic Features for details).
• Upgrade - perform the actual upgrade.
• Cleanup

Before you start your upgrade, you will need to backup the hosts you are going to upgrade. Then do any preparation of code or applications that is necessary prior to the upgrade (see Interaction with Other MarkLogic Features) for possible preparations.

When you have completed the upgrade, you may need to perform some clean up.

Rolling Upgrade Status in Admin UI

You can view the status of a rolling upgrade in the Admin UI. To view the rolling upgrade status, complete the following procedure:

1. Click the Configure icon in the left tree menu.

   

2. Click the Upgrade tab to view the upgrade progress of each host in the cluster. The displayed version number is the highest version number on any host in the cluster.

   

   If a rolling upgrade is in progress, a sync icon will appear to the right of the version number from which you are upgrading, which is located above the left tree menu. Click the sync icon to navigate to the Upgrade tab.

   If a rolling upgrade is not in progress, all hosts in the cluster are running the same version. Click the Upgrade tab to verify the version number.

Effective version and software version

Until you commit the upgrade, the effective version of the hosts in the cluster is the earlier version, not the newer version (for example 8.0-6, not 9.0-1). The effective version is the version that the cluster as a whole is running. The software version is the version of MarkLogic Server that is installed on each host. You will be prompted to upgrade the Security database when you log into the Admin UI.

After commiting a rolling upgrade you can only restore to the later version (for example, 9.0-12), not to the earlier version (for example, 8.0-6). Running your cluster in an uncommited state is equivalent to running in the previous (earlier) version of MarkLogic. No 9.0-12 features are available until the upgrade has been commited.

An upgrade of the Security database is required after you have committed the new version of MarkLogic.

Rolling Upgrades Using REST Management APIs

After backing up your hosts and preparing your applications, you can perform a rolling upgrade using the REST Management APIs. The following example assumes a three node cluster with 8.0-6 installed, upgrading to 9.0-x.

Upgrade your cluster to Red Hat Enterprise Linux 7 (x64) before starting the MarkLogic upgrade. MarkLogic 9 will not work on Red Hat Enterprise Linux 6. See Supported Platforms in the Release Notes for more information.

The following code sample can be used as a model to script an upgrade of a single three-node cluster.

(: This is an end-to-end scenario to orchestrate a rolling upgrade on a 3-node 8.0 cluster to a 9.0 build. :)

(: Iterate over each host in the cluster :)

GET:/manage/v2/hosts

(: Remove host from load-balancer rotation if necessary :)

PUT:/manage/v2/hosts/{id|name}/properties 

(: Disable any local-disk forests on the host to force a failover :)

PUT:/manage/v2/forests/{id|name}/properties 

(: Change primary host for any shared-disk forests :)

PUT:/manage/v2/forests/{id|name}/properties 

(: Restart any failover forests that are open on the host :)

PUT:/manage/v2/forests/{id|name}?state=restart 

$ curl --anyauth --user user:password -i -X POST \
  -d '{"operation": "restart-local-cluster"}'\
  http://localhost:8002/manage/v2

(: Wait for task-server and app servers to become idle :)

GET:/manage/v2/servers, GET:/manage/v2/servers/{id|name}?view=status 

(: Stop the host :)

$ curl -v -X POST --anyauth --user admin:admin \
  -H "Content-Type:application/x-www-form-urlencoded" \
  -d '{"operation": "shutdown-local-cluster"}' \
  "http://localhost:8002/manage/v2"

(: Start the host :)

$ curl -v -X POST --anyauth --user admin:admin \
  -H "Content-Type:application/x-www-form-urlencoded" \
  -d '{"operation": "restart-local-cluster"}'\
  "http://localhost:8002/manage/v2"

(: Enable any local-disk failover forest on the host :)

PUT:/manage/v2/forests/{id|name}/properties 

(: Restore primary host for any shared-disk forests :) 	

PUT:/manage/v2/forests/{id|name}/properties 

(: Restart any failover forests that should fail back. :)

PUT:/manage/v2/forests/{id|name}?state=restart 

PUT:/manage/v2/forests/{id|name}/properties 

(: upgrade security db :)
curl -v -X POST --anyauth --user admin:admin \
  --header "Content-Type:application/x-www-form-urlencoded" \
  -d '{"operation": "security-database-upgrade-local-cluster"}'\
  "http://localhost:8002/manage/v2"

(: verify cluster version :)
curl -v -X GET --anyauth --user admin:admin 
  --header "Content-Type:application/json"   http://localhost:8002/manage/v2?format=json | tools/jq/jq  '.
  ["local-clusterlocalhost-default"]["effective-version"]'`

The jq tool is used to parse out the json properties. It is a free download from https://stedolan.github.io/jq/

Upgrading an EC2 Instance

The process for performing a rolling upgrade in EC2 (AWS) is fairly simple. It is very similar to a normal update of the Cloud Formation templates. See Upgrading MarkLogic on AWS in the MarkLogic Server on Amazon Web Services (AWS) Guide for details about a normal update.

This example assumes an existing 3-node cluster running MarkLogic 8.0 from Cloud Formation templates. Before you upgrade your instance, you need to upgrade your Cloud Formation template to reference the new AMI (9.0 CF template). See http://developer.marklogic.com/products/aws for details about upgrading your templates.

Here are the additional steps:

1. Backup any important data before performing the upgrade.
2. Update stack with your updated Cloud Formation template. Make sure the stack update is complete.

   

   We do not recommend that you automatically swap out the Cloud Formation template. Instead, make a copy of your existing template (if it contains the AMI IDs), edit just the AMI IDs, and then use that for the update. (If the AMI ID is passed as a parameter or other means, use those means).

3. In the EC2 dashboard, terminate one instance at a time and wait for it to be replaced with a new one. Starting with the master instance or node that contains the Security database. The host will automatically be restarted by the managed cluster feature.

   

   Wait for the host to come back up (with new host name).

4. In the EC2 dashboard, repeat the process and terminate node2.
5. When all nodes have been updated, check the cluster state from the Query Console using this query:

   xquery version "1.0-ml";
    
     <hosts>{
     for $i in xdmp:host-name(xdmp:hosts())
       return (
          let $response :=        xdmp:http-get(concat("http://localhost:8002/manage/v2/hosts/",
            $i,"?view=status&amp;format=json"),
           <options xmlns="xdmp:http">
             <authentication method="digest">
               <username>admin</username>
               <password>admin</password>
             </authentication>
             <headers>
               <content-type>application/json</content-type>
             </headers>
           </options>)
           return (
             <host>
             <name>{$response[2]//*:name/data()}</name>
             <host-software-version>
             {$response[2]//*:software-version/value/data()}
               </host-software-version>
             <cluster-effective-version>{$response[2]//*:
                effective-version/value/data()}
              </cluster-effective-version>
             </host>
           ) ) }</hosts>

   

6. Make a call from 8001 to check host status. The exact URL to check this, is http://hostname:8001/host-summary.xqy?section=host.

   

7. Repeat these steps for node3.
8. When the node3 update is complete, check to verify that the upgrade is complete by checking the cluster effective version.

   xquery version "1.0-ml";
    
     <hosts>{
   	  for $i in xdmp:host-name(xdmp:hosts())
       return (
          let $response :=       xdmp:http-get(concat("http://localhost:8002/manage/v2/hosts/",
           $i,"?view=status&amp;format=json"),
           <options xmlns="xdmp:http">
             <authentication method="digest">
               <username>admin</username>
               <password>admin</password>
             </authentication>
             <headers>
               <content-type>application/json</content-type>
             </headers>
           </options>)
           return (
             <host>
             <name>{$response[2]//*:name/data()}</name>
             <host-software-version>
             {$response[2]//*:software-version/value/data()}
             </host-software-version>
             <cluster-effective-version>
             {$response[2]//*:effective-version/value/data()}
             </cluster-effective-version>
             </host>
           )       
       )
       }</hosts>

   

9. Navigating anywhere in the port 8001 Admin UI will prompt you to upgrade your Security database. To upgrade the Security database, go to http://ec2-52-12-84-72.us-west-2.compute.amazonaws.com:8001/security-upgrade.xqy. When that has been done, then the upgrade is complete.

Rolling Upgrades Using XQuery

The XQuery Admin APIs can be used to set up and perform a rolling upgrade through the Query Console. This section contains sample code that you can use from the Query Console.

To get the host versions via REST:

xquery version "1.0-ml";
 
  <hosts>{
  for $i in xdmp:host-name(xdmp:hosts())
    return (
       let $response := xdmp:http-get(concat("http://localhost:8002/manage/v2/hosts/",$i,"?view=status&amp;format=json"),
        <options xmlns="xdmp:http">
          <authentication method="digest">
            <username>admin</username>
            <password>admin</password>
          </authentication>
          <headers>
            <content-type>application/json</content-type>
          </headers>
        </options>)
        return (
          <host>
          <name>{$response[2]//*:name/data()}</name>
<host-software-version>{$response[2]//*:software-version/value/data()}</host-software-version>
<cluster-effective-version>{$response[2]//*:effective-version/value/data()}</cluster-effective-version>
          </host>
        )       
    )
    }</hosts>

To complete the upgrade, log onto the Admin UI to upgrade the Security database.

Committing the upgrade results in the updated configuration being saved with a re-read delay of 5 seconds to ensure that all online hosts have received the new file before XDQP connections start dropping.

See step #10 in Upgrading an EC2 Instance. If the servers don't have the correct version, there may be a host that is in maintenance mode. The admin:can-commit-upgrade function will return true if all servers have the correct software version. See Admin APIs for more about the XQuery Admin APIs available.

Rolling Upgrades on Both Production and DR Clusters

Upgrade the disaster recovery cluster first. It is important to upgrade the disaster recovery cluster first, since the newer version of the software will be able to receive fragments and journal frames encoded on the master cluster.

Once the disaster recovery cluster has been upgraded, then upgrade the production cluster.

Admin APIs

These Admin API functions are available for rolling upgrades:

• admin:cluster-get-effective-version

  Returns the cluster's effective MarkLogic version (for example, 8000600 for 8.0-6).

• admin:can-commit-upgrade

  Returns true if the cluster is ready to commit the upgrade, returns false otherwise.

REST Management APIs

The following REST Management endpoints provide useful information and functionality when performing a Rolling Upgrade operation.

• GET /manage/v2/properties includes effective version
• GET /manage/v2/hosts?view=status - includes version and effective-version.
• GET /manage/v2/hosts/{id|name}?view=status - includes version and effective-version.

SQL

In MarkLogic 9, an updated version of SQL using the triple index is being introduced. The existing version (pre-MarkLogic 9) will continue to work in a mixed cluster, and after the cluster has been upgraded to 9.0-x. The updated version of SQL will not work in a mixed cluster. You will need to upgrade to the newer version of MarkLogic and commit the upgrade before those features are available.

The earlier version of SQL based on range indexes will work in the mixed cluster (prior to committing the upgrade), and it will also work with MarkLogic 9.

Server-Side JavaScript

In the new version of Server-Side JavaScript, ValueIterator has been replaced by Sequence. The ValueIterator interface used to represent sequences of value in MarkLogic 8 has been replaced by the new Sequence interface. A Sequence is a JavaScript Iterable object. All functions which previously operated on or returned a ValueIterator now use a Sequence instead.

In many cases, this change is transparent to your code. However, code that depends on the following ValueIterator properties and methods must be changed:

• ValueIterator.next - Use a for..of loop to iterate over a Sequence. Use fn.head if you just want to pick off the first or only value in a Sequence.
• ValueIterator.count - Use fn.count instead.
• ValueIterator.clone - No longer needed. You can iterate over the same Sequence multiple times.

To prepare your code for a possible mixed environment, you might use a safe coding pattern similar to this:

var list = xdmp.arrayValues(...);
if (list instanceof Sequence) {
    ... ML9 idiom ...
} else {
    ... ML8 idiom ...
}

See Sequence in the JavaScript Reference Guide and Sequence in the MarkLogic Server-Side JavaScript Function Reference for more information.

Java Client API

You can upgrade a Java application from Java Client API 3.x to 4.x before upgrading from MarkLogic 8 to MarkLogic 9. However, you must first upgrade your JRE to version 1.8 or later. The Java Client API version 4.x only supports JRE 1.8 or later.

Custom UDFs

Plugins will not work in a mixed cluster because the interface for UDFs has changed. You cannot have the same code compiled against two different sets of definitions from two different releases. You must recompile and redeploy your UDF libraries for MarkLogic 9.

Reverse Queries Involving Circles

In MarkLogic 9, a change was made to store circle radii as kilometers instead of miles. When operating in a mixed cluster consisting of 9.0-1 and 9.0-2 nodes, you may receive unexpected results for reverse queries involving circles. No issues exist when upgrading from MarkLogic 8.0-x.