Loading TOC...
Administrator's Guide (PDF)

MarkLogic Server 11.0 Product Documentation
Administrator's Guide
— Chapter 21

Rolling Upgrades

Users with highly available clusters under heavy transaction loads may want to upgrade to a newer version of MarkLogic in a seamless manner. A rolling upgrade, where hosts in a cluster are upgraded one by one, is one approach to addressing this need. Rolling upgrades are used to upgrade a large cluster with many hosts to a newer version of MarkLogic Server without incurring any downtime in availability or interruption of transactions. A rolling upgrade may also be used to apply patches to multiple hosts.

The goal in performing a rolling upgrade is to have zero downtime of your server availability or transactional data. This is most useful for large high availability (HA) clusters that have a large number of ongoing transactions. A rolling upgrade can be performed on both a primary cluster and a disaster recovery (DR) cluster.

Your cluster must have MarkLogic 8.0-6 or later installed to perform a rolling upgrade. The rolling upgrade feature works on all supported platforms. Rolling upgrades will only work when upgrading from MarkLogic 8.0-6 or later to MarkLogic 10.0-x or later.

Do not change your application to take advantage of any 10.0-1 features until all the nodes in your cluster have been upgraded to 10.0-1. In addition, you should avoid making any configuration changes to your cluster during a rolling upgrade.

This chapter describes rolling upgrades and includes the following sections:

Understanding Rolling Upgrades

A rolling upgrade incrementally installs a later version of MarkLogic Server (host by host), rather than having to take down the whole cluster to install the newer version. Performing a rolling upgrade means that your cluster may be in a mixed state (where more than one version of MarkLogic Server is running) for some period of time during the upgrade process. During the process, the features in the newer version of MarkLogic will not be available until the whole cluster has been committed to the new version. Because of this you may have to change or modify some of your application code prior to starting the rolling upgrade, so that code will work in a mixed environment. For example, JavaScript code may need modification (because of the new version of V8 for server-side JavaScript) before you commit the upgrade.

The security database and the schemas database must be on the same host, and that host should be the first host you upgrade when upgrading a cluster.

In a mixed node cluster, before the upgrade has been committed, the node that has been upgraded to MarkLogic 10.0-1 will be read-only. This is to prevent any configuration changes from the 10.0-1 node. We strongly recommend that you not make any configuration changes until you have finished upgrading the entire cluster.

When Cluster Has Nodes at Different Software Version Levels

The rolling upgrade feature is designed is to enable business continuity while a cluster is being upgraded. The window of time when a cluster has nodes of varying versions should be small. During this time, do not make application code changes and/or configuration changes.

Configuration changes involve the following:

  • Changes to index, forest, database, application server, host, group, and cluster setting
  • Changes to security settings such adding/changing/deleting roles, users, privileges, credentials, certificates, etc.
  • Adding/removing/updating TDE templates
  • Adding/removing/updating redaction rules

In addition, do not perform any manual merges and disable reindexing while the cluster has nodes that are at different software version levels. Changing error log settings and adding trace events to debug issues should be fine.

Rolling Upgrade Process

You can upgrade your cluster with a minimal amount of transactional downtime (less than 5-10 minutes) without using the rolling upgrade feature. Consider whether the tradeoff in added complexity warrants using rolling upgrades instead of the regular upgrade process. See Upgrading from Previous Releases in the Installation Guide for information about the regular upgrade process.

Here are the steps in the rolling upgrade process:

Before you start your upgrade, you will need to backup the hosts you are going to upgrade. Then do any preparation of code or applications that is necessary prior to the upgrade (see Interaction with Other MarkLogic Features) for possible preparations.

When you have completed the upgrade, you may need to perform some clean up.

Rolling Upgrade Status in Admin UI

To view the rolling upgrade status in the Admin UI:

  1. Click the Configure icon in the left tree menu.

  2. Click the Upgrade tab to view the upgrade progress of each host in the cluster. The displayed version number is the highest version number on any host in the cluster.

    If a rolling upgrade is in progress, a Sync icon will appear to the right of the version number from which you are upgrading, which is located above the left tree menu. Click the Sync icon to navigate to the Upgrade tab.

    If a rolling upgrade is not in progress, all hosts in the cluster are running the same version. Click the Upgrade tab to verify the version number.

Effective version and software version

Until you commit the upgrade, the effective version of the hosts in the cluster is the earlier version, not the newer version (for example 9.0-6, not 10.0-1). The effective version is the version that the cluster as a whole is running. The software version is the version of MarkLogic Server that is installed on each host. You will be prompted to upgrade the Security database when you log into the Admin UI.

After committing a rolling upgrade you can only restore to the later version (for example, 10.0-1), not to the earlier version (for example, 9.0-6). Running your cluster in an uncommitted state is equivalent to running in the previous (earlier) version of MarkLogic. No 10.0-1 features are available until the upgrade has been committed.

An upgrade of the Security database is required after you have committed the new version of MarkLogic.

Example--Rolling Upgrade

The following is a simplified step-by-step process for a rolling upgrade, on a small, three host cluster. The general outline is to first backup all of your hosts, make any changes to software applications, then proceed with the rolling upgrade, failing over and upgrading each node. When all nodes in the cluster have been upgraded, verify that you can commit the upgrade and change the cluster effective version to the new version. Finish by doing any cleanup that is necessary.

In addition, prior to starting the upgrade, you may need to modify some of your existing software to run in mixed version cluster. See Interaction with Other MarkLogic Features for details.

  1. Backup all hosts in your existing cluster. See Backing Up and Restoring a Database for details on backing up your hosts.
  2. Modify any code that will need to be modified. See Interaction with Other MarkLogic Features for a list of potential software issues.
  3. Upgrade your cluster to Red Hat Enterprise Linux 7 (x64). See Supported Platforms in the Installation Guide for details.

    MarkLogic 9 will not work on Red Hat Enterprise Linux 6. See Supported Platforms in the Release Notes for more information.

  4. Take down the first host and start the upgrade. Use these commands from the command line.
    1. Stop MarkLogic. Use this curl command so that you can also take advantage of the fast failover feature.
      curl -X POST --anyauth --user admin:admin -d "state=shutdown&failover=true" "http://node1:8002/manage/v2/hosts/node3"

      The failover parameter was added to POST /manage/v2/hosts/{id|name} in MarkLogic version 9.0-5. The above call will fail in previous versions of MarkLogic.

    2. Uninstall the existing RPM
      rpm uninstall MarkLogic-8.0-1.x86_64.rpm
    3. Install the new RPM
      rpm install MarkLogic-10.0-3.x86_64.rpm
    4. Bring the host back up, and start MarkLogic.
      sudo /sbin/service MarkLogic start
  5. Repeat steps 3 through 5 for each of the hosts in the cluster. (You will need to perform the upgrade process node-by-node.)
  6. When you have completed all of the host upgrades, check the software version and the effective version for the cluster, and then commit the upgrade.

    Use this XQuery command to check the cluster's effective version:

    xquery version "1.0-ml"; 
    import module namespace admin = "http://marklogic.com/xdmp/admin" 
      at "/MarkLogic/admin.xqy";
    
    let $config := admin:get-configuration()
    return 
      admin:cluster-get-effective-version($config)
    
    =>
    (: returns the effective software version of this cluster :) 

    Use this query to check if the cluster is ready to commit the upgrade:

    xquery version "1.0-ml"; 
    import module namespace admin = "http://marklogic.com/xdmp/admin" 
      at "/MarkLogic/admin.xqy";
    
    admin:can-commit-upgrade()
    
    =>
    (: returns true if the cluster is ready to commit the upgrade :)

    The cluster version should be 9000100 or later for the upgrade to commit.

    Upgrade the Security database on the host cluster.

    After committing the upgrade, verify the upgrade with this query:

    xquery version "1.0-ml"; 
    import module namespace admin = "http://marklogic.com/xdmp/admin" 
      at "/MarkLogic/admin.xqy";
    
    let $config := admin:get-configuration()
    return 
      admin:cluster-get-effective-version($config) 
    
    =>
    (: returns the effective software version of the cluster :)

This step-by-step example for a simple rolling upgrade can also be scripted. For an example model for a script, see Rolling Upgrades Using REST Management APIs.

Performing Rolling Upgrades

You can perform a rolling upgrade via scripting through the REST Management APIs or by using the XQuery APIs. You can also perform a rolling upgrade on an AWS cluster. This section describes the different options for configuring and performing a rolling upgrade.

Rolling Upgrades Using REST Management APIs

After backing up your hosts and preparing your applications, you can perform a rolling upgrade using the REST Management APIs. The following example assumes a three node cluster with 8.0-6 installed, upgrading to 10.0-x.

Upgrade your cluster to Red Hat Enterprise Linux 7 (x64) before starting the MarkLogic upgrade. MarkLogic 10 will not work on Red Hat Enterprise Linux 6. See Supported Platforms in the Release Notes for more information.

The following code sample can be used as a model to script an upgrade of a single three-node cluster.

Please note that, when doing a rolling upgrade from 8.0 to 10.0 (with 9.0 being skipped), you must not use the Management REST API at all. You may use the Admin API, but only to read cluster configuration. This does not affect rolling upgrades from 8.0 to 9.0 or upgrades from 9.0 to 10.0.

(: This is an end-to-end scenario to orchestrate a rolling upgrade on a 3-node 9.0 cluster to a 10.0 build. :)

(: Iterate over each host in the cluster :)

GET:/manage/v2/hosts

(: Remove host from load-balancer rotation if necessary :)

PUT:/manage/v2/hosts/{id|name}/properties 

(: Disable any local-disk forests on the host to force a failover :)

PUT:/manage/v2/forests/{id|name}/properties 

(: Change primary host for any shared-disk forests :)

PUT:/manage/v2/forests/{id|name}/properties 

(: Restart any failover forests that are open on the host :)

PUT:/manage/v2/forests/{id|name}?state=restart 

$ curl --anyauth --user user:password -i -X POST \
  -d '{"operation": "restart-local-cluster"}'\
  http://localhost:8002/manage/v2

(: Wait for task-server and app servers to become idle :)

GET:/manage/v2/servers, GET:/manage/v2/servers/{id|name}?view=status 

(: Stop the host :)

$ curl -v -X POST --anyauth --user admin:admin \
  -H "Content-Type:application/x-www-form-urlencoded" \
  -d '{"operation": "shutdown-local-cluster"}' \
  "http://localhost:8002/manage/v2"

(: Start the host :)

$ curl -v -X POST --anyauth --user admin:admin \
  -H "Content-Type:application/x-www-form-urlencoded" \
  -d '{"operation": "restart-local-cluster"}'\
  "http://localhost:8002/manage/v2"

(: Enable any local-disk failover forest on the host :)

PUT:/manage/v2/forests/{id|name}/properties 

(: Restore primary host for any shared-disk forests :) 	

PUT:/manage/v2/forests/{id|name}/properties 

(: Restart any failover forests that should fail back. :)

PUT:/manage/v2/forests/{id|name}?state=restart 

PUT:/manage/v2/forests/{id|name}/properties 

(: upgrade security db :)
curl -v -X POST --anyauth --user admin:admin \
  --header "Content-Type:application/x-www-form-urlencoded" \
  -d '{"operation": "security-database-upgrade-local-cluster"}'\
  "http://localhost:8002/manage/v2"

(: verify cluster version :)
curl -v -X GET --anyauth --user admin:admin 
  --header "Content-Type:application/json"   http://localhost:8002/manage/v2?format=json | tools/jq/jq  '.
  ["local-clusterlocalhost-default"]["effective-version"]'`

The jq tool is used to parse out the json properties. It is a free download from https://stedolan.github.io/jq/

Upgrading an EC2 Instance

The process for performing a rolling upgrade in EC2 (AWS) is fairly simple. It is very similar to a normal update of the Cloud Formation templates. See Upgrading MarkLogic on AWS in the MarkLogic Server on Amazon Web Services (AWS) Guide for details about a normal update.

This example assumes an existing 3-node cluster running MarkLogic 8.0 from Cloud Formation templates. Before you upgrade your instance, you need to upgrade your Cloud Formation template to reference the new AMI (9.0 CF template). See http://developer.marklogic.com/products/aws for details about upgrading your templates.

Here are the additional steps:

  1. Backup any important data before performing the upgrade.
  2. Update stack with your updated Cloud Formation template. Make sure the stack update is complete.

    We do not recommend that you automatically swap out the Cloud Formation template. Instead, make a copy of your existing template (if it contains the AMI IDs), edit just the AMI IDs, and then use that for the update. (If the AMI ID is passed as a parameter or other means, use those means).

  3. In the EC2 dashboard, terminate one instance at a time and wait for it to be replaced with a new one. Starting with the master instance or node that contains the Security database. The host will automatically be restarted by the managed cluster feature.

    Wait for the host to come back up (with new host name).

  4. In the EC2 dashboard, repeat the process and terminate node2.
  5. When all nodes have been updated, check the cluster state from the Query Console using this query:
    xquery version "1.0-ml";
     
      <hosts>{
      for $i in xdmp:host-name(xdmp:hosts())
        return (
           let $response :=        xdmp:http-get(concat("http://localhost:8002/manage/v2/hosts/",
             $i,"?view=status&amp;format=json"),
            <options xmlns="xdmp:http">
              <authentication method="digest">
                <username>admin</username>
                <password>admin</password>
              </authentication>
              <headers>
                <content-type>application/json</content-type>
              </headers>
            </options>)
            return (
              <host>
              <name>{$response[2]//*:name/data()}</name>
              <host-software-version>
              {$response[2]//*:software-version/value/data()}
                </host-software-version>
              <cluster-effective-version>{$response[2]//*:
                 effective-version/value/data()}
               </cluster-effective-version>
              </host>
            ) ) }</hosts>

  6. Make a call from 8001 to check host status. The exact URL to check this, is http://hostname:8001/host-summary.xqy?section=host.

  7. Repeat these steps for node3.
  8. When the node3 update is complete, check to verify that the upgrade is complete by checking the cluster effective version.
    xquery version "1.0-ml";
     
      <hosts>{
    	  for $i in xdmp:host-name(xdmp:hosts())
        return (
           let $response :=       xdmp:http-get(concat("http://localhost:8002/manage/v2/hosts/",
            $i,"?view=status&amp;format=json"),
            <options xmlns="xdmp:http">
              <authentication method="digest">
                <username>admin</username>
                <password>admin</password>
              </authentication>
              <headers>
                <content-type>application/json</content-type>
              </headers>
            </options>)
            return (
              <host>
              <name>{$response[2]//*:name/data()}</name>
              <host-software-version>
              {$response[2]//*:software-version/value/data()}
              </host-software-version>
              <cluster-effective-version>
              {$response[2]//*:effective-version/value/data()}
              </cluster-effective-version>
              </host>
            )       
        )
        }</hosts>

  9. Navigating anywhere in the port 8001 Admin UI will prompt you to upgrade your Security database. To upgrade the Security database, go to http://ec2-52-12-84-72.us-west-2.compute.amazonaws.com:8001/security-upgrade.xqy. When that has been done, then the upgrade is complete.

Rolling Upgrades Using XQuery

The XQuery Admin APIs can be used to set up and perform a rolling upgrade through the Query Console. This section contains sample code that you can use from the Query Console.

To get the host versions via REST:

xquery version "1.0-ml";
 
  <hosts>{
  for $i in xdmp:host-name(xdmp:hosts())
    return (
       let $response := xdmp:http-get(concat("http://localhost:8002/manage/v2/hosts/",$i,"?view=status&amp;format=json"),
        <options xmlns="xdmp:http">
          <authentication method="digest">
            <username>admin</username>
            <password>admin</password>
          </authentication>
          <headers>
            <content-type>application/json</content-type>
          </headers>
        </options>)
        return (
          <host>
          <name>{$response[2]//*:name/data()}</name>
<host-software-version>{$response[2]//*:software-version/value/data()}</host-software-version>
<cluster-effective-version>{$response[2]//*:effective-version/value/data()}</cluster-effective-version>
          </host>
        )       
    )
    }</hosts>

To complete the upgrade, log onto the Admin UI to upgrade the Security database.

Committing the upgrade results in the updated configuration being saved with a re-read delay of 5 seconds to ensure that all online hosts have received the new file before XDQP connections start dropping.

See step #10 in Upgrading an EC2 Instance. If the servers don't have the correct version, there may be a host that is in maintenance mode. The admin:can-commit-upgrade function will return true if all servers have the correct software version. See Admin APIs for more about the XQuery Admin APIs available.

Rolling Upgrades on Both Production and DR Clusters

Upgrade the disaster recovery cluster first. It is important to upgrade the disaster recovery cluster first, since the newer version of the software will be able to receive fragments and journal frames encoded on the master cluster.

Once the disaster recovery cluster has been upgraded, then upgrade the production cluster.

Rolling Back a Partial Upgrade

As long as you have not committed your upgrade to MarkLogic 9 or later, you can reinstall the earlier version of the server (MarkLogic 8.0-6) on each node.

In the event that you need to roll back an upgrade that has not been completed and committed, you can roll back the partial upgrade by re-installing the previous version of MarkLogic (for example 8.0-6) on the machines that have been upgraded.

APIs for Rolling Upgrades

These APIs are available for managing rolling upgrades in a MarkLogic cluster.

Admin APIs

These Admin API functions are available for rolling upgrades:

REST Management APIs

The following REST Management endpoints provide useful information and functionality when performing a Rolling Upgrade operation.

Interaction with Other MarkLogic Features

For existing features that will work as expected with MarkLogic 9, a rolling upgrade will not have any impact. Some existing features may not work as expected until the rolling upgrade is complete and the cluster has been committed to the newer version.

One possible example of this would be semantic triples, where the triple count may be increased after inserting same data twice in during a rolling upgrade in mixed mode. During a rolling upgrade, the MarkLogic 9 triple index is not able to return triples in the correct order for MarkLogic 8 semantics. A user would need to have multiple triples that are identical, except for the types of the values for this situation to occur.

Features introduced in MarkLogic 9 may or may not work in a mixed cluster (a cluster that has not been completely upgraded to MarkLogic 9, and has an effective version of 8.0-x). The following is a list of features that may need to be monitored while performing rolling upgrade.

SQL

Beginning in MarkLogic 9, an updated version of SQL using the triple index is introduced. The existing version (pre-MarkLogic 9) will continue to work in a mixed cluster, and after the cluster has been upgraded to 9.0-x or 10.0-x. The updated version of SQL will not work in a mixed cluster. You will need to upgrade to the newer version of MarkLogic and commit the upgrade before those features are available.

The earlier version of SQL based on range indexes will work in the mixed cluster (prior to committing the upgrade), and it will also work with MarkLogic 9 and later.

Server-Side JavaScript

In the new version of Server-Side JavaScript, ValueIterator has been replaced by Sequence. The ValueIterator interface used to represent sequences of value in MarkLogic 8 has been replaced by the new Sequence interface. A Sequence is a JavaScript Iterable object. All functions which previously operated on or returned a ValueIterator now use a Sequence instead.

In many cases, this change is transparent to your code. However, code that depends on the following ValueIterator properties and methods must be changed:

To prepare your code for a possible mixed environment, you might use a safe coding pattern similar to this:

var list = xdmp.arrayValues(...);
if (list instanceof Sequence) {
    ... ML9 idiom ...
} else {
    ... ML8 idiom ...
}

See Sequence in the JavaScript Reference Guide and Sequence in the MarkLogic Server-Side JavaScript Function Reference for more information.

Java Client API

When upgrading to MarkLogic 9, you can upgrade a Java application from Java Client API 3.x to 4.x before upgrading from MarkLogic 8 to MarkLogic 9. However, you must first upgrade your JRE to version 1.8 or later. The Java Client API version 4.x only supports JRE 1.8 or later.

Custom UDFs

When upgrading from MarkLogic 8, plugins will not work in a mixed cluster because the interface for UDFs has changed. You cannot have the same code compiled against two different sets of definitions from two different releases. You must recompile and redeploy your UDF libraries for MarkLogic 9 and later.

Reverse Queries Involving Circles

In MarkLogic 9, a change was made to store circle radii as kilometers instead of miles. When operating in a mixed cluster consisting of 9.0-1 and 9.0-2 nodes, you may receive unexpected results for reverse queries involving circles. No issues exist when upgrading from MarkLogic 8.0-x.

Important Points to Note Before Performing Rolling Upgrades

  • Your cluster must have MarkLogic 8.0-6 or later installed to perform a rolling upgrade.
  • The cluster needs to be configured for High Availability (HA).
  • Do not change your application to take advantage of any 10.0-x features until all the nodes in your cluster have been upgraded to 10.0-x.
  • You may have to change or modify some of your application code prior to starting the rolling upgrade, so that code will work in a mixed environment.
  • The security database and the schemas database must be on the same host, and that host should be the first host you upgrade when upgrading a cluster.
  • In a mixed node cluster, before the upgrade has been committed, the node that has been upgraded to MarkLogic 10.0-x will be read-only. This is to prevent any configuration changes from the 10.0-x node. We strongly recommend that you not make any configuration changes until you have finished upgrading the entire cluster.
  • The window of time when a cluster has nodes of varying versions should be small. During this time, do not make application code changes and/or configuration changes. Configuration changes involve the following:
    • Changes to index, forest, database, application server, host, group, and cluster settings
    • Changes to security settings such adding/changing/deleting roles, users, privileges, credentials, certificates, etc.
    • Adding/removing/updating TDE templates
    • Adding/removing/updating redaction rules
  • Do not perform any manual merges and disable reindexing while the cluster has nodes that are at different software version levels. Changing error log settings and adding trace events to debug issues should be fine.
  • After committing a rolling upgrade, you can only restore to the later version (for example, 10.0-1), not to the earlier version (for example, 9.0-6). Running your cluster in an uncommitted state is equivalent to running in the previous (earlier) version of MarkLogic. No 10.0-1 features are available until the upgrade has been committed.
  • An upgrade of the Security database is required after you have committed the new version of MarkLogic.
  • Please note that, when doing a rolling upgrade from 8.0 to 10.0 (with 9.0 being skipped), you must not use the Management REST API at all. You may use the Admin API, but only to read cluster configuration. This does not affect rolling upgrades from 8.0 to 9.0 or upgrades from 9.0 to 10.0.
  • Upgrade your cluster to Red Hat Enterprise Linux 7 (x64) before starting the MarkLogic upgrade. Both MarkLogic 9 and MarkLogic 10 will not work on Red Hat Enterprise Linux 6.
  • It is important to upgrade the disaster recovery cluster first, since the newer version of the software will be able to receive fragments and journal frames encoded on the master cluster.

Other Upgrade Options

There are alternatives to rolling upgrades for applying patches or upgrading your hosts. You can perform an upgrade to hosts in a cluster will very minimal downtime. See Upgrading from Previous Releases in the Installation Guide for more information.

« Previous chapter
Next chapter »