MarkLogic Server 11.0 Product Documentation

Scripting Administrative Tasks Guide — Chapter 3

Procedure: Set Up the First Host in a Cluster

Setting up the first (or only) host in a cluster involves the following Management REST API requests to the host:

• POST http://bootstrap-host:8001/admin/v1/init
• POST http://bootstrap-host:8001/admin/v1/instance-admin

The following procedure outlines the scriptable steps:

1. Install MarkLogic Server. For details, see Installing MarkLogic in Installation Guide. For example:

   sudo rpm -i /your/location/MarkLogic-8.0-1.x86_64.rpm

2. Start MarkLogic Server. For details, see Starting MarkLogic Server in Installation Guide. For example:

   sudo /sbin/service MarkLogic start

3. Initialize MarkLogic Server with a POST /admin/v1/init request. You can optionally install a license during this step. This step causes a restart. For example:

   curl -X POST -d "" http://${BOOTSTRAP_HOST}:8001/admin/v1/init

4. Initialize security with a POST /admin/v1/instance-admin request. This step causes a restart. Note that this request passes the Admin password in cleartext, so you should only perform this operation in a secure environment. For example:

   curl -X POST -H "Content-type: application/x-www-form-urlencoded" \
      --data "admin-username=${USER}" --data "admin-password=${PASS}" \
      --data "wallet-password=${WPASS}" --data "realm=${SEC_REALM}" \
      http://${BOOTSTRAP_HOST}:8001/admin/v1/instance-admin

When this procedure is completed, MarkLogic Server is fully operational on the host, and you can configure forests, databases, and App Servers, or add additional hosts to the cluster.

Once you successfully complete POST /admin/v1/instance-admin, security is initialized, and all subsequent requests require authentication.

Example Script: Set Up the First Host in a Cluster

The following bash shell script assumes

• MarkLogic Server has already been installed and started on the host. You can include these steps in your script. They are omitted here for simplicity.
• You are not installing a license.

Use the script by specifying at least two hosts on the command line.

this_script [options] bootstrap_host

Use the command line options in the following table to tailor the script to your environment:

-a auth_mode

-p password

-r sec_realm

-u username

This script makes use of the restart checking technique described in Setting Up the First Host in a Cluster. This script performs only minimal error checking and is not meant for production use.

#!/bin/bash
################################################################
# Use this script to initialize the first (or only) host in
# a MarkLogic Server cluster. Use the options to control admin
# username and password, authentication mode, and the security
# realm. If no hostname is given, localhost is assumed. Only
# minimal error checking is performed, so this script is not
# suitable for production use.
#
# Usage:  this_command [options] hostname
#
################################################################

BOOTSTRAP_HOST="localhost"
USER="admin"
PASS="password"
WPASS="wpass"
AUTH_MODE="anyauth"
SEC_REALM="public"
N_RETRY=5
RETRY_INTERVAL=10

#######################################################
# restart_check(hostname, baseline_timestamp, caller_lineno)
#
# Use the timestamp service to detect a server restart, given a
# a baseline timestamp. Use N_RETRY and RETRY_INTERVAL to tune
# the test length. Include authentication in the curl command
# so the function works whether or not security is initialized.
#   $1 :  The hostname to test against
#   $2 :  The baseline timestamp
#   $3 :  Invokers LINENO, for improved error reporting
# Returns 0 if restart is detected, exits with an error if not.
#
function restart_check {
  LAST_START=`$AUTH_CURL "http://$1:8001/admin/v1/timestamp"`
  for i in `seq 1 ${N_RETRY}`; do
    if [ "$2" == "$LAST_START" ] || [ "$LAST_START" == "" ]; then
      sleep ${RETRY_INTERVAL}
      LAST_START=`$AUTH_CURL "http://$1:8001/admin/v1/timestamp"`
    else 
      return 0
    fi
  done
  echo "ERROR: Line $3: Failed to restart $1"
  exit 1
}


#######################################################
# Parse the command line

OPTIND=1
while getopts ":a:p:r:u:" opt; do
  case "$opt" in
    a) AUTH_MODE=$OPTARG ;;
    p) PASS=$OPTARG ;;
    w) WPASS=$OPTARG ;;
    r) SEC_REALM=$OPTARG ;;
    u) USER=$OPTARG ;;
    \?) echo "Unrecognized option: -$OPTARG" >&2; exit 1 ;;
  esac
done
shift $((OPTIND-1))

if [ $# -ge 1 ]; then
  BOOTSTRAP_HOST=$1
  shift
fi

# Suppress progress meter, but still show errors
CURL="curl -s -S"
# Add authentication related options, required once security is initialized
AUTH_CURL="${CURL} --${AUTH_MODE} --user ${USER}:${PASS}" --wpass ${WPASS}


#######################################################
# Bring up the first (or only) host in the cluster. The following
# requests are sent to the target host:
#   (1) POST /admin/v1/init
#   (2) POST /admin/v1/instance-admin?admin-user=W&admin-password=X&wallet-password=Y&realm=Z
# GET /admin/v1/timestamp is used to confirm restarts.

# (1) Initialize the server
echo "Initializing $BOOTSTRAP_HOST..."
$CURL -X POST -d "" http://${BOOTSTRAP_HOST}:8001/admin/v1/init
sleep 10

# (2) Initialize security and, optionally, licensing. Capture the last
#     restart timestamp and use it to check for successful restart.
TIMESTAMP=`$CURL -X POST \
   -H "Content-type: application/x-www-form-urlencoded" \
   --data "admin-username=${USER}" --data "admin-password=${PASS}" \
   --wallet-password "wpass=${WPASS}" --data "realm=${SEC_REALM}" \
   http://${BOOTSTRAP_HOST}:8001/admin/v1/instance-admin \
   | grep "last-startup" \
   | sed 's%^.*<last-startup.*>\(.*\)</last-startup>.*$%\1%'`
if [ "$TIMESTAMP" == "" ]; then
  echo "ERROR: Failed to get instance-admin timestamp." >&2
  exit 1
fi

# Test for successful restart
restart_check $BOOTSTRAP_HOST $TIMESTAMP $LINENO

echo "Initialization complete for $BOOTSTRAP_HOST..."
exit 0

Procedure: Add a Host to a Cluster

Once you configure the first host in a cluster, add additional hosts to the cluster by using the following series of Management REST API requests for each host:

• POST http://joining-host:8001/admin/v1/init
• GET http://joining-host:8001/admin/v1/server-config
• POST http://bootstrap-host:8001/admin/v1/cluster-config
• POST http://joining-host:8001/admin/v1/cluster-config

The following procedure outlines the scriptable steps:

1. Install MarkLogic Server on the joining host. For details, see Installing MarkLogic in Installation Guide. For example:

   sudo rpm -i /your/location/MarkLogic-8.0-1.x86_64.rpm

2. Start MarkLogic Server on the joining host. For details, see Starting MarkLogic Server in Installation Guide. For example:

   sudo /sbin/service MarkLogic start

3. Initialize MarkLogic Server on the joining host by sending a POST request to /admin/v1/init. Authentication is not required. This step causes a restart.

   curl -X POST -d "" http://${JOINING_HOST}:8001/admin/v1/init

4. Fetch the configuration of the joining host with a GET /admin/v1/server-config request. Authentication is not required. The following example saves the config to a shell variable. You can also save it to a file by using curl's -o option.

   JOINER_CONFIG=`curl -s -S -X GET -H "Accept: application/xml" \
     http://${JOINING_HOST}:8001/admin/v1/server-config`

5. Send the joining host configuration data to a host already in the cluster with a POST /admin/v1/cluster-config request. The cluster host responds with cluster configuration data in ZIP format.

   The following example command assumes the input server config is in the shell variable JOINER_CONFIG and saves the output cluster configuration to cluster-config.zip.

   curl -s -S --digest --user admin:password -X POST 
     -o cluster-config.zip -d "group=Default" \
     --data-urlencode "server-config=${JOINER_CONFIG}" \
     -H "Content-type: application/x-www-form-urlencoded" \
     http://${BOOTSTRAP_HOST}:8001/admin/v1/cluster-config

6. Send the cluster configuration ZIP data to the joining host with a POST /admin/v1/cluster-config request. This completes the join sequence. This step causes a restart.

   curl -s -S -X POST -H "Content-type: application/zip" \
     --data-binary @./cluster-config.zip \
     http://${JOINING_HOST}:8001/admin/v1/cluster-config

Once this procedure completes, the joining host is a fully operational member of the cluster.

Example Script: Add Hosts to a Cluster

The following bash shell script assumes

• MarkLogic Server has already been fully initialized on the bootstrap host, using either the Admin Interface or the Management REST API. You can include this step in your script. It is omitted here for simplicity.
• MarkLogic Server has already been installed and started on each host joining the cluster. You can include these steps in your script. They are omitted here for simplicity.
• You are not installing a license.
• The joining host should be in the Default group.

The example script completes the cluster join sequence for each host serially. However, you can also add hosts concurrently. For details, see Adding Hosts to a Cluster Concurrently.

Use the script by specifying at least two hosts on the command line. A fully initialized host that is already part of the cluster must be the first parameter.

this_script [options] bootstrap_host joining_host [joining_host...]

Use the command line options in the following table to tailor the script to your environment:

-a auth_mode

-p password

-u username

The script makes use of the restart checking technique described in Using the Timestamp Service to Verify a Restart. This script performs only minimal error checking and is not meant for production use.

#!/bin/bash
################################################################
# Use this script to initialize and add one or more hosts to a
# MarkLogic Server cluster. The first (bootstrap) host for the
# cluster should already be fully initialized.
#
# Use the options to control admin username and password, 
# authentication mode, and the security realm. At least two hostnames
# must be given: A host already in the cluster, and at least one host
# to be added to the cluster. Only minimal error checking is performed, 
# so this script is not suitable for production use.
#
# Usage:  this_command [options] cluster-host joining-host(s)
#
################################################################

USER="admin"
PASS="password"
AUTH_MODE="anyauth"
N_RETRY=5
RETRY_INTERVAL=10

#######################################################
# restart_check(hostname, baseline_timestamp, caller_lineno)
#
# Use the timestamp service to detect a server restart, given a
# a baseline timestamp. Use N_RETRY and RETRY_INTERVAL to tune
# the test length. Include authentication in the curl command
# so the function works whether or not security is initialized.
#   $1 :  The hostname to test against
#   $2 :  The baseline timestamp
#   $3 :  Invokers LINENO, for improved error reporting
# Returns 0 if restart is detected, exits with an error if not.
#
function restart_check {
  LAST_START=`$AUTH_CURL "http://$1:8001/admin/v1/timestamp"`
  for i in `seq 1 ${N_RETRY}`; do
    if [ "$2" == "$LAST_START" ] || [ "$LAST_START" == "" ]; then
      sleep ${RETRY_INTERVAL}
      LAST_START=`$AUTH_CURL "http://$1:8001/admin/v1/timestamp"`
    else 
      return 0
    fi
  done
  echo "ERROR: Line $3: Failed to restart $1"
  exit 1
}


#######################################################
# Parse the command line

OPTIND=1
while getopts ":a:p:u:" opt; do
  case "$opt" in
    a) AUTH_MODE=$OPTARG ;;
    p) PASS=$OPTARG ;;
    u) USER=$OPTARG ;;
    \?) echo "Unrecognized option: -$OPTARG" >&2; exit 1 ;;
  esac
done
shift $((OPTIND-1))

if [ $# -ge 2 ]; then
  BOOTSTRAP_HOST=$1
  shift
else
  echo "ERROR: At least two hostnames are required." >&2
  exit 1
fi
ADDITIONAL_HOSTS=$@

# Curl command for all requests. Suppress progress meter (-s), 
# but still show errors (-S)
CURL="curl -s -S"
# Curl command when authentication is required, after security
# is initialized.
AUTH_CURL="${CURL} --${AUTH_MODE} --user ${USER}:${PASS}"


#######################################################
# Add one or more hosts to a cluster. For each host joining
# the cluster:
#   (1) POST /admin/v1/init (joining host)
#   (2) GET /admin/v1/server-config (joining host)
#   (3) POST /admin/v1/cluster-config (bootstrap host)
#   (4) POST /admin/v1/cluster-config (joining host)
# GET /admin/v1/timestamp is used to confirm restarts.

for JOINING_HOST in $ADDITIONAL_HOSTS; do
  echo "Adding host to cluster: $JOINING_HOST..."

  # (1) Initialize MarkLogic Server on the joining host
  TIMESTAMP=`$CURL -X POST -d "" \
     http://${JOINING_HOST}:8001/admin/v1/init \
     | grep "last-startup" \
     | sed 's%^.*<last-startup.*>\(.*\)</last-startup>.*$%\1%'`
  if [ "$TIMESTAMP" == "" ]; then
    echo "ERROR: Failed to initialize $JOINING_HOST" >&2
    exit 1
  fi
  restart_check $JOINING_HOST $TIMESTAMP $LINENO

  # (2) Retrieve the joining host's configuration
  JOINER_CONFIG=`$CURL -X GET -H "Accept: application/xml" \
        http://${JOINING_HOST}:8001/admin/v1/server-config`
  echo $JOINER_CONFIG | grep -q "^<host"
  if [ "$?" -ne 0 ]; then
    echo "ERROR: Failed to fetch server config for $JOINING_HOST"
    exit 1
  fi

  # (3) Send the joining host's config to the bootstrap host, receive
  #     the cluster config data needed to complete the join. Save the
  #     response data to cluster-config.zip.
  $AUTH_CURL -X POST -o cluster-config.zip -d "group=Default" \
        --data-urlencode "server-config=${JOINER_CONFIG}" \
        -H "Content-type: application/x-www-form-urlencoded" \
        http://${BOOTSTRAP_HOST}:8001/admin/v1/cluster-config
  if [ "$?" -ne 0 ]; then
    echo "ERROR: Failed to fetch cluster config from $BOOTSTRAP_HOST"
    exit 1
  fi
  if [ `file cluster-config.zip | grep -cvi "zip archive data"` -eq 1 ]; then
    echo "ERROR: Failed to fetch cluster config from $BOOTSTRAP_HOST"
    exit 1
  fi

  # (4) Send the cluster config data to the joining host, completing 
  #     the join sequence.
  TIMESTAMP=`$CURL -X POST -H "Content-type: application/zip" \
      --data-binary @./cluster-config.zip \
      http://${JOINING_HOST}:8001/admin/v1/cluster-config \
      | grep "last-startup" \
      | sed 's%^.*<last-startup.*>\(.*\)</last-startup>.*$%\1%'`
  restart_check $JOINING_HOST $TIMESTAMP $LINENO
  rm ./cluster-config.zip

  echo "...$JOINING_HOST successfully added to the cluster."
done

Specifying Input Format

Most methods of the REST Management API accept input as XML or JSON. Some methods accept URL-encoded form data (MIME type application/x-www-form-urlencoded). Use the HTTP Content-type request header to indicate the format of your input.

For details, see the REST Management API Reference.

Specifying Expected Output Format

Many methods can return data as XML or JSON. The monitoring GET methods, such as GET /manage/v2/clusters, also support HTML.

The response data format is controlled through either the HTTP Accept header or a format request parameter (where available). When both the header and the parameter are present, the format parameter takes precedence.

For details, see the REST Management API Reference.

How Error Format is Determined

If a request results in an error, the body of the response includes error details. The MIME type of error details in the response is determined by the format request parameter (where supported), Accept header, or request Content-type header, in that order of precedence.

For example, if a request supplies XML input (request Content-type set to application/xml), but specifies JSON output using the format parameter, then error details are returned as JSON. If a request supplies JSON input, but no Accept header or format parameter, then error details are returned as JSON.

The default error detail format is XML.