MarkLogic Server 11.0 Product Documentation

MarkLogic Server on Amazon Web Services (AWS) Guide — Chapter 4

Accessing a MarkLogic Server Instance

This section describes how to use the Elastic Load Balancer (ELB) URL to access an instance of MarkLogic Server in EC2. The ELB URL will direct you to any available instance of MarkLogic Server in your cluster. If you want to access a specific instance, as you would when running the mlcmd script described in Using the mlcmd Script, then use the Public DNS for that instance.

You can access the MarkLogic Admin Interface through the ELB by clicking on the URL in the Outputs portion of the Cloud Formation Console, as described in 13 in Creating a CloudFormation Stack using the AWS Console.

This section describes how to access MarkLogic Server through the ELB from the EC2 Dashboard.

1. In the EC2 Dashboard, click on Load Balancers in the left-hand navigation menu.
2. Select a Load Balancer from the list and copy URL from the DNS name.

You use the URL to access the MarkLogic Server. You can access any of the ports you have defined as an ELB port. For example, if the URL is DLEE-CF-L-ElasticL-OCCR192PW0OO-925510329.us-east-1.elb.amazonaws.com, then, to access MarkLogic port 8000, the URL you enter into the browser would be:

http://DLEE-CF-L-ElasticL-OCCR192PW0OO-925510329.us-east-1.elb.amazonaws.com:8000

Accessing an EC2 Instance

You may need to SSH into an EC2 instance for certain task, such as checking the log files for that instance, as described in Detecting EC2 Errors.

You cannot SSH to the load balancer, you must SSH to a specific EC2 instance. To SSH into an EC2 instance, you must have the key pair used by the instance downloaded to your local host.

To SSH into an instance, do the following:

1. Open the EC2 Dashboard.
2. Select Instances from the left-hand navigation section.
3. Select the instance to which you want to connect.
4. Click on the Connect button, or select Connect from the Actions pull-down menu.

1. Select A Java SSH Client directly from my browser (Java required). Specify ec2-user as the User Name, and provide the path to your copy of the key pair you downloaded to your local host. Click Launch SSH Client.

   

2. This will open up a shell window to the EC2 instance. When you first connect in this manner, you may be prompted to create various directories. Respond by clicking Yes for each prompt.

Alternatively you can open a shell window and SSH into an instance using the following command:

ssh -i /path/to/keypair.pem ec2-user@<Public DNS>

For example, if your keypair, named newkey.pem, is stored in your c:/stuff/ directory, you can access the instance with a public DNS of ec2-54-242-94-98.compute-1.amazonaws.com as follows:

ssh -i c:/stuff/newkey.pem ec2-user@ec2-54-242-94-98.compute-1.amazonaws.com

Detecting EC2 Errors

Start up errors are stored in the /var/log/messages file in each instance. To view the messages file, SSH into an instance as described in Accessing an EC2 Instance.

To access the messages file, you must be super user. For example, if you want to tail the messages file, enter:

sudo tail -f /var/log/messages

You can also capture errors related to Cloud Formation stack by means of the SNS Topic, as described in Creating a Simple Notification Service (SNS) Topic.

Using the mlcmd Script

The mlcmd script supports startup operations and advanced use of the Managed Cluster features. The mlcmd script is installed as an executable script in /opt/MarkLogic/bin/mlcmd.

In order to run mlcmd, you must be logged into the host and running as root or with root privileges. You must also have Java installed and the java command in the PATH or JAVA_HOME set to the JRE or JDK home directory. The first time you start MarkLogic on your server the /var/local/mlcmd.conf file is created, which is required to use the mlcmd script. Once the /var/local/mlcmd.conf file is created, it is not necessary to start MarkLogic to use the mlcmd script.

The syntax of mlcmd is as follows:

mlcmd command

The mlcmd commands are listed below:

Configuring MarkLogic for Amazon Simple Storage Service (S3)

Amazon S3 storage can be used for backups and as a read-only forest in a tiered storage configuration. MarkLogic does not support regular or replica forests on Amazon S3 storage.

Amazon S3 support is built into MarkLogic Server as an available file system type. You configure S3 access at the group level. Once you have configured a group for S3, any forest in the group can be placed on S3 by specifying an S3 Path. Additionally, any host in the group can do backups to S3, restore from S3, as well as read and write directories and files on S3.

Transaction journaling does not work on S3 because the S3 file system cannot do the file operations necessary to maintain a journal. Unless your S3 forest is configured with a fast data directory or updates allowed is set to read-only, you must set the journaling option on your database to off before attaching the forest to the database. This is not a requirement for backup/restore operations on a database, however.

To configure MarkLogic to access Amazon S3, do the following:

• Set up an S3 Bucket
• Configure the S3 Endpoint for your Group
• Configure AWS Credentials
• Set an S3 Path in Forest Data Directory
• Load Content into MarkLogic to Test

s3://bucket/directory/file

s3://my-company-bucket
s3://my-company-bucket/directory
s3://my-company-bucket/dir1/dir2/dir3

s3://my-company-bucket/file.xml
s3://my-company-bucket/directory/file.txt
s3://my-company-bucket/dir1/dir2/dir3/file.txt

Configuring a VPC for MarkLogic Telemetry

If you have an existing VPC or if you are not using the VPC template described in Deploying MarkLogic on EC2 Using CloudFormation, follow the procedure in this section to configure your VPC for the MarkLogic telemetry feature described in Telemetry in the Monitoring MarkLogic Guide.

MarkLogic telemetry requires that the subnets containing your EC2 instances be configured for outbound access. To configure your VPC to enable your EC2 instances to access the outside internet, do the following:

1. Create a NAT Gateway, as described in:

   https://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/vpc-nat-gateway.html#nat-gateway-working-with

   Note the ID of the NAT Gateway.

2. Create a route table for the subnets containing your EC2 instances, as described in:

   https://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Route_Tables.html#WorkWithRouteTables

3. In the route table, add the following route:
   • Destination: 0.0.0.0/0
   • Target: NAT Gateway ID

Configuring a VPC for MarkLogic Clients

If you have an existing VPC or if you are not using the VPC template described in Deploying MarkLogic on EC2 Using CloudFormation, follow the procedure in this section to configure your VPC to enable access by clients, such as mlcp, Java Client API, DMSDK, Node.js Client API, and so on.

MarkLogic clients require that the subnets containing your EC2 instances be configured for inbound access. The procedures for this are as follows.

1. Create an Internet Gateway and attach it to your VPC, as described in:

   https://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Internet_Gateway.html

   Note the ID of the Internet Gateway.

If your EC2 instance is in a public subnet, do the following:

1. Enable the public IP address on the EC2 instance, as described in:

   https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-instance-addressing.html#concepts-public-addresses

2. Create a route table for the subnets containing your EC2 instances, as described in:

   https://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Route_Tables.html#WorkWithRouteTables

3. In the route table, add the following route:
   • Destination: 0.0.0.0/0
   • Target: Internet Gateway ID

If your EC2 instance is in a private subnet, do the following:

1. Create a public subnet, as described in:

   https://docs.aws.amazon.com/appstream2/latest/developerguide/managing-network-internet-default.html

2. Follow Steps 2 and 3, above.

Scaling Cluster Resources on EC2

If you have created your stack using the 3+ Cloud Formation template, you can temporarily add nodes and forests to scale up your cluster for periods of heavy use and then remove them later when less resources are needed.

Adding more hosts to a cluster is simple. Simply use the Update Stack feature to reapply the 3+ Cloud Formation template and provide a larger number for the NodesPerZone setting. Alternatively, you can add hosts by means of your Auto Scaling Groups. The recommended way to scale up the data capacity of your cluster is to add additional volumes, as described in Creating an EBS Volume and Attaching it to an Instance.

Scaling a cluster down involves some manual intervention. The procedure is as follows:

1. Use MarkLogic and AWS tools to identify equal number of hosts in each ASG to delete. Never delete the host with the Security database, or any of the other built-in MarkLogic databases, such as Meters, App-Services, Modules, and so on.
2. Delete or move the data from the hosts to be removed to other hosts. This can be done by using the REST Management API or XQuery tieredstorage API to migrate partitions or forests to a volume on another host. For details on migrating data, see Migrating Forests and Partitions in the Administrator's Guide.
3. As a super user, run the leave-cluster -terminate command on each host to be removed. This will cause the node to leave the cluster, and adjust the AutoScaling Group DesiredCount setting. For details, see leave-cluster.
4. Delete any unused volumes.
5. Update the Cloud Formation template to represent downsized cluster and use the Update Stack feature to reapply the template to the stack to alert AWS of the updated configuration.

Upgrading MarkLogic on AWS

You can upgrade from any version of MarkLogic instance on AWS to a later version. To upgrade a MarkLogic instance on AWS, you must update the AMI IDs in your original CloudFormation template.

You cannot upgrade your CloudFormation template to a different version. You must keep your original CloudFormation template. Upgrades are supported by updating the AMI IDs in your original CloudFormation template with the AMI IDs from the MarkLogic version to which you are upgrading.

To use a later version of the CloudFormation template, you can set up a new cluster, move data and configuration to the new template, and switch to the new cluster after testing.

The following procedure describes how to upgrade your stack to use AMI IDs for a later version of MarkLogic:

1. Download the CloudFormation template for the later version of MarkLogic at https://developer.marklogic.com/products/cloud/aws.
2. Navigate to your AWS CloudFormation console, and select your original CloudFormation template used to create the stack. Download your original template currently used by the stack to ensure any modifications made to the CloudFormation template are preserved after the upgrade.
   • Click Actions.
   • In the Actions menu, click View/Edit template in Designer.
   • To download the template, click the file icon > Save.
   • Select Local File, name the file, and click Save.
3. Locate the AMI IDs in your downloaded original template, and find the corresponding AMI IDs in the new template. For example, the LicenseRegion2AMI and AWSRegionArch2AMI definitions in your original template might look like the following:

   CloudFormation templates for MarkLogic 9.0-5 or a later version of MarkLogic are in YAML data format. CloudFormation templates for MarkLogic 9.0-4 or an earlier version of MarkLogic are in JSON data format.

   YAML

   LicenseRegion2AMI
    us-east-1:
     Enterprise: ami-08dcfd988a2988e01
     BYOL: ami-078c0f5757e1ffdd7

   JSON

   "AWSRegionArch2AMI": {
    "us-east-1": {
     "HVM": "ami-4464743e"
    },
   ...
   }

   If, for example, your instance is located in the us-east-1 region, open the new template, locate LicenseRegion2AMI, and copy the AMI IDs for the us-east-1 region. For example, the new template contains the following:

    us-east-1
     Enterprise: ami-0ac019c39cac73c89
     BYOL: ami-0ea837234c4c34363

   In the original template, you can modify LicenseRegion2AMI or AWSRegionArch2AMI as follows:

   YAML

   LicenseRegion2AMI
    us-east-1:
     Enterprise: ami-0ac019c39cac73c89
     BYOL: ami-0ea837234c4c34363

   JSON

   "AWSRegionArch2AMI": {
    "us-east-1": {
     "HVM": "ami-0ac019c39cac73c89"
    },
   ...
   }

4. Take a backup for each database on your cluster. If the backup fails, use MarkLogic database backup and restore. For details, see Backup and Restore Overview in the Administrator's Guide.

   MarkLogic database backup and restore does not back up your configuration.

   Do not use EBS Volume snapshot for backup purposes unless MarkLogic is completely stopped at the time of the snapshot.

5. Navigate to your AWS CloudFormation console, and select the CloudFormation template used to create the stack.
   • Click Update.
   • Upload the modified CloudFormation template.
6. In the EC2 Dashboard, terminate one instance at a time, called a rolling upgrade. After terminating an instance, wait for the terminated instance to come up and reconnect without any UI interaction. After the instance comes up, navigate to the Admin Interface to check the software version installed on each host. For details, see Understanding Rolling Upgrades in the Administrator's Guide.

   Before terminating your instance, MarkLogic recommends you stop all traffic to the cluster during the upgrade.

   You must revert forest replicas back to the original primary hosts. Otherwise, you will experience degraded performance. For details, see Reverting a Failed Over Forest Back to the Primary Host in the Scalability, Availability, and Failover Guide.

7. After terminating your last instance, navigate to the Admin Interface (8001 port) where the security database upgrade prompt will be displayed. (security-upgrade.xqy screen)
8. Click OK and wait for the upgrade to complete on the instance.

Monitoring (CloudWatch)

AWS provides robust monitoring of EC2 instances, EBS volumes, and other services via the CloudWatch service. You can use CloudWatch to set thresholds on individual AWS services and send notifications via SMS or Email when these thresholds have been exceeded. For example, you can set a threshold on excessive storage throughput. You can also create your own metrics to monitor with CloudWatch. For example, you might write a custom metric to monitor the current free memory on your instances and to alarm or trigger an automatic response should a memory threshold be exceeded.

For details on the use of CloudWatch, see http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/WhatIsCloudWatch.html.

Migrating from Enterprise Data Center to EC2

This section describes to steps for migrating your data and configuration from a data center to EC2.

There are a number of ways you could migrate from a local data center to EC2. The following is one possible procedure.

1. Copy an existing backup of your databases to S3 storage. If you prefer to backup your databases directly to S3, set your S3 security credentials, as described in Configure AWS Credentials, on your local MarkLogic cluster and, for your backup directory, provide the path to your S3 bucket, as described in Set an S3 Path in Forest Data Directory.
2. Export all of the configuration data for your cluster.
3. Create a Cloud Formation template, as described in Deploying MarkLogic on EC2 Using CloudFormation, to recreate the hosts for your cluster on EC2.
4. Import your configuration data into your EC2 cluster.
5. Restore your backed-up data from S3 to your configured EC2 forests.

   For more detail about exporting and importing a configuration, see https://help.marklogic.com/Knowledgebase/Article/View/686/0/transporting-configuration-to-a-new-cluster.

Creating an EBS Volume and Attaching it to an Instance

This section describes how to create an EBS volume and attach it to your MarkLogic Server instance.

In general, it is a best practice to have one volume per node and one forest per volume. The recommendation is to use large EBS volumes as opposed to multiple smaller ones because larger EBS volumes (gp2) have faster IO as described by the Amazon EBS Volume types and you have to keep enough spare capacity on each EBS volume to allow for merges. Additionally, it is recommended that you have one large EBS data volume per node - while it's possible to have multiple volumes per instance, it is not typically worth the additional administrative complexity. When resizing, adopt a vertical scaling approach (growing into a single bigger EBS volume versus adding multiple smaller volumes per node). Note that S3 storage is eventually consistent, therefore S3 can only be used for backups or read-only forests in MarkLogic Server (otherwise you risk the possibility of data loss).

Hibernating a MarkLogic Cluster

At any time, you can hibernate the cluster by using the Update Stack feature to reapply the Cloud Formation template to your stack and setting the NodesPerZone value to 0. You must later restart the cluster by resetting the NodesPerZone value to the previously set value. For example, assume the NodesPerZone value is 3. Set the value to 0 to hibernate the cluster, and reset the value back to 3 to restart the cluster.

You must revert all forest replicas back to their original primary hosts. To do this, restart the server. For details, see Restarting the Server in the Administrator's Guide.

Hibernating a MarkLogic cluster deletes all nodes and related root volumes in your cluster. Restarting a hibernated MarkLogic cluster recreates nodes and related root volumes. In addition, data volumes are preserved and reattached to nodes.

Do not manually stop your MarkLogic instances from the EC2 dashboard, as each AutoScaling Group will detect that they have stopped and will automatically recreate them. The same is true if you shutdown MarkLogic from the Admin Interface, by means of a MarkLogic API call, or with the appropriate system command for your platform.

Resizing a MarkLogic Cluster

At any time, you can resize the cluster by using the Update Stack feature to change the instance type. Before you resize the cluster, you must hibernate the cluster by setting the NodesPerZone value to 0. After you resize the cluster, you must restart the cluster by resetting the NodesPerZone value to the previously set value, as described in Hibernating a MarkLogic Cluster. Resizing a MarkLogic Cluster on AWS can be done horizontally and not just vertically (as described above), similar to how it is done with on-premise deployments. For changing the number of instances (horizontal scaling), use the Update Stack feature by changing the NodesPerZone setting on the CFT. Alternatively, you can use the auto-scaling groups. For more details, see Scaling Cluster Resources on EC2. Data capacity can be resized in two different ways: resizing using AWS snapshot and resizing using MarkLogic's rebalancing feature. While vertical scale out is significantly easier on AWS versus on-premise deployments, note that MarkLogic requires at least some degree of horizontal scaling as high availability (HA) requires at least three nodes in a cluster. Whether you are scaling nodes or data capacity, horizontally or vertically, it is recommended to: test your scale out procedure thoroughly before implementing as well as take full backups of your data before making changes to your cluster.

MarkLogic recommends taking a full backup before changing the instance type.

Terminating a MarkLogic Cluster

To terminate your MarkLogic cluster, you can delete the stack, as described in Deleting a CloudFormation Stack.