MarkLogic 9 Product Documentation

MarkLogic Server on Amazon Web Services (AWS) Guide — Chapter 1

AWS Terminology

The following are the definitions for the terms used in this guide:

Amazon Web Services (AWS) is the Amazon Cloud Computing service. For details, see http://aws.amazon.com/.

Elastic Compute Cloud (EC2) is an AWS service that enables you to launch and manage server instances in Amazon's data centers using APIs or available tools and utilities. The AWS EC2 website is available at: http://aws.amazon.com/ec2/.

Virtual Private Cloud (VPC) lets you provision a logically isolated section of the AWS Cloud where you can launch AWS resources in a virtual network defined by you. For details, see https://aws.amazon.com/vpc/.

CentOS is a community-supported, free and open source operating system based on Red Hat Enterprise Linux.

An Elastic Load Balancer (ELB) is a service that automatically distributes and balances application traffic among multiple EC2 instances. For details, see http://docs.aws.amazon.com/gettingstarted/latest/wah/getting-started-create-lb.html.

Elastic Network Interface (ENI) is a virtual network interface that you can attach to an instance in a VPC. Network interfaces are available only for instances running in a VPC. For details, see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html.

AWS Lambda lets you run code without provisioning or managing servers. You pay only for the compute time you consume; there is no charge when your code is not running. For details, see https://aws.amazon.com/lambda/.

Amazon Machine Image (AMI) is an encrypted machine image that contains all information necessary to boot instances of software. Instances of MarkLogic Server are created from the stock Amazon Linux and Linux 2 AMI and have been pre-installed with MarkLogic and the necessary dependancies.

Elastic Block Store (EBS) is a type of storage designed specifically for Amazon EC2 instances. Amazon EBS allows you to create volumes that can be mounted as devices by Amazon EC2 instances. Amazon EBS volumes behave like raw unformatted external block devices. They are attached to user-specified block devices and provide a block device interface. You can load a file system on top of Amazon EBS volumes, or use them just as you would use a block device. Amazon EBS volumes exist separately from the actual instances and persist until you delete them. This allows you to store your data without leaving an Amazon EC2 instance running. Each Amazon EBS volume can be up to 16 TiB in size.

An Instance is the running system after an AMI is launched. Instances remain running unless they fail or are terminated. When this happens, the data on the instance is no longer available. Once launched, an instance looks very much like a traditional host.

An Instance Type defines the size of an Amazon EC2 instance. The MarkLogic Server instance types are shown in the table at the end of 5 in Creating a CloudFormation Stack using the AWS Console.

An Instance Store (sometimes referred to as Ephemeral Storage) is a fixed amount of storage space for an instance. An instance store is not designed to be a permanent storage solution. If an instance reboots, either intentionally or unintentionally, the data on the instance store will survive. If the underlying drive fails or the instance is terminated, the data will be lost.

AWS Cloud Storage (S3) is an Amazon web services interface that can be used to store and retrieve any amount of data, at any time, from anywhere on the web. For details, see Configuring MarkLogic for Amazon Simple Storage Service (S3) and https://aws.amazon.com/s3/.

Cloud Formation (CF) is the AWS Cloud Formation service for provisioning startup of AWS resources. For details, see Deploying MarkLogic on EC2 Using CloudFormation and http://aws.amazon.com/cloudformation/. The MarkLogic CloudFormation templates are available from http://developer.marklogic.com/products/cloud/aws.

Key Management Service (KMS) is a an AWS service that provides secure location, known as a keystore, where the encryption keys used to encrypt data are stored. This AWS KMS is not to be confused with the internal MarkLogic KMS described in Overview of MarkLogic Server on AWS in the Security Guide. The AWS KMS can be used as an external KMS. To access AWS KMS, MarkLogic must be configured to use AWS Credentials, as described in Configure AWS Credentials. A MarkLogic cluster inside a VPC (with or without a public DNS) will work with AWS KMS. For details on AWS KMS, see https://aws.amazon.com/kms/.

Managed Clusters is a MarkLogic feature that works with AWS features to automatically create and provision the necessary AWS resources and provide MarkLogic with the information needed to manage your cluster. For details, see The Managed Cluster Feature.

MarketPlace is the AWS service for publishing pay-per-use and free (no extra charge) public AMI's on amazon. For details, see https://aws.amazon.com/marketplace.

An EC2 Compute Unit (ECU) provides the equivalent CPU capacity of a 1.0-1.2 GHz 2007 Opteron or 2007 Xeon processor.

Metadata Database is the database that stores and indexes all of the configuration data required to manage a cluster of one or more MarkLogic Servers. For AWS, the DynamoDB service is used to implement the Metadata Database. For details, see http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/GettingStartedDynamoDB.html.

The Managed Cluster Feature

Running MarkLogic Server in AWS has some challenges that you may not experience in traditional IT data centers. The Managed Clusters feature helps you mitigate these challenges with support for reliability, scalability and high availability, as well as with tools that automatically handle some of the more problematic issues. It is highly recommended that you use the CloudFormation templates and follow the procedures in this guide to launch your MarkLogic AMI in AWS as they are provided to help you leverage AWS and MarkLogic features especially designed for a reliable and easy cloud deployment.

The Managed Cluster feature now supports SSL-enabled clusters. For details, see Configuring SSL on App Servers in the Security Guide.

On AWS, the following should be considered before deploying MarkLogic nodes or clusters:

• Instance Hostnames -- An EC2 instance starts life with a unique hostname. If this instance is stopped for any reason (such as a hardware/software failure or manual stopping to avoid EC2 charges), when it restarts it will start with a new hostname. This causes configuration problems, especially in clusters, but also in single-node installations. If you stop a cluster then bring it back online without reconfiguring it, the nodes will not rejoin the cluster because all the hostnames are different.

  The Managed Cluster feature automatically detects hostname changes and propagates the changes throughout the cluster.

• Transient Data -- When an EC2 instance is terminated, the root volume is released and all of the data on it is lost. This includes any pre-installed software or OS configurations. MarkLogic installations should not be put on the root volume, but rather on an attached EBS volume.

  The Managed Cluster feature automatically keeps track of each EBS volume, along with its related EC2 instance and mount directory. When you restart your EC2 instances, the Managed Cluster feature automatically re-attached and mounts your volumes to the appropriate locations to ensure that your Forests and Databases are intact.

• AWS Security -- If you want to access any AWS services from within an EC2 node you need authentication. One way to do this is to provide your AWS credentials and store them on the EC2 instance. However, this is both inconvenient and a security risk. The Managed Cluster feature and CloudFormation templates use IAM Roles so that you never need to expose your AWS Credentials.
• Securing your Infrastructure -- EC2 instances by default are open to the world. You need to consider your security requirements before launching an EC2 instance and putting sensitive data on it. The Cloud Formation Templates create baseline security groups to allow easy startup. You can edit these security groups to your particular security needs.
• Volatile Instances -- EC2 instances can fail at any time. This is true of on-premise hardware as well, but in the cloud you are not there to fix this yourself. Instead you need to rely on AWS features to restart failed instances to keep your database up and running at all times. The CloudFormation templates create a suggested topology with AutoScaling groups across zones as needed that distribute your cluster and restart unhealthy instances automatically. The Managed Cluster support makes sure that restarted instances rejoin the cluster.
• Datacenter Failures -- Datacenters can and do fail. Often do to unpredictable issues, like hurricanes or earthquakes. If you want high availability you need to plan for failure by distributing your cluster across datacenters, so that any single failure does not take your cluster offline. The CloudFormation templates create a suggested topology with AutoScaling groups distributed across zones in a single region for maximum fault tolerance.
• Load Balancing and Routing -- Your MarkLogic cluster should only be accessible when it is healthy. Use of AWS ELBs is highly recommended even for single nodes, but especially when running a cluster. The ELBs not only balance traffic across healthy nodes, but automatically notify the AutoScaling groups when a node is unhealthy, even if the hardware and OS are running fine, so that traffic is diverted to healthy nodes and the unhealthy nodes are terminated and restarted.

  The Managed Cluster feature provides a Health Check application on each server that is used by the Load Balancer to detect if your MarkLogic instance is healthy.

  If you are using a XCC client (such as mlcp) with MarkLogic running on AWS, you must enable the xcc.httpcompliant setting to work with AWS ELBs. For details, see Using a Load Balancer or Proxy Server with an XCC Application in the XCC Developer's Guide.

• Scaling Up and Down -- Sometimes you want more capacity sometimes you want less. Cloud Computing provides the raw tools to enable scaling up and down, but your software needs to understand and integrate with provisioning changes. The CloudFormation templates allow an easy one-step process for scaling your cluster capacity up and down.
• License Application on a Cluster -- When launching a cluster, you traditionally need to apply your license to each node manually after they come up for the first time. The Managed Cluster feature automatically applies your license keys to all of the nodes in the cluster.
• Cluster Formation -- When running a MarkLogic cluster, you need to configure the first node with your Administrator credentials, then configure remaining nodes to connect to the cluster. The Managed Cluster feature automatically handles this for you so clusters (even clusters of one) come up ready to run without further manual intervention.
• Hibernating Clusters -- A great benefit of Cloud Computing is that you only pay for what you use. If you are running a development server, or a site that doesn't need to be running 24/7, you can hibernate your entire cluster so you don't incur EC2 charges while it is not running. The Managed Cluster feature along with CloudFormation enable you to quickly restart the cluster and have your resources re-attach to all of your data, so that your cluster will be up and running where left off.

CloudFormation is not required to make use of the Managed Clusters feature, instead you can choose to manually or programmatically configure the AWS resources using other tools, but it is a challenging task without strong cloud orchestration and management tools. CloudFormation allows you to both document and implement a managed cluster configuration using a simple declarative template that can grow with your needs.

Running MarkLogic without the Managed Clusters feature is also supported (with or without our provided AMI's) and is the simplest configuration. However it is also the least reliable and is not recommended.

Launching a MarkLogic AMI outside of CloudFormation

This section describes the minimum steps you need to take should you insist on running MarkLogic without either CloudFormation or your own or third party cloud management tools. These steps do not enable the Managed Cluster feature and are not recommended.

• Launch an EC2 AMI, either one we provide in MarketPlace or your own AMI on which you have pre-installed MarkLogic, following the OS-specific installation instructions.
• Create a Key Pair, as described in Creating a Key Pair. You will need the Key Pair name to launch the instance.
• Create a Security Group that opens at port 22 (for ssh) and ports (8000-8002) for the Admin API. You may also need to configure the Security Group to open additional ports for your own applications or for foreign clusters.
• Create an EBS volume for your data and attach it to the logical device, /dev/sdf. On startup, MarkLogic will detect this volume (or wait for it to be attached) then create a filesystem and mount it as /var/opt/MarkLogic. The MarkLogic AMI will have the data volume pre-defined and it must be associated with /dev/sdf.

You may now go to the admin page on port 8001 and continue configuring your MarkLogic instance.