MarkLogic Server 11.0 Product Documentation

Database Replication Guide — Chapter 1

Terms Used in this Guide

The following are the definitions for the Database Replication terms used in this guide:

• To Replicate is to create a copy of a document in another database and to keep that copy in sync (possibly with some time-lag/latency) with the original.
• A Local Cluster is the cluster of MarkLogic Server hosts at your current location.
• A Foreign Cluster is a remote cluster of MarkLogic Server hosts.
• A Master Database is the database being replicated. In any Database Replication scheme there is a Master database and at least one Replica database.
• A Master Forest is the forest being replicated. In any Database Replication scheme there is a Master forest and at least one Replica forest.
• A Master Cluster is shorthand for the cluster that hosts a Master Database.
• A Replica Database is the database that receives replicated data from the Master database.
• A Replica Forest is the forest that receives replicated data from the Master forest.
• A Replica Cluster is shorthand for the cluster that hosts a Replica Database.
• A Bootstrap Host is the MarkLogic Server host machine used by a foreign cluster to initiate communication with the local cluster.
• A Foreign Bind Port is the port used on each host to handle XDQP communication with foreign clusters.
• Asynchronous Replication refers to a configuration in which the Master does not wait for confirmation that the update has been received by the Replica before committing the transaction and proceeding with additional transactions. Database Replication is asynchronous.
• Transaction-aware refers to a configuration in which all updates that make up a transaction on the Master are applied as a single transaction on the Replica.
• Zero-day Replication refers to replicating the data from the Master database that existed before replication was configured.

Understanding Database Replication

Database Replication is the process of maintaining copies of forests on databases in multiple MarkLogic Server clusters. At a minimum, there will be a Master database and one Replica database. A Master database can replicate to multiple databases. A Replica database cannot serve as a Master database to another Replica.

All hosts participating in Database Replication must:

• Run the same maintenance release of MarkLogic Server.
• Use the same operating system.

This section contains the following topics:

• Overview of Database Replication
• Bulk Replication
• Bootstrap Hosts
• Inter-cluster Communication
• Replication Lag
• Master and Replica Database Index Settings

Example Database Replication Configurations

This section describes some of the possible Database Replication configurations.

The most basic form of Database Replication consists of a two clusters, each containing a single database with a single forest, as shown below.

Master and Replica forests can be distributed differently across hosts in each cluster. For example, as shown below, all of the Master forests may reside on a single host and the Replica forests on two hosts.

A Master cluster can replicate forests to multiple Replica clusters. In the example shown below, the Master maintains a copy of each forest on two clusters:

In the example shown below, the Master maintains a single copy of Database 1 on Replica Cluster 1 and a single copy of Database 2 on Replica Cluster 2:

A cluster can contain both Master and Replica databases. On Cluster A in the example shown below, Database 1 is the Master database and Database 2 is the Replica database.

Database Replication for Disaster Recovery

The purpose of Database Replication is to make data continuously available to mission-critical applications with minimal impact to application performance. In a typical disaster recovery scheme, clusters are located in data centers at different geographical locations. In the event of a disaster in the location of the Master cluster, the data can be made available by the Replica cluster

In this section, the original Master and Replica databases are referred to as Database A and Database B, respectively.

There are two basic approaches to a disaster recovery operation, each of which involves redirecting the applications to Database B. Before executing your applications on Database B, the database must be rolled back, as described in Rolling Back to the Non-Blocking Timestamp.

• Simply disable Database Replication on Database B so that it is no longer a Replica and can accept updates.

  

• Reconfigure Database B as the Master database and replicate updates to a third foreign database (Database C).

  

When Database A is returned to service, you can resynchronize it with Database B by either:

• Configuring Database A as a Replica of Database B. In this case, Database B will automatically bulk replicate to Database A.

  

• Running a backup operation on Database B and restoring Database A from the backup file, as described in Backing Up and Restoring a Database in the Administrator's Guide.

  

Once the data has been restored on the Database A, you can redirect the applications from Database B back to Database A.

Database Replication can be used in combination with the local-disk failover feature described in High Availability of Data Nodes With Failover and Configuring Local-Disk Failover for a Forest in the Scalability, Availability, and Failover Guide. For example, the Master and Replica clusters shown below are configured for both local-disk failover and Database Replication. Should, for example, the F1 forest in the Master cluster become unavailable, then the R1 forest resumes replication to the F1 forest in the Replica cluster.

Rolling Back to the Non-Blocking Timestamp

After you fail over your applications to a Replica database, each Replica forest will likely have committed its last transaction at different timestamps. If the Replica database has multiple forests and relationships exist between those forests, this inconsistency may cause problems. In order to return the database to a transactionally consistent state, all forests must be rolled back to the minimum non-blocking timestamp.

For example, the illustration below shows four forests and their committed transactions. Updates for each transaction are identified by the convention 'T#-u#' and commits are identified by a 'C'. Each forest completed its last commit at a different point in time when the failure of the Master database occurred. In this example, Forest A has only committed transactions up to timestamp 3 while Forest B has committed transactions up to timestamp 6. This means that, in order to return the database to a transactionally consistent state, all forests must be rolled back to at least timestamp 3.

The procedure described in this section can also be applied to the Master database, should you want to roll it back to some point after the last merge timestamp. When the Master database is rolled back, the rollback will be replicated to the Replica databases.

For example, the database you want to restore has four forests, as shown below. You use the xdmp:forest-status function to locate the nonblocking-timestamp value for each forest and roll back all of your forests to the minimum nonblocking-timestamp value. In this example, the minimum nonblocking-timestamp is the timestamp of the last committed transaction in Forest A.

The following procedure describes how to restore to the minimum non-blocking timestamp using the XQuery API.

1. Use the admin:database-get-merge-timestamp function to get the current merge timestamp. Save this value so it can be reset after you have completed the rollback operation.
2. Use the admin:database-set-merge-timestamp function to set the merge timestamp to any time before the failure of the master database. This will preserve fragments in merge after this timestamp until you have rolled back your forest data.
3. Use the xdmp:forest-rollback function to roll back the forests to the minimum nonblocking-timestamp returned by the xdmp:forest-status function for each forest that is in the open or open replica state.

   For example, you can use the following query to rollback your forest data for the Documents database:

   xquery version "1.0-ml";
   let $db := xdmp:database("Documents")
   let $timestamp := xdmp:database-nonblocking-timestamp($db)
   let $rollback := xdmp:forest-rollback(
       xdmp:forest-open-replica(
       xdmp:database-forests($db)),
       $timestamp)
   return
       "Roll back done"

4. Use admin:database-set-merge-timestamp function to set the merge timestamp back to the value you saved in Step 1.

   When the Security database is replicated, amps and triggers refer to Modules database IDs in the Master cluster. When failing over to the Security database on the Replica cluster, use the sec:amps-change-modules-database and trgr:triggers-change-modules-database functions to map Modules database IDs from the Master cluster IDs to the Replica cluster IDs.

   Starting in 9.0-7 for triggers and 10.0-2 for amps, Database names can be used in the trigger and amp creation apis, thus making it easy to support the same functionality on replica clusters for databases with the same names.

Upgrading Clusters Configured with Database Replication

If the Security database isn't replicated, then there shouldn't be anything special you need to do other than upgrade the two clusters.

If the security database is replicated, do the following:

1. Upgrade the Replica cluster and run the upgrade scripts. This will update the Replica's Security database to indicate that it is current. It will also do any necessary configuration upgrades.
2. Upgrade the Master cluster and run the upgrade scripts. This will update the Master's Security database to indicate that it is current. It will also do any necessary configuration upgrades.

   If the effective version of the replica database is higher than the master database, database replication should still work. If the effective version of replica database is lower than the master database, database replication will be paused.

Comparison of Database Replication with Flexible Replication

MarkLogic Server provides two types of replication: Database Replication, as described in this guide, and Flexible Replication, as described in the Flexible Replication Guide. You should choose a replication type based on your application and high-availability requirements

Depending on your requirements, one type of replication might have an advantage over the other.

The key differences between the these two types of replication are:

• Replicate all or part of the Master database -- Database Replication replicates the entire Master database to the Replica database. Flexible Replication replicates Content Processing Framework (CPF) domains and allows you to write XQuery filters that enable you to modify the replicated documents in some manner, determine whether to replicate a change, or select which parts of a document will be replicated. As a consequence, Flexible Replication enables you to replicate select content from the Master database.
• Performance -- Flexible Replication uses CPF and generates information in the properties of each replicated document. Database Replication simply copies and replays journal frames. If you replicate a complete database, you can expect less overhead with Database replication than with Flexible Replication.
• Transactional -- Database Replication replicates as soon as a document is inserted or updated and ensures that a single multi-document transaction in the Master database replicates as a single multi-document transaction to the Replica database. Flexible Replication replicates documents after each transaction without consideration of transactional grouping in the Master database. For example, if you have a transaction that commits 10 documents to the Master database and some sort of failure occurs, with Flexible Replication it is possible for the Replica to end up with 5 documents from that transaction on the Replica. With Database Replication, the Replica will either have no documents or all 10 of them.