Scripting Administrative Tasks Guide — Chapter 7

Preliminary Configuration Procedures

The procedures described in this chapter assume you have done the following configuration on MarkLogic Server:

1. Create three new forests:
   • MasterForest
   • MyTriggersForest
   • ReplicaForest

     The MasterForest and MyTriggersForest must be on the same server. For details on how to create forests, see Creating Forests and Databases.

2. Create three new databases:
   • Master
   • MyTriggers
   • Replica

     The Master and MyTriggers databases must be on the same server. For details on how to create databases, see Creating Forests and Databases.

3. Attach the forests to the databases:
   • MasterForest to Master
   • MyTriggersForest to MyTriggers
   • ReplicaForest to Replica

     For details on how to attach forests to databases, see Attaching Forests to Databases.

4. Configure the Master database to use the MyTriggers database as its Triggers Database:

   xquery version "1.0-ml";
   import module namespace admin = "http://marklogic.com/xdmp/admin" 
     at "/MarkLogic/admin.xqy";
   let $config := admin:get-configuration()
   let $config := admin:database-set-triggers-database(
     $config, xdmp:database("Master"),
     xdmp:database("MyTriggers"))
   return admin:save-configuration($config)

5. For the MyTriggers database:
   • Insert a Flexible Replication pipeline and a Status Change Handling pipeline, as described in Inserting Existing CPF Pipelines.
   • Create a CPF domain, named Replicated Content, that uses the Flexible Replication and Status Change Handling pipelines, as described in Creating a CPF Domain.
6. Create two HTTP App Servers with the following settings

   Server Name	Root	Port	Database
   Master-flexrep	FlexRep	8010	Master
   Replica-flexrep	FlexRep	8011	Replica

   For details on how to use the Admin API to create App Servers, see Creating an App Server.

Configuring the Master Database

The flexrep:configure-databasefunction creates the indexes needed by the Master database for CPF based replication.

xquery version "1.0-ml"; 
import module namespace flexrep =
  "http://marklogic.com/xdmp/flexible-replication" 
  at "/MarkLogic/flexrep.xqy";
import module namespace admin = "http://marklogic.com/xdmp/admin"
  at "/MarkLogic/admin.xqy";
let $config := admin:get-configuration() 
let $config := flexrep:configure-database(
  $config,
  xdmp:database("Master")) 
return admin:save-configuration($config)

Creating a Replication Configuration Element

Most of the Flexible Replication API functions require a replication configuration element for each replicated domain. You create a replication configuration element by calling the flexrep:configuration-createfunction and insert it into the database by calling the flexrep:configuration-insertfunction.

The following query creates a new replication configuration element for the Replication Content domain and inserts it into the database. This query is executed against the Master database, so an xdmp:evalfunction is used to obtain the domain ID from the MyTriggers database.

xquery version "1.0-ml";
import module namespace flexrep =
  "http://marklogic.com/xdmp/flexible-replication" 
    at "/MarkLogic/flexrep.xqy";
(: Obtain the id of the replicated CPF domain from the 
   Triggers database. :)
let $domain:= xdmp:eval(
    'xquery version "1.0-ml";
    import module namespace dom = "http://marklogic.com/cpf/domains" 
      at "/MarkLogic/cpf/domains.xqy";
    fn:data(dom:get( "Replicated Content" )//dom:domain-id)',
    (),
    <options xmlns="xdmp:eval">
      <database>{xdmp:database("MyTriggers")}</database>
    </options>)
(: Create a replication configuration for the Replicated 
   Content domain. :)
let $cfg := flexrep:configuration-create($domain)
(: Insert the replication configuration element into the database. :)
return flexrep:configuration-insert($cfg) 

Creating a Replication Target

This section describes how to use the flexrep:target-createfunction to create a replication target. The following query is executed against the Master database, so an xdmp:eval function is used to obtain the domain id from the MyTriggers database.

xquery version "1.0-ml";
import module namespace flexrep =
  "http://marklogic.com/xdmp/flexible-replication" 
    at "/MarkLogic/flexrep.xqy";
(: Obtain the id of the replicated CPF domain from the 
   Triggers database. :)
let $domain:= xdmp:eval(
    'xquery version "1.0-ml";
    import module namespace dom = "http://marklogic.com/cpf/domains" 
      at "/MarkLogic/cpf/domains.xqy";
    fn:data(dom:get( "Replicated Content" )//dom:domain-id)',
    (),
    <options xmlns="xdmp:eval">
      <database>{xdmp:database("MyTriggers")}</database>
    </options>)
(: Obtain the replication configuration. :)
let $cfg := flexrep:configuration-get($domain, fn:true()) 
(: Specify the HTTP options for the replication target. :)
let $http-options := 
  <flexrep:http-options     xmlns:flexrep="http://marklogic.com/xdmp/flexible-replication">
    <http:authentication xmlns:http="xdmp:http">
      <http:username>admin</http:username>
      <http:password>admin</http:password>
    </http:authentication>
    <http:client-cert xmlns:http="xdmp:http"/>
    <http:client-key xmlns:http="xdmp:http"/>
    <http:pass-phrase xmlns:http="xdmp:http"/>
  </flexrep:http-options>
(: Create the replication target. :)
let $cfg := flexrep:target-create(
  $cfg,
  "Replica",
  "http://localhost:8011/",
  60,
  300,
  10,
  fn:true(),
  $http-options,
  fn:false(),
  (),
  () ) 
(: Insert the changes to the replication configuration. :)
return flexrep:configuration-insert($cfg) 

Creating a Push Replication Scheduled Task

This section describes how to use the scheduler functions to create a scheduled replication push task. The following query is executed against the Master database.

xquery version "1.0-ml";
import module namespace admin = "http://marklogic.com/xdmp/admin" 
  at "/MarkLogic/admin.xqy";
let $config := admin:get-configuration()
(: Define a minutely scheduled task to push replication 
   each minute. :)
let $task := admin:group-minutely-scheduled-task(
  "/MarkLogic/flexrep/tasks/push.xqy",
  "Modules",
  1,
  xdmp:database("Master"),
  0,
  xdmp:user("admin"), 
  admin:host-get-id($config, xdmp:host-name())
)
(: Add the scheduled task to the Default group. :)
let $config:= admin:group-add-scheduled-task(
  $config, 
  admin:group-get-id($config, "Default"), 
  $task)
return admin:save-configuration($config)

Disabling Push Replication on the Master Database

Before configuring Pull Replication on the Replica database, you must disable Push Replication on the Master database. The following query is executed against the Master database.

xquery version "1.0-ml";
import module namespace flexrep =
  "http://marklogic.com/xdmp/flexible-replication" 
    at "/MarkLogic/flexrep.xqy";
(: Obtain the id of the replicated CPF domain from the 
   Triggers database. :)
let $domain:= xdmp:eval(
    'xquery version "1.0-ml";
    import module namespace dom = "http://marklogic.com/cpf/domains" 
      at "/MarkLogic/cpf/domains.xqy";
    fn:data(dom:get( "Replicated Content" )//dom:domain-id)',
    (),
    <options xmlns="xdmp:eval">
      <database>{xdmp:database("MyTriggers")}</database>
    </options>)
(: Obtain the replication configuration. :)
let $cfg := flexrep:configuration-get($domain, fn:true()) 
(: Obtain the replication target id. :)
let $target-id := flexrep:configuration-target-get-id(
  $cfg,
  "Replica")
(: Disable Push Replication on the replication target. :)
let $cfg := flexrep:configuration-target-set-enabled(
  $cfg,
  $target-id,
  fn:false())
(: Insert the replication configuration element into the database. :)
return flexrep:configuration-insert($cfg) 

Creating a Pull Replication Configuration

Pull Replication requires a pull replication configuration element for each replicated domain. You create a pull replication configuration element by calling the flexrep:pull-create function and insert it into the database by calling the flexrep:pull-insert function.

The following query is executed against the Replica database, which is most likely running on a different server than the Master database. As a consequence, you will need to first obtain the domain ID from the master database and the target ID from the master's triggers database.

xquery version "1.0-ml";
import module namespace flexrep =
  "http://marklogic.com/xdmp/flexible-replication" 
    at "/MarkLogic/flexrep.xqy";
(: Specify the id of the replicated CPF domain obtained from the 
   Master's Triggers database. :)
let $domain:= 9535475951259984368
(: Specify the id of the replication target obtained from the 
   Master database. :)
let $target-id := 18130470845627037840
(: Specify the HTTP options for the replication target. :)
let $http-options := 
  <flexrep:http-options     xmlns:flexrep="http://marklogic.com/xdmp/flexible-replication">
    <http:authentication xmlns:http="xdmp:http">
      <http:username>admin</http:username>
      <http:password>admin</http:password>
    </http:authentication>
    <http:client-cert xmlns:http="xdmp:http"/>
    <http:client-key xmlns:http="xdmp:http"/>
    <http:pass-phrase xmlns:http="xdmp:http"/>
  </flexrep:http-options>
let $pullconfig := flexrep:pull-create(
  "Master",
  $domain,
  $target-id,
  "http://localhost:8010/",
  $http-options)
(: Insert the pull configuration into the Replica database. :)
return flexrep:pull-insert($pullconfig) 

Creating a Pull Replication Scheduled Task

This section describes how to use the scheduler functions to create a scheduled replication pull task. The following query is executed against the Replica database.

xquery version "1.0-ml";
import module namespace admin = "http://marklogic.com/xdmp/admin" 
  at "/MarkLogic/admin.xqy";
let $config := admin:get-configuration()
(: Define a minutely scheduled task to pull updates from the
   Master database each minute. :)
let $task := admin:group-minutely-scheduled-task(
  "/MarkLogic/flexrep/tasks/pull.xqy",
  "Modules",
  1,
  xdmp:database("Replica"),
  0,
  xdmp:user("admin"), 
  admin:host-get-id($config, xdmp:host-name())
)
(: Add the scheduled task to the Default group. :)
let $config:= admin:group-add-scheduled-task(
  $config, 
  admin:group-get-id($config, "Default"), 
  $task)
return admin:save-configuration($config)

Preliminary Configuration Procedures

This section describes how to create the databases, forests and App Servers to be used for Flexible Replication.

1. Use POST /manage/v2/databases to create a Master, Replica, and Triggers database:

   curl -v -X POST  --anyauth -u admin:admin --header "Content-Type:application/json" \
   -d '{"database-name":"Master"}' \
   http://localhost:8002/manage/v2/databases
   curl -v -X POST  --anyauth -u admin:admin --header "Content-Type:application/json" \
   -d '{"database-name":"MyTriggers"}' \
   http://localhost:8002/manage/v2/databases
   curl -v -X POST  --anyauth -u admin:admin --header "Content-Type:application/json" \
   -d '{"database-name":"Replica"}' \
   http://localhost:8002/manage/v2/databases

2. Use POST /manage/v2/forests to create forests for the databases and attach them to the databases.

   The host property must specify the name of the host, not localhost. In this example, the name of the host is myhost.

   curl --anyauth --user admin:admin -X POST \
   -d '{"forest-name": "MasterForest", 
        "host": "myhost.marklogic.com", 
        "database": "Master"}' \
   -i -H "Content-type: application/json" \
   http://localhost:8002/manage/v2/forests
   curl --anyauth --user admin:admin -X POST \
   -d '{"forest-name":"MyTriggersForest",
        "host":"myhost.marklogic.com",
        "database":"MyTriggers"}' \
   -i -H "Content-type: application/json" \
   http://localhost:8002/manage/v2/forests
   curl --anyauth --user admin:admin -X POST \
   -d '{"forest-name":"ReplicaForest",
        "host":"myhost.marklogic.com",
        "database":"Replica"}' \
   -i -H "Content-type: application/json" \
   http://localhost:8002/manage/v2/forests

3. Use PUT /manage/v2/databases/{id|name}/properties to configure the Master database to use the MyTriggers database:

   curl -v -X PUT --anyauth -u admin:admin --header "Content-Type:application/json" \
   -d '{"triggers-database" : "MyTriggers"}' \
   http://localhost:8002/manage/v2/databases/Master/properties

4. Use POST /manage/v2/servers to create the Master and Replica App Servers, configured with their respective databases.

   curl -X POST  --anyauth -u admin:admin --header "Content-Type:application/json" \
   -d '{
      "server-name":"Master-flexrep",
      "server-type":"http",
      "group-name":"Default",
      "root":"FlexRep",
      "port":8010,
      "content-database":"Master"
      }' \
   http://localhost:8002/manage/v2/servers?group-id=Default 
   curl -X POST  --anyauth -u admin:admin --header "Content-Type:application/json" \
   -d '{
      "server-name":"Replica-flexrep",
      "server-type":"http",
      "group-name":"Default",
      "root":"FlexRep",
      "port":8011,
      "content-database":"Replica"
      }' \
   http://localhost:8002/manage/v2/servers?group-id=Default

Installing and Configuring CPF

This section describes how to install CPF and create a domain with the pipelines required for Flexible Replication.

1. Use POST /manage/v2/databases/{id|name}/pipelines to load the Flexible Replication Pipleines

   curl -X POST  --anyauth -u admin:admin --header "Content-Type:application/json" \
   -d'{"operation": "load-default-cpf-pipelines"}' \
   http://localhost:8002/manage/v2/databases/MyTriggers/pipelines

2. Use POST /manage/v2/databases/{id|name}/domains to create a domain

   curl -X POST  --anyauth -u admin:admin --header "Content-Type:application/json" \
   -d '{
      "domain-name": "Replicated Content",
      "description": "Flexible Replication Domain",
      "scope": "directory",
      "uri": "/",
      "depth": "infinity",
      "eval-module": "Modules",
      "eval-root": "/",
      "pipeline": ["Flexible Replication","Status Change Handling"]
      }' \
   http://localhost:8002/manage/v2/databases/MyTriggers/domains

3. Use POST /manage/v2/databases/{id|name}/cpf-configs to install CPF:

   curl -X POST  --anyauth --user admin:admin --header "Content-Type:application/json" \
   -d '{
      "domain-name": "Replicated Content",
      "restart-user-name": "admin",
      "eval-module": "Modules",
      "eval-root": "/",
      "conversion-enabled": true,
      "permission": [{
          "role-name": "app-user",
          "capability": "read"
          }]
      }' \
   http://localhost:8002/manage/v2/databases/MyTriggers/cpf-configs?format=json

Creating a Replication Configuration Element

Use POST /manage/v2/databases/{id|name}/flexrep/configs to configure the Replicated Content domain to be used for Flexible Replication:

curl -v -X POST  --anyauth --user admin:admin --header "Content-Type:application/json" \
-d '{"domain-name": "Replicated Content"}' \
http://localhost:8002/manage/v2/databases/Master/flexrep/configs

Creating a Replication Target

Use POST /manage/v2/databases/{id|name}/flexrep/configs/{id|name}/targets to create a flexible replication target on Master-flexrep which defines url to Replica-flexrep

curl -v -X POST  --anyauth --user admin:admin --header "Content-Type:application/json" \
-d '{
   "target-name": "replica",
   "url": [ "http://localhost:8011/" ],
   "retry-seconds-min": 60,
   "immediate-push": true,
   "retry-seconds-max": 300,
   "documents-per-batch": 10,
   "enabled": true,
   "replicate-cpf": false,
   "http-options": {
       "username": "admin",
       "password": "admin",
       "client-cert": "",
       "client-key": "",
       "client-pass-phrase": ""},
   "filter-module": "",
   "filter-option": [""]
   }' \
http://localhost:8002/manage/v2/databases/Master/flexrep/configs/Replicated%20Content/targets?format=json

Creating a Push Replication Scheduled Task

Use POST /manage/v2/tasksto create a flexrep scheduled task:

curl -v -X POST  --anyauth --user admin:admin --header "Content-Type:application/json" \
-d '{
   "task-enabled":true, 
   "task-path":"/MarkLogic/flexrep/tasks/push-local-forests.xqy", 
   "task-root":"Modules/", 
   "task-type":"minutely", 
   "task-period":1, 
   "task-timestamp":"2014-11-24T13:44:53.143178-08:00", 
   "task-database":"Master", 
   "task-modules":"", 
   "task-user":"admin", 
   "task-priority":"higher"}' \
http://localhost:8002/manage/v2/tasks?group-id=Default

Preliminary Configuration Procedures

1. Use POST /manage/v2/databases to create the following databases:

   master - the flexrep master database

   master-triggers - the triggers database for 'master'

   master-modules - the modules database for master's triggers

   push - the target database for push replication

   pull - the target database for pull replication

   curl -v -X POST  --anyauth -u admin:admin --header "Content-Type:application/json" \
     -d '{"database-name":"master"}' http://localhost:8002/manage/v2/databases
   curl -v -X POST  --anyauth -u admin:admin --header "Content-Type:application/json" \
     -d '{"database-name":"master-triggers"}' http://localhost:8002/manage/v2/databases
   curl -v -X POST  --anyauth -u admin:admin --header "Content-Type:application/json" \
     -d '{"database-name":"master-modules"}' http://localhost:8002/manage/v2/databases
   curl -v -X POST  --anyauth -u admin:admin --header "Content-Type:application/json" \
     -d '{"database-name":"push"}' http://localhost:8002/manage/v2/databases
   curl -v -X POST  --anyauth -u admin:admin --header "Content-Type:application/json" \
     -d '{"database-name":"pull"}' http://localhost:8002/manage/v2/databases

2. Use POST /manage/v2/forests to create forests for the databases and attach them to the databases.

   The host property must specify the name of the host, not localhost. In this example, the name of the host is myhost.

   curl --anyauth --user admin:admin -X POST \
   -d '{"forest-name": "master", 
        "host": "myhost.marklogic.com", 
        "database": "master"}' \
   -i -H "Content-type: application/json" \
   http://localhost:8002/manage/v2/forests
   curl --anyauth --user admin:admin -X POST \
   -d '{"forest-name":"master-triggers",
        "host":"myhost.marklogic.com",
        "database":"master-triggers"}' \
   -i -H "Content-type: application/json" \
   http://localhost:8002/manage/v2/forests
   curl --anyauth --user admin:admin -X POST \
   -d '{"forest-name":"master-modules",
        "host":"myhost.marklogic.com",
        "database":"master-modules"}' \
   -i -H "Content-type: application/json" \
   http://localhost:8002/manage/v2/forests
   curl --anyauth --user admin:admin -X POST \
   -d '{"forest-name":"push",
        "host":"myhost.marklogic.com",
        "database":"push"}' \
   -i -H "Content-type: application/json" \
   http://localhost:8002/manage/v2/forests
   curl --anyauth --user admin:admin -X POST \
   -d '{"forest-name":"pull",
        "host":"myhost.marklogic.com",
        "database":"pull"}' \
   -i -H "Content-type: application/json" \
   http://localhost:8002/manage/v2/forests

3. Use PUT /manage/v2/databases/{id|name}/properties to set master-triggers as the triggers database

   curl -v -X PUT --anyauth -u admin:admin \
   --header "Content-Type:application/json" \
   -d '{"triggers-database" : "master-triggers"}' \
   http://localhost:8002/manage/v2/databases/master/properties

Installing and Configuring CPF

This section describes how to install CPF and create a domain with the pipelines required for Flexible Replication.

1. Use POST /manage/v2/databases/{id|name}/pipelines to load the Flexible Replication Pipleines into the master-triggers database.

   curl -X POST  --anyauth -u admin:admin --header "Content-Type:application/json" \
   -d'{"operation": "load-default-cpf-pipelines"}' \
   http://localhost:8002/manage/v2/databases/master-triggers/pipelines

2. Use POST /manage/v2/databases/{id|name}/domains to create a domain with the Flexible Replication and Status Change Handling pipelines.

   curl -X POST  --anyauth -u admin:admin --header "Content-Type:application/json" \
   -d '{
        "domain-name": "my-cpf-domain",
        "description": "My domain of documents",
        "scope": "directory",
        "uri": "/",
        "depth": "infinity",
        "eval-module": "master-modules",
        "eval-root": "/",
        "pipeline": ["Flexible Replication","Status Change Handling"]
      }' \
   http://localhost:8002/manage/v2/databases/master-triggers/domains

3. Use POST /manage/v2/databases/{id|name}/cpf-configs to install CPF:

   curl -X POST  --anyauth --user admin:admin --header "Content-Type:application/json" \
   -d '{
       "domain-name": "my-cpf-domain",
       "restart-user-name": "admin",
       "eval-module": "master-modules",
       "eval-root": "/",
       "conversion-enabled": true,
       "permission": [{
           "role-name": "admin",
           "capability": "execute"
          }]
      }' \
   http://localhost:8002/manage/v2/databases/master-triggers/cpf-configs?format=json

Configuring the Master Database and Creating App Servers

1. Use PUT /manage/v2/databases/{id|name}/properties to configure the range indexes needed for flexible replication.

   The range indexes will replace any existing range indexes, so if the application needs additional range indexes you'll need to include them here as well.

   curl -X PUT  --anyauth --user admin:admin --header "Content-Type:application/json" \
   -d '{"word-positions": true,
        "element-word-positions": true,
        "range-element-index":
       [ { "scalar-type": "dateTime",
           "namespace-uri": "http://marklogic.com/cpf",
           "localname": "last-updated",
           "collation": "",
           "range-value-positions": false,
           "invalid-values": "reject"
         }, 
         { "scalar-type": "dateTime",
           "namespace-uri": "http://marklogic.com/xdmp/flexible-replication",
           "localname": "last-success",
           "collation": "",
           "range-value-positions": true,
           "invalid-values": "reject"
         }, 
         { "scalar-type": "dateTime",
           "namespace-uri": "http://marklogic.com/xdmp/flexible-replication",
           "localname": "next-try",
           "collation": "",
           "range-value-positions": true,
           "invalid-values": "reject"
         }, 
         { "scalar-type": "unsignedLong",
           "namespace-uri": "http://marklogic.com/xdmp/flexible-replication",
           "localname": "target-id",
           "collation": "",
           "range-value-positions": true,
           "invalid-values": "reject"
         }, 
         { "scalar-type": "unsignedLong",
           "namespace-uri": "http://marklogic.com/xdmp/flexible-replication",
           "localname": "domain-id",
           "collation": "",
           "range-value-positions": true,
           "invalid-values": "reject"
         }, 
         {"scalar-type": "dateTime",
           "namespace-uri": "http://marklogic.com/xdmp/flexible-replication",
           "localname": "received-at",
           "collation": "",
           "range-value-positions": true,
           "invalid-values": "reject"
         }]}' \
   http://localhost:8002/manage/v2/databases/master/properties

2. Use POST /manage/v2/servers to create a master-flexrep App Server to service the master database. The master-flexrep App Server is also used to service the alerting rules, as described in Creating Alerting Rules.

   curl -X POST  --anyauth -u admin:admin --header "Content-Type:application/json" \
   -d '{
      "server-name":"master-flexrep",
      "server-type":"http",
      "group-name":"Default",
      "root":"FlexRep",
      "port":8010,
      "content-database":"master",
      "log-errors": true,
      "default-error-format": "compatible",
      "error-handler": "/error-handler.xqy",
      "url-rewriter": "/rewriter.xml",
      "rewrite-resolves-globally": false
      }' \
   http://localhost:8002/manage/v2/servers?group-id=Default 

3. Create a push-flexrep App Server for the replica database to receive push replicated content. Each target cluster would need to have one of these configured for inbound replication.

   curl -X POST  --anyauth -u admin:admin --header "Content-Type:application/json" \
   -d '{
      "server-name":"push-flexrep",
      "server-type":"http",
      "group-name":"Default",
      "root":"FlexRep",
      "port":8011,
      "content-database":"push",
      "log-errors": true,
      "default-error-format": "compatible",
      "error-handler": "/error-handler.xqy",
      "url-rewriter": "/rewriter.xml",
      "rewrite-resolves-globally": false
      }' \
   http://localhost:8002/manage/v2/servers?group-id=Default

4. Create a pull-flexrep App Server for the replica database to receive pulled replicated content. This would be used for status information on the replica database.

   curl -X POST  --anyauth -u admin:admin --header "Content-Type:application/json" \
   -d '{
      "server-name":"pull-flexrep",
      "server-type":"http",
      "group-name":"Default",
      "root":"FlexRep",
      "port":8012,
      "content-database":"pull",
      "log-errors": true,
      "default-error-format": "compatible",
      "error-handler": "/error-handler.xqy",
      "url-rewriter": "/rewriter.xml",
      "rewrite-resolves-globally": false
      }' \
   http://localhost:8002/manage/v2/servers?group-id=Default

Creating Users

Use POST /manage/v2/users to create user1 for the query-based push target and user2 for the query-based pull target. Create flexrep-admin-user for admin access to the master-flexrep App Server.

curl -X POST  --anyauth --user admin:admin \
--header "Content-Type:application/json" \
-d '{"user-name": "user1",
     "password": "password1",
     "description": "first user",
     "role": [ "flexrep-user", "alert-user" ]
    }' \
http://localhost:8002/manage/v2/users
curl -X POST  --anyauth --user admin:admin \
--header "Content-Type:application/json" \
-d '{"user-name": "user2",
     "password": "password2",
     "description": "second user",
     "role": [ "flexrep-user", "alert-user" ]
    }' \
http://localhost:8002/manage/v2/users
curl -X POST  --anyauth --user admin:admin \
--header "Content-Type:application/json" \
-d '{"user-name": "flexrep-admin-user",
     "password": "flexrep-admin-password",
     "description": "FlexRep administrative user",
     "role": [ "flexrep-admin", "flexrep-user" ]
    }' \
http://localhost:8002/manage/v2/users

Configuring Alerting

Use POST /manage/v2/databases/{id|name}/alert/configs to create the alerting configuration for the master database.

curl -X POST  --anyauth --user admin:admin \
--header "Content-Type:application/json" \
-d '{
  "uri": "http://acme.com/alerting",
  "name": "qbfr",
  "description": "alerting rules for query-based flexrep",
  "trigger": [],
  "domain": [],
  "action": [],
  "option": []
   }' \
http://localhost:8002/manage/v2/databases/master/alert/configs

Use POST /manage/v2/databases/{id|name}/alert/actions to create the alert action and apply it to the alert configuration.

curl -X POST  --anyauth --user admin:admin \
--header "Content-Type:application/json" \
-d '{
  "name": "log",
  "description": "QBFR log action",
  "module": "/log.xqy",
  "module-db": "master-modules",
  "module-root": "/",
  "option": []
   }' \
http://localhost:8002/manage/v2/databases/master/alert/actions?uri=http://acme.com/alerting

Creating a Replication Configuration Element

Use POST /manage/v2/databases/{id|name}/flexrep/configs to configure the my-cpf-domain domain to be used for Flexible Replication:

curl -v -X POST  --anyauth --user admin:admin \
--header "Content-Type:application/json" \
-d '{"domain-name": "my-cpf-domain",
     "alerting-uri": "http://acme.com/alerting"}' \
http://localhost:8002/manage/v2/databases/master/flexrep/configs 

Creating Replication Targets

Use POST /manage/v2/databases/{id|name}/flexrep/configs/{id|name}/targets to create flexible replication targets for the push and pull databases on master database.

The user1 user is associated with user1-target to ensure only documents intended for user1 are replicated to that target. The same is done for user2 and user1-target. The alerting rules described in Creating Alerting Rules control which documents are replicated to which target.

1. Create the push target, user1-target. The user setting is required for a query-based target.

   curl -X POST  --anyauth --user admin:admin \
   -H "Content-Type: application/json" \
   -d '{"target-name": "user1-target",
        "url": [ "http://localhost:8011/" ],
        "documents-per-batch": 10,
        "http-options": {
           "username": "admin",
           "password": "admin"},
        "user": "user1"}' \
   http://localhost:8002/manage/v2/databases/master/flexrep/configs/my-cpf-domain/targets

2. Create the pull target, user2-target. The user setting is required for a query-based target

   curl -X POST  --anyauth --user admin:admin \
   -H "Content-Type: application/json" \
   -d '{"target-name": "user2-target",
        "url": [],
        "documents-per-batch": 10,
        "user": "user2"}' \
   http://localhost:8002/manage/v2/databases/master/flexrep/configs/my-cpf-domain/targets

Creating Alerting Rules

Use POST /v1/domains/{domain-id-or-default-domain-name}/targets/{id|name}/rules to create the alerting rules for the two users, via the master-flexrep App Server (port 8010) on the master database.

1. The first two rules match documents readable by user1 and contain either the word apple or orange. In this way, only documents with these attributes will be pushed to the push replica database.

   curl -X POST  --anyauth --user admin:admin \
   --header "Content-Type:application/json" \
   -d '{ "name": "apple",
       "query": [ 
         {"wordQuery":{"text":["apple"], 
         "user-name": "user1",
          "options":["lang=en"]}} ],
          "action-name": "log" }' \
   http://localhost:8002/manage/v2/databases/master/alert/actions/log/rules?uri=http://acme.com/alerting
   curl -X POST  --anyauth --user admin:admin \
   --header "Content-Type:application/json" \
   -d '{ "name": "orange",
       "query": [ 
         {"wordQuery":{"text":["orange"], 
         "user-name": "user2",
          "options":["lang=en"]}} ],
          "action-name": "log" }' \
   http://localhost:8002/manage/v2/databases/master/alert/actions/log/rules?uri=http://acme.com/alerting

2. The second two rules match documents readable by user2 and contain either the word apple or banana. In this way, only documents with these attributes will be pulled into the pull replica database.

   curl -X POST  --anyauth --user admin:admin \
   --header "Content-Type:application/json" \
   -d '{ "name": "apple",
         "query": [ 
            {"wordQuery":{"text":["apple"], 
            "user-name": "user1",
             "options":["lang=en"]}} ],
             "action-name": "log" }' \
   http://localhost:8002/manage/v2/databases/master/alert/actions/log/rules?uri=http://acme.com/alerting
   curl -X POST  --anyauth --user admin:admin \
   --header "Content-Type:application/json" \
   -d '{ "name": "orange",
         "query": [ 
            {"wordQuery":{"text":["orange"], 
            "user-name": "user2",
             "options":["lang=en"]}} ],
         "action-name": "log" }' \
   http://localhost:8002/manage/v2/databases/master/alert/actions/log/rules?uri=http://acme.com/alerting

Configuring Pull Replication

To configure pull replication, you need the domain id and target id of the user2-target.

1. You can return the domain id and target id by doing a GET on the following endpoints (note that these are executed on port 8010, which is the master-flexrep App Server):

   http://localhost:8010/v1/domains/my-cpf-domain/targets?format=json
   http://localhost:8010/v1/domains/my-cpf-domain/targets/user2-target?format=json

2. In this example, the domain id returned is 7860365094706821763 and the target id of user2-target is 15425733952606449897. Use POST /manage/v2/databases/{id|name}/flexrep/pulls to configure pull replication on the pull database:

   curl -X POST  --anyauth --user admin:admin \
   -H "Content-type: application/json" \
   -d '{"pull-name": "from-master",
        "domain-id": "7860365094706821763",
        "target-id": "15425733952606449897",
        "url": [ "http://localhost:8010/" ],
        "http-options": {
             "username": "user2",
             "password": "password2"}
        }' \
   http://localhost:8002/manage/v2/databases/pull/flexrep/pulls

Creating a Push and Pull Replication Scheduled Task

The master database needs a scheduled task to handle retries and zero-day replication. The pull target's database also needs a scheduled task to pull content from the master database.

1. Use POST /manage/v2/tasks to create a flexrep scheduled task to push replicated content to the push database:

   curl -X POST --anyauth --user admin:admin \
   -H "Content-type: application/json" \
   -d '{"task-priority": "higher",
        "task-user": "admin",
        "task-database": "master",
        "task-path": "/MarkLogic/flexrep/tasks/push-local-forests.xqy",
        "task-root": "Modules/",
        "task-type": "minutely",
        "task-period": "1"}' \
   http://localhost:8002/manage/v2/tasks?group-id=Default

2. Use POST /manage/v2/tasks to create a flexrep scheduled task to pull replicated content from the master database into the pull database:

   curl -X POST --anyauth --user admin:admin \
   -H "Content-type: application/json" \
   -d '{"task-priority": "higher",
        "task-user": "admin",
        "task-database": "pull",
        "task-path": "/MarkLogic/flexrep/tasks/pull.xqy",
        "task-root": "Modules/",
        "task-type": "minutely",
        "task-period": "1"}' \
   http://localhost:8002/manage/v2/tasks?group-id=Default

Inserting Some Documents to Test

Use PUT /v1/documents to insert some JSON and XML documents into the master database. (Note that these are executed on port 8000 and that single quotes are used in the URI because of the multiple parameters.)

These calls require that you have the eval-in privilege, as described in Security Requirements in the REST Application Developer's Guide. Also, you must insert each document with the alert-user:read permission.

1. Insert some documents that include the word 'apple.' For example:

curl -X PUT --anyauth  --user admin:admin -H "Content-type: application/json" \
-d '{"produce":{"fruit": "apple"}}' \
'http://localhost:8000/LATEST/documents?uri=/json/apple.json&database=master&perm:alert-user=read'

curl --anyauth  --user admin:admin -X PUT -H "Content-type: application/xml" \
-d '<produce>
        <fruit>apple</fruit>
    </produce>' \
'http://localhost:8000/LATEST/documents?uri=/xml/apple.xml&database=master&perm:alert-user=read'

1. Insert some documents that include the word 'orange.' For example:

curl -X PUT --anyauth  --user admin:admin -H "Content-type: application/json" \
-d '{"produce":{"fruit": "orange"}}' \
'http://localhost:8000/LATEST/documents?uri=/json/orange.json&database=master&perm:alert-user=read'

curl --anyauth  --user admin:admin -X PUT -H "Content-type: application/xml" \
-d '<produce>
        <fruit>orange</fruit>
    </produce>' \
'http://localhost:8000/LATEST/documents?uri=/xml/orange.xml&database=master&perm:alert-user=read'

1. Insert some documents that include the word 'banana.' For example:

curl -X PUT --anyauth  --user admin:admin -H "Content-type: application/json" \
-d '{"produce":{"fruit": "banana"}}' \
'http://localhost:8000/LATEST/documents?uri=/json/banana.json&database=master&perm:alert-user=read'

curl --anyauth  --user admin:admin -X PUT -H "Content-type: application/xml" \
-d '<produce>
        <fruit>banana</fruit>
    </produce>' \
'http://localhost:8000/LATEST/documents?uri=/xml/banana.xml&database=master&perm:alert-user=read'

Based on the rules created in Creating Alerting Rules, the documents containing apple and orange should be replicated to the push database and the documents containing apple and banana should be replicated to the pull database.