MarkLogic 9 Product Documentation

mlcp User Guide — Chapter 6

Basic Steps

To copy one database to another with mclp:

1. Set -input_host, -input_port, -input_username, and -input_password to identify the source MarkLogic Server instance and user.
2. Set -output_host, -output_port, -output_username, and -output_password to identify the destination MarkLogic Server instance and user.
3. Select what documents to copy. For details, see Filtering Archive and Copy Contents.
   • To select documents in one or more collections, set -collection_filter to a comma separated list of collection URIs.
   • To select documents in one or more database directories, set -directory_filter to a comma separated list of directory URIs.
   • To select documents matching an XPath expression, use -document_selector. To use namespace prefixes in the XPath expression, define the prefix binding using -path_namespace.
   • To select document matching a query, use -query_filter. You can use this option alone or in combination with a directory, collection or document selector filter. False positives are possible; for details, see Understanding When Filters Are Accurate.
   • To select all documents in the database, leave -collection_filter, -directory_filter, -document_selector, and -query_filter unset.
4. If you want to exclude some or all source document metadata:
   • Set -copy_collections to false to exclude document collections metadata.
   • Set -copy_permissions to false to exclude document permissions metadata.
   • Set -copy_properties to false to exclude document properties.
   • Set -copy_quality to false to exclude document quality metadata.
   • Set -copy_metadata to false to exclude document key-value metadata.
5. If you want to add or override document metadata in the destination database:
   • Set -output_collections to add destination documents to a collection.
   • Set -output_permissions to add permissions to destination documents.
   • Set -output_quality to set the quality of destination documents.
6. If you want the destination documents to have database URIs different from the source URIs, set -output_uri_replace, -output_uri_prefix, and/or -output_uri_suffix. For details, see Controlling Database URIs During Ingestion.

For a complete list of mlcp copy command options, see Copy Command Line Options.

Examples

The following example copies all documents and their metadata from the source database to the destination database:

# Windows users, see Modifying the Example Commands for Windows 
$ mlcp.sh copy -mode local -input_host srchost -input_port 8000 \
    -input_username user1 -input_password password1 \
    -output_host desthost -output_port 8010 -output_username user2 \
    -output_password password2

The following example copies selected documents, excluding the source permissions and adding the documents to 2 new collections in the destination database:

# Windows users, see Modifying the Example Commands for Windows 
$ mlcp.sh copy -mode local -input_host srchost -input_port 8000 \
    -input_username user1 -input_password password1 \
    -output_host desthost -output_port 8000 -output_username user2 \
    -output_password password2 -copy_permissions false \
    -output_collections shakespeare,plays

For an example of using -query_filter, see Example: Exporting Documents Matching a Query.

Redacting Content During a Copy

Redaction is the process of eliminating or obscuring portions of a document when retrieving the document from MarkLogic. For example, you can eliminate or mask sensitive personal information such as credit card numbers, phone numbers, or email addresses from documents. You can only redact document content, not document properties.

Redaction is performed as documents are read from the source database. For example, if you copy documents between databases in two different MarkLogic installations, the unredacted content never leaves the source installation.

Redaction support in MarkLogic is covered in detail in Redacting Content During Export or Copy Operations and Redacting Document Content in the Application Developer's Guide.

Use the -redaction option to apply redaction rules during a copy. For example, the following command copies documents in the my_docs collection from one database to another, and applies the redaction rules in the rule collections hipaa-rules and biz-rules to the source documents before copying them to the destination database.

# Windows users, see Modifying the Example Commands for Windows 
$ mlcp.sh copy -mode local -input_host srchost -input_port 8000 \
    -input_username user1 -input_password password1 \
    -output_host desthost -output_port 8000 -output_username user2 \
    -output_password password2 -collection_filter my_docs \
    -redaction "hipaa-rules,biz-rules"

For more details, see Redacting Content During Export or Copy Operations.

Advanced Document Selection for Copy

The mlcp tool uses the MarkLogic Connector for Hadoop to distribute work across your MarkLogic cluster, even when run in local mode. When you use the mlcp copy command, the source MarkLogic Server instance acts as an input source for a Hadoop MapReduce job. Similarly, the destination MarkLogic Server instance acts as the output sink for the job. You can take low level control of the job by setting connector and Hadoop configuration properties.

This is an advanced technique. You should understand how to use the MarkLogic Connector for Hadoop before attempting this.

The following list describes some use cases in which you might choose to set low level configuration properties:

• The options discussed in Controlling What is Exported, Copied, or Extracted do not enable you to select the desired set of documents to copy.

Similar use cases and techniques apply to export operations. For details, see Advanced Document Selection and Transformation.

The following table lists some connector and Hadoop configuration properties relevant to advanced configuration for copy.

When you take low-level control of a copy operation, you can no longer use options such as -copy_collections, -copy_permissions, and -copy_properties to copy the various categories of metadata from the source database to the destination database. If you include the -copy_* options on the mlcp command line, they will be ignored.

You can pass a connector configuration file through mlcp with the -conf option. The -conf option must appear after -options_file (if present) and before any other mlcp options. The following example command demonstrates using the -conf option in a copy operation.

$ mlcp.sh copy -conf conf.xml -input_host srchost -input_port 8000 \
    -input_username user -input_password password \
    -output_host desthost -output_port 8000 \
    -output_username user -output_password password \
    -mode local

The following example connector configuration file includes an XQuery split query that selects documents from a specific collection (similar to what the -collection_filter option does), and an XQuery input query that selects specific elements of each document.

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
  <property>
    <name>mapreduce.job.inputformat.class</name>
    <value>com.marklogic.mapreduce.DocumentInputFormat</value>
  </property>
  <property>
    <name>mapreduce.job.outputformat.class</name>
    <value>com.marklogic.mapreduce.ContentOutputFormat</value>
  </property>
  <property>
    <name>mapreduce.marklogic.input.mode</name>
    <value>advanced</value>
  </property>
  <property>
    <name>mapreduce.marklogic.input.splitquery</name>
    <value><![CDATA[
xquery version "1.0-ml";
declare namespace wp="http://www.mediawiki.org/xml/export-0.4/";
import module namespace admin ="http://marklogic.com/xdmp/admin" 
  at "/MarkLogic/admin.xqy";
let $conf := admin:get-configuration()
for $forest in xdmp:database-forests(xdmp:database())
let $host_id :=admin:forest-get-host($conf,$forest)
let $host_name := admin:host-get-name($conf,$host_id)
let $cnt := xdmp:estimate(
  cts:search(fn:collection("mycoll"),
             cts:and-query(()),(),0.0,$forest))
return
($forest,$cnt,$host_name)
    ]]></value>
  </property>
  <property>
    <name>mapreduce.marklogic.input.query</name>
    <value><![CDATA[
xquery version "1.0-ml";
declare default element namespace "http://HadoopTest";
fn:collection("mycoll")//*:bar/*:foo
    ]]></value>
  </property>
</configuration>

For more details and examples, see the MarkLogic Connector for Hadoop Developer's Guide.

Copy Command Line Options

This section summarizes the command line options available with the mlcp copy command. The following command line options define your connection to MarkLogic:

-input_host comma-list

-input_port number

-input_username string

-input_password string

-output_host comma-list

-output_port number

-output_username string

-output_password string

The following table lists command line options that define the characteristics of the copy operation:

-batch_size number

-collection_filter comma-list

-conf filename

-copy_collections boolean

-copy_metadata boolean

-copy_permissions boolean

-copy_properties boolean

-copy_quality boolean

-D property=value

-directory_filter comma-list

-document_selector string

-fastload boolean

-hadoop_conf_dir string

-input_database string

-input_ssl boolean

-input_ssl_protocol string

-max_split_size number

-mode string

-path_namespace comma-list

-options_file string

-output_collections comma-list

-output_database string

-output_permissions comma-list

-output_quality string

-output_partition string

-output_ssl boolean

-output_ssl_protocol string

-output_uri_prefix string

-output_uri_replace comma-list

-output_uri_suffix string

-query_filter string

-redaction comma-list

-restrict_input_hosts boolean

-restrict_output_hosts boolean

-snapshot boolean

-temporal_collection string

-thread_count number

-transform_function string

-transform_module string

-transform_namespace string

-transform_param string

-transaction_size number