MarkLogic Server 11.0 Product Documentation

mlcp User Guide — Chapter 5

Filtering Document Exports

This section covers options available for filtering what is exported by the mlcp export command when -output_type is document.

By default, mlcp exports all documents in the database. That is, mlcp exports the equivalent of fn:collection(). The following options allow you to filter what is exported. These options are mutually exclusive.

• -directory_filter - export only the documents in the listed database directories. You cannot use this option with -collection_filter or -document-selector.
• -collection_filter - export only the documents in the listed collections. You cannot use this option with -directory_filter or -document_selector.
• -document_selector - export only documents selected by the specified XPath expression. You cannot use this option with -directory_filter or -collection_filter. Use -path_namespace to define namespace prefixes.
• -query_filter - export only documents matched by the specified cts query. You can use this option alone or in combination with a directory, collection or document selector filter. You can only use this filter with the export and copy commands. Results may not be accurate; for details, see Understanding When Filters Are Accurate.

  When filtering with a document selector, the XPath filtering expression should select fragment roots only. An XPath expression that selects nodes below the root is very inefficient.

When using -document_selector to filter by XPath expression, you can define namespace prefixes using the -path_namespace option. For example:

-path_namespace 'ex1,http://marklogic.com/example,ex2,http://my/ex2'
-document_selector '/ex1:elem[ex2:attr > 10]'

Filtering Archive and Copy Contents

This section covers options available for controlling what is exported by mlcp export when -output_type is archive, or what is copied by the mlcp copy command.

By default, all documents and metadata are exported/copied. The following options allow you to modify this behavior:

• -directory_filter - export/copy only the documents in the listed database directories, including related metadata. You cannot use this option with -collection_filter or -document_selector.
• -collection_filter - export/copy only the documents in the listed collections, including related metadata. You cannot use this options with -directory_filter or -document_selector.
• -document_selector - export/copy only documents selected by the specified XPath expression.You cannot use this option with -directory_filter or -collection_filter. Use -path_namespace to define namespace prefixes.
• -query_filter - export/copy only documents matched by the specified cts query. You can use this option alone or in combination with a directory, collection or document selector filter. Results may not be accurate; for details, see Understanding When Filters Are Accurate.
• -copy_collections - whether to include collection metadata
• -copy_permissions - whether to include permissions metadata
• -copy_properties - whether to include naked and document properties
• -copy_quality - whether to include document quality metadata
• -copy_metadata - whether to include document key-value metadata

If you set all the -copy_* options to false when exporting to an archive, the archive contains no metadata. When you import an archive with no metadata, you must set -archive_metadata_optional to true.

When filtering with a document selector, the XPath filtering expression should select fragment roots only. An XPath expression that selects nodes below the root is very inefficient.

When using -document_selector to filter by XPath expression, you can define namespace prefixes using the -path_namespace option. For example:

-path_namespace 'ex1,http://marklogic.com/example,ex2,http://my/ex2'
-document_selector '/ex1:elem[ex2:attr > 10]'

Understanding When Filters Are Accurate

When you use -directory_filter, -collection_filter, or -document_selector without -query_filter, the set of documents selected by mlcp exactly matches your filtering criteria.

The query you supply with -query_filter is used in an unfiltered search, which means there can be false positives among the selected documents. When you combine -query_filter with -directory_filter, -collection_filter, or -document_selector, mlcp might select documents that do not meet your directory, collection, or path filter criteria.

The interaction between -query_filter and the other filtering options is similar to the following. In this example, the search can match documents that are not in the parts collection.

-collection_filter parts 
-query_filter yourSerializedQuery
==> selects the documents to export similar to the following:

cts:search(
  fn:collection("parts"), 
  yourQuery, 
  ("unfiltered"))

For a complete example using -query_filter, see Example: Exporting Documents Matching a Query.

To learn more about the implications of unfiltered searches, see Fast Pagination and Unfiltered Searches in the Query Performance and Tuning Guide.

Example: Exporting Documents Matching a Query

This example demonstrates how to use -query_filter to select documents for export. You can apply the same technique to filtering the source documents when copying documents from one database to another.

The -query_filter option accepts a serialized XML cts:query or JSON cts.query as its value. For example, the following table shows the serialization of a cts word query, prettyprinted for readability:

<cts:word-query xmlns:cts="http://marklogic.com/cts">
  <cts:text xml:lang="en">mark</cts:text>
</cts:word-query>

{"wordQuery":{
  "text":["huck"], 
  "options":["lang=en"]
}}

For details on how to obtain the serialized representation of a cts query, see Serializations of cts:query Constructors in the Search Developer's Guide.

Using an options file is recommended when using -query_filter because both XML and JSON serialized queries contain quotes and other characters that have special meaning to the Unix and Windows command shells, making it challenging to properly escape the query. If you use -query_filter on the command line, you must quote the serialized query and may need to do additional special character escaping.

For example, you can create an options file similar to the following. It should contain at least 2 lines: One for the option name and one for the serialized query. You can include other options in the file. For details, see Options File Syntax.

-query_filter
<cts:word-query xmlns:cts="http://marklogic.com/cts"><cts:text xml:lang="en">mark</cts:text></cts:word-query>

-query_filter
{"wordQuery":{"text":["huck"], "options":["lang=en"]}}

If you save the above option in a file named query_filter.txt, then the following mlcp command exports files from the database that contain the word huck:

# Windows users, see Modifying the Example Commands for Windows 
$ mlcp.sh export -host localhost -port 8000 -username user \
    -password password -mode local -output_file_path \
    /space/mlcp/export/files -options_file query_filter.txt

You can combine -query_filter with another filtering option. For example, the following command combines the query with a collection filter. The command exports only documents containing the word huck in the collection named classics:

$ mlcp.sh export -host localhost -port 8000 -username user \
    -password password -mode local -output_file_path \
    /space/mlcp/export/files -options_file query_filter.txt
    -collection_filter classics

The documents selected by -query_filter can include false positives, including documents that do not match other filter criteria. For details, see Understanding When Filters Are Accurate.

The following example demonstrates generating a serialized XML cts:and-query or JSON cts.andQuery using the wrapper technique. Copy either example into Query Console, select the appropriate query type, and run it to see the output.

xquery version "1.0-ml";
let $query := cts:and-query((
  cts:word-query("mark"), 
  cts:word-query("twain")
))
let $q := xdmp:quote(
  <query>{$query}</query>/*, 
  <options xmlns="xdmp:quote"><indent>no</indent></options>
)
return $q

(: Output: (whitespace added for readability)
<cts:and-query xmlns:cts="http://marklogic.com/cts">
  <cts:word-query>
    <cts:text xml:lang="en">mark</cts:text>
  </cts:word-query>
  <cts:word-query>
    <cts:text xml:lang="en">twain</cts:text>
  </cts:word-query>
</cts:and-query>
:)

var wrapper = 
  { query:
      cts.andQuery([
        cts.wordQuery("huck"),
        cts.wordQuery("tom")])
  };
xdmp.quote(wrapper.query.toObject())

/* Output: (whitespace added for readability)
{"andQuery":{
  "queries":[
    {"wordQuery":{"text":["huck"], "options":["lang=en"]}},
    {"wordQuery":{"text":["tom"], "options":["lang=en"]}}
  ]
}}
*/

Notice that in the XML example, the xdmp:quote indent option is used to disable XML prettyprinting, making the output better suited for inclusion on the mlcp command line:

xdmp:quote(
  <query>{$query}</query>/*, 
  <options xmlns="xdmp:quote"><indent>no</indent></options>
)

Notice that in the JavaScript example, it is necessary to call toObject on the wrapped query to get the proper JSON serialization. Using toObject converts the value to a JavaScript object which xdmp.quote will serialize as JSON.

xdmp.quote(wrapper.query.toObject())

If you want to test your serialized query before using it with mlcp, you can round-trip your XML query with cts:search in XQuery or your JSON query with cts.search or the JSearch API in Server-Side JavaScript, as shown in the following examples.

xquery version "1.0-ml";
let $wrapper := 
  <query>{
    cts:and-query((
      cts:word-query("tom"),
      cts:word-query("huck")))
  }</query>
let $q := xdmp:quote(
  $wrapper/*, 
  <options xmlns="xdmp:quote"><indent>no</indent></options>)
return cts:search(
  fn:doc(), 
  cts:query(xdmp:unquote($q)/*[1])
)

var wrapper = 
  { query:
      cts.andQuery([
        cts.wordQuery("huck"),
        cts.wordQuery("tom")])
  };
var serializedQ = xdmp.quote(wrapper.query.toObject())
cts.search(
  cts.query(fn.head(xdmp.unquote(serializedQ)).root))

Note that xdmp:unquote returns a document node in XQuery, so you need to use XPath to address the underlying query element root node when reconstructing the query:

cts:query(xdmp:unquote($q)/*[1])

Similarly, xdmp.unquote in JavaScript returns a Sequence on document nodes, so you must dereference both the iterator and the document node when reconstructing the query:

cts.query(fn.head(xdmp.unquote(serializedQ)).root)

Filtering Forest Contents

This section covers options available for filtering what is extracted from a forest when you use Direct Access. That is, when you use the mlcp import command with -input_file_type forest or the mlcp extract command.

By default, mlcp extracts all documents in the input forests. That is, mlcp extracts the equivalent of fn:collection(). The following options allow you to filter what is extracted from a forest with Direct Access. These options can be combined.

• -type_filter: Extract only documents with the listed content type (text, XML, or binary).
• -directory_filter: Extract only the documents in the listed database directories.
• -collection_filter: Extract only the documents in the listed collections.

For example, following combination of options extracts only XML documents in the collections named 2004 or 2005.

mlcp.sh extract -type_filter xml -collection_filter "2004,2005" ...

Similarly, the following options import only binary documents in the source database directory /images/:

mlcp.sh import -input_file_type forest \
    -type_filter binary -directory_filter /images/

When you use Direct Access, filtering is performed in the process that reads the forest files rather than being performed by MarkLogic Server. For example, in local mode, filters are applied by mlcp on the host where you run it.

In addition, filtering cannot be applied until after a document is read from the forest. When you import or extract files from a forest file, mlcp must touch every document in the forest.

For details, see Using Direct Access to Extract or Copy Documents.

Extracting a Consistent Database Snapshot

By default, when you export or copy database contents, content is extracted from the source database at multiple points in time. You get whatever is in the database when mlcp accesses a given document. If the database contents are changing while the job runs, the results are not deterministic relative to the starting time of the job. For example, if a new document is inserted into the database while an export job is running, it might or might not be included in the export.

If you require a consistent snapshot of the database contents during an export or copy, use the -snapshot option to force all documents to be read from the database at a consistent point in time. The submission time of the job is used as the timestamp. Any changes to the database occurring after this time are not reflected in the output.

If a merge occurs while exporting or copying a consistent snapshot, and the merge eliminates a fragment that is subsequently accessed by the mlcp job, you may get an XDMP-OLDSTAMP error. If this occurs, the documents included in the same batch or task may not be included in the export/copy result. If the source database is on MarkLogic Server 7 or later, you may be able to work around this problem by setting the merge timestamp to retain fragments for a time period longer than the expected running time of the job; for details, see Understanding and Controlling Database Merges in the Administrator's Guide.

Basic Steps for Redacting Documents

Use the -redaction option of mlcp to apply redaction rules to an export or copy operation. This option accepts a comma-separated list of redaction rule collection URIs. For example:

-redaction "pii-rules,sec-rules"

Before you can use redaction, you must install one or more redaction rule sets in the Schemas database. For details on defining and installing redaction rules, see Redacting Document Content in the Application Developer's Guide.

Preparing to redact documents with mlcp requires the following steps. For a complete example, see Example: Using mlcp for Redaction.

1. Install one or more redaction rules in the Schemas database. Each rule must be part of at least one collection. For details, see Defining Redaction Rules and Installing Redaction Rules in the Application Developer's Guide.
2. If you create a rule that uses a user-defined redaction function, install the implementation of your redaction function in the modules database associated with the App Server you will connect to using mlcp. For details, see User-Defined Redaction Functions in the Application Developer's Guide.
3. Add the -redaction option to your mlcp command line. For example, the following command applies the rules in the collections pii-rules and sec-rules to all exported documents.

   # Windows users, see Modifying the Example Commands for Windows 
   $ mlcp.sh export -host localhost -port 8000 -username user \
       -password password -mode local -output_file_path \
       /space/mlcp/export/files -directory_filter /people/ \
       -redaction "pii-rules,sec-rules"

The -redaction option works similarly for copy operations. For details, see Redacting Content During a Copy.

The user who extracts redacted documents must have read permissions on the source documents and the rules, but need not be able to modify the rule collection or rule definitions. For details, see Security Considerations in Application Developer's Guide.

The following behaviors apply when exceptional conditions occur. You should be aware of these behaviors so you understand when content might not be redacted as expected:

• If a rule collection is empty, mlcp issues a warning and continues with the job.
• If any of the rules contain errors, an error is reported and mlcp aborts the export or copy operation.
• If a rule is valid, but an error occurs when applying the rule, the rule is skipped for the current document and a warning is logged. The job continues.

Example: Using mlcp for Redaction

This example walks you through using mlcp to install and apply redaction rules based on the built-in redaction functions. For a similar example using XQuery and Query console, see Example: Getting Started With Redaction in the Application Developer's Guide.

The example has the following parts:

• Creating a Work Area
• Installing the Source Documents
• Installing the Redaction Rules
• Understanding the Example Rules
• Applying the Redaction Rules

This example uses rules based on built-in redaction functions. For an example of using user-defined redaction functions, see User-Defined Redaction Functions in the Application Developer's Guide.

redact-gs/
  data/
  rules/

$ mkdir -p redact-gs/data redact-gs/rules

>mkdir redact-gs\data redact-gs\rules

<rule xml:lang="zxx" xmlns="http://marklogic.com/xdmp/redaction">
  <description>Obscure phone numbers.</description>
  <path>//summary</path>
  <method>
    <function>redact-us-phone</function>
  </method>
  <options>
    <level>partial</level>
  </options>
</rule>

{ "rule": {
    "description": "Remove customer ids.",
    "path": "//id",
    "method": { "function": "conceal" }
}}

$ mlcp.sh export -host localhost -port 8000 \
    -username user -password password -mode local \
    -collection_filter "gs-samples" \
    -output_file_path ./output/ \
    -redaction "gs-rules"

redact-gs/
  output/
    redact-gs/
      sample1.xml
      sample2.json

Original Document

<personal>
  <name>Little Bopeep</name>
  <summary>Seeking lost sheep. Please call 123-456-7890.</summary>
  <id>12-3456789</id>
</personal>

Redacted Result

<personal>
  <name>Little Bopeep</name>
  <summary>Seeking lost sheep. Please call ###-###-7890.</summary>
</personal>

Original Document

{"personal": {
  "name": "Jack Sprat", 
  "summary": "Free nutrition advice! Call (234)567-8901 now!",
  "id": "45-6789123"
}}

Redacted Result

{"personal": {
  "name": "Jack Sprat", 
  "summary": "Free nutrition advice! Call (###)###-8901 now!"
}}