MarkLogic 10 Product Documentation

mlcp User Guide — Chapter 4

Default Document URI Construction

The default database URI assigned to ingested documents depends on the input source. Loading content from the local filesystem can create different URIs than loading the same content from a ZIP file or archive. Command line options are available for you to modify this behavior. You can use options to generate different URIs; for details, see Transforming the Default URI.

The following table summarizes the default behavior with several input sources:

For example, the following command loads all files from the file systemdirectory /space/bill/data into the database attached to the App Server on port 8000. The documents inserted into the database have URIs of form /space/bill/data/filename.

# Windows users, see Modifying the Example Commands for Windows 
$ mlcp.sh import -host localhost -port 8000 -username user \
    -password passwd -input_file_path /space/bill/data -mode local

If the /space/bill/data directory is zipped up into bill.zip, such that bill/ is the root directory in zip file, then the following command inserts documents with URIs of the form bill/data/filename:

# Windows users, see Modifying the Example Commands for Windows 
$ cd /space; zip -r bill.zip bill
$ mlcp.sh import -host localhost -port 8000 -username user \
    -password passwd -input_file_path /space/bill.zip \
    -mode local -input_compressed true

When you use the -generate_uri option to have mlcp generate URIs for you, the generated URIs follow the same pattern as for aggregate XML and line delimited JSON:

/path/filename-split_start-seqnum

The generated URIs are unique across a single import operation, but they are not globally unique. For example, if you repeatedly import data from some file /tmp/data.csv, the generated URIs will be the same each time (modulo differences in the number of documents inserted by the job).

Transforming the Default URI

Use the following options to tailor the database URI of inserted documents:

• -output_uri_replace performs one or more string substitutions on the default URI.
• -output_uri_prefix prepends a string to the URI after substitution.
• -output_uri_suffix appends a string to the URI after substitution.

The -output_uri_replace option accepts a comma delimited list of regular expression and replacement string pairs. The string portion must be enclosed in single quotes:

-output_uri_replace pattern,'string',pattern,'string'

For details on the regular expression language supported by -output_uri_replace, see Regular Expression Syntax.

These options are applied after the default URI is constructed and encoded, so if the option values contain characters not allowed in a URI, you must encode them yourself. See Character Encoding of URIs.

The following example loads documents from the filesystem directory /space/bill/data. The default output URIs would be of the form /space/bill/data/filename. The example uses -output_uri_replace to replace bill/data with will and strip off /space/, and then adds a /plays prefix using -output_uri_prefix. The end result is output URIs of the form /plays/will/filename.

# Windows users, see Modifying the Example Commands for Windows 
$ mlcp.sh import -host localhost -port 8000 -username user \
    -password passwd -input_file_path /space/bill/data -mode local \
    -output_uri_replace "/space,'',/bill/data/,'/will/'" \
    -output_uri_prefix /plays

Loading a Single File

Use the following procedure to load all the files in a native directory and its sub-directories. To load selected files, see Filtering Documents Loaded From a Directory.

1. Set -input_file_path to the path to the input file.
2. Set -input_file_type if your input files are not documents. For example, if loading from delimited text files, sequence files, aggregate XML files, RDF triples files, or database archives.
3. Set -document_type if -input_file_type is not documents and the content type cannot be accurately deduced from the file suffixes as described in How mlcp Determines Document Type.
4. Set -mode:
   • To perform the work locally, set -mode to local.

By default, the imported document has a database URI based on the input file path. For details, see Controlling Database URIs During Ingestion.

The following example command loads a single XML file:

# Windows users, see Modifying the Example Commands for Windows 
$ mlcp.sh import -host localhost -port 8000 -username user \
    -password password -input_file_path /space/bill/data/hamlet.xml

Loading All the Files in a Directory

Use the following procedure to load all the files in a native directory and its sub-directories. To load selected files, see Filtering Documents Loaded From a Directory.

1. Set -input_file_path to the input directory.
2. Set -input_file_type if your input files are not documents. For example, if loading from delimited text files, sequence files, aggregate XML files, or database archives.
3. Set -document_type if -input_file_type is not documents and the content type cannot be accurately deduced from the file suffixes as described in How mlcp Determines Document Type.
4. Set -mode:
   • To perform the work locally, set -mode to local.

By default, the imported documents have database URIs based on the input file path. For details, see Controlling Database URIs During Ingestion.

The following example command loads all the files in /space/bill/data:

# Windows users, see Modifying the Example Commands for Windows 
$ mlcp.sh import -host localhost -port 8000 -username user \
    -password password -input_file_path /space/bill/data

Filtering Documents Loaded From a Directory

If -input_file_path names a directory, mlcp loads all the documents in the input directory and subdirectories by default. Use the -input_file_pattern option to filter the loaded documents based on a regular expression.

Input document filtering is handled differently for -input_file_type forest. For details, see Filtering Forest Contents.

For example, the following command loads only files with a .xml suffix from the directory /space/bill/data:

# Windows users, see Modifying the Example Commands for Windows 
$ mlcp.sh import -host localhost -port 8000 -username user \
    -password password -input_file_path /space/bill/data \
    -mode local -input_file_pattern '.*\.xml'

The mlcp tool uses Java regular expression syntax. For details, see Regular Expression Syntax.

Example: Generating Documents From a CSV File

When you import content from delimited text files, mlcp creates an XML or JSON document for each line of input after the initial header line.

The default document type is XML. To create JSON documents, use -document_type json.

When creating XML documents, each document has a root node of <root> and child elements with names corresponding to each column title. You can override the default root element name using the -delimited_root_name option; for details, see Customizing XML Output.

When creating JSON documents, each document is rooted at an unnamed object containing JSON properties with names corresponding to each column title. By default, the values for JSON are always strings. Use -data_type to override this behavior; for details, see Controlling Data Type in JSON Output.

For example, if you have the following data and mlcp command:

# Windows users, see Modifying the Example Commands for Windows 
$ cat example.csv
first,last
george,washington
betsy,ross
$ mlcp.sh ... -mode local -input_file_path /space/mlcp/data \
    -input_file_type delimited_text ...

Then mlcp creates the XML output shown in the table below. To generate the JSON output, add -document_type json to the mlcp command line.

<root>
  <first>george</first>
  <last>washington</last>
</root>

<root>
  <first>betsy</first>
  <last>ross</last>
</root>

{
  "first": "george",
  "last": "washington"
}

{
  "first": "betsy",
  "last": "ross"
}

Expected Input Format

A delimited text input file must have the following characteristics:

• The first line in the input file contains column names that are used to create the XML element or JSON property names of each document created from the file.
• The same delimiter is used to separate each value, as well as the column names. The default separator is a comma; use -delimiter to override it; for details, see Specifying the Field Delimiter.
• Every line has the same number of fields (values). Empty fields are represented as two delimiters in a row, such as a,b,,d.

For example, the following data meets the input format requirements:

first,last
george,washington
betsy,ross

This data produces documents with XML elements or JSON properties named first and last.

Customizing XML Output

When creating XML documents, each document has a root node of <root> and child elements with names corresponding to each column title. You can override the default root element name using the -delimited_root_name option. You can use the -namespace option to specify a root namespace.

The following example produces documents with root element <person> in the namespace http://my.namespace.

$ mlcp.sh ... -mode local -input_file_path /space/mlcp/data \
    -input_file_type delimited_text -namespace http://my.namespace \
    -delimited_root_name person
...
<person xmlns="http://my.namespace">
  <first>george</first>
  <last>washington</last>
</person>
...

Controlling Data Type in JSON Output

When creating JSON documents, the default value type is string. You can use the -data_type option to specify explicit data types for some or all columns. The options accepts comma-separated list of columnName,typeName pairs, where the typeName can be one of number, boolean, or string.

For example, if you have an input file called catalog.csv that looks like the following:

id, price, in-stock
12345, 8.99, true
67890, 2.00,false

Then the default output documents look similar to the following. Notice that all the property values are strings.

{ "id": "12345",
  "price": "8.99",
  "in-stock: "true"
}

The following example command uses the -data_type option to make the price property a number value and the in-stock property a boolean value. Since the id field is not specified in the -data_type option, it remains a string.

$ mlcp.sh ... -mode local -input_file_path catalog.csv \
    -input_file_type delimited_text -document_type json \
    -data_type "price,number,in-stock,boolean"
...
{ "id": "12345",
  "price": 8.99,
  "in-stock: true
}

Controlling the Output Document URI

By default, the document URIs use the value in the first column. For example, if your input data looks like the following:

first,last
george,washington
betsy,ross

Then importing this data with no URI related options creates two documents with name corresponding to the first value. The URI will be george and betsy.

Use -uri_id to choose a different column or -generate_uri to have MarkLogic Server automatically generate a unique URI for each document. For example, the following command creates the documents washington and ross:

# Windows users, see Modifying the Example Commands for Windows 
$ mlcp.sh ... -mode local -input_file_path /space/mlcp/data \
    -input_file_type delimited_text -uri_id last

Note that URIs generated with -generate_uri are only guaranteed to be unique across your import operation. For details, see Default Document URI Construction.

You can further tailor the URIs using -output_uri_prefix and -output_uri_suffix. These options apply even when you use -generate_uri. For details, see Controlling Database URIs During Ingestion.

If your URI id's are not unique, you can overwrite one document in your input set with another. Importing documents with non-unique URI id's from multiple threads can also cause deadlocks.

Specifying the Field Delimiter

The default delimiter between fields is a comma (,). You can override this using the -delimiter option. If the delimiter is a character that is difficult to specify on the command line, specify the delimiter in an options file instead. For details, see Options File Syntax.

For example, the Linux bash shell parser makes it difficult to specify a tab delimiter on the command line, so you can put the options in a file instead. In the example options file below, the string literal after -delimiter should contain a tab character.

$ cat delim.opt
-input_file_type
delimited_text
-delimiter
"tab"
$ mlcp.sh import ... -mode local -input_file_path /space/mlcp/data \
    -options_file delim.opt

Optimizing Ingestion of Large Files

If your delimited text files are very large, consider using the -split_input option. When this option is true, mlcp attempts to break each input file into multiple splits, enabling more documents to be loaded in parallel. For details, see Improving Throughput with -split_input.

Line-Delimited JSON Overview

A line-delimited JSON file is a type of aggregate file where each line is a self-contained piece of JSON data, such as an object or array.

Usually, each line of input has similar structure, such as the following:

{"id": "12345","price":8.99, "in-stock": true}
{"id": "67890","price":2.00, "in-stock": false}

However, the JSON data on each line is independent of the other lines, so the lines do not have to contain JSON data of the same shape. For example, the following is a valid input file:

{"first": "george", "last": "washington"}
{"id": 12345, "price": 8.99, "in-stock": true}

Given the input shown below, the following command creates 2 JSON documents. Each document contains the data from a single line of input.

$ cat example.json
{"id": "12345","price":8.99, "in-stock": true}
{"id": "67890","price":2.00, "in-stock": false}

# Windows users, see Modifying the Example Commands for Windows 
$ mlcp.sh import -host localhost -port 8000 -username user \
    -password password -mode local -input_file_path example.json \
    -input_file_type delimited_json

The example command creates documents whose contents precisely mirror each of input:

{"id": "12345","price":8.99, "in-stock": true}

{"id": "67890","price":2.00, "in-stock": false}

Controlling the Output Document URI

The default document URI is generated from the input file name, the split number, and a sequence number within the split, as described in Default Document URI Construction. For example, if the input file absolute path is /space/data/example.json, then the default output document URIs have the following form:

/space/data/example.json-0-1
/space/data/example.json-0-2
...

You can base the URI on values in the content instead by using the -uri_id option to specify the name of a property found in the data. You can further tailor the URIs using -output_uri_prefix and -output_uri_suffix. For details, see Controlling Database URIs During Ingestion.

For example, the following command uses the value in the id field as the base of the URI and uses -output_uri_suffix to add a .json suffix to the URIs:

# Windows users, see Modifying the Example Commands for Windows 
$ mlcp.sh ... -mode local -input_file_path /space/data/example.json \
    -input_file_type delimited_json 
    -uri_id id -output_uri_suffix ".json"

Given these options, an input line of the form shown below produces a document with the URI 12345.json instead of /space/data/example.json-0-1.

{"id": "12345","price":8.99, "in-stock": true}

If the property name specified with -uri_id is not unique in your data, mlcp will use the first occurrence found in a breadth first search. The value of the specified property should be a valid number or string.

If you use -uri_id, any records (lines) that do not contain the named property are skipped. If the property is found but the value is null or not a number or string, the record is skipped.

Basics of Triple Loading

To load semantic triples, use -input_file_type rdf and follow the instructions for loading a single file, all files in a directory, or a compressed file. For example, the following command loads triples files from the directory /my/data.

# Windows users, see Modifying the Example Commands for Windows 
$ mlcp.sh import -username user -password password -host localhost \
    -port 8000 -input_file_path /my/data -mode local \
    -input_file_type rdf 

You can use mlcp to load triples files in several formats, including RDF/XML, Turtle, and N-Quads. For a full list of supported formats, see Supported RDF Triple Formats in Semantic Graph Developer's Guide.

Each time you load triples from a file, mlcp inserts new documents into the database. That is, multiple loads of the same input inserts new triples each time, rather than overwriting. Only the XQuery and REST API allow you replace triples.

Load triples data embedded within other content according to the instructions for the enclosing input file type, rather than with -input_file_type rdf. For example, if you have an XML input document that happens to have some triples embedded in it, load the document using -input_file_type documents.

You cannot combine loading triples files with other input file types.

If you do not include any graph selection options in your mlcp command, Quads are loaded into the graph specified in the data. Quads with no explicit graph specification and other kinds of triple data are loaded into the default graph. You can change this behavior with options. For details, see Graph Selection When Loading Quads or Graph Selection for Other Triple Types.

For details, see Loading Triples with mlcp in Semantic Graph Developer's Guide.

Graph Selection When Loading Quads

When loading quads, you can use the following command line options to control the graph into which your quads are loaded:

• -output_graph
• -output_override_graph
• -output_collections

You can use -output_collections by itself or with the other two options. You cannot use -output_graph and -output_override_graph together.

If your semantic data is not in a quad format like N-Quads, see Graph Selection for Other Triple Types.

Quads interact with these options differently than other triple formats because quads can include a graph IRI in each quad. The following table summarizes the affect of various option combinations when importing quads with mlcp:

For more details, see Loading Triples with mlcp in the Semantic Graph Developer's Guide.

For example, suppose you load the following N-Quad data with mlcp. There are 3 quads in the data set. The first and last quad include a graph IRI, the second quad does not.

<http://one.example/subject1> <http://one.example/predicate1>
    <http://one.example/object1> <http://example.org/graph3> .
_:subject1 <http://an.example/predicate1> "object1"  .
_:subject2 <http://an.example/predicate2> "object2"
    <http://example.org/graph5> .

If you use a command similar to the following load the data:

# Windows users, see Modifying the Example Commands for Windows 
$ mlcp.sh import -username user -password password -host localhost \
    -port 8000 -input_file_path /my/data.nq -mode local \
    -input_file_type rdf 

Then the table below illustrates how the various graph related options affect how the triples are loaded into the database:

none

-output_graph /my/graph

-output_override_graph /my/graph

-output_collections "aa,bb,cc"

-output_graph /my/graph
-output_collections "bb,cc"

-output_override_graph /my/graph
-output_collections "bb,cc"

Graph Selection for Other Triple Types

When loading triples (rather than quads), you can use the following command line options to control the graph into which your triples are loaded:

• -output_graph
• -output_collections

The following table summarizes the affect of various option combinations when importing triples with mlcp. For quads, see Graph Selection When Loading Quads.

For more details, see Loading Triples with mlcp in the Semantic Graph Developer's Guide.

For example, if you use a command similar to the following load triples data:

# Windows users, see Modifying the Example Commands for Windows 
$ mlcp.sh import -username user -password password -host localhost \
    -port 8000 -input_file_path /my/data.nt -mode local \
    -input_file_type rdf

Then the table below illustrates how the various graph related options affect how the triples are loaded into the database:

-output_graph /my/graph

-output_collections "aa,bb,cc"

-output_graph /my/graph
-output_collections "bb,cc"

Time vs. Space: Configuring Batch and Transaction Size

You can tune the document insertion throughput and memory requirements of your job by configuring the batch size and transaction size of the job.

• -batch_size controls the number of updates per request to the server.
• -transaction_size controls the number of requests to the server per transaction.

The default batch size is 100 and the maximum batch size is 200. (However, some options can affect the default). The default transaction size is 1 and the maximum transaction size is 4000/actualBatchSize. This means that the default maximum number of updates per transaction is 1000, and updates per transaction can range from 20 to 4000.

Selecting a batch size is a speed vs. memory tradeoff. Each request to the server introduces overhead because extra work must be done. However, unless you use -streaming or -document_type mixed, all the updates in a batch stay in memory until a request is sent, so larger batches consume more more memory.

Transactions introduce overhead on MarkLogic Server, so performing multiple updates per transaction can improve insertion throughput. However, an open transaction holds locks on fragments with pending updates, potentially increasing lock contention and affecting overall application performance.

It is also possible to overwhelm MarkLogic Server if you have too many concurrent sessions active.

Time vs. Correctness: Understanding -fastload Tradeoffs

The -fastload option can significantly speed up ingestion during import and copy operations, but it can also cause problems if not used properly. This section describes how -fastload affects the behavior of mlcp and some of the tradeoffs associated with enabling it.

The optimizations described by this section are only enabled if you explicitly specify the -fastload or -output_directory options. (The -output_directory option implies -fastload).

The -fastload option work slightly different when used with -restrict_hosts. For details, see How -restrict_hosts Affects -fastload. The limitations of -fastload described in this section still apply.

By default, mlcp inserts documents into the database by distributing work across the e-nodes in your MarkLogic cluster. Each e-node inserts documents into the database according to the configured document assignment policy.

This means the default insertion process for a document is similar to the following:

1. mlcp selects Host A from the available e-nodes in the cluster and sends it the document to be inserted.
2. Using the document assignment policy configured for the database, Host A determines the document should be inserted into Forest F on Host B.
3. Host A sends the document to Host B for insertion.

When you use -fastload (or -output_directory), mlcp attempts to cut out the middle step by applying the document assignment policy on the client. The interaction becomes similar to the following:

1. Using the document assignment policy, mlcp determines the document should be inserted into Forest F on Host B.
2. mlcp sends the document to Host B for insertion, with instructions to insert it into a specific forest.

Pre-determining the destination host and forest can always be done safely and consistently if the all of the following conditions are met:

• Your forest topology is stable.
• You are creating rather than updating documents.

To make forest assignment decisions locally, mlcp gathers information about the database assignment policy and forest topology at the beginning of a job. If you change the assignment policy or forest topology while an mlcp import or copy operation is running, mlcp might make forest placement decisions inconsistent with those MarkLogic Server would make. This can cause problems such as duplicate document URIs and unbalanced forests.

Similar problems can occur if mlcp attempts to update a document already in the database, and the forest topology or assignment policy changes between the time the document was originally inserted and the time mlcp updates the document. Using user-specified forest placement when initially inserting a document creates the same conflict.

Therefore, it is not safe to enable -fastload optimizations in the following situations:

• A document mlcp inserts already exists in the database and any of the following conditions are true:
  • The forest topology has changed since the document was originally inserted.
  • The assignment policy has changed since the document was originally inserted.
  • The assignment policy is not Legacy (default) or Bucket. For details, see How Assignment Policy Affects Optimization.
  • The document was originally inserted using user-specified forest placement.
• A document mlcp inserts does not already exist in the database and any of the following conditions are true:
  • The forest topology changes while mlcp is running.
  • The assignment policy changes while mlcp is running.

Assignment policy is a database configuration setting that affects how MarkLogic Server selects what forest to insert a document into or move a document into during rebalancing. For details, see Rebalancer Document Assignment Policies in Administrator's Guide.

Assignment policy was introduced with MarkLogic 7 and mlcp v1.2. If you use an earlier version of mlcp with MarkLogic 7 or later, the database you import data into with -fastload or -output_directory must be using the legacy assignment policy.

Any operation that changes the forests available for updates changes your forest topology, including the following:

• Adding or an employing a new forest
• Removing or retiring an existing forest
• Changing the updates-allowed state of forest. For example, calling admin:forest-set-updates-allowed
• Changing the database assignment policy

In most cases, it is your responsibility to determine whether or not you can safely use -fastload (or -output_directory, which implies -fastload). In cases where mlcp can detect -fastload is unsafe, it will disable it or give you an error.

How Assignment Policy Affects Optimization

This section describes how your choice of document assignment policy can introduce additional limitations and risks. Assignment policy is a database configuration setting that affects how MarkLogic Server selects what forest to insert a document into or move a document into during rebalancing. For details, see Rebalancer Document Assignment Policies in Administrator's Guide.

Assignment policy was introduced with MarkLogic 7 and mlcp v1.2. If you use an earlier version of mlcp with MarkLogic 7 or later, the database you import data into with -fastload or -output_directory must be using the legacy assignment policy.

The following table summarizes the limitations imposed by each assignment policy. If you do not explicitly set assignment policy, the default is Legacy or Bucket.

Tuning Split Size and Thread Count for Local Mode

You can tune split size only when importing documents in local mode from one of the following input file types:

• Whole documents (-input_file_type documents), whether from flat or compressed files.
• Composite file types that support -split_input, such as delimited_text.

You cannot tune split size when creating documents from composite files that do not support -split_input, such as sequence files and aggregate XML files.

You can tune thread count for both whole documents and all composite files types. Thread count and split size can interact to affect job performance.

In local mode, a split defines the unit of work per thread devoted to a session with MarkLogic Server. The ideal split size is one that keeps all mlcp session threads busy. The default split size is 32M for local mode. Use the -max_split_size, -thread_count, and -thread_count_per_split options to tune your load.

By default, threads are assigned to splits in a round-robin fashion. For example, consider a loading 120 small documents of length 1M. Since the default split size is 32M, the load is broken into 4 splits. If -thread_count is 10, each split is assigned to at least 2 threads (10 / 4 = 2). The remaining 2 threads are each assigned to a split, so the number of threads per split are distributed as follows:

Split 1: 3 threads
Split 2: 3 threads
Split 3: 2 threads
Split 4: 2 threads

This distribution could result in two of the splits completing faster, leaving some threads idle. If you set -max_split_size to 12M, the load has 10 splits, which can be evenly distributed across the threads and may result in better thread utilization.

Prior to 10.0-4.2, mlcp uses 4 as the default thread count. For mlcp versions equal to or higher than 10.0-4.2, mlcp conducts initial polling to identify the available server threads on the port that handles mlcp requests. Mlcp then uses this value as the default thread count. Users can overwrite the default value by specifying -thread_count in the command line.

If -thread_count is less than the number of splits, the default behavior is one thread per split, up to the total number of threads. The remaining splits must wait until a thread becomes available.

If you specify -thread_count_per_split, each input split will run with the specified number. The total number of thread count, however, is controlled by the newly calculated thread count or -thread_count if it is specified.

If MarkLogic Server is not I/O bound, then raising the thread count, and possibly threads per split, can improve throughput when the number of splits is small but each split is very large. This is often applicable to loading from zip files, aggregate files, and delimited text files. Note that if MarkLogic Server is already I/O bound in your environment, increasing the concurrency of writes will not necessarily improve performance.

Reducing Memory Consumption With Streaming

The streaming protocol allows you to insert a large document into the database without holding the entire document in memory. Streaming uploads documents to MarkLogic Server in 128k chunks.

Streaming content into the database usually requires less memory on the host running mlcp, but ingestion can be slower because it introduces additional network overhead. Streaming also does not take advantage of mlcp's builtin retry mechanism. If an error occurs that is normally retryable, the job will fail.

Streaming is only usable when -input_file_type is documents. You cannot use streaming with delimited text files, sequence files, or archives.

To use streaming, enable the -streaming option. For example:

# Windows users, see Modifying the Example Commands for Windows 
$ mlcp.sh import -username user -password password -host localhost \
    -port 8000 -input_file_path /my/dir -streaming

Improving Throughput with -split_input

If you are loading documents from very large files, you might be able to improve throughput using the -split_input option. When -split_input is true, mlcp attempts to break large input files that would otherwise by processed in a single split into multiple splits. This enables portions of the input file to be loaded by threads (local mode).

This option can only be applied to composite input file types that logically produce multiple documents and for which mlcp can efficiently identify document boundaries, such as delimited_text. Not all composite file types are supported and files containing multi-byte characters must be UTF-8 encoded; for details, see Import Command Line Options.

In local mode, -split_input is false by default.

The -split_input option affects local mode as follows: Suppose you are importing a very large delimited text file in local mode with -split_input set to false and the data processed as a single split. The work might be performed by multiple threads (depending on the job configuration), but these threads read records from the input file synchronously. This can cause some read contention. If you set -split_input to true, then each thread is assigned its own chunk of input, resulting in less contention and greater concurrency.

The number of subdivisions is determined by the formula file-size / max-split-size, so you should also consider tuning split size to match your input data characteristics. For example, if your data consists of 1 delimited text file containing 16M of data, you can observe the following interactions between -split_input and -max_split_size:

Tuning the split size in this case potentially enables greater concurrency because the multiple splits can be assigned to different threads or tasks.

Split size is tunable using -max_split_size, -min_split_size, and block size. For details, see Tuning Split Size and Thread Count for Local Mode.

MLCP Concurent Jobs

We do not recommend using concurrent mlcp jobs. Regardless of the version, mlcp doesn't support concurrent jobs if mlcp is importing from/exporting to the same data file. In addition, beginning in 10.0-4.2, each mlcp job uses the maximum number of threads available on the server as the default thread count (more about this can be found in the 10.0-4.2 release notes). Therefore, using concurrent mlcp jobs will not improve performance, as one job is already using full concurrent capacity.

Creating a Custom XQuery Transformation

The following topics describe how to implement a server-side content transformation function in XQuery:

• Function Signature
• Input Parameters
• Expected Output
• Example Implementation

declare function yourNamespace:transform(
  $content as map:map,
  $context as map:map)
as map:map*

document-node

document-node

binary-node

text-node

declare function example:transform(
  $content as map:map,
  $context as map:map
) as map:map*
{
  let $attr-value := 
   (map:get($context, "transform_param"), "UNDEFINED")[1]
  let $the-doc := map:get($content, "value")
  return
    if (fn:empty($the-doc/element()))
    then $content
    else
      let $root := $the-doc/*
      return (
        map:put($content, "value",
          document {
            $root/preceding-sibling::node(),
            element {fn:name($root)} {
              attribute { fn:QName("", "NEWATTR") } {$attr-value},
              $root/@*,
              $root/node()
            },
            $root/following-sibling::node()
          }
        ), $content
      )
};

Creating a Custom JavaScript Transformation

The following topics describe how to implement a server-side content transformation function in Server-Side JavaScript:

• Function Signature
• Input Parameters
• Expected Output
• Example Implementation

function yourTransform(content, context)

{ uri: string,
  value: node
}

document-node

document-node

binary-node

text-node

{ transform_param: string,
  collections: [ string, ... ],
  permissions: [ object, ... ],
  quality: number,
  temporalCollection: string}

// Add a property named "NEWPROP" to any JSON input document.
// Otherwise, input passes through unchanged.

function addProp(content, context)
{
  const propVal = (context.transform_param == undefined)
                 ? "UNDEFINED" : context.transform_param;

  if (xdmp.nodeKind(content.value) == 'document' &&
      content.value.documentFormat == 'JSON') {
    // Convert input to mutable object and add new property
    const newDoc = content.value.toObject();
    newDoc.NEWPROP = propVal;

    // Convert result back into a document
    content.value = xdmp.unquote(xdmp.quote(newDoc));
  }
  return content;
};

exports.addProp = addProp;

Implementation Guidelines

You should be aware of the following guidelines and limitations when implementing your transformation function:

• If you use a server-side transform with -fastload (or -output_directory, which enables -fastload), your transformation function only has access to database content in the same forest as the input document. If your transformation function needs general access to the database, do not use -fastload or -output_directory.

Installing a Custom Transformation

Install the XQuery library module containing your function into the modules database or modules root directory of the XDBC App Server associated with the destination database. For import operations, this is the App Server identified by -host and -port mlcp command line options. For copy operations, this is the App Server identified by -output_host and -output_port mlcp command line options.

Best practice is to install your libraries into the modules database of your XDBC App Server. If you install your module into the modules database, MarkLogic Server automatically makes the implementation available throughout your MarkLogic Server cluster. If you choose to install dependent libraries into the Modules directory of your MarkLogic Server installation, you must manually do so on each node in your cluster.

MarkLogic Server supports several methods for loading modules into the modules database:

• Run an XQuery or JavaScript query in Query Console. For example, you can run a query similar to the following to install a module using Query Console. Note: First select your modules database in the Query Console Content Source dropdown.

  xquery version "1.0-ml";
  xdmp:document-load("/space/mlcp/transform.xqy",
      <options xmlns="xdmp:document-load">
        <uri>/example/mlcp-transform.xqy</uri>
        <repair>none</repair>
        <permissions>{xdmp:default-permissions()}</permissions>
      </options>)

• If you use the App Server on port 8000 or have a REST API instance, you can use any of the following Client APIs:
  • Java: ResourceExtensionsManager.write. For details, see Managing Dependent Libraries and Other Assets in the Java Application Developer's Guide.
  • Node.js: DatabaseClient.config.extlibs. For details, see Managing Assets in the Modules Database in the Node.js Application Developer's Guide.
  • REST: PUT /v1/ext/{directories}/{asset}. For details, see Managing Dependent Libraries and Other Assets in the REST Application Developer's Guide.

If you use the filesystem instead of a modules database, you can manually install your module into the Modules directory. Copy the module into MARKLOGIC_INSTALL_DIR/Modules or into a subdirectory of this directory. The default location of this directory is:

• Unix: /opt/MarkLogic/Modules
• Windows: C:\Program Files\MarkLogic\Modules

If your transformation function requires other modules, you should also install the dependent libraries in the modules database or the modules directory.

For a complete example, see Example: Server-Side Content Transformation.

Using a Custom Transformation

Once you install a custom transformation function on MarkLogic Server, you can apply it to your mlcp import or copy job using the following options:

• -transform_module - The path to the module containing your transformation. Required.
• -transform_namespace - The namespace of your transformation function. If omitted, no namespace is assumed.
• -transform_function - The local name of your transformation function. If omitted, the name transform is assumed.
• -transform_param - Optional additional string data to be passed through to your transformation function.

Take note of the following limitations:

• When -fastload is in effect, your transform function runs in the scope of a single forest (the forest mlcp determines is the appropriate destination for the file being inserted). This means if you change the document URI as part of your transform, you can end up creating documents with duplicate URIs.
• When you use a transform function, all the documents in each batch are transformed and inserted into the database as a single statement. This means, for example, that if the (transformed) batch contain more than one document with the same URI, you will get an XDMP-CONFLICTINGUPDATES error.

The following example command assumes you previously installed a transform module with path /example/mlcp-transform.xqy, and that the function implements a transform function (the default function) in the namespace http://marklogic.com/example. The function expects a user-defined parameter value, supplied using the -transform_param option.

# Windows users, see Modifying the Example Commands for Windows 
$ mlcp.sh import -mode local -host mlhost -port 8000 \
    -username user -password password \
    -input_file_path /space/mlcp-test/data \
    -transform_module /example/mlcp-transform.xqy \
    -transform_namespace "http://marklogic.com/example" \
    -transform_param "my-value"

For a complete example, see Example: Server-Side Content Transformation.

Example: Server-Side Content Transformation

This example walks you through installing and using an XQuery or Server-Side JavaScript transform function to modify content ingested with mlcp. The example XQuery transform function modifies XML documents by adding an attribute named NEWATTR, with an attribute value specified on the mlcp command line. The example JavaScript transform function modifies JSON documents by adding a new property named NEWPROP, with a value specified on the mlcp command line.

This example assumes you have already created an XDBC App Server, configured to use "/" as the root and a modules database of Modules.

1. Create the sample input files
2. Create the XQuery transform module
3. Create the JavaScript transform module
4. Install the transformation module
5. Apply the transformation

xquery version "1.0-ml";
module namespace example = "http://marklogic.com/example";

(: If the input document is XML, insert @NEWATTR, with the value
 : specified in the input parameter. If the input document is not
 : XML, leave it as-is.
 :)
declare function example:transform(
  $content as map:map,
  $context as map:map
) as map:map*
{
  let $attr-value := 
    (map:get($context, "transform_param"), "UNDEFINED")[1]
  let $the-doc := map:get($content, "value")
  return
    if (fn:empty($the-doc/element()))
    then $content
    else
      let $root := $the-doc/*
      return (
        map:put($content, "value",
          document {
            $root/preceding-sibling::node(),
            element {fn:name($root)} {
              attribute { fn:QName("", "NEWATTR") } {$attr-value},
              $root/@*,
              $root/node()
            },
            $root/following-sibling::node()
          }
        ), $content
      )
};

// Add a property named "NEWPROP" to any JSON input document.
// Otherwise, input passes through unchanged.

function addProp(content, context)
{
  var propVal = (context.transform_param == undefined)
                 ? "UNDEFINED" : context.transform_param;

  var docType = xdmp.nodeKind(content.value);
  if (xdmp.nodeKind(content.value) == 'document' &&
      content.value.documentFormat == 'JSON') {
    // Convert input to mutable object and add new property
    var newDoc = content.value.toObject();
    newDoc.NEWPROP = propVal;

    // Convert result back into a document
    content.value = xdmp.unquote(xdmp.quote(newDoc));
  }
  return content;
};

exports.transform = addProp;

# Windows users, see Modifying the Example Commands for Windows 
$ mlcp.sh import -mode local -host mlhost -port 8000 \
    -username user -password password \
    -input_file_path /space/mlcp/txform/data \
    -transform_module /example/mlcp-transform.xqy \
    -transform_namespace "http://marklogic.com/example" \
    -transform_param "my-value"

# Windows users, see Modifying the Example Commands for Windows 
$ mlcp.sh import -mode local -host mlhost -port 8000 \
    -username user -password password \
    -input_file_path /space/mlcp/txform/data \
    -transform_module /example/mlcp-transform.sjs \
    -transform_function transform \
    -transform_param "my-value"

... INFO contentpump.LocalJobRunner: OUTPUT_RECORDS: 2
... INFO contentpump.LocalJobRunner: Total execution time: 1 sec

<parent NEWATTR="my-value">
  <child/>
</parent>

{ "key": "value", "NEWPROP": "my-value"}

Example: Changing the URI and Document Type

This example demonstrates changing the type of a document from binary to XML and changing the document URI to match.

Transforms that change the document URI should not be combined with the -fastload or -output_directory options as they can cause duplicate document URIs. For details, see Time vs. Correctness: Understanding -fastload Tradeoffs.

As described in How mlcp Determines Document Type, the URI extension and MIME type mapping are used to determine document type when you use -document_type mixed. However, transform functions do not run until after document type selection is completed. Therefore, if you want to affect document type in a transform, you must convert the document node, as well as optionally changing the output URI.

Suppose your input document set generates an output document URI with the unmapped extension .1, such as /path/doc.1. Since 1 is not a recognized URI extension, mlcp creates a binary document node from this input file by default. The example transform function in this section intercepts such a document and transforms it into an XML document.

• XQuery Implementation
• JavaScript Implementation

Note that if you define a MIME type mapping that maps the extension 1 to XML (or JSON) in your MarkLogic Server configuration, then mlcp creates a document of the appropriate type to begin with, and this conversion becomes unnecessary.

xquery version "1.0-ml";
module namespace example = "http://marklogic.com/example";

declare function example:mod_doc_type(
  $content as map:map,
  $context as map:map
) as map:map*
{
  let $orig-uri := map:get($content, "uri")
  return
  if (fn:substring-after($orig-uri, ".") = "1") then
    let $doc-type := xdmp:node-kind(map:get($content, "value"))
    return (
      (: change the URI to an xml suffix :)
      map:put($content, "uri", 
        fn:concat(fn:substring-before($orig-uri, "."), ".xml")
      ),
      (: convert the input from binary node to xml document node :)
      if ($doc-type = "binary") then
        map:put(
          $content, "value",
          xdmp:unquote(xdmp:quote(map:get($content, "value")))
        )
      else (),
      $content
    )
  else $content
};

function modDocType(content, context)
{
  var uri = String(content.uri);
  var dot = uri.lastIndexOf('.');
  if (dot > 0) {
    var suffix = uri.slice(dot);
    if (suffix == '.1') {
      content.uri = uri.substring(0,dot+1) + 'json';
      if (xdmp.nodeKind(content.value) == 'binary') {
        // convert the content to a JSON document
        content.value = xdmp.unquote(xdmp.quote(content.value));
      }
    }
  }
  return content;
};

exports.transform = modDocType;

How mlcp Uses the Host List

You must specify at least one host with -host command line option. You can specify multiple hosts.

If any hostname listed in the value of the -host option is not resolvable by mlcp at the beginning of a job, then mlcp will abort the job with an IllegalArgumentException.

Assuming all hostnames are resolvable, mlcp uses the first of these hosts to gather information about the target database. If mlcp is unable to connect to the first host in the -host list, then mlcp will move on to the next host in the list. If mlcp cannot connect to any of the listed hosts, then the job will fail with an IOException.

If mlcp successfully retrieves a list of forest hosts, then mlcp subsequently connects directly to these hosts when distributing work across the cluster, whether or not these hosts are specified in the -host option. In this way, your job does not need to be aware cluster topology.

This behavior applies to the import, export, and copy commands. (For a copy job, you specify hosts through -input_host and -output_host, rather than -host.)

You can also restrict mlcp to just the hosts listed by the -host option. For details, see Restricting the Hosts mlcp Uses to Connect to MarkLogic.

Restricting the Hosts mlcp Uses to Connect to MarkLogic

You can restrict the hosts to which mlcp distributes work using the -restrict_hosts and -host command line options. You might find this option combination useful in situations such as the following:

• Limit the host working set to just the e-nodes in your cluster.
• The public and private DNS names of a host differ, such as can occur for an AWS instance.

  MarkLogic automatically sets -restrict_hosts to true when it detects the presence of a load balancer.

When -restrict_hosts is set to true, mlcp will only connect to the hosts listed in the -host option, rather than using the approach described in How mlcp Uses the Host List.

Using -restrict_hosts will usually degrade the performance of an mlcp job because mlcp cannot distribute work as efficiently.

For example, if you're using mlcp with a load balancer between your client and your MarkLogic cluster, you can specify the load balancer with -host and set -restrict_hosts to true to prevent mlcp from attempting to bypass the load balancer and connect directly to the forest hosts.

You can restrict mlcp's host list when using the import, export, and copy commands. For import and export, use the -host and -restrict_hosts options. For copy, use -input_host and -restrict_input_hosts and/or -output_host and -restrict_output_hosts.

How -restrict_hosts Affects -fastload

You can use -fastload with -restrict_hosts. The performance improvement from -fastload will be less than if you did not use -restrict_hosts, but better than if you do not use -fastload. The usual cautions about -fastload apply; see Time vs. Correctness: Understanding -fastload Tradeoffs.

The -fastload and -restrict_hosts options interact as follows:

Without -restrict_hosts, mlcp figures out which hosts contains the destination forest for a document, and then connects directly to that host. When -restrict_hosts is true, a connection to the forest host might not possible. In this case, mlcp connects to an allowed e-node, and includes the detailed destination information along with the document. The destination details makes an insertion faster than it would otherwise be.

How MLCP Adjusts Client Concurrency

When running an import job, mlcp periodically send polling requests to the server through the XCC layer to obtain the maximum server threads. When the DHS cluster adds more dynamic e-nodes, server has more available concurrency. Then mlcp decides whether to scale-out or scale-in, its own thread pool based on the result.

The following command line options can be used to tune this process:

• -max_thread_percentage: The percentage (between 0 and 100) of maximum available server threads mlcp will use to run import jobs.
• -polling_period: The time interval (in minutes) mlcp sends the polling requests to the server.
• -polling_init_delay: The initial delay (in minutes) before mlcp starts sending the polling requests.

How Other Command Line Options Affect Auto-scaling

The following existing command line options also affect the auto-scaling feature:

• -thread_count and -thread_count_per_split: When these two options are specified, mlcp will use a fixed number of threads and auto-scaling will not happen.
• -max_threads: When -max_threads is specified, mlcp will cap the maximum thread count, and auto-scaling cannot go beyond this number. This is to prevent the client-side from running out of memory as the DHS cluster may have a huge number of nodes. By default, -max_threads is not set.

How MLCP Assigns Threads in Auto-Scaling Process

When mlcp scales-out or scales-in, new threads are assigned to or removed from the existing input splits using round-robin fashion, same as the logic discussed in Tuning Split Size and Thread Count for Local Mode.

MLCP Logs for Auto-Scaling

When mlcp scales-out or scales-in, there will be a log message on INFO level to notify user about the scaling process. If the thread count has reached the maximum value, it will also be logged on INFO level. For every periodic polling, mlcp will log new available server threads on DEBUG level. If mlcp decides to scale-out or scale-in, the assigned or deducted threads for each input split will also be logged on DEBUG level.

The following messages are an example of common log messages a user may get in an auto-scaling process. Timestamps have been removed.

DEBUG contentpump.ThreadManager: Initial thread pool size: 32
DEBUG contentpump.ThreadManager: Thread pool will auto-scale based on available server threads.
DEBUG contentpump.ThreadManager: Running with MultithreadedMapper. Initial thread count for split #0: 11
DEBUG contentpump.ThreadManager: Running with MultithreadedMapper. Initial thread count for split #1: 11
DEBUG contentpump.ThreadManager: Running with MultithreadedMapper. Initial thread count for split #2: 10
INFO contentpump.LocalJobRunner:  completed 0%
DEBUG contentpump.ThreadManager: New available server threads: 32
DEBUG contentpump.ThreadManager: New available server threads: 32
DEBUG contentpump.ThreadManager: New available server threads: 16
INFO contentpump.ThreadManager: Thread pool is scaling-in. New thread pool size: 16
DEBUG contentpump.ThreadManager: Running with MultithreadedMapper. New thread count for split #0: 6
DEBUG contentpump.ThreadManager: Running with MultithreadedMapper. New thread count for split #1: 5
DEBUG contentpump.ThreadManager: Running with MultithreadedMapper. New thread count for split #2: 5
DEBUG contentpump.ThreadManager: New available server threads: 16