MarkLogic Server 11.0 Product Documentation

mlcp User Guide — Chapter 3

Prepare to Run the Examples

This section leads you through creating a work area and sample data with the following file system layout:

gs/
  import/
    one.xml
    two.json
  export/

Follow this procedure to set up the example work area

1. Download and install mlcp according to the instructions in Installation and Configuration.
2. Ensure the mlcp bin directory and the java commands are on your path. For example, the following example command places the mlcp bin directory on your path if mlcp is installed in MLCP_INSTALL_DIR:

   Linux: export PATH=${PATH}:MLCP_INSTALL_DIR/bin
   Windows: set PATH=%PATH%;MLCP_INSTALL_DIR\bin

3. Create a directory to serve as your work area and change directories to this work area. For example:

   mkdir gs
   cd gs

4. Create a sub-directory to hold the sample input and output data. For example:

   mkdir import

5. Create the sample input files in the import/ directory.
   1. Use the following commands on Linux:

      echo '<data>1</data>' > import/one.xml
      echo '{"two": 2}' > import/two.json

   2. Use the following commands on Windows:

      echo ^<data^>1^</data^> > import\one.xml
      echo {"two":2} > import\two.json

Optional: Create an Options File

You can encapsulate mlcp command line options in an options file; for details, see Options File Syntax. An options file is convenient for re-use of commonly used options. Also, using an options file can help you avoid command line interpolation of quotes by the shell.

The examples use an options file to save MarkLogic connection related options so that you can easily re-use them across multiple commands. This section describes how to create this file.

If you prefer to pass the connection options directly on the command line instead, add -username, -password, -host, and possibly -port options to the example mlcp commands in place of -options_file.

Use the following procedure to create the example options file.

1. If you are not already at the top level of your work area, change directory to this location. That is, the gs folder created in Prepare to Run the Examples.

   cd gs

2. Create a file named conn.txt with the following contents. Each line is either an option name or a value for the preceding option.

   -username
   your_username
   -password
   your_password
   -host
   localhost
   -port
   8000

3. Edit conn.txt and modify the values of the -username and -password options to match your environment.
4. Optionally, modify the -host and/or -port option values. The host and port must identify a MarkLogic Server App Server that supports the XDBC protocol. MarkLogic Server comes with an App Server pre-configured on port 8000 that supports XDBC, attached to the Documents database. You can choose a different App Server.

You should now have the following file structure:

gs/
  conn.txt
  import/
    one.xml
    two.json

Load Documents

Load documents into a MarkLogic Server database using the mlcp import command. The examples in this section load documents from flat files into the default database associated with the App Server on port 8000 (the Documents database).

Other input options include compressed files, delimited text files, aggregate XML data, and line-delimited JSON data. See Importing Content Into MarkLogic Server for details. You can also load document into a different database using the -database option.

To load a single file, specify the path to the file as the value of -input_file_path. For example:

-input_file_path import

When you load documents, a default URI is generated based on the type of input data. For details, see Controlling Database URIs During Ingestion.

We will import documents from flat files, so the default URI is the absolute pathname of the input file. For example, if your work area is /space/gs on Linux or C:\gs on Windows, then the default URI when you import documents from gs/import is as follows:

Linux: /space/gs/import/filename
Windows: /c:/gs/import/filename

You can use the -output_uri_replace option to strip off the portion of the URI that comes from the path steps before gs. The option argument is of the form pattern,replacement_text. For example, given the default URIs shown above, we'll add the following option to create URIs that begin with /gs:

Linux: -output_uri_replace "/space,''"
Windows: -output_uri_replace "/c:,''"

Run the following command from the root of your work area (gs) to load all the files in the import directory. Modify the argument to -output_uri_replace to match your environment.

Linux: 
  mlcp.sh import -options_file conn.txt \
    -output_uri_replace "/space,''" -input_file_path import

Windows:
  mlcp.bat import -options_file conn.txt ^
    -output_uri_replace "/c:,''" -input_file_path import

The output from mlcp should look similar to the following (but with a timestamp prefix on each line). OUTPUT_RECORDS_COMITTED: 2 indicates mlcp loaded two files. For more details, see Understanding mlcp Output.

INFO contentpump.LocalJobRunner: Content type is set to MIXED.  The format of 
  the  inserted documents will be determined by the MIME  type specification 
  configured on MarkLogic Server.
INFO input.FileInputFormat: Total input paths to process : 2
INFO contentpump.LocalJobRunner:  completed 100%
INFO contentpump.LocalJobRunner: com.marklogic.mapreduce.MarkLogicCounter:
INFO contentpump.LocalJobRunner: INPUT_RECORDS: 2
INFO contentpump.LocalJobRunner: OUTPUT_RECORDS: 2
INFO contentpump.LocalJobRunner: OUTPUT_RECORDS_COMMITTED: 2
INFO contentpump.LocalJobRunner: OUTPUT_RECORDS_FAILED: 0
INFO contentpump.LocalJobRunner: Total execution time: 0 sec

Optionally, use Query Console's Explore feature to examine the contents of the Documents database and see that the documents were created. You should see documents with the following URIs:

/gs/import/one.xml
/gs/import/two.json

You can also create documents from files in a compressed file and from other types of input archives. For details, see Importing Content Into MarkLogic Server.

Export Documents

Use the mlcp export command to export documents from a MarkLogic Server database into files on your filesystem. You can export documents to several formats, including files, compressed files, and database archives. For details, see Exporting Content from MarkLogic Server.

You can identify the documents to export in several ways, including by URI, by directory, by collection, and by XPath expression. This example uses a directory filter. Recall that the input documents were loaded with URIs of the form /gs/import/filename. Therefore we can easily extract the files by database directory using -directory_filter /gs/import/.

This example exports documents from the default database associated with the App Server on port 8000. Use the -database option to export documents from a different database.

Use the following procedure to export the documents inserted in Load Documents.

1. If you are not already at the top level of your work area, change directory to this location. That is, the gs folder created in Prepare to Run the Examples. For example:

   cd gs

2. Extract the previously inserted documents into a directory named export. The export directory must not already exist.

   Linux:
     mlcp.sh export -options_file conn.txt -output_file_path export \
       -directory_filter /gs/import/
   
   Windows:
     mlcp.bat export -options_file conn.txt -output_file_path export ^
       -directory_filter /gs/import/

You should see output similar to the following, but with a timestamp prefix on each line. The OUTPUT_RECORDS: 2 line indicates mlcp exported 2 files.

INFO mapreduce.MarkLogicInputFormat: Fetched 1 forest splits.
INFO mapreduce.MarkLogicInputFormat: Made 1 splits.
INFO contentpump.LocalJobRunner:  completed 100%
INFO contentpump.LocalJobRunner: com.marklogic.mapreduce.MarkLogicCounter:
INFO contentpump.LocalJobRunner: INPUT_RECORDS: 2
INFO contentpump.LocalJobRunner: OUTPUT_RECORDS: 2
INFO contentpump.LocalJobRunner: Total execution time: 0 sec

The exported documents are in gs/export. A filesystem directory is created for each directory step in the original document URI. Therefore, you should now have the following directory structure:

gs/
  export/
    gs/
      import/
        one.xml
        two.json

Understanding mlcp Output

The output from mlcp varies depending on the operation (import, export, copy, extract), but usually looks similar to the following (with a timestamp prefix on each line). The following example is output from an import job.

INFO contentpump.LocalJobRunner: Content type is set to MIXED.  The format of 
  the  inserted documents will be determined by the MIME  type specification 
  configured on MarkLogic Server.
INFO input.FileInputFormat: Total input paths to process : 2
INFO contentpump.LocalJobRunner:  completed 100%
INFO contentpump.LocalJobRunner: com.marklogic.mapreduce.ContentPumpStats:
INFO contentpump.LocalJobRunner: INPUT_RECORDS: 2
INFO contentpump.LocalJobRunner: OUTPUT_RECORDS: 2
INFO contentpump.LocalJobRunner: OUTPUT_RECORDS_COMMITTED: 2
INFO contentpump.LocalJobRunner: OUTPUT_RECORDS_FAILED: 0
INFO contentpump.LocalJobRunner: Total execution time: 0 sec

The following table summarizes the purpose of key pieces of information reported by mlcp:

Content type is set to format X.

Total input paths to process : N

INPUT_RECORDS: N

ESTIMATED_INPUT_RECORDS: N

OUTPUT_RECORDS: N

OUTPUT_RECORDS_COMMITTED: N

OUTPUT_RECORDS_FAILED: N

Stopping an mclp Job Prematurely

In local mode, an interrupted job will shutdown gracefully as long as it can finish within 30 seconds. If that time period expires, mlcp prints a warning.