MarkLogic 9 Product Documentation

mlcp User Guide — Chapter 2

Specifying the Hadoop Configuration File Location

You must tell mlcp where to find the Hadoop configuration files on the host where you run mlcp. Hadoop does not need to be installed on this host, but the Hadoop configuration files must be reachable.

Use one of the following methods to tell mlcp where to find your Hadoop configuration files locally:

• Set the mlcp command line option -hadoop_conf_dir. For example:

  $ mlcp.sh command -hadoop_conf_dir /etc/hadoop/conf

• Set the environment variable HADOOP_CONF_DIR. For example:

  $ export HADOOP_CONF_DIR=/etc/hadoop/conf

If your Apache Hadoop installation is on a remote host, you can copy the configuration files locally and set HADOOP_CONF_DIR (or -hadoop_conf_dir) to that directory.

Setting Custom Hadoop Options and Properties

Use the following mlcp options to pass Hadoop-specific configuration information through mlcp to your Hadoop installation. You can use these options to control mlcp's use of Hadoop in distributed mode.

• -conf conf_filename : Pass in a Hadoop configuration properties file.
• -D property=value : Pass one Hadoop configuration property setting.

The property names and configuration file syntax is as dictated by Hadoop. For details, consult the documentation for your Hadoop distribution.

These Hadoop options must appear on the command line after -options_file (if present) and before any other mlcp-specific options.

Required Hadoop User Privileges

When you use distributed mode for import, the user your Hadoop tasks run as must have permission to access the directories or files specified by -input_file_path. Similarly, when you use distributed mode for export or extract, the user must have permission to create directories and files in the directory specified by -output_file_path.

Using mlcp With MapR

To use MapR as mlcp's Hadoop distribution, you must download the mlcp-mapr-version-bin bundle instead of the standard mlcp bundle. For example, download mlcp-mapr-9.0-bin.zip from developer.marklogic.com.

You must also separately download the maprfs Java JAR file from MapR and make it available to mlcp. You can use the following procedure:

1. Download maprfs-version-mapr.jar from http://repository.mapr.com/nexus/content/groups/mapr-public/com/mapr/hadoop/maprfs/version-mapr/. Modify version to match your MapR version.

   For example, download maprfs-5.1.0-mapr.jar from http://repository.mapr.com/nexus/content/groups/mapr-public/com/mapr/hadoop/maprfs/5.1.0-mapr/.

2. Make the JAR file available to mlcp:
   1. If you have write access to your mlcp installation directory, place the JAR file in MLCP_INSTALL_DIR/lib.
   2. If you do not have write access to your mlcp installation directory, then ensure the JAR file is on your Java classpath when running mlcp.

To avoid warnings about JCE policy files when using mlcp with MapR, you should also install the Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files compatible with your JRE.

For example, you can download the Java 8 policy files for the Oracle JRE from the following location:

http://www.oracle.com/technetwork/java/javase/downloads/jce8-download-2133166.html

Enabling SSL on Your App Server

You can only use SSL to connect to MarkLogic through an SSL-enabled App Server. For more details, see Configuring SSL on App Servers in the Security Guide.

If you want to use SSL with both the source (input) and destination (output) App Servers during an mlcp copy job, both App Servers must be SSL enabled.

Configuring mlcp to Use SSL

By default, mlcp does not connect to MarkLogic using SSL. Use one of the following options to specify that mlcp should connect via SSL:

All these options accept a boolean argument value. As described in Command Line Summary, true is assumed if you leave the argument off.

If you have disabled the default SSL protocol on your App Server, you must also use one of the following options to explicitly specify the SSL protocol that mlcp should use when connecting to MarkLogic:

The above SSL protocol options are ignored in some cases when you use the SSL configuration technique describe in Advanced SSL Configuration.

Advanced SSL Configuration

An advanced user can specify detailed SSL configuration options by following the procedure in this section.

Internally, mlcp uses the MarkLogic Connector for Hadoop to connect to MarkLogic and move data as required by your job. If you make an implementation of com.marklogic.mapreduce.SslConfigOptions available to the connector (through mlcp), then your mlcp job will use the configuration specified by that implementation.

Use the following procedure to create an SSL config class and use it with mlcp:

1. Create a class that implements com.marklogc.mapreduce.SslConfigOptions from the MarkLogic Connector for Hadoop. Your implementation should use one of the techniques described in Accessing SSL-Enabled XDBC App Servers in the XCC Developer's Guide to supply an SSLContext object through the getSslContext method.
2. Create a configuration file that tells the connector to use your SslConfigOptions class to configure SSL connections. See the example below.
3. Ensure your SslConfigOptions implementation is on the classpath for your mlcp job.
4. Use the -conf mlcp command line option to tell mlcp where to find the configuration file from Step 2. For example:

   mlcp import -conf yourConfFile otherMlcpOptions

The following is an example of a MarkLogic Connector for Hadoop configuration for using a custom SSL configuration for an mlcp import job. Substitute your class name for yourSslOptions in the property value.

<property>
  <name>mapreduce.marklogic.output.ssloptionsclass</name>
  <value>yourSslOptions.class</value>
</property>

The following is an example configuration file for an export job. Substitute your class name for yourSslOptions in the property value.

<property>
  <name>mapreduce.marklogic.input.ssloptionsclass</name>
  <value>yourSslOptions.class</value>
</property>

For a copy job, set either or both of mapreduce.marklogic.output.ssloptionsclass or mapreduce.marklogic.input.ssloptionsclass, depending on which connections you wish to secure with SSL.

If you set the property mapreduce.marklogic.input.usessl or mapreduce.marklogic.output.usessl in the configuration file as well as using the -ssl mlcp option, the mlcp command line options take precedence over the configuration file property setting.

When you set the property mapreduce.marklogic.input.ssloptionsclass, mlcp ignores the -ssl_protocol (for import) and -input_ssl_protocol (for copy) command line options. Similarly, if you set the property mapreduce.marklogic.output.ssloptionsclass, mlcp ignores the -ssl_protocol (for export) and -output_ssl_protocol (for copy) command line options.

For an example of a custom SSL configuration implementation, see the ContentReader example for the MarkLogic Connector for Hadoop. The example source code is available in the following ways:

• Download the MarkLogic Connector for Hadoop distribution from http://developer.marklogic.com/products/hadoop,
• Browse or clone a copy of the sources in the GitHub marklogic-contentpump project at http://github.com/marklogic/marklogic-contentpump.

Creating Users

Before you can use Kerberos for authentication, you must create at least one MarkLogic user with which mlcp can use Kerberos authentication to connect to MarkLogic Server, as described in Assigning an External Name to a User in the Security Guide.

This user must also be assigned roles and privileges required to enable your mlcp operations.

For example, if you're using mlcp to import documents into a database, then the user must have update privileges on the target database, as well as the minimum privileges required by mlcp. For details on the minimum privileges required by mlcp, see Security Considerations.

Configuring an XDBC App Server for Kerberos Authentication

The mlcp tool communicates with MarkLogic through an XDBC App Server. Configure your XDBC App Server to use Kerberos for external security, as described in Configuring an App Server for External Authentication in the Security Guide.

Configure your XDBC App Server to use kerberos-ticket authentication.

For example, if you create a configuration named kerb-conf, then configure your XDBC App Server with the following values for the authentication, internal security, and external security configuration settings in the Admin Interface:

You can use an existing XDBC App Server or create a new one. To create a new XDBC App Server, use the Admin Interface, the Admin API, or the REST Management API. For details, see Procedures for Creating and Managing XDBC Servers in the Administrator's Guide.

Configure the App Server to use kerberos-ticket authentication and the Kerberos external security configuration object you created following the instructions in Creating an External Authentication Configuration Object in the Security Guide.

When you install MarkLogic, an XDBC App Server and other services are available port 8000. Changing the security configuration for the App Server on port 8000 affects all the MarkLogic services available through this port, including the HTTP App Server and REST Client API instance.

Invoking mlcp

Once you configure your XDBC App Server and user for Kerberos external security, then you can do the following to use Kerberos authentication with mlcp:

• Use kinit or a similar program on your mlcp client host to create and cache a Kerberos Ticket to Get Tickets (TGT) for a principal you assigned to a MarkLogic user.
• Invoke mlcp with no -username and no -password option from the environment in which you cached the TGT.

For example, suppose you configured an XDBC App Server on port 9010 of host ml-host to use kerberos-ticket authentication. Further, suppose you associated the Kerberos principal name kuser with the user mluser. Then the following commands result in mlcp authenticating with Kerberos as user kuser, and importing documents into the database as mluser.

kinit kuser
...
mlcp.sh import -host ml-host -port 9010 -input_file_path src_dir

You do not necessarily need to run kinit every time you invoke mlcp. The cached TGT typically has a lifetime over which it is valid.