MarkLogic Connector for Hadoop Developer's Guide — Chapter 2

Required Software

The MarkLogic Connector for Hadoop is a Java-only API and is only available on Linux. You can use the connector with any of the Hadoop distributions listed below. Though the Hadoop MapReduce Connector is only supported on the Hadoop distributions listed below, it may work with other distributions, such as an equivalent version of Apache Hadoop.

The following software is required to use the MarkLogic Connector for Hadoop:

• Linux
• MarkLogic 6.0-1 or later
• MarkLogic XML Content Connector for Java (XCC/J) 5.0 or later
• An installation of one of the following Hadoop MapReduce distributions. You might be able to use the Connector with other distributions based on Apache Hadoop v2.6.
  • Cloudera's Distribution Including Apache Hadoop (CDH) version 5.8
  • Hortonworks Data Platform (HDP) version 2.4
  • MapR version 5.1
• Oracle/Sun Java JDK 1.7 or later.

  Apache Hadoop only supports the Oracle/Sun JDK, though other JDK's may work. For details, see http://wiki.apache.org/hadoop/HadoopJavaVersions.

Security Requirements for MapReduce Jobs

The user with which a MapReduce job accesses MarkLogic Server must have appropriate privileges for the content accessed by the job, such as permission to read or update documents in the target database. Specify the user in the mapreduce.marklogic.input.username and mapreduce.marklogic.output.username job configuration properties. See Configuring a MapReduce Job.

In addition, the input and output user must use one of the pre-defined roles listed below:

The hadoop-internal role is for internal use only. Do not assign this role to any users. This role is used to amp special privileges within the context of certain functions of the Hadoop MapReduce Connector. Assigning this role to users gives them privileges on the system that you typically do not want them to have.

For details about roles and privileges, see the Understanding and Using Security Guide.

Selecting the App Server and Database

The MarkLogic Connector for Hadoop requires a MarkLogic Server installation configured with an XDBC App Server. When you install MarkLogic Server, a suitable XDBC App Server attached to the Documents database comes pre-configured on port 8000.

The example commands in this guide assume you're using this port 8000 App Server and database, and therefore no additional setup is required.

However, you can choose to use a different database or App Server and database:

• To use the pre-configured App Server on port 8000 with a different database, set the com.marklogic.output.databasename configuration property when you follow the steps in Configuring the Job. (A similar property exists for overriding the default database when using MarkLogic for output.)
• To create your own XDBC App Server on a different port, attached to a different database, see the Administrator's Guide, then configure your job appropriately when you get to Configuring the Job.

Loading the Sample Data

This section covers loading the sample data in two ways: Using Query Console to load the data using simple XQuery, or using the MarkLogic Content Pump (mlcp) command.

• Loading Sample Data with mlcp
• Loading Sample Data Manually

$ mlcp.sh import -username user -password password -host localhost \
    -port 8000 -input_file_path hdfs:/user/me/hello/data \
    -output_uri_replace "/user/me/hello/data/,''"

Configuring the Job

Before running the HelloWorld sample job, set the connector configuration properties that identify the MarkLogic Server user and instance for input and output.

Although the input and output MarkLogic Server instances and users can be different, this example configures the job to use the same host, port, and database for both input and output.

Configuration also includes an input and an output user name and password. Choose (or create) a MarkLogic user with sufficient privileges to access your XDBC App Server, and read and insert documents in the attached database. If using a non-admin user, assign the user to the hadoop-user-all role. For details, see Security Requirements for MapReduce Jobs.

To configure the job:

1. Copy the marklogic-hello-world.xml configuration file from $CONNECTOR_HOME/conf to your work area. For example:

   $ cp $CONNECTOR_HOME/conf/marklogic-hello-world.xml /space/examples/hello

2. Edit your local copy of marklogic-hello-world.xml to configure your input and output host name, port, user name, and password. Set the following parameters to match your environment:

   mapreduce.marklogic.input.username
   mapreduce.marklogic.input.password
   mapreduce.marklogic.input.host
   mapreduce.marklogic.input.port
   mapreduce.marklogic.output.username
   mapreduce.marklogic.output.password
   mapreduce.marklogic.output.host
   mapreduce.marklogic.output.port

   The configured input user must have sufficient privileges to access the XDBC App Server identified by the input host/port and to read documents from the input database.

   The configured output user must have sufficient privileges to access the XDBC App Server identified by the output host/port and to insert documents in the output database.

For example, if your MarkLogic installation is on localhost and you use the pre-configured App Server on port 8000 with the username and password 'my-user' and 'my-password' for input, then your input connection related property settings should be similar to the following after editing:

<property>
  <name>mapreduce.marklogic.input.username</name>
  <value>my-user</value>
</property>
<property>
  <name>mapreduce.marklogic.input.password</name>
  <value>my-password</value>
</property>
<property>
  <name>mapreduce.marklogic.input.host</name>
  <value>localhost</value>
</property>
<property>
  <name>mapreduce.marklogic.input.port</name>
  <value>8000</value>
</property>

Your output connection related property settings should have similar values.

Running the Job

The HelloWorld sample reads the first word of text from the input documents, concatenates the words into a string, and saves the result as HelloWorld.txt. Assuming the database contains only the documents created in Loading the Sample Data, the output document contains the phrase 'hello world'. If your database contains additional documents, you get different results.

To view the sample code, see $CONNECTOR_HOME/src/com/marklogic/mapreduce/examples.

Use the following procedure to run the example MapReduce job:

1. If you are not already in your work area, change to that directory. For example:

   cd /space/examples/hello

2. Ensure the hadoop command is in your path.
3. Run the HelloWorld job using the following command. Modify the connector JAR file name as needed for your installation.

   hadoop jar \
     $CONNECTOR_HOME/lib/marklogic-mapreduce-examples-version.jar \
     com.marklogic.mapreduce.examples.HelloWorld -libjars $LIBJARS \
     -conf marklogic-hello-world.xml

The -conf command line option tells Hadoop where to get application-specific configuration information. You can also add a configuration directory to HADOOP_CLASSPATH.

As the job runs, Hadoop reports the job progress to stdout. If the sample job does not run or does not produce the expected results, see Troubleshooting and Debugging.

Near the end of the job output, you should see text similar to the following. Notice there are 2 map input records (hello.xml and world.xml), 2 map output records (the first word from each input record), and 1 reduce output record (HelloWorld.txt).

timestamp INFO mapreduce.Job:  map 100% reduce 100%
timestamp INFO mapreduce.Job: Job jobId completed successfully
timestamp mapreduce.Job: Counters: 33
        File System Counters
                ...
        Map-Reduce Framework
                Map input records=2
                Map output records=2
                Map output bytes=20
                Map output materialized bytes=30
                Input split bytes=91
                Combine input records=0
                Combine output records=0
                Reduce input groups=1
                Reduce shuffle bytes=30
                Reduce input records=2
                Reduce output records=1

Use Query Console to explore the output database and examine the output document, HelloWorld.txt. The document should contain the phrase 'hello world'.

If you do not see the expected output, see the tips in Troubleshooting and Debugging.