Loading TOC...
Security Guide (PDF)

Security Guide — Chapter 11

Encryption at Rest

Encryption at rest protects your data on media - which is 'data at rest' as opposed to data moving across a communications channel, otherwise known as 'data in motion.' Increasing security risks and compliance requirements sometimes mandate the use of encryption at rest to prevent unauthorized access to data on disk.

To enable encryption at rest, a license key that includes encryption at rest security is required. For details on purchasing a license key for the encryption at rest feature, contact your MarkLogic sales representative.

Encryption at rest can be configured to encrypt data, log files, and configuration files separately. Encryption is only applied to newly created files once encryption at rest is enabled, and does not apply to existing files without further action by the user. For existing data, a merge or re-index will trigger encryption of data, a configuration change will trigger encryption of configuration files, and log rotation will initiate log encryption.

This chapter describes encryption at rest security and includes the following sections:

Terms and Definitions

The following terms and definitions are associated with encryption at rest.

TermDefinition
Encryption at restEncryption of data that is stored on digital media
KMSKey Management System
walletThe PKCS #11 secured wallet provided and managed by MarkLogic, that functions as the default standalone KMS
KEKA Key Encryption Key used to encrypt or ‘wrap' another encryption key
keystoreRepository for crytographic keys in the PKCS #11 secured wallet or any external KMS that is KMIP-server conformant
KMIPKey Management Interoperability Protocol (KMIP specification) - governed by OASIS standards body. There are two different currently available versions: 1.1.x and 1.2. MarkLogic Encryption supports 1.2
PKCS #11One of the Public-Key Cryptography Standards, and also the programming interface to create and manipulate cryptographic tokens. See the OASIS PKCS TC for details
MKEKMaster Key Encryption Key, resides in the keystore, and is used to generate the CKEK, which is enveloped (encrypted) with the MKEK
CKEKCluster Key Encryption Key, resides in the keystore and is used to encrypt the data (CDKEK), configuration(CCKEK), and log CLKEK) encryption keys
CDKEKCluster Data Key Encryption Key, used to directly encrypt FRKEYs for stands, forest journals, and large files
CCKEKCluster Configuration Key Encryption Key, used to encrypt (wrap) the object key encryption keys (OKEY) for configuration files
CLKEKCluster Log Key Encryption Key, used to encrypt (wrap) the object key encryption keys (OKEY) for log files
FRKEYFast Rotation Key Encryption Key, used to encrypt (wrap) the object key encryption keys (OKEY) for stands, forest journals, and large files
OKEYObject Encryption Key, otherwise known as the data object encryption key, a symmetric key used to directly encrypt objects like stands, forest journals, large files, configuration files, or log files
BKEKBackup Key Encryption Key, used to encrypt backups, both full and incremental
HSMHardware Security Module or other hardware device is a physical computing device that safeguards and manages digital key materials
Key strengthThe size of key in bits. Usually the more bits, the stronger the key and more difficult to break; for example 128-bits, 256 bits, or 512-bits, and so on
Key rotationThe process of aging out and replacing encryption keys over time

Understanding Encryption at Rest

Encryption at rest enables you to transparently and selectively encrypt your data residing on disk (locally or in the cloud) in MarkLogic clusters. You can set your options at the cluster level to encrypt data on all the hosts in that cluster.

Three types of data can be encrypted:

  • User data - data ingested into MarkLogic as databases, along with derived data such as indexes, user dictionaries, journals, backups, and so on
  • Configuration files - all configuration files generated by MarkLogic (for example, whenever a change is made to the configuration file)
  • Log files - all log files generated by MarkLogic, such as error logs, access logs, service dumps, server error logs, logs for each application server, and the task server logs

    There are both app-level and system-level logs, and both types of logs will be encrypted as part of log encryption.

These types of data can each be encrypted separately. You can configure encryption for databases individually, or at the cluster level. Encryption at rest is 'off' by default. To use encryption at rest, you need to configure and enable encryption for your database(s), configuration files, and/or log files.

Encryption at rest provides data confidentiality, but not authentication of identity or access control (permissions). See Authenticating Users and Protecting Documents for information about authentication and other forms of security in MarkLogic Server.

If you cannot access your PKCS #11 secured wallet (or external KMS if you are using one), or lose your encryption keys, you will not be able to decrypt any of your encrypted data. There is no 'back-door' to recover the encrypted data. We recommend that you backup your encryption keys in a secure location. See Backup and Restore for more details.

Keystores - PKCS #11 Secured Wallet or External KMS

A keystore is a secure location where the actual encryption keys used to encrypt data are stored. The keystore for encryption at rest is a key management system (KMS). This keystore can be either the MarkLogic embedded PKCS #11 secured wallet, or an external KMS that conforms to the KMIP-standard interface. The embedded keystore is installed by default when you install MarkLogic 9.0-x or later.

The MarkLogic embedded wallet uses a standard PKCS #11 protocol, using the PKCS #11 APIs. The wallet or another KMS, must be available during the MarkLogic startup process (or be bootstrapped from MarkLogic during start-up). You can also use any KMIP-compliant external keystore with MarkLogic.

To configure an external KMS you will need the following information for your cluster:

  • Host name
  • Port number
  • Client certificate
  • Server certificate
  • Data encryption key ID
  • Configuration encryption key ID
  • Logs encryption key ID

    If you plan to use an external key management system, configure the external KMS first, and then turn on encryption in the MarkLogic server.

For details, see Configuring a External Keystore.

Encryption Key Hierarchy Overview

The following section provides an overview of the encryption key hierarchy used by MarkLogic encryption at rest to secure data. Keys in the encryption hierarchy wrap (or encrypt) those keys below them in the hierarchy. Three possible configurations of the encryption key hierarchy are described. The first is an idealized key hierarchy that provides a generic example. The second is an embedded KMS (the PKCS #11 secured wallet) configuration, and the third shows an external keystore management system (KMS) configuration.

You do not need to completely understand the details of the key hierarchy to use the encryption feature, but this section will help to understand the general concepts involved.

The keystore contains the Master Key Encryption Key (MKEK). The keystore generates the Cluster Key Encryption Key (CKEK), which is enveloped (encrypted) with or derived from the Master Key Encryption Key. Both the Master Key Encryption Key and the Cluster Key Encryption Key reside in the keystore (key management system or KMS). These keys never leave the keystore and MarkLogic Server has no knowledge or control over these keys. The keys are referenced from the keystore by their key IDs.

The KMS can be either the internal keystore provided by MarkLogic or a external KMIP-compliant KMS; the same mechanism is used by both types of keystores. The configuration happens at the cluster level because there is one keystore per cluster. The encryption feature is fully compliant with the KMIP standard.

The external KMS provides even higher security. The key IDs are provided by the KMS and returned through an SSL/TLS tunnel after the MarkLogic-generated keys have been sent to the KMS and wrapped (encrypted). The actual encryption keys never leave the KMS.

There are multiple levels to the key hierarchy, each level wrapping (encrypting) the level below it. The KMS generates the Cluster Level Encryption Keys for data (CDKEK), configuration files (CCKEK), and log files (CLKEK). The Fast Rotation Key Encryption Key (FRKEY) is used to encrypt (wrap) all the Object Encryption Keys (OKEY) generated by MarkLogic Server for each file, so that a unique key protects each file, no matter what category (data, configuration files, logs).

There are three types of OKEY encryption keys at the forest level, for stands, forest journal, and large files. The individual, Object Encryption Keys (OKEY) are randomly generated per file (for stands, journals, config files, and log files, etc.) wrapped (encrypted) with the Fast Rotation Key Encryption Key (FRKEY). So a unique key protects each file within a category (data, configuration files, logs).

For example, the Master Key Encryption Key (MKEK) wraps (encrypts) the Cluster Key Encryption Keys (CKEK), which in turn wraps (encrypts) the Data Key Encryption Key (CDKEK). The Data Key Encryption Key encrypts the Fast Rotation Key Encryption Key (FRKEY), which then wraps (encrypts) the Object Encryption Key (OKEY) for a file such as a stand. The keys at the bottom of the diagram are encrypted as headers in each file, wrapped (encrypted) with each of the keys above them in the hierarchy. Each of the three categories of objects (data, configuration files, and logs) has its own key encryption hierarchy.

Database backups are encrypted using the CDKEK. This key is then encrypted with the cluster key (CKEK). See Backup and Restore for more information about backups.

Embedded KMS Key Hierarchy

When you use the embedded PKCS #11 secured wallet provided with MarkLogic Server, the recommended key hierarchy would be similar to this illustration.

MarkLogic Server generates the Data Key Encryption Key (CDKEK), the Configuration Key Encryption key (CCKEK) and the Logs Key Encryption Key (CLKEK). The Data Key Encryption Key is then used to generate the Fast Rotation Key Encryption Key (FRKEK) to encrypt database stands. This key is then used to wrap the OKEYs for the database objects (journals, data files, etc.). These keys are stored in the wallet (internal KMS), the actual encryption keys never leave the KMS. The key IDs are provided to MarkLogic Server for encryption and decryption by the KMS (the PKCS #11 secured wallet in this case). The configuration happens at the cluster level because there is one keystore per cluster.

The individual Object Encryption Keys (OKEYs) are then randomly generated and used to directly encrypt individual files (journals, config files, and log files, etc.). These keys (the OKEYs) are wrapped (encrypted) with the Fast Rotation Key Encryption Keys (FRKEY). A unique key protects (encrypts) each file. The keys at the object levels are wrapped (encrypted by the keys above them) for each category.

For example, the Data Key Encryption Key (CDKEK) wraps (encrypts) the Fast Rotation Key Encryption Key (FRKEY), which wraps (encrypts) the Object Encryption Key (OKEY) for a file such as a journal. The keys at the bottom of the diagram are encrypted (wrapped) by all the keys above them in the hierarchy, and then placed in the header for each file.

Database backups are encrypted using the CDKEK. This key is then encrypted with the cluster key (CKEK). See Backup and Restore for more information about backups.

External KMS Key Hierarchy

The external KMS provides even higher security, along with additional key management features. When you use an external key management system (KMS or keystore), the recommended key hierarchy deployment might look like this illustration.

The keystore contains the Master Key Encryption Key (MKEK). The KMS generates or derives the Cluster Key Encryption Key (CKEK), which is enveloped (encrypted) with the Master Key Encryption Key. Both the Master Key Encryption Key and the Cluster Key Encryption Key reside in the KMS keystore. These keys never leave the keystore. MarkLogic Server has no knowledge or control over these keys. The keys are referenced from the keystore by their key IDs. The actual encryption keys never leave the KMS.

There are multiple levels to the key hierarchy in this deployment, each level wrapping (encrypting) the level below it. The KMS generates the cluster level encryption keys for data (CDKEK), configuration files (CCKEK), and log files (CLKEK). The Fast Rotation Key Encryption Key (FRKEY) is used to encrypt (wrap) all the Object Encryption Keys (OKEY) generated by MarkLogic Server for each file, so that a unique key protects each file, no matter what category (data, configuration files, logs). A unique key protects each file within a category (data, configuration files, logs).

The Fast Rotation Key Encryption Key (FRKEY) is used to encrypt (wrap) all the Object Encryption Keys (OKEY) generated by MarkLogic Server for each file, so that a unique key protects each file, no matter what category (data, configuration files, logs).

For example, the Master Key Encryption Key (MKEK) wraps (encrypts) the Cluster Key Encryption Keys (CKEK), which in turn wraps (encrypts) the Data Key Encryption Key (CDKEK), then wraps (encrypts) the Fast Rotation Key Encryption Key (FRKEY), which wraps the Object Encryption Key (OKEY) for a file such as a stand. The keys at the bottom of the diagram are encrypted (wrapped) by all the keys above them in the hierarchy, and then placed in the header for each file.

Database backups are encrypted using the CDKEK. This key is then encrypted with the cluster key (CKEK). See Backup and Restore for more information about backups.

If you plan to use an external key management system, configure the external KMS first, and then turn on encryption in the MarkLogic server.

Encryption Algorithms Used

Symmetric encryption, AES-256, is used for data encryption (databases, logs, and config files). For the internal MarkLogic keystore, key IDs are secured using PKCS #11 provided by the MarkLogic PKSC #11 secured wallet. For an external keystore or KMS, key IDs may be secured using PKCS #11 or KMIP, which can be provided by an external key management system.

Example --Encryption at Rest

This section describes a scenario using encryption at rest to encrypt a database. This example is for informational purposes only. It is not meant to demonstrate the correct way to set up and use encryption at rest, as your situation is likely to be unique. However, it demonstrates how encryption at rest works and may give you ideas for how to configure your own encryption at rest security model.

Description:

To set up encryption at rest for this scenario, you will need Admin privileges. You will need access to both MarkLogic Admin Interface and Query Console.

To run through the example, perform the steps in each of the following sections:

Set Up Encryption Example

Install MarkLogic 9.0-1 or later. Encryption at rest options are not available in earlier versions of MarkLogic Server. You must explicitly select which data (databases, configuration files, log files, or entire clusters) you want to have encrypted. This example shows how to set up encryption for a single database.

The Security database or other databases used by MarkLogic will not be encrypted by default. Existing data can be encrypted by forcing a merge or a reindex of the database.

See Configuring Encryption at Rest for more details.

Encrypt a Database

For this example, we will use the Admin UI to set up encryption for the Documents database.

  1. Select Databases from the left tree menu in the Admin UI.
  2. Click on the Documents database.
  3. On the Database Configuration page, next to data encryption, select on from the drop-down menu. (The other options are default-cluster and off.)

  4. Click ok.

If you select default-cluster, encryption for that database will default to whatever encryption option has been set for the cluster as a whole. If the cluster is set to encrypt data, this database will be encrypted. If encryption has not been turned on for the cluster, this database will not be encrypted if default-cluster is selected. See Cluster Encryption Options for details.

As you access data in your database, it will be encrypted when it is written back to disk. You can view the encryption progress on the Database Status page by looking at the Size and Encrypted Size numbers.

To encrypt the existing data in your database, you will need to re-index your database. On the Database Configuration page, click the reindex button at the top of the page (below the 'OK' button), and then click ok. You can also force a merge of the database to encrypt the data.

Encryption of large databases will take some time initially. Updates and changes to the database will be fairly transparent to the user after initial encryption. The Size and Encrypted Size numbers will be equal when the encryption process is complete.

Test It Out

Using Query Console, you can run a simple query to verify that the Documents database has encryption turned on.

xquery version "1.0-ml";
import module namespace admin = "http://marklogic.com/xdmp/admin"		
  at "/MarkLogic/admin.xqy";

let $config := admin:get-configuration()
return admin:database-get-data-encryption($config, xdmp:database("Documents"))

=>
on

You can also check the Size and Encrypted Size numbers on the Database Status page. These numbers will be equal when the encryption process is complete and the entire database is encrypted.

Turn Off Encryption for a Database

  1. Select Databases from the left tree menu in the Admin UI.
  2. Click on the Documents database to turn off encryption.
  3. On the Database Configuration page, next to data encryption, select off from the drop-down menu.

  4. Click ok.

To verify that encryption is turned off, run this query in Query Console:

xquery version "1.0-ml";
import module namespace admin = "http://marklogic.com/xdmp/admin" 
		  at "/MarkLogic/admin.xqy";

let $config := admin:get-configuration()
return admin:database-get-data-encryption($config, xdmp:database("Documents"))

=>
off

To decrypt the existing data in your database, you will need to re-index your database. On the Database Configuration page, click the reindex button and then click ok.

You can also decrypt the data by forcing a merge on the database to decrypt its contents. This process may take a while.

Configuring Encryption at Rest

Install MarkLogic Server version 9.0-x or later. The encryption at rest feature and the PKCS #11 secured wallet are installed by default. You can configure encryption at rest for databases (data encryption), log files (log encryption) and configuration files (config encryption). The encryption feature will need to be configured and enabled for your data to be encrypted.

When you start up MarkLogic for the first time after installation, the keystore.xml file will be loaded first. It contains the encrypted keys. After loading the keystore.xml configuration, MarkLogic validates connectivity to the KMS (local or external) and the validity of the keys stored in keystore.xml. Once validated, per host encryption keys will be loaded and decrypted. Normal startup then continues. If configuration files are encrypted, the file layer will decrypt them as they are being loaded, making the encryption transparent to the cluster.

If you have an encrypted host, the DR cluster for that host must also be encrypted. See Disaster Recovery/Shared Disk Failover for more information.

This section contains the following topics:

Database Encryption Options

You can configure encryption for each database on the Database Configuration page in the Admin UI. Encryption at rest can be separately enabled per database, or at the cluster level by setting the database encryption to default to the cluster encryption settings. The encryption options for databases are shown in the following table.

Encryption Encryption Options
Default-ClusterOnOff
Database encryptionencryption defaults to cluster settingencryption enabled for database

encryption off,

unless cluster encryption is set to force encryption

With encryption enabled, files are encrypted as they are ingested into the database, or when those files are written back to disk. If you want to encrypt existing data in a database either reindex the database or force a merge on the database. This will take a few minutes depending on the size of database. See Cluster Encryption Options

Large binary files are only encrypted during initial ingestion into the database. If you want to encrypt existing large binary files already loaded into MarkLogic Server prior to turning on encryption, you must reindex the database or force a merge.

  1. To configure database encryption, go to the Admin UI and click Databases in the left navigation tree.
  2. Click on the database you want to encrypt.
  3. On the Database Configuration page, next to data encryption, select on from the drop-down menu. (The other options are default-cluster and off.)

  4. Click ok when you are done.

Configure Cluster Encryption

You can set cluster encryption options for configuration files and log files, and also set or override the encryption options for databases on the Cluster Configuration page.

Configuration File and Log File Encryption Options

Encryption at rest for configuration files and/or log files is done on the Cluster Configuration page in the Admin UI. Navigate to this page by choosing Clusters from the left tree menu, clicking the cluster name, and then clicking the Configure tab.

The encryption options are shown in the following table.

File TypeCluster Encryption Settings
Default OnDefault Off
Configuration filesencryptdo not encrypt
Log filesencryptdo not encrypt

Cluster configuration settings for encryption at rest interact with the encryption settings for databases. You can separately configure encryption for each database on the Database Configuration page in the Admin UI or set database encryption to default to the cluster encryption settings.

The database encryption configuration settings take precedence unless the cluster Force Encryption option is set. Please check all database encryption settings to ensure that they are set correctly.

The following table shows the interaction between the cluster configuration options and the database configuration options. There are three possible database encryption settings and three possible cluster encryption settings. The cell where the row and column intersect shows the outcome of that configuration combination.

Database Encryption SettingCluster Encryption Settings
Force EncryptionDefault OnDefault Off
Default to clusterencryptencryptdo not encrypt
Onencryptencryptencrypt
Offencryptdo not encryptdo not encrypt

The Force Encryption option in the Cluster Encryption Settings will force encryption for all of the databases in the cluster. If the Cluster Encryption Setting is Force Encryption (or Default On), or the Database Encryption Setting is On, then the database will be encrypted.

Cluster Encryption Options

You can either configure encryption for the embedded keystore (the PKCS #11 secured wallet) or for a external KMIP-compliant keystore using the Admin UI. Use the Cluster Configuration page to configure encryption at rest on a cluster.

  1. To configure encryption using the embedded keystore in the Admin UI, click Clusters in the left navigation tree and click the name of the cluster you want to configure.
  2. When the page opens, navigate to the Edit Local Cluster Configuration page.

  3. Use the drop-down menus to configure encryption for data, config files, and/or log files.
  4. Click ok when you are done.

You can configure data encryption, configuration file encryption, or encryption of log files on this page.

Adding or changing any encryption information will require a restart of all of the hosts in the cluster.

Using an Alternative PKCS #11 Device

MarkLogic uses SoftHSM as its default cryptographic store, accessible through the PKCS #11 interface. You can configure SoftHSM for use with a PKCS #11 compliant hardware security module (HSM) like a plug-in card. If you want to use an HSM or any other PKCS #11 compliant storage device for your encryption keys, you need to configure SoftHSM using the steps in this section.

No data should be encrypted before starting this process, and that includes credentials.

The HSM device must not be initialized and no PIN should be set prior to configuring it for use with MarkLogic encryption. MarkLogic will initialize the device and set a PIN as part of this process. These steps assume that you have already set up encryption using the internal KMS (the PKCS #11 secured wallet).

  1. Update the library path for the keystore using the admin function admin:cluster-set-keystore-p11-driver-path.

    For example:

    xquery version "1.0-ml"; 
    import module namespace admin = "http://marklogic.com/xdmp/admin" 
      at "/MarkLogic/admin.xqy";
    
    let $config := admin:get-configuration()
    return 
      admin:cluster-set-keystore-p11-driver-path($config,"/opt/HSM/libhsm.so")
    													 
    =>
    (: returns the new configuration element -- use admin:save-configuration to save the changes
    to the configuration or pass the configuration to other Admin API functions 
    to make other changes.  :)
  2. Rotate all encryption keys using these admin functions: admin:cluster-rotate-data-encryption-key-id, admin:cluster-rotate-config-encryption-key-id and admin:cluster-rotate-logs-encryption-key-id.

    For example:

    xquery version "1.0-ml";
    import module namespace admin = "http://marklogic.com/xdmp/admin" 
      at "/MarkLogic/admin.xqy";
    
    let $config := admin:get-configuration()
    return 
      admin:cluster-rotate-data-encryption-key-id($config)
    
    =>
    (: returns the new configuration element -- use admin:save-configuration to save the changes
    to the configuration or pass the configuration to other Admin API functions 
    to make other changes.  :)

    The rotate function will create a new key even though the function says 'rotate'. This usually means there is already an existing key that is being changed.

  3. Restart MarkLogic.

See Key Rotation for more about key rotation.

Saving the Embedded KMS to a Different Location

Use the options available in admin:cluster-set-keystore-wallet-location to change the location of the backup for the internal wallet.

let $dir-name := "/sotfhsm/wallet"
let $config := admin:get-configuration()
return 
  admin:cluster-set-keystore-wallet-location($config,$dir-name)

The admin:cluster-set-keystore-wallet-location function will set the backup location for embedded KMS.

Configure Encryption Using XQuery

Instead of using the Admin UI, you can configure encryption for your MarkLogic instance using XQuery.

In Query Console, you can use admin:cluster-set-data-encryption to turn on data encryption for the current database:

xquery version "1.0-ml";
import module namespace admin = "http://marklogic.com/xdmp/admin" 
		  at "/MarkLogic/admin.xqy";

let $config := admin:get-configuration()
return 
  admin:cluster-set-data-encryption($config,"on")

For example, to set the encryption for log files at cluster level:

xquery version "1.0-ml"; 
import module namespace admin = "http://marklogic.com/xdmp/admin" 
  at "/MarkLogic/admin.xqy";

let $config := admin:get-configuration()
return 
  admin:cluster-set-logs-encryption($config, "on")
													 
(: returns the new configuration element -- use admin:save-configuration 
to save the changes to the configuration or pass the configuration to
other Admin API functions to make other changes.  :)

To see whether encryption is turned on for log files, you can run this XQuery in the Query Console:

xquery version "1.0-ml"; 
import module namespace admin = "http://marklogic.com/xdmp/admin" 
  at "/MarkLogic/admin.xqy";

let $config := admin:get-configuration()
return 
  admin:cluster-get-logs-encryption($config)
=>
on
 (: returns the encryption setting for log files:)

Configure Encryption Using REST

You can use REST Management APIs to work with encryption at rest.

GET /manage/v2/databases/{id|name}/properties

This command gets the current properties of the Documents database, including the encryption status and encryption key ID in JSON format:

$ curl -GET --anyauth -u admin:admin \
  -H "Accept:application/json,Content-Type:application/json" \
  http://localhost:8002/manage/v2/databases/Documents/properties

Returns

{"database-name":"Documents", "forest":["Documents"],
"security-database":"Security", "schema-database":"Schemas",
"triggers-database":"Triggers", "enabled":true,
"data-encryption":"off", "encryption-key-id":"", 

The same command in XML format:

$ curl -GET --anyauth -u admin:admin \
  -H "Accept:application/xml,Content-Type:application/xml" \
  http://localhost:8002/manage/v2/databases/Documents/properties

Returns

<database-properties xmlns="http://marklogic.com/manage">
  <database-name>Documents</database-name>
  <forests>
    <forest>Documents</forest>
  </forests>
  <security-database>Security</security-database>
  <schema-database>Schemas</schema-database>
  <triggers-database>Triggers</triggers-database>
  <enabled>true</enabled>
  <data-encryption>on</data-encryption>
  <encryption-key-id/>
...
</database-properties>

GET /manage/v2/security/properties

This command returns the current encryption status, along with other properties including encryption key ID, for localhost in JSON format:

$ curl -GET --anyauth -u admin:admin \
  -H "Accept:application/json,Content-Type:application/json" \
  http://localhost:8002/manage/v2/security/properties

Returns:

{"keystore":{"data-encryption":"default-off",
"data-encryption-key-id":"091fd9a0-f090-4c7e-91ca-fedfe21dbfef",
"config-encryption":"off", "config-encryption-key-id":"",
"logs-encryption":"off", "logs-encryption-key-id":"",
"host-name":"LOCALHOST", "port":9056}}

Here is the same version of the command, this time returning XML:

$ curl -GET --anyauth -u admin:admin \
  -H "Accept:application/xml,Content-Type:application/xml" \
  http://localhost:8002/manage/v2/security/properties

Returns:

<security-properties xsi:schemaLocation="http://marklogic.com/manage/security/properties
manage-security-properties.xsd" xmlns="http://marklogic.com/manage/security/properties"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <keystore>
    <data-encryption>default-off</data-encryption>
    <data-encryption-key-id>8d0b07d8-b655-4408-affd-e49a2ece0af3
      </data-encryption-key-id>
    <config-encryption>off</config-encryption>
    <config-encryption-key-id/>
    <logs-encryption>off</logs-encryption>
    <logs-encryption-key-id/>
    <host-name>LOCALHOST</host-name>
    <port>9056</port>
  </keystore>
</security-properties>

PUT /manage/v2/databases/{id|name}/properties

This command will turn on encryption for the Documents database:

$ curl -X PUT --anyauth -u admin:admin -d '{"data-encryption":"on"}' \
  -H "Content-Type:application/json" \
  http://localhost:8002/manage/v2/databases/Documents/properties
Export Wallet

To export the embedded KMS (the PKCS #11 secured wallet) using REST, you can use this form in XQuery:

POST manage/v2/security?
operation=export-wallet&filename=/my/test..wallet&password=test

As a curl command (using MANAGEADMIN="admin" and MANAGEPASS="admin") it would look like this:

curl -v -X POST  --anyauth --user $MANAGEADMIN:$MANAGEPASS \
  --header "Content-Type:application/xml" \
-d@data/security/export-wallet.xml \
  http://$host:8002/manage/v2/security

Where export-wallet.xml is:

<export-wallet-operation xmlns="http://marklogic.com/manage/security">
  <operation>export-wallet</operation>
  <filename>/tmp/mywallet.txt</filename>
  <password>mypassword</nassword>
</export-wallet-operation>

Or you can use this form for JavaScript:

POST manage/v2/security
{"operation":"export-wallet","filename":"/my/test.wallet","password":"test"}

As a curl command (using MANAGEADMIN="admin" and MANAGEPASS="admin") it would look like this:

curl -v -X POST  --anyauth --user $MANAGEADMIN:$MANAGEPASS \
  --header "Content-Type:application/json" \
-d@data/security/export-wallet.json \
  http://$host:8002/manage/v2/security

Where export-wallet.json is:

{
    "operation":"export-wallet",
    "filename":"/tmp/mywallet.tmp",
    "password":"mypassword"
}

The export wallet operation saves the wallet to a directory on the server on which MarkLogic is running. Similarly, the import wallet operation imports from the filesystem on which MarkLogic is running.

Import Wallet

To import the embedded KMS (the PKCS #11 secured wallet) using REST, you can use this form in XQuery:

POST manage/v2/security?
operation=import-wallet&filename=/my/test.wallet&password=test

As a curl command (using MANAGEADMIN="admin" and MANAGEPASS="admin") it would look like this:

curl -v -X POST  --anyauth --user $MANAGEADMIN:$MANAGEPASS \
  --header "Content-Type:application/xml" \
-d@data/security/import-wallet.xml \
  http://$host:8002/manage/v2/security

Where import-wallet.xml is:

<import-wallet-operation xmlns="http://marklogic.com/manage/security">
  <operation>import-wallet</operation>
  <filename>/tmp/mywallet.txt</filename>
  <password>mypassword</password>
</import-wallet-operation>

Or you can use this form for JavaScript:

POST manage/v2/security
{"operation":"import-wallet","filename":"/my/test.wallet","password":"test"}

As a curl command (using MANAGEADMIN="admin" and MANAGEPASS="admin") it would look like this:

curl -v -X POST  --anyauth --user $MANAGEADMIN:$MANAGEPASS \
  --header "Content-Type:application/json" \
-d@data/security/import-wallet.json \
  http://$host:8002/manage/v2/security

Where import-wallet.json is:

{
    "operation":"import-wallet",
    "filename":"/tmp/mywallet.tmp",
    "password":"mypassword"
}

MarkLogic will only import keys generated by the embedded MarkLogic KMS.

Key Management

Encryption key management for the embedded KMS (the PKCS #11 secured wallet) is handled automatically by MarkLogic. Keys are never purged from the wallet, which is encrypted by a MarkLogic-generated key activated by a passphrase. The administrator's password is used as the initial passphrase.

By default the keystore passphrase is set to the admin password. We strongly recommend that you set a new, different passphrase before turning on encryption. Using a separate passphrase for admin and the keystore helps support the strong security principle called 'Separation of Duties'.

This passphrase can be changed using either the XQuery (xdmp:keystore-set-kms-passphrase) or JavaScript (xdmp.keystoreSetKmsPassphrase) built-ins. As part of key management, you may want to export, import, or rotate encryption keys. MarkLogic provides built-in functions for exporting and importing encryption keys, and manually rotating encryption keys. If you require additional key management functionality, you may want to consider a external key management system. See Configuring a External Keystore for more information.

If you believe that an encryption key has been compromised, you should force a merge or start a re-index of your data to change/update the encryption keys. See Key Rotation for more about updating encryption keys.

This section includes the following topics:

Key Rotation

Every time data is written to disk, it is re-encrypted with a new key. Data encryption keys are automatically rotated during merges or reindexing because MarkLogic always generates a one-time-use object key for every new file that it writes (stand files, configuration files, logs, journals, etc.). Because of this, MarkLogic is constantly re-encrypting keys. You can also re-encrypt data that has not be touched in a while (old stale forests) by forcing a full merge or re-index of the data.

For the internal PKCS #11 secured wallet (KMS), key encryption keys (KEK) can be manually rotated. Keys can be manually rotated at regular intervals or if an encryption key has been compromised. This type of key rotation can be triggered on individual encryption categories (configuration, data, logs) using using MarkLogic built-in functions.

There are two steps to key rotation. First, rotating the KEK keys (using AES 256 symmetric encryption) used to envelope the object file encryption keys, and second, re-encrypting the object file encryption keys (also using AES 256 symmetric encryption).

After calling the built-in function to rotate encryption keys, all new data will be written to disk using the new key encryption key. Old data will be migrated as it is re-written to disk. If you wish to force re-encryption using the new key, you can either force a merge or re-index the forest.

Manual rotation of the data keys, configuration keys, and the logs keys (CDKEK, CCKEK, CLKEK) can be done using these APIs:

To manually rotate the cluster-level keys, use these APIs:

Fast Keys

The intermediate fast rotation keys enable immediate envelope key rotation with a minimum of I/O. File level keys can be rotated at any time by forcing a merge. Log rotation and configuration file updates use new keys. Old logs, backups, and configuration files are not re-encrypted.

The internal KMS (the PKCS #11 secured wallet) follows these steps for fast key rotation:

  1. User sends rotation key command to MarkLogic (for example, admin:cluster-rotate-data-encryption-key-id).
  2. MarkLogic requests a new data encryption key (CDKEK, CCKEK, CLKEK - the cluster-level encryption keys) from the internal KMS.
  3. Only the fast rotation keys are re-encrypted with the new data encryption keys (CDKEK, CCKEK, CLKEK).

An external KMS, follows these steps for fast key rotation:

  1. The external KMS creates new KEK key (CDKEK, CCKEK, CLKEK - the cluster-level encryption keys).
  2. User updates the UUIDs in MarkLogic. See Setting Up an External KMS with MarkLogic Encryption for UUID details.
  3. MarkLogic sends sends a Fast Rotation Key (FRKEK) to the KMS.
  4. The external KMS sends new enveloped key back to MarkLogic.
  5. The enveloped key is saved to disk, per file.

    Expired keys can be used for decryption, but not encryption. Expired keys may be needed for backups.

Export and Import Encryption Keys

The ability to export and import key encryption keys (KEK) from the PKCS #11 secured wallet (the embedded KMS) is useful when you want to clone a cluster. Exporting a key encryption key (KEK) is restricted to cluster-level keys (CDKEK, CCKEK, CLKEK) and requires a passphrase and a filepath. The data will be exported (encrypted with the passphrase) into a file at the location specified by the filepath.

To export a keystore from the embedded KMS:

xquery version "1.0-ml";
import module namespace admin = "http://marklogic.com/xdmp/admin" 
  at "/MarkLogic/admin.xqy";

xdmp:keystore-export("Unique passphrase", "/backups/MarkLogic.wallet.bak")
=>
true

To import a keystore into the embedded KMS:

xquery version "1.0-ml";
import module namespace admin = "http://marklogic.com/xdmp/admin" 
  at "/MarkLogic/admin.xqy";

xdmp:keystore-import("Unique passphrase", "/backups/MarkLogic.wallet.bak")
=>  true

Key encryption keys can only be imported from MarkLogic exported files. Imported keys can only be used for decryption. The import requires the passphrase that was provided at the time of the export.

If a duplicate key is supplied during the import, the import will be rejected. Duplicate keys can be caused by importing the keystore twice.

Key Deletion and Key Revocation

For these functions you will need to use a external keystore (KMS).

Configuring a External Keystore

An external key management system (KMS) or keystore offers additional security for your encryption keys, along with key management capabilities like automatic key rotation, key revocation, and key deletion. If you want the ability to perform these tasks, you will need an external KMS. MarkLogic Encryption at Rest supports KMIP 1.2 compliant KMS servers.

When using an external KMS, usually there is a security administrator role separate from the MarkLogic administrator. The security administrator would be the role setting up and configuring the external keystore. The MarkLogic administrator can also perform this task, but for greater security it is recommended that the separate security administrator configure the KMS.

Having a separate security administrator follows an important security principle called 'Separation of Duties' and is recommended by security experts.

This section covers setting up MarkLogic encryption for use with an external key management system from the MarkLogic Admin UI on the MarkLogic host. You don't need to have MarkLogic encryption turned on for your cluster while you are setting up and configuring the external key management system.

If you plan to use an external key management system, we recommend that you configure the external keystore first, and then turn on encryption in the MarkLogic server.

The installation process for the external keystore will vary depending on the type of external KMIP-compliant KMS you plan to use. A security administrator must configure the external keystore using the administration set up tools that come with the external KMS. This section provides a high-level overview of the process from the MarkLogic Server point of view.

Types of External KMS Deployments

There are a variety of types of external key management systems. An external key management system deployment may be one of the following types:

  1. A virtual KMS instance running in a VM (virtual machine) environment, or in a private or public cloud
  2. A physical appliance running a KMS server
  3. A dedicated FIPS 140-2 Level 3 appliance
  4. A dedicated hardened FIPS 140-2 Level 4 appliance

These systems are listed by increasing levels of security. For more detailed information about configuring external key management systems to work with MarkLogic, see Third Party KMS Setup.

Setting Up an External KMS with MarkLogic Encryption

To configure the external key management system using the MarkLogic Admin UI on the MarkLogic host, you will need the following information for your external KMS:

  • Host name - the hostname of the key management system
  • Port number - the port number used to communicate with KMS
  • Data encryption key ID (UUID generated by external KMS)
  • Configuration encryption key ID (UUID generated by external KMS)
  • Logs encryption key ID (UUID generated by external KMS)

The SSL/TLS certificates, used to secure the communication with the KMS, must be stored locally on each host in the MarkLogic data directory (/var/opt/MarkLogic/Data). By default, the files are expected to be located in the MarkLogic data directory and must have the following names:

  • kmip-cert.pem - PEM-encoded KMIP SSL/TLS certificate
  • kmip-key.pem - PEM-encoded private key for the client (optional for some KMS servers)
  • kmip-CA.pem - PEM-encoded CA of the KMS

These certificates are the Certificate Authority (CA) for the root of the certificate chain for the kmip-cert.pem. A certificate could be a self-signed root used by an enterprise or an external CA. Copy these files into the MarkLogic data directory (/var/opt/MarkLogic/Data). The location and name of these files can be changed by calling the admin functions. See Admin APIs for Encryption at Rest for details.

These settings are cluster wide, so each individual host must have a local copy at the location specified.

Set Up the External KMS

In most cases, an external KMS is configured by security administrator, a separate role from the MarkLogic admin role. However, in some cases the security administrator may also be the MarkLogic admin role.

If you don't already have the external KMS configured and running, set up the external KMS using the appliance's interface before turning on MarkLogic encryption. The steps in the process for setting up the external KMS will depend on the type of KMIP-compliant external KMS you are using.

Make sure that:

  • The external key management system is set up, running, and provisioned first to use KMIP 1.2, before you configur MarkLogic encryption.
  • To secure communications between the KMS and MarkLogic Server obtain the required certificates; KMIP SSL/TLS certificate, CA of the KMS, private key for the client (optional for some KMS servers).

The security administrator can enable encryption for user data, configuration files, and/or logs, either per cluster or per database. You must use the administration tools that come with the external KMS to set up the external keystore.

The external key management system (KMS) must be available during the MarkLogic startup process. Access to the external KMS must be granted to all nodes in the cluster.

Set up MarkLogic Encryption

Before you set up encryption at rest, be sure that your cluster has upgraded to MarkLogic 9. If the cluster has not been upgraded, the encryption feature will not be available.

  1. Set up your external KMS, if not already set up. See Setting Up an External KMS with MarkLogic Encryption for details.
  2. Get the generated encryption key IDs from the external KMS (for data, config, and logs as needed). If you are using data encryption, configuration file encryption, and log encryption, you will need three encryption key IDs (UUIDs).
  3. On the Local Cluster Configuration page in the Admin UI, click the external radio button next to Key Management System (KMS). Additional fields for setting up the KMS are displayed.
  4. Provide the host name and port number for your external KMS in the appropriate fields.

    Replace the existing host name and port and any existing encryption key IDs, with the information for the external KMS.

  5. Add the encryption key IDs (generated by the external KMS) for the types of encryption you are configuring, to the appropriate fields on the Local Cluster Configuration page in the Admin UI.
  6. Click ok.

    Adding the encryption information will require a restart of all of the hosts in your cluster.

  7. Turn on the types of encryption you wish from Admin UI (data encryption, configuration file encryption, and/or log file encryption).

When using an external KMS, key encryption keys (KEK) might be rotated according to the policy set in the KMS. Each time that the keys are rotated in an external KMS, you will have to update the new KEK IDs (UUIDs - i.e. key encryption keys - KEKs) to MarkLogic. Data will then start to be encrypted with new KEK ID, as described in Key Rotation. The object keys (OKEYs) with be enveloped by the external KMS and the new keys as MarkLogic provides the new IDs to the KMS.

Encryption at rest may be configured using REST, XQuery, or JavaScript APIs. See APIs for Encryption at Rest for details.

Transitioning from PKCS #11 Secured Wallet to an External KMS

Transitioning from the internal PKCS #11 secured wallet to an external KMS will re-encrypt of all configuration files and forest labels. Re-encryption will happen the next time a file is written to disk. If a you want to force re-encryption of all data, start a re-index of the database.

Customer provided cluster KEK IDs will be validated against the KMS for encryption/decryption. If any KEK ID validation fails or MarkLogic cannot connect to the KMS, there will be no changes to the configuration files.

Even after you have migrated to an external KMS, the PKCS #11 secured wallet will retain and manage any encryption keys that were generated before the migration to the external keystore.

To migrate from the PKCS #11 secured wallet to an external keystore (KMS) do the following:

  1. Confirm that the external KMS is running and available. SeeSetting Up an External KMS with MarkLogic Encryption .
  2. Enable the desired encryption options from the MarkLogic Admin UI. MarkLogic encryption will now used the encryption keys supplied by the external KMS.

Transitioning From an External KMS to PKCS #11 Secured Wallet

If for some reason you want to stop using your external KMS and revert back to using the internal PKCS #11 secured wallet, use the steps in this section to transition to the internal PKCS #11 wallet.

To migrate encryption to internal the PKCS #11 wallet, do the following:

  1. Turn off encryption on all categories and force decryption of all encrypted forests by issuing a merge command.
  2. Ensure that all data is un-encrypted, forest status reports encryption size.

    Encrypted read-only forests will need to be set to updates-allow all and merge or they will be inaccessible.

  3. Set the configuration back to the internal PKCS #11 KMS and rotate the key encryption keys. See Key Rotation for more information.
  4. Re-index or force a merge of the database to re-encrypt your data.

    Moving from an external KMS to the internal KMS will downgrade your overall security, as the external KMS is more secure than the internal PKCS #11 secured wallet.

Administration and Maintenance

This section covers additional tasks you may want to perform once you have configured encryption.

Backup and Restore

The individual backup files are encrypted with the cluster data encryption key (CDKEK). Backups are forest driven, so data from an encrypted forest will also be encrypted in backups. Configuration files included in a backup will be encrypted if the cluster is enabled for configuration file encryption. This encryption works with full backups, incremental backups, and journal archiving.

The encryption keys residing in the PKCS #11 secured wallet (the embedded KMS) will be exported as part of a full backup. This is true whether encryption is configured to use the internal KMS or an external KMS. Keys may be from the PKCS #11 secured wallet or the external KMS. Full backups will include this exported copy of the keystore, encrypted using the embedded KMS passphrase.

If you cannot access your PKCS #11 secured wallet (or external KMS if you are using one), or lose your encryption keys, you will not be able to decrypt any of your encrypted data (including backups). There is no 'back-door' to recover the encrypted data. We recommend that you backup your encryption keys in a secure location.

The built-in function admin:cluster-set-keystore-passphrase can be used to change the KMS passphrase. When you first set up encryption, we strongly recommend that you change the KMS passphrase to something other than the admin passphrase. This is to ensure that you utilize the Separation of Duties security principle as much as possible.

By default the keystore passphrase is automatically set to the admin password. We strongly recommend that you set a new, different passphrase before turning on encryption.

During an internal keystore backup/restore, data is added to the embedded PKCS #11 secured wallet; nothing is deleted. The encrypted file containing the keys is named kms.exp. The internal keystore is not imported during a restore from a backup. If you need to restore the keys, use the xdmp:keystore-import function. The keystore passphrase will be required to decrypt the exported keystore file when restoring backups on another MarkLogic instance.

To change the keystore passphrase, the current password or passphrase is required.

To restore an encrypted backup to the same cluster:

  1. Import the backup as usual. See Backing Up and Restoring a Database in the Administrator's Guide for details.

To restore an encrypted backup to a different cluster:

  1. Use the xdmp:keystore-import function to import the keystore. The function requires the keystore passphrase to decrypt the keystore.

    xdmp:keystore-import("strong passphrase", "/backups/MarkLogic.wallet.bak")

    The import process will reject duplicate keys and log a warning that includes the ID of the rejected keys. Imported keys can only be used for decryption.

  2. Import the backup as usual. See Backing Up and Restoring a Database in the Administrator's Guide for details.

    Even if you have enabled encryption at the time of backup and restore, the restored files will not be encrypted. The files will need to be merged in order to be encrypted. This is the expected behavior.

Using this process you can move your encrypted backups from one system to another and restore them, as long as you have the passphrase and import the keystore into the new system before restoring the backup. See Backup and Restore Overview in the Administrator's Guide for more information about backup and restore procedures.

If you lose the cluster configuration information, you must first manually restore the keystore before an encrypted backup can be restored.

To export your keystore, use the xdmp:keystore-export function.

xdmp:keystore-export("strong passphrase", "/backups/MarkLogic.wallet.bak")

This function exports all of the encryption keys stored in the MarkLogic embedded KMS (the PKCS #11 secured wallet) and stores them at the location provided to the function.

By default the MarkLogic embedded KMS (the PKCS #11 secured wallet) is automatically included in a backup. You can exclude the embedded wallet using the options in admin:cluster-set-keystore-backup-option. The include or exclude options enable you to choose whether to have the embedded KMS included as part of backups.

xquery version "1.0-ml"; 
import module namespace admin = "http://marklogic.com/xdmp/admin" 
  at "/MarkLogic/admin.xqy";

let $option := "exclude"
let $config := admin:get-configuration()
return 
  admin:cluster-set-keystore-backup-option($config,$option)

Setting the option to exclude prevents the embedded KMS from being included in the backup.

If you set the backup option to exclude and turn off the automatic inclusion of the keystore, you are responsible for saving keystore (the embedded KMS) to a secure location. If you cannot access your PKCS #11 secured wallet (or external KMS if you are using one), or lose your encryption keys, you will not be able to decrypt any of your encrypted data (including backups).

Backups Using a Secondary Key

MarkLogic encryption at rest includes the ability to use a secondary backup KEK (key encryption key) for encrypting backups when encryption is configured with an external KMS (the BDKEK). Using this BDKEK you can restore your backup to a new system, one that might not have access to the CDKEK and/or CCKEK.

For example, with this XQuery statement you can backup your Documents database using the BDKEK:

xdmp:database-backup(xdmp:database-forests(xdmp:database("Documents")),????"/backups/Data", fn:true(), 
"/backups/JournalArchiving", 15,"bf44aab-3f7a-41d2-a6a5-fc41a0e5e0cf")

Or you could use server-side JavaScript:

xdmp.databaseBackup(xdmp.databaseForests(xdmp.database("Documents")),?? "/backups/Data", fn:true(),
"/backups/JournalArchiving", 15,"bf44aab-3f7a-41d2-a6a5-fc41a0e5e0cf");

In these examples 'bf44aab-3f7a-41d2-a6a5-fc41a0e5e0cf' is the secondary backup key (BDKEK).

The built-ins xdmp:database-backup and xdmp:database-incremental-backup have an optional argument to take advantage of the BDKEK from the external KMS. The REST API can also take advantage of a secondary backup key as part of the backup operations.

Viewing Encrypted Log Files Outside of the Server

MarkLogic encryption at rest includes the mlecat command line tool, which can be used to view encrypted log files. Log files are encrypted with the keystore passphrase by default. We recommend that when you turn on encryption, you set an additional alternative passphrase using admin:cluster-set-keystore-logs-encryption-passphrase to encrypt the log files if you want to view encrypted log files outside of MarkLogic Server.

The mlecat tool should be run by a user with sufficient OS privileges to access the PKCS#11 wallet (located by default at /var/opt/MarkLogic). It is suggested that the user be a member of group running MarkLogic (by default daemon).

The log files must be encrypted with an alternative passphrase in order to view the encrypted log files outside of the MarkLogic server. To encrypt log files with an additional alternative passphrase, use the admin:cluster-set-keystore-logs-encryption-passphrase built-in function

xquery version "1.0-ml"; 
import module namespace admin = "http://marklogic.com/xdmp/admin" 
  at "/MarkLogic/admin.xqy";

let $config := admin:get-configuration()
let $passphrase := "dazzling zebras"
return 
  admin:cluster-set-keystore-logs-encryption-passphrase($config,$passphrase)

=>
<configuration>
 <keystore xmlns="http://marklogic.com/xdmp/keystore">
  <keystore-id>18281011442190322418</keystore-id>
  <data-encryptionxmlns:ks="http://marklogic.com/xdmp/keystore">default-off
    </data-encryption>
  <config-encryption xmlns:ks="http://marklogic.com/xdmp/keystore">on
    </config-encryption>
  <logs-encryption xmlns:ks="http://marklogic.com/xdmp/keystore">on
    </logs-encryption>
  <kms-type>internal</kms-type>
  <host-name>localhost</host-name>
  <port>9056</port>
  <data-encryption-key-id>7f8b1e33-4956-4361-a0d7-877e8eeeea37
    </data-encryption-key-id>
  <config-encryption-key-id>459c7781-3d5a-497d-b54d-2057d89dcd17
    </config-encryption-key-id>
  <logs-encryption-key-id>
ce975984-067c-4cfd-b0c9-db61622bd1ae
    </logs-encryption-key-id>
  <kmip-certificate-path></kmip-certificate-path>
  <kmip-key-path></kmip-key-path>
  <kmip-CA-path></kmip-CA-path>
  <key-size>256</key-size>
  <p11-driver-path></p11-driver-path>
  <kms-init-id xmlns:ks="http://marklogic.com/xdmp/keystore">VUZ8CO9KPxhXeE1ONqxy00
    </kms-init-id>
 </keystore>
</configuration>

Log file encryption must be enabled for this passphrase to be used.

In the case where MarkLogic is configured to use the PKCS #11 embedded wallet, the tool requires a passphrase (either the default or preferably the alternative one that you have set). If MarkLogic encryption at rest is configured with an external KMS, the tool requires a KMS user ID and password.

You can always set the alternative passphrase after you set up encryption, however it will only work on logs generated after the passphrase was set. Older logs will require the keystore passphrase in order to be decrypted.

To use the mlecat command line tool, your environment must be set to include the MarkLogic libraries.

  • On Linux, LD_LIBRARY_PATH must include $MARKLOGIC_INSTALL_DIR/lib
  • On Windows, PATH must include %MARKLOGIC_INSTALL_DIR%

For example, to run the mlecat tool on Linux, set this environment:

MARKLOGIC_DATA_DIR=/var/opt/MarkLogic
MARKLOGIC_INSTALL_DIR=/opt/MarkLogic
LD_LIBRARY_PATH=$MARKLOGIC_INSTALL_DIR/lib:$LD_LIBRARY_PATH
PATH=$PATH:$MARKLOGIC_INSTALL_DIR/bin
export MARKLOGIC_INSTALL_DIR MARKLOGIC_DATA_DIR PATH LD_LIBRARY_PATH

To use the mlecat tool to view an encrypted log file, you would use:

MARKLOGIC_KMS_PASSPHRASE=dazzling zebras
export MARKLOGIC_KMS_PASSPHRASE
mlecat ErrorLog.txt;
mlecat -p admin /var/opt/MarkLogic/Logs/ErrorLog.txt

These command options are available for the mlecat tool:

mlecat [option] filepath(s )
option:
  -I iDIR, iDir is MarkLogic's Install directory, alternatively the environmental variable
MARKLOGIC_INSTALL_DIR can be used to set this value.
  -d dDIR, dDIR is MarkLogic's Data directory, alternatively the environmental variable
MARKLOGIC_DATA_DIR can be used to set this value
  -u KMS_USER, KMS_USER is the user id to authenticate with the external KMS
  -p PASS, PASS is the passphrase to authenticate with the KMS (internal or external); 
    alternatively the environmental variable MARKLOGIC_KMS_PASSPHRASE can be used to provide 
    this value. If MarkLogic is configured to use an additional passphrase for logs 
    such passphrase can be provided here
  [-f] filepath(s), one or more file paths (-f can be specified before each file for explicit file list)

Alternative Hardware Security Module (HSM) Configuration

The MarkLogic PKCS #11 wallet (the embedded KMS) uses SoftHSM as its default hardware security module (HSM). This section describes the process of setting up an alternate hardware security module if you want to use a PKCS #11 HSM or any other PKCS #11 compliant storage device for your keystore.

To use this procedure, you must not have any encryption enabled. This means that you should have turned off or disabled encryption and ensured that all data, configuration, and log files have been decrypted at both the cluster and database levels. In addition, the PKCS #11 device must not be initialized and no PIN should be set. MarkLogic will initialize the device and set a PIN.

Do the following:

  1. Update the library path by calling the admin function admin:cluster-set-keystore-p11-driver-path
  2. Rotate all encryption keys by calling the admin functions: admin:cluster-rotate-data-encryption-key-id, admin:cluster-rotate-config-encryption-key-id and admin:cluster-rotate-logs-encryption-key-id
  3. Restart MarkLogic Server.
  4. Before proceeding, validate the status of keystore using xdmp:keystore-status. The status field must be 'normal' and not 'error'.
    xquery version "1.0-ml"; 
    import module namespace admin = "http://marklogic.com/xdmp/admin" 
      at "/MarkLogic/admin.xqy";
    
    xdmp:keystore-status()
    =>
    <keystore-status xmlns="http://marklogic.com/xdmp/status/keystore">
      <status>normal</status>
      <kms-type>internal</kms-type>
      <data-encryption>default-off</data-encryption>
      <config-encryption>on</config-encryption>
      <logs-encryption>on</logs-encryption>
      <data-encryption-key-id>7f8b1e33-4956-4361-a0d7-877e8eeeea37
        </data-encryption-key-id>
      <config-encryption-key-id>459c7781-3d5a-497d-b54d-2057d89dcd17
        </config-encryption-key-id>
      <logs-encryption-key-id>ce975984-067c-4cfd-b0c9-db61622bd1ae
        </logs-encryption-key-id>
    </keystore-status>

Disaster Recovery/Shared Disk Failover

Unless you have suffered a complete loss of your host, disaster recovery should work just fine with encryption at rest. See High Availability and Disaster Recovery in the Concepts Guide for information about setting up shared disk failover and steps for disaster recovery.

If you have experienced a complete loss of your host, you will need to do the following:

  1. Reinstall and configure a new MarkLogic host.
  2. Import the keystore and keys from a backup (using xdmp:keystore-import). See Export and Import Encryption Keys for details.
  3. Perform a restore from backup as usual. See Backing Up and Restoring a Database in the Administrator's Guide for more information.

APIs for Encryption at Rest

The encryption at rest feature includes APIs for working with encryption, using either the default keystore (the interal PKCS #11 secured wallet) or a KMIP-compliant external KMS.

This section includes:

Built-ins for Encryption at Rest

These functions will work with both the internal PKCS #11 secured wallet, or a external KMIP-compliant keystore. Using these functions you can encrypt data and check the status of encryption in your clusters using either JavaScript or XQuery.

The Server-Side JavaScript built-ins are:

The Server-Side XQuery built-ins are:

Using a Credential ID with http-options

The xdmp:http-options function now accepts a credential-id when used with XQuery. The schema looks like this:

<xs:complexType name="options">
    <xs:sequence>
      <xs:element ref="timeout" minOccurs="0"/>
      <xs:element ref="data" minOccurs="0"/>
      <xs:element ref="headers" minOccurs="0"/>
      <xs:element ref="credential-id" minOccurs="0"/>
      <xs:element ref="authentication" minOccurs="0"/>
      <xs:element ref="client-cert" minOccurs="0"/>
      <xs:element ref="client-key" minOccurs="0"/>
      <xs:element ref="pass-phrase" minOccurs="0"/>
    </xs:sequence>
  </xs:complexType>

Admin APIs for Encryption at Rest

These functions are used to set the mode and descriptions for the host, set the keystore host name and the keystore host port. You can also set the keystore data key ID, config key ID, or logs key ID, along with setting the keystore serve certificate and enabling encryption.

These server-side XQuery functions work with either the PKCS #11 secured wallet or a third-party KMIP-compliant keystore:

The admin:cluster-rotate-xxxx-encryption-key-id APIs are only for use with the embedded KMS provided by MarkLogic (the PKCS #11 secured wallet). Using these functions with an external KMS will cause an error.

These next two APIs are used in transitioning from an internal keystore (the PKCS #11 secured wallet) to an external KMIP-compliant keystore. If these functions are set to external, MarkLogic Server will first look for the external keystore to verify the keys.

These functions are designed to work with a external KMIP-compliant keystore:

REST Management APIs for Encryption

You can manage encryption using the REST Management APIs. Some of the tasks you can do with these APIs include:

  • Encryption configuration
  • Keystore configuration
  • Database configuration
  • Database status, including database encryption (encrypted size, total size)
  • Cluster status
  • Forest status
  • Security
  • Backups, status (encrypted or not)
  • Restore (with property for using private key)

The REST Management APIs that are used to query and manage the cluster security properties include encryption information for database, cluster, and forest.

Below is a XML payload example for the security endpoint:

<security-properties xmlns="http://marklogic.com/manage/security/properties"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://marklogic.com/manage/security/properties
manage-security-properties.xsd"> 
<keystore> 
<data-encryption>default-off</data-encryption>
<config-encryption>off</config-encryption>
<logs-encryption>off</logs-encryption>
<kms-type>internal</kms-type>
<host-name>localhost</host-name>
<port>9056</port>
<data-encryption-key-id>92ed7360-458a-427e-abad-c6595b192cb7</data-encryption-key-id>
<config-encryption-key-id>8b9a9bdb-7b0e-41eb-9aa6-ed6e8cb23ad5</config-encryption-key-id>
<logs-encryption-key-id>01c50d02-b43f-46bc-bbe5-6d4111d1180b</logs-encryption-key-id>
</keystore>
</security-properties>

And here is a JSON payload example for the security endpoint:

{
  "keystore": {
    "data-encryption": "default-off",
    "config-encryption": "off",
    "logs-encryption": "off",
    "kms-type": "internal",
    "host-name": "localhost",
    "port": 9056,
    "data-encryption-key-id": 
      "92ed7360-458a-427e-abad-c6595b192cb7",
    "config-encryption-key-id": 
      "8b9a9bdb-7b0e-41eb-9aa6-ed6e8cb23ad5",
    "logs-encryption-key-id": 
      "01c50d02-b43f-46bc-bbe5-6d4111d1180b"
  }
}

These operations are available for encryption key rotation:

curl -v -X POST --anyauth --user admin:admin \
  --header "Content-Type:application/json" -d \
  '{"operation":"rotate-config-encryption-key"}' \
  http://localhost:8002/manage/v2/security
curl -v -X POST --anyauth --user admin:admin \
  --header "Content-Type:application/json" -d \
  '{"operation":"rotate-data-encryption-key"}' \
  http://localhost:8002/manage/v2/security
curl -v -X POST --anyauth --user admin:admin \
  --header "Content-Type:application/json" -d \
  '{"operation":"rotate-logs-encryption-key"}' \
  http://localhost:8002/manage/v2/security

Interactions with Other MarkLogic Features

In most cases the encryption at rest feature will be transparent to the user, that is data on disk will be encrypted, decrypted during use (by users with the appropriate security permissions), and re-encrypted when the data is written back to disk.

Rolling Upgrades

During a rolling upgrade, the encryption at rest feature is not available for use until cluster is committed to the new version (admin:commit-upgrade). See Rolling Upgrades in the Administrator's Guide for more about rolling upgrades.

During upgrades, the default passphrase for the upgraded system is not set. You will need to reset the default passphrase after an upgrade.

Telemetry.

The telemetry feature is not available for use until the cluster is upgraded to MarkLogic 9.0-1 or later. See Telemetry in the Monitoring MarkLogic Guide for more about telemetry.

« Previous chapter
Next chapter »
Powered by MarkLogic Server 7.0-4.1 and rundmc | Terms of Use | Privacy Policy