MarkLogic 10 Product Documentation

Security Guide — Chapter 14

Embedded KMS Key Hierarchy

When you use the embedded PKCS #11 secured wallet provided with MarkLogic Server, the recommended key hierarchy would be similar to this illustration.

MarkLogic Server generates the Data Key Encryption Key (CDKEK), the Configuration Key Encryption key (CCKEK) and the Logs Key Encryption Key (CLKEK). The Data Key Encryption Key is then used to wrap the OKEYs for the database objects (journals, data files, etc.). These keys are stored in the wallet (internal KMS). The key IDs are generated in the MarkLogic Server for encryption and decryption by the KMS (the PKCS #11 secured wallet in this case). The configuration happens at the cluster level because there is one keystore per cluster.

The individual Object Encryption Keys (OKEYs) are then randomly generated and used to directly encrypt individual files (journals, config files, and log files, etc.). These keys (the OKEYs) are wrapped (encrypted) with the corresponding KEK for data, config, and logs. A unique key protects (encrypts) each file. The keys at the object levels are wrapped (encrypted by the keys above them) for each category.

For example, the Data Key Encryption Key (CDKEK) wraps (encrypts) the Object Encryption Key (OKEY) for a file such as a journal. The keys at the bottom of the diagram are encrypted (wrapped) by all the keys above them in the hierarchy, and then placed in the header for each file. In the case of the embedded KMS, there is only one CDKEK for the entire cluster - all databases in the cluster will use that key. When using the embedded KMS, it is not possible to use per database keys for encryption.

Database backups are encrypted using the locally generated backup key (BKEK) that is used to encrypt all of the files in the backup. The BKEK is then encrypted with the cluster data key (CDKEK) and then encrypted with the cluster key (CKEK). Additionally you could encrypt this key with the BDKEY and a passphrase. See Backup and Restore for more information about backups.

External KMS Key Hierarchy

The external KMS provides even higher security, along with additional key management features. When you use an external key management system (KMS or keystore), the recommended key hierarchy deployment might look like this illustration.

The keystore contains the Master Key Encryption Key (MKEK). The KMS generates or derives the Cluster Key Encryption Key (CKEK), which is enveloped (encrypted) with the Master Key Encryption Key. Both the Master Key Encryption Key and the Cluster Key Encryption Key reside in the KMS keystore. These keys never leave the keystore. MarkLogic Server has no knowledge or control over these keys. The keys are referenced from the keystore by their key IDs. The actual encryption keys never leave the KMS.

There are multiple levels to the key hierarchy in this deployment, each level wrapping (encrypting) the level below it. The KMS generates the cluster level encryption keys for data (CDKEK), configuration files (CCKEK), and log files (CLKEK). The corresponding KEK is used is used to encrypt (wrap) all the Object Encryption Keys (OKEY) generated by MarkLogic Server for each file, so that a unique key protects each file, no matter what category (data, configuration files, logs). A unique key protects each file within a category (data, configuration files, logs).

The corresponding KEK (for data, config, or logs) is used to encrypt (wrap) all the Object Encryption Keys (OKEY) generated by MarkLogic Server for each file, so that an encryption key protects each file, no matter what category (data, configuration files, logs).

For example, the Master Key Encryption Key (MKEK) wraps (encrypts) the Cluster Key Encryption Keys (CKEK), which in turn wraps (encrypts) the Data Key Encryption Key (CDKEK), then wraps (encrypts) the Object Encryption Key (OKEY) for a file such as a stand. The keys at the bottom of the diagram are encrypted (wrapped) by all the keys above them in the hierarchy, and then placed in the header for each file.

Database backups are encrypted using the BKEK, the locally generated backup KEK, the BKEK is encrypted with the CDKEK. Then the CDKEY may be encrypted or derived from the cluster key (CKEK). This last step is outside of the control of MarkLogic. You can also use a password or passphrase to encrypt and secure your backup. See Backup and Restore for more information about backups and the use of a passphrase to secure your backup.

If you plan to use an external key management system, configure the external KMS first, and then turn on encryption in the MarkLogic server.

Set Up Encryption Example

Install MarkLogic 9.0-1 or later. Encryption at rest options are not available in earlier versions of MarkLogic Server. You must explicitly select which data (databases, configuration files, log files, or entire clusters) you want to have encrypted. This example shows how to set up encryption for a single database.

The Security database or other databases used by MarkLogic will not be encrypted by default. Existing data can be encrypted by forcing a merge or a reindex of the database.

See Configuring Encryption at Rest for more details.

Encrypt a Database

For this example, we will use the Admin UI to set up encryption for the Documents database.

1. Select Databases from the left tree menu in the Admin UI.
2. Click on the Documents database.
3. On the Database Configuration page, next to data encryption, select on from the drop-down menu. (The other options are default-cluster and off.)

   

4. Click ok.

If you select default-cluster, encryption for that database will default to whatever encryption option has been set for the cluster as a whole. If the cluster is set to encrypt data, this database will be encrypted. If encryption has not been turned on for the cluster, this database will not be encrypted if default-cluster is selected. See Cluster Encryption Options for details.

As you access data in your database, it will be encrypted when it is written back to disk. You can view the encryption progress on the Database Status page by looking at the Size and Encrypted Size numbers.

To encrypt the existing data in your database, you will need to re-index your database. On the Database Configuration page, click the reindex button at the top of the page (below the OK button), and then click ok. You can also force a merge of the database to encrypt the data.

Encryption of large databases will take some time initially. Updates and changes to the database will be fairly transparent to the user after initial encryption. The Size and Encrypted Size numbers will be equal when the encryption process is complete.

Test It Out

Using Query Console, you can run a simple query to verify that the Documents database has encryption turned on.

xquery version "1.0-ml";
import module namespace admin = "http://marklogic.com/xdmp/admin"		
  at "/MarkLogic/admin.xqy";

let $config := admin:get-configuration()
return admin:database-get-data-encryption($config, xdmp:database("Documents"))

=>
on

You can also check the Size and Encrypted Size numbers on the Database Status page. These numbers will be equal when the encryption process is complete and the entire database is encrypted.

Turn Off Encryption for a Database

1. Select Databases from the left tree menu in the Admin UI.
2. Click on the Documents database to turn off encryption.
3. On the Database Configuration page, next to data encryption, select off from the drop-down menu.

   

4. Click ok.

To verify that encryption is turned off, run this query in Query Console:

xquery version "1.0-ml";
import module namespace admin = "http://marklogic.com/xdmp/admin" 
		  at "/MarkLogic/admin.xqy";

let $config := admin:get-configuration()
return admin:database-get-data-encryption($config, xdmp:database("Documents"))

=>
off

To decrypt the existing data in your database, you will need to re-index your database. On the Database Configuration page, click the reindex button and then click ok.

You can also decrypt the data by forcing a merge on the database to decrypt its contents. This process may take a while.

Database Encryption Options

You can configure encryption for each database on the Database Configuration page in the Admin UI. Encryption at rest can be separately enabled per database, or at the cluster level by setting the database encryption to default to the cluster encryption settings. The encryption options for databases are shown in the following table.

With encryption enabled, files are encrypted as they are ingested into the database, or when those files are written back to disk. If you want to encrypt existing data in a database either reindex the database or force a merge on the database. This will take a few minutes depending on the size of database. See Cluster Encryption Options

Large binary files are only encrypted during initial ingestion into the database. If you want to encrypt existing large binary files already loaded into MarkLogic Server prior to turning on encryption, you must reindex the database or force a merge.

1. To configure database encryption, go to the Admin UI and click Databases in the left navigation tree.
2. Click on the database you want to encrypt.
3. On the Database Configuration page, next to data encryption, select on from the drop-down menu. (The other options are default-cluster and off.)

   

4. Click ok when you are done.

Configure Cluster Encryption

You can set cluster encryption options for configuration files and log files, and also set or override the encryption options for databases on the Cluster Configuration page.

Cluster Encryption Options

You can either configure encryption for the embedded keystore (the PKCS #11 secured wallet) or for a external KMIP-compliant keystore using the Admin UI. Use the Edit Keystore Configuration page to configure encryption at rest for a cluster. Using this page you can configure data encryption, configuration file encryption, or encryption of log files.

1. To configure encryption using the embedded keystore in the Admin UI, click Clusters in the left navigation tree and click the name of the cluster you want to configure.
2. Click the Keystore tab to configure the keystore for encryption at rest.

   

3. Use the drop-down menus to configure encryption for data, config files, and/or log files.

   Setting	Description
   data encryption	Specifies whether or not encryption is enabled for user data. The options are: force -- Force encryption for all data in the cluster. The database configuration cannot overwrite this setting. default-on -- By default encryption is on. The database configuration can overwrite this setting. default-off -- By default encryption is off. The database configuration can overwrite this setting.
   config encryption	Specifies whether or not encryption is enabled for configuration files
   logs encryption	Specifies whether or not encryption is enabled for log files.
   kms type	Specifies whether the KMS is internal to MarkLogic or an external KMS A keystore is a secure location where the actual encryption keys used to encrypt data are stored. The keystore for encryption at rest is a key management system (KMS). This keystore can be either the MarkLogic embedded PKCS #11 secured wallet, or an external third party KMS

   Beneath these options on the Edit Keystore Configuration page, there are two tabs for specifying further options for either the Internal KMS or the External KMS. For the Internal KMS there are these options:

   Setting	Description
   backup option	The internal KMS is automatically included in backups unless you change the default setting of include to exclude.
   internal data encryption key id	The UUID that identifies the encryption key from the internal KMS that is to be used to encrypt data files.
   internal config encryption id	The UUID that identifies the encryption key from the internal KMS that is to be used to encrypt config files.
   internal logs encryption id	The UUID that identifies the encryption key from the internal KMS that is to be used to encrypt log files.

4. Click ok when you are done.

   Adding or changing any encryption information will require a restart of all of the hosts in the cluster.

Using an Alternative PKCS #11 Device

MarkLogic uses SoftHSM as its default hardware security module (HSM). This section describes the process of setting up an alternate hardware security module if you want to use a PKCS #11 HSM (or any other PKCS #11-compliant HSM) by following these steps before starting MarkLogic for the first time.

This process will only work on a clean data directory with a first time install.

1. The PKCS#11 devices must not be initialized and no PIN should be set, MarkLogic will initialize it and set a PIN.
2. Set environment variable=MARKLOGIC_P11_DRIVER_PATH to the PKCS#11 library you want to use.
3. Start MarkLogic for the first time
4. Verify no error messages are logged during startup.

let $dir-name := "/sotfhsm/wallet"
let $config := admin:get-configuration()
return 
  admin:cluster-set-keystore-wallet-location($config,$dir-name)

Configure Encryption Using XQuery

Instead of using the Admin UI, you can configure encryption for your MarkLogic instance using XQuery.

In Query Console, you can use admin:cluster-set-data-encryption to turn on data encryption for the current database:

xquery version "1.0-ml";
import module namespace admin = "http://marklogic.com/xdmp/admin" 
		  at "/MarkLogic/admin.xqy";

let $config := admin:get-configuration()
return 
  admin:cluster-set-data-encryption($config,"default-on")

For example, to set the encryption for log files at cluster level:

xquery version "1.0-ml"; 
import module namespace admin = "http://marklogic.com/xdmp/admin" 
  at "/MarkLogic/admin.xqy";

let $config := admin:get-configuration()
return 
  admin:cluster-set-logs-encryption($config, "default-on")
													 
(: returns the new configuration element -- use admin:save-configuration 
to save the changes to the configuration or pass the configuration to
other Admin API functions to make other changes.  :)

To see whether encryption is turned on for log files, you can run this XQuery in the Query Console:

xquery version "1.0-ml"; 
import module namespace admin = "http://marklogic.com/xdmp/admin" 
  at "/MarkLogic/admin.xqy";

let $config := admin:get-configuration()
return 
  admin:cluster-get-logs-encryption($config)
=>
on
 (: returns the encryption setting for log files:)

Configure Encryption Using REST

You can use REST Management APIs to work with encryption at rest.

GET /manage/v2/databases/{id|name}/properties

This command gets the current properties of the Documents database, including the encryption status and encryption key ID in JSON format:

$ curl -GET --anyauth -u admin:admin \
  -H "Accept:application/json,Content-Type:application/json" \
  http://localhost:8002/manage/v2/databases/Documents/properties

Returns

{"database-name":"Documents", "forest":["Documents"],
"security-database":"Security", "schema-database":"Schemas",
"triggers-database":"Triggers", "enabled":true,
"data-encryption":"off", "encryption-key-id":"", 

The same command in XML format:

$ curl -GET --anyauth -u admin:admin \
  -H "Accept:application/xml,Content-Type:application/xml" \
  http://localhost:8002/manage/v2/databases/Documents/properties

Returns

<database-properties xmlns="http://marklogic.com/manage">
  <database-name>Documents</database-name>
  <forests>
    <forest>Documents</forest>
  </forests>
  <security-database>Security</security-database>
  <schema-database>Schemas</schema-database>
  <triggers-database>Triggers</triggers-database>
  <enabled>true</enabled>
  <data-encryption>on</data-encryption>
  <encryption-key-id/>
...
</database-properties>

GET /manage/v2/security/properties

This command returns the current encryption status, along with other properties including encryption key ID, for localhost in JSON format:

$ curl -GET --anyauth -u admin:admin \
  -H "Accept:application/json,Content-Type:application/json" \
  http://localhost:8002/manage/v2/security/properties

Returns:

{"keystore":{"data-encryption":"default-off",
"data-encryption-key-id":"091fd9a0-f090-4c7e-91ca-fedfe21dbfef",
"config-encryption":"off", "config-encryption-key-id":"",
"logs-encryption":"off", "logs-encryption-key-id":"",
"host-name":"LOCALHOST", "port":9056}}

Here is the same version of the command, this time returning XML:

$ curl -GET --anyauth -u admin:admin \
  -H "Accept:application/xml,Content-Type:application/xml" \
  http://localhost:8002/manage/v2/security/properties

Returns:

<security-properties xsi:schemaLocation="http://marklogic.com/manage/security/properties
manage-security-properties.xsd" xmlns="http://marklogic.com/manage/security/properties"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <keystore>
    <data-encryption>default-off</data-encryption>
    <data-encryption-key-id>8d0b07d8-b655-4408-affd-e49a2ece0af3
      </data-encryption-key-id>
    <config-encryption>off</config-encryption>
    <config-encryption-key-id/>
    <logs-encryption>off</logs-encryption>
    <logs-encryption-key-id/>
    <host-name>LOCALHOST</host-name>
    <port>9056</port>
  </keystore>
</security-properties>

POST:/manage/v2/security/properties

This command sets the protected path for //d with read permissions for manage-user:

$ curl -POST --anyauth -u admin:admin \
  -d @file.xml -H "Content-Type:application/xml" \
  http://localhost:8002/manage/v2/protected-paths

Here is the payload (file.xml):

<protected-path-properties xmlns="http://marklogic.com/manage/protected-path/properties">
  <path-expression>//d</path-expression>
  <path-namspaces/>
  <permissions>
    <permission>
      <role-name>manage-user</role-name>
      <capability>read</capability>
    </permission>
  </permissions>
</protected-path-properties>

Here is the same operation in JSON:

curl -X POST --anyauth -u admin:admin \
-d @file.json -H "Content-Type:application/json" \
http://localhost:8002/manage/v2/protected-paths

Here is the payload (file.json):

{
"path-expression": "//e",
"path-namespace": [],
"permission": [{
"role-name": ["manage-user"],
"capability": "read"
}]
}

PUT /manage/v2/databases/{id|name}/properties

This command will turn on encryption for the Documents database:

$ curl -X PUT --anyauth -u admin:admin -d '{"data-encryption":"on"}' \
  -H "Content-Type:application/json" \
  http://localhost:8002/manage/v2/databases/Documents/properties

POST manage/v2/security?
operation=export-wallet&filename=/my/test..wallet&password=test

curl -v -X POST  --anyauth --user $MANAGEADMIN:$MANAGEPASS \
  --header "Content-Type:application/xml" \
-d@data/security/export-wallet.xml \
  http://$host:8002/manage/v2/security

<export-wallet-operation xmlns="http://marklogic.com/manage/security">
  <operation>export-wallet</operation>
  <filename>/tmp/mywallet.txt</filename>
  <password>mypassword</nassword>
</export-wallet-operation>

POST manage/v2/security
{"operation":"export-wallet","filename":"/my/test.wallet","password":"test"}

curl -v -X POST  --anyauth --user $MANAGEADMIN:$MANAGEPASS \
  --header "Content-Type:application/json" \
-d@data/security/export-wallet.json \
  http://$host:8002/manage/v2/security

{
    "operation":"export-wallet",
    "filename":"/tmp/mywallet.tmp",
    "password":"mypassword"
}

POST manage/v2/security?
operation=import-wallet&filename=/my/test.wallet&password=test

curl -v -X POST  --anyauth --user $MANAGEADMIN:$MANAGEPASS \
  --header "Content-Type:application/xml" \
-d@data/security/import-wallet.xml \
  http://$host:8002/manage/v2/security

<import-wallet-operation xmlns="http://marklogic.com/manage/security">
  <operation>import-wallet</operation>
  <filename>/tmp/mywallet.txt</filename>
  <password>mypassword</password>
</import-wallet-operation>

POST manage/v2/security
{"operation":"import-wallet","filename":"/my/test.wallet","password":"test"}

curl -v -X POST  --anyauth --user $MANAGEADMIN:$MANAGEPASS \
  --header "Content-Type:application/json" \
-d@data/security/import-wallet.json \
  http://$host:8002/manage/v2/security

{
    "operation":"import-wallet",
    "filename":"/tmp/mywallet.tmp",
    "password":"mypassword"
}

Key Rotation

For the internal wallet, key encryption keys (KEK) can be manually rotated. Keys can be manually rotated at regular intervals or if an encryption key has been compromised. This type of key rotation can be triggered on individual encryption categories (configuration, data, logs) using MarkLogic built-in functions.

There are two steps to key rotation. First, rotating the KEK keys (using AES 256 symmetric encryption) used to envelope the object file encryption keys, and second, re-encrypting the object file encryption keys (also using AES 256 symmetric encryption).

After calling the built-in function to rotate encryption keys, all new data will be written to disk using the new key encryption key. Old data will be migrated as it is re-written to disk. If you wish to force re-encryption using the new key, you can either force a merge or re-index the forest.

At the local, host level, you can manually rotate the data keys, configuration keys, and the logs keys (CDKEK, CCKEK, CLKEK) using these APIs:

• admin:cluster-rotate-config-encryption-key-id
• admin:cluster-rotate-data-encryption-key-id
• admin:cluster-rotate-logs-encryption-key-id

  These key rotation functions are only available for the MarkLogic internal KMS (the PKCS #11 secured wallet) and not for any keys that are managed by an external KMS.

At the cluster level, to manually rotate the cluster-level keys use these APIs:

• admin:group-get-rotate-audit-files
• admin:group-get-rotate-log-files
• admin:group-set-rotate-audit-files
• admin:group-set-rotate-log-files

  When you are using an external KMS, MarkLogic does not have access to the envelope key, it only has access to the key ID, and asks for the KMS to open the envelope.

Export and Import Encryption Keys

The ability to export and import key encryption keys (KEK) from the PKCS #11 secured wallet (the embedded KMS) is useful when you want to clone a cluster. Exporting a key encryption key (KEK) is restricted to cluster-level keys (CDKEK, CCKEK, CLKEK) and requires a passphrase and a filepath. The data will be exported (encrypted with the passphrase) into a file at the location specified by the filepath.

To export a keystore from the embedded KMS:

xquery version "1.0-ml";
import module namespace admin = "http://marklogic.com/xdmp/admin" 
  at "/MarkLogic/admin.xqy";

xdmp:keystore-export("Unique passphrase", "/backups/MarkLogic.wallet.bak")
=>
true

To import a keystore into the embedded KMS:

xquery version "1.0-ml";
import module namespace admin = "http://marklogic.com/xdmp/admin" 
  at "/MarkLogic/admin.xqy";

xdmp:keystore-import("Unique passphrase", "/backups/MarkLogic.wallet.bak")
=>  true

Key encryption keys can only be imported from MarkLogic exported files. Imported keys can only be used for decryption. The import requires the passphrase that was provided at the time of the export.

If a duplicate key is supplied during the import, the import will be rejected. Duplicate keys can be caused by importing the keystore twice.

Key Deletion and Key Revocation

For these functions you will need to use a external keystore (KMS).

Types of External KMS Deployments

There are a variety of types of external key management systems. An external key management system deployment may be one of the following types:

1. A virtual KMS instance running in a VM (virtual machine) environment, or in a private or public cloud
2. A physical appliance running a KMS server
3. A dedicated FIPS 140-2 Level 3 appliance
4. A dedicated hardened FIPS 140-2 Level 4 appliance

These systems are listed by increasing levels of security.

Using MarkLogic Encryption with AWS Key Management System

Amazon Web Services (AWS) provides a key management system (KMS) that you can use with MarkLogic encryption at rest to encrypt your data. The AWS KMS is supported for customers running their cluster on AWS in the cloud. You must set up your AWS KMS encryption keys and configure the encryption key IDs in your MarkLogic server before using the AWS KMS.

To set up the AWS key management system, first set up your AWS instance. See Getting Started with MarkLogic Server on AWS and Overview of MarkLogic Server on AWS in the MarkLogic Server on Amazon Web Services (AWS) Guide for details.

The AWS KMS keys (data, config, and log encryption keys) needed to encrypt and decrypt data must be configured in MarkLogic before using encryption.

You cannot use the master key and roles from the MarkLogic KMS to access the AWS KMS, so you will need to have a Key Administrator specify access to the AWS KMS keys on a per-key basis tied to the user's IAM role. The Key Administrator can specify access using the Encryption Keys section of the IAM AWS management console. See the next section (AWS KMS on EC2) for details and the AWS documentation regarding key policies for more information.

If an encryption key stored in the AWS KMS is disabled for any reason, it cannot be used for encryption or decryption, and MarkLogic loses access to any data encrypted with the disabled key. Deleting a key will lead to permanent data loss as deleted keys can never be recovered. Any keys created in the AWS KMS are cluster management keys and should never be deleted. See https://docs.aws.amazon.com/kms/latest/developerguide/enabling-keys.html for more information.

Using MarkLogic Encryption with Microsoft Azure Key Vault

Microsoft Azure Key Vault can encrypt your data in MarkLogic. Azure Key Vault is supported for customers running their cluster on Microsoft Azure. You must set up your Azure Key Vault, create the encryption keys in Key Vault, and configure the encryption key IDs in your MarkLogic server before using the keys to encrypt data in MarkLogic.

To set up the Microsoft Azure Key Vault, first set up your Azure instance. See Getting Started with MarkLogic Server on Azure and Overview of MarkLogic Server on Azure for details. Keys are governed by access policies created by the Key Administrator. See the next section (Microsoft Azure Key Vault) for details and the for more information.

If an encryption key stored in the Azure Key Vault is disabled, it cannot be used for encryption or decryption, and MarkLogic loses access to any data encrypted with the disabled key. Deleting a key will lead to permanent data loss as deleted keys can never be recovered.

Set Up an External KMIP KMS with MarkLogic Encryption

To configure the external key management system using the MarkLogic Admin UI on the MarkLogic host, you will need the following information for your external KMS:

• Host name - the hostname of the key management system
• Port number - the port number used to communicate with KMS
• Data encryption key ID (UUID generated by external KMS)
• Configuration encryption key ID (UUID generated by external KMS)
• Logs encryption key ID (UUID generated by external KMS)

The TLS certificates, used to secure the communication with the KMS, must be stored locally on each host in the MarkLogic data directory (/var/opt/MarkLogic). By default, the files are expected to be located in the MarkLogic data directory and must have the following names:

• kmip-CA.pem - The root/certificate of the CA that signed the certificate request for MarkLogic.
• kmip-cert.pem - The certificate that was issued to MarkLogic and the one that was signed by the CA.
• kmip-key.pem - The private key that was generated for MarkLogic and is associated with the Certificate issued to MarkLogic (kmip-cert). (Optional for some KMS servers.)

These certificates are the Certificate Authority (CA) for the root of the certificate chain for the kmip-cert.pem. A certificate could be a self-signed root used by an enterprise or an external CA. Copy these files into the MarkLogic data directory (/var/opt/MarkLogic). The location and name of these files can be changed by calling the admin functions. See Admin APIs for Encryption at Rest for details.

These settings are cluster wide, so each individual host must have a local copy at the location specified.

High Availability and Failover with External KMS

Encryption at rest enables the you to specify multiple hosts, multiple ports, and multiple KMIP credentials to connect to KMIP servers. The information to connect to these servers are specified in the fields on the external Key Management Service (KMS) section of the Edit Keystore Configuration page. The information must be validated at configuration time. For each host with specified, if there must exist a PEM-encoded Certificate Authority file and a PEM-encoded KMIP certificate file accessible to each node of the MarkLogic server.

For each host specified, there must exist a PEM-encoded Certificate Authority file and PEM-encoded KMIP certificate file accessible to each node of MarkLogic server.

The pem files are looked up with the user-specified path or default location for the first host. For subsequent hosts, the file names are expected to be accessible through the original file name pre-pended by the host's index in the configuration sequence.

For example, if the configured host names are kms1.marklogic.com and kms2.marklogic.com. The configured port is 9010. The specified CA file is at path/CA.pem. The specified certificate file is at /path/cert.pem. The configuration must be validated through the following:

1. File /path/CA.pem, /path/1-CA.pem, /path/cert.pem, and /path/1-cert.pem all exist.
2. The user-specified encryption keys can be validated through connecting to kms1.marklogic.com at port 9010.
3. The user-specified encryption keys can be validated through connecting to kms2.marklogic.com at port 9010.

If the first specified KMIP host stops responding, the program will try to connect to each of the other hosts on the user-specified list in turn until it successfully connects.

If for some reason the programs is unable to connect with a valid KMIP server after multiple attempts, it will report exception.

Set up MarkLogic Encryption

Before you set up encryption at rest, be sure that your cluster has upgraded to MarkLogic 9. If the cluster has not been upgraded, the encryption feature will not be available.

1. Set up your external KMS, if not already set up. See Set Up an External KMIP KMS with MarkLogic Encryption for details.
2. Get the generated encryption key IDs from the external KMS (for data, config, and logs as needed). If you are using data encryption, configuration file encryption, and log encryption, and you want different encryption keys for each, you will need three encryption key IDs (UUIDs).
3. Click Clusters in the left navigation tree, then click the name of the cluster to configure.
4. Click the Keystore tab, then click the external radio button next to Key Management System (KMS). Additional fields for setting up the external KMS are displayed.
5. Provide the host name and port number for your external KMS in the appropriate fields.

   Replace the existing host name and port and any existing encryption key IDs, with the information for the external KMS.

   

6. Add the encryption key IDs (generated by the external KMS) for the types of encryption you are configuring (data, configuration, and/or logs), to the appropriate fields on the Edit Keystore Configuration page in the Admin UI.
7. Click ok.

   Adding the encryption information will require a restart of all of the hosts in your cluster.

8. Turn on the types of encryption you wish from Admin UI (data encryption, configuration file encryption, and/or log file encryption).

When using an external KMS, key encryption keys (KEK) might be rotated according to the policy set in the KMS. Each time that the keys are rotated in an external KMS, you will have to update the new KEK IDs (UUIDs - i.e. key encryption keys - KEKs) to MarkLogic. Data will then start to be encrypted with new KEK ID, as described in Key Rotation. The object keys (OKEYs) with be enveloped by the external KMS and the new keys as MarkLogic uses the IDs to request that the OKEY be enveloped with the corresponding KEK ID.

Encryption at rest may be configured using REST, XQuery, or JavaScript APIs. See APIs for Encryption at Rest for details.

Transitioning from PKCS #11 Secured Wallet to an External KMS

Transitioning from the internal PKCS #11 secured wallet to an external KMS will re-encrypt of all configuration files and forest labels. Re-encryption will happen the next time a file is written to disk. If a you want to force re-encryption of all data, start a re-index of the database.

Customer-provided cluster KEK IDs will be validated against the KMS for encryption/decryption. If any KEK ID validation fails or MarkLogic cannot connect to the KMS, there will be no changes to the configuration files.

Even after you have migrated to an external KMS, the PKCS #11 secured wallet will retain and manage any encryption keys that were generated before the migration to the external keystore.

To migrate from the PKCS #11 secured wallet to an external keystore (KMS) do the following:

1. Important: Before you start the transition to an external KMS, backup the wallet that contains all of the internal keys.
2. Confirm that the external KMS is running and available. See Set Up an External KMIP KMS with MarkLogic Encryption .
3. Enable the desired encryption options from the MarkLogic Admin UI. MarkLogic encryption will now used the encryption keys supplied by the external KMS.

Transitioning From an External KMS to PKCS #11 Secured Wallet

If for some reason you want to stop using your external KMS and revert back to using the internal PKCS #11 secured wallet, use the steps in this section to transition to the internal PKCS #11 wallet.

To migrate encryption to internal the PKCS #11 wallet, do the following:

1. Important: Before you start the transition to an external KMS, backup the wallet that contains all of the internal keys.
2. Turn off encryption on all categories and force decryption of all encrypted forests by issuing a merge command.
3. Ensure that all data is un-encrypted, forest status reports encryption size.

   Encrypted read-only forests will need to be set to updates-allow all and merge or they will be inaccessible.

4. Set the configuration back to the internal PKCS #11 KMS and rotate the key encryption keys. See Key Rotation for more information.
5. Re-index or force a merge of the database to re-encrypt your data.

   Moving from an external KMS to the internal KMS will downgrade your overall security, as the external KMS is more secure than the internal PKCS #11 secured wallet.

Backup and Restore

Individual backup files are encrypted with the cluster data encryption key (CDKEK). Backups are forest driven, so data from an encrypted forest will also be encrypted in backups. Configuration files included in a backup will be encrypted if the cluster is enabled for configuration file encryption. This encryption works with full backups, incremental backups, and journal archiving.

If any forest in the backup has encryption enabled, then the entire backup will be encrypted.

The encryption keys residing in the PKCS #11 secured wallet (the embedded KMS) will be exported as part of a full backup by default. This is true whether encryption is configured to use the internal KMS or an external KMS. Full backups will include this exported copy of the keystore, encrypted using the embedded KMS passphrase, unless you specify otherwise. See Excluding the Embedded KMS from a Backup.

If you cannot access your PKCS #11 secured wallet (or external KMS if you are using one), or lose your encryption keys, you will not be able to decrypt any of your encrypted data (including backups). There is no workaround to recover the encrypted data. We recommend that you backup your encryption keys in a secure location.

The built-in function admin:cluster-set-keystore-passphrase can be used to change the KMS passphrase. When you first set up encryption, we strongly recommend that you change the KMS passphrase to something other than the admin passphrase. This is to ensure that you utilize the Separation of Duties security principle as much as possible.

By default the keystore passphrase is automatically set to the admin password. We strongly recommend that you set a new, different passphrase before turning on encryption.

During an internal keystore backup/restore, data is added to the embedded PKCS #11 secured wallet; no keys are deleted. The encrypted file containing the keys is named kms.exp. The exported keystore is not imported during a restore from a backup. If you need to restore the keys, use the xdmp:keystore-import function. The keystore passphrase will be required to decrypt the exported keystore file when restoring backups on another MarkLogic instance.

To change the keystore passphrase, the current password or passphrase is required.

To restore an encrypted backup to the same cluster:

1. Restore the backup as usual. See Backing Up and Restoring a Database in the Administrator's Guide for details.

To restore an encrypted backup to a different cluster:

1. Use the xdmp:keystore-import function to import the keystore. The function requires the keystore passphrase of the cluster where the backup was created, to decrypt the keystore.

   xdmp:keystore-import("keystore passphrase", "/backup-directory/kms.exp")

   The import process will reject duplicate keys and log a warning that includes the ID of the rejected keys. Imported keys can only be used for decryption.

2. Restore the backup as usual. See Backing Up and Restoring a Database in the Administrator's Guide for details.

   As long as the current database being restored is encrypted, the restored database will also be encrypted.

Using this process you can move your encrypted backups from one system to another and restore them, as long as you have the passphrase and import the keystore into the new system before restoring the backup. See Backup and Restore Overview in the Administrator's Guide for more information about backup and restore procedures.

If you lose the cluster configuration information, you must first manually restore the keystore before an encrypted backup can be restored.

To export your keystore, use the xdmp:keystore-export function.

xdmp:keystore-export("strong passphrase", "/backups/MarkLogic.wallet.bak")

This function exports all of the encryption keys stored in the MarkLogic embedded KMS (the PKCS #11 secured wallet) and stores them at the location provided to the function.

xquery version "1.0-ml"; 
import module namespace admin = "http://marklogic.com/xdmp/admin" 
  at "/MarkLogic/admin.xqy";

let $option := "exclude"
let $config := admin:get-configuration()
return 
  admin:cluster-set-keystore-backup-option($config,$option)

xdmp:database-backup(xdmp:database-forests(xdmp:database("Documents")),"/backups/Data", fn:true(), 
"/backups/JournalArchiving", 15,"bf44aab-3f7a-41d2-a6a5-fc41a0e5e0cf")

xdmp.databaseBackup(xdmp.databaseForests(xdmp.database("Documents")), "/backups/Data", fn:true(),
"/backups/JournalArchiving", 15,"bf44aab-3f7a-41d2-a6a5-fc41a0e5e0cf");

Tool to View Encrypted Log Files Outside of the Server

MarkLogic encryption at rest includes the mlecat command line tool, which can be used to view encrypted log files outside of the server. The mlecat tool can be used successfully in either of these conditions:

• If the mlecat tool is given access to the MarkLogic data directory and the .pem files.
• If the log files are encrypted with a user-specified logs passphrase and the same logs passphrase is passed to mlecat with -p option.

  The mlecat tool should be run by a user with sufficient OS privileges to access the PKCS#11 wallet (located by default at /var/opt/MarkLogic). It is suggested that the user be a member of group running MarkLogic (by default daemon).

If you want to decrypt log files without having access to your KMS, you must set a logs-encryption-passphrase. To set this passphrase, use the admin:cluster-set-keystore-logs-encryption-passphrase function. For example:

xquery version "1.0-ml"; 
import module namespace admin = "http://marklogic.com/xdmp/admin" 
  at "/MarkLogic/admin.xqy";

let $config := admin:get-configuration()
let $passphrase := "dazzling zebras"
let $config := admin:cluster-set-keystore-logs-encryption-passphrase
   ($config,$passphrase)
return admin:save-configuration($config) 

Log file encryption must be enabled for this passphrase to be used.

For every OS you must add MARKLOGIC_INSTALL_DIR and MARKLOGIC_INSTALL_DIR/bin to your PATH. For example,

PATH=$MARKLOGIC_INSTALL_DIR:$MARKLOGIC_INSTALL_DIR/bin:$PATH

For more about setting environment variables on various platforms, see the information about installation and data directories as part of Installing MarkLogic in the Installation Guide.

To see the command line options for the mlecat tool, invoke mlecat (or mlecat.bat) with no arguments.

mlecat
==>
mlecat [option] filepath(s)
option:
  -i iDIR, iDir is MarkLogic's Install directory, alternatively the environmental variable
MARKLOGIC_INSTALL_DIR can be used to set this value.
  -d dDIR, dDIR is MarkLogic's Data directory, alternatively the environmental variable
MARKLOGIC_DATA_DIR can be used to set this value
  -p PASS, PASS is your logs-encryption-passphrase (if you are using one); 
  [-f] filepath(s), one or more file paths (-f can be specified before each file for explicit file list)

For example:

mlecat -p admin /var/opt/MarkLogic/Logs/ErrorLog.txt

Defaults for the MarkLogic data and install directories are shown in the following

Windows users will use mlecat.bat, instead of mlecat.

For more about setting environment variables on various platforms, see the information about installation and data directories as part of Installing MarkLogic in the Installation Guide.

Disaster Recovery/Shared Disk Failover

Unless you have suffered a complete loss of your host, disaster recovery should work just fine with encryption at rest. See High Availability and Disaster Recovery in the Concepts Guide for information about setting up shared disk failover and steps for disaster recovery.

If you have experienced a complete loss of your host, you will need to do the following:

1. Reinstall and configure a new MarkLogic host.
2. Import the keystore and keys from a backup (using xdmp:keystore-import). See Export and Import Encryption Keys for details.
3. Perform a restore from backup as usual. See Backing Up and Restoring a Database in the Administrator's Guide for more information.

Built-ins for Encryption at Rest

These functions will work with both the internal PKCS #11 secured wallet, or a external KMIP-compliant keystore. Using these functions you can encrypt data and check the status of encryption in your clusters using either JavaScript or XQuery.

The Server-Side JavaScript built-ins are:

• xdmp.keystoreExport
• xdmp.keystoreImport
• xdmp.filesystemFileEncryptionStatus
• xdmp.databaseEncryptionAtRest
• xdmp.databaseEncryptionKeyId
• xdmp.keystoreValidateExported

The Server-Side XQuery built-ins are:

• xdmp:keystore-export
• xdmp:keystore-import
• xdmp:filesystem-file-encryption-status
• xdmp:database-encryption-at-rest
• xdmp:database-encryption-key-id
• xdmp:keystore-validate-exported

<xs:complexType name="options">
    <xs:sequence>
      <xs:element ref="timeout" minOccurs="0"/>
      <xs:element ref="data" minOccurs="0"/>
      <xs:element ref="headers" minOccurs="0"/>
      <xs:element ref="credential-id" minOccurs="0"/>
      <xs:element ref="authentication" minOccurs="0"/>
      <xs:element ref="client-cert" minOccurs="0"/>
      <xs:element ref="client-key" minOccurs="0"/>
      <xs:element ref="pass-phrase" minOccurs="0"/>
    </xs:sequence>
  </xs:complexType>

Admin APIs for Encryption at Rest

These functions are used to set the mode and descriptions for the host, set the keystore host name and the keystore host port. You can also set the keystore data key ID, config key ID, or logs key ID, along with setting the keystore serve certificate and enabling encryption.

These server-side XQuery functions work with either the PKCS #11 secured wallet or a third-party KMIP-compliant keystore:

• admin:cluster-get-config-encryption
• admin:cluster-get-data-encryption
• admin:cluster-get-logs-encryption
• admin:cluster-set-config-encryption
• admin:cluster-set-data-encryption
• admin:cluster-set-logs-encryption
• admin:database-get-data-encryption
• admin:database-set-data-encryption
• admin:cluster-set-keystore-passphrase
• admin:cluster-set-keystore-logs-encryption-passphrase
• admin:cluster-get-keystore-backup-option
• admin:cluster-get-keystore-wallet-location

The admin:cluster-rotate-xxxx-encryption-key-id APIs are only for use with the embedded KMS provided by MarkLogic (the PKCS #11 secured wallet). Using these functions with an external KMS will cause an error.

• admin:cluster-rotate-config-encryption-key-id
• admin:cluster-rotate-data-encryption-key-id
• admin:cluster-rotate-logs-encryption-key-id
• admin:group-get-rotate-audit-files
• admin:group-get-rotate-log-files
• admin:group-set-rotate-audit-files
• admin:group-set-rotate-log-files

These next two APIs are used in transitioning from an internal keystore (the PKCS #11 secured wallet) to an external KMIP-compliant keystore. If these functions are set to external, MarkLogic Server will first look for the external keystore to verify the keys.

• admin:cluster-set-keystore-kms-type
• admin:cluster-get-keystore-kms-type

These functions are designed to work with a external KMIP-compliant keystore:

• admin:cluster-get-config-encryption-key-id
• admin:cluster-set-config-encryption-key-id
• admin:cluster-get-data-encryption-key-id
• admin:cluster-set-data-encryption-key-id
• admin:cluster-get-keystore-host-name
• admin:cluster-set-keystore-host-name
• admin:cluster-get-keystore-port
• admin:cluster-set-keystore-port
• admin:cluster-get-logs-encryption-key-id
• admin:cluster-set-logs-encryption-key-id
• admin:cluster-get-keystore-kmip-CA-path
• admin:cluster-set-keystore-kmip-CA-path
• admin:cluster-get-keystore-kmip-certificate-path
• admin:cluster-set-keystore-kmip-certificate-path
• admin:cluster-get-keystore-kmip-key-path
• admin:cluster-set-keystore-kmip-key-path
• admin:database-get-encryption-key-id
• admin:database-set-encryption-key-id

  The functions designed to work with a external KMS will return an error if you try to use them with the PKCS #11 secured wallet (the default built-in KMS).

REST Management APIs for Encryption

You can manage encryption using the REST Management APIs. Some of the tasks you can do with these APIs include:

• Encryption configuration
• Keystore configuration
• Database configuration
• Database status, including database encryption (encrypted size, total size)
• Cluster status
• Forest status
• Security
• Backups, status (encrypted or not)
• Restore (with property for using private key)

The REST Management APIs that are used to query and manage the cluster security properties include encryption information for database, cluster, and forest.

Below is a XML payload example for the security endpoint:

<security-properties xmlns="http://marklogic.com/manage/security/properties"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://marklogic.com/manage/security/properties
manage-security-properties.xsd"> 
<keystore> 
<data-encryption>default-off</data-encryption>
<config-encryption>off</config-encryption>
<logs-encryption>off</logs-encryption>
<kms-type>internal</kms-type>
<host-name>localhost</host-name>
<port>9056</port>
<data-encryption-key-id>92ed7360-458a-427e-abad-c6595b192cb7</data-encryption-key-id>
<config-encryption-key-id>8b9a9bdb-7b0e-41eb-9aa6-ed6e8cb23ad5</config-encryption-key-id>
<logs-encryption-key-id>01c50d02-b43f-46bc-bbe5-6d4111d1180b</logs-encryption-key-id>
</keystore>
</security-properties>

And here is a JSON payload example for the security endpoint:

{
  "keystore": {
    "data-encryption": "default-off",
    "config-encryption": "off",
    "logs-encryption": "off",
    "kms-type": "internal",
    "host-name": "localhost",
    "port": 9056,
    "data-encryption-key-id": 
      "92ed7360-458a-427e-abad-c6595b192cb7",
    "config-encryption-key-id": 
      "8b9a9bdb-7b0e-41eb-9aa6-ed6e8cb23ad5",
    "logs-encryption-key-id": 
      "01c50d02-b43f-46bc-bbe5-6d4111d1180b"
  }
}

These operations are available for encryption key rotation:

curl -v -X POST --anyauth --user admin:admin \
  --header "Content-Type:application/json" -d \
  '{"operation":"rotate-config-encryption-key"}' \
  http://localhost:8002/manage/v2/security
curl -v -X POST --anyauth --user admin:admin \
  --header "Content-Type:application/json" -d \
  '{"operation":"rotate-data-encryption-key"}' \
  http://localhost:8002/manage/v2/security
curl -v -X POST --anyauth --user admin:admin \
  --header "Content-Type:application/json" -d \
  '{"operation":"rotate-logs-encryption-key"}' \
  http://localhost:8002/manage/v2/security

Rolling Upgrades

Encryption at rest is a feature introduced in MarkLogic 9. Clusters running older versions need to be completely upgraded to MarkLogic 9 before using this feature. See Rolling Upgrades in the Administrator's Guide for more about rolling upgrades.

During upgrades, the default passphrase for the upgraded system is not set. You will need to reset the default passphrase after an upgrade.

Telemetry

The telemetry feature is not available for use until the cluster is upgraded to MarkLogic 9.0-1 or later. See Telemetry in the Monitoring MarkLogic Guide for more about telemetry.