Encryption at rest protects your data on media - which is data at rest as opposed to data moving across a communications channel, otherwise known as data in motion. Increasing security risks and compliance requirements sometimes mandate the use of encryption at rest to prevent unauthorized access to data on disk.
No additional license is required to use encryption at rest with the built-in internal keystore. To use encryption at rest with an external key management system (KMS), an Advanced Security license is required. For details on purchasing an Advanced Security license, contact your MarkLogic sales representative. See Licensing for more information.
Encryption at rest can be configured to encrypt data, log files, and configuration files separately. Encryption is only applied to newly created files once encryption at rest is enabled, and does not apply to existing files without further action by the user. For existing data, a merge or re-index will trigger encryption of data, a configuration change will trigger encryption of configuration files, and log rotation will initiate log encryption.
This chapter describes encryption at rest security and includes the following sections:
The use of an external Key Management System (KMS) or keystore with encryption at rest requires an Advanced Security License, in addition to the regular license. See Licensing for more details.
The following terms and definitions are associated with encryption at rest.
Term | Definition |
---|---|
Encryption at rest | Encryption of data that is stored on digital media |
KMS | Key Management System |
wallet | The PKCS #11 secured wallet provided and managed by MarkLogic, that functions as the default standalone KMS |
KEK | A Key Encryption Key used to encrypt or 'wrap' another encryption key |
keystore | Repository for crytographic keys in the PKCS #11 secured wallet or any external KMS that is KMIP-server conformant |
KMIP | Key Management Interoperability Protocol (KMIP specification) - governed by OASIS standards body. There are multiple versions of KMIP currently available. MarkLogic Encryption supports KMIP version 1.2 |
PKCS #11 | One of the Public-Key Cryptography Standards, and also the programming interface to create and manipulate cryptographic tokens. See the OASIS PKCS TC for details |
MKEK | Master Key Encryption Key, resides in the keystore, and is used to generate the CKEK, which is enveloped (encrypted) with the MKEK |
CKEK | Cluster Key Encryption Key, resides in the keystore and is used to encrypt the data (CDKEK), configuration(CCKEK), and log CLKEK) encryption keys |
CDKEK | Cluster Data Key Encryption Key, used to directly encrypt (wrap) the object key encryption keys (OKEY) for stands, forest journals, and large files |
CCKEK | Cluster Configuration Key Encryption Key, used to encrypt (wrap) the object key encryption keys (OKEY) for configuration files |
CLKEK | Cluster Log Key Encryption Key, used to encrypt (wrap) the object key encryption keys (OKEY) for log files |
OKEY | Object Encryption Key, otherwise known as the data object encryption key, a symmetric key used to directly encrypt objects like stands, forest journals, large files, configuration files, or log files |
BKEK | Backup Key Encryption Key, used to encrypt backups, both full and incremental. The BKEK is a locally generated backup KEK, that is used to encrypt all files in the backup. The BKEK is encrypted with the CDKEY and the BDKEY. |
BDKEK | Backup Database Key, (alternative) only applicable to external KMS configurations. It is used to encrypt a backup in addition to the CDKEK. |
HSM | Hardware Security Module or other hardware device is a physical computing device that safeguards and manages digital key materials |
Key strength | The size of key in bits. Usually the more bits, the stronger the key and more difficult to break; for example 128-bits, 256 bits, or 512-bits, and so on |
Key rotation | The process of aging out and replacing encryption keys over time |
Encryption at rest enables you to transparently and selectively encrypt your data residing on disk (locally or in the cloud) in MarkLogic clusters. You can set your options at the cluster level to encrypt data on all the hosts in that cluster.
Three types of data can be encrypted:
There are both MarkLogic Application Server logs and MarkLogic Server logs; both types of logs will be encrypted as part of log encryption.
If you are using the Default Conversion Option described in The Default Conversion Option in the Content Processing Framework Guide, note that the MarkLogic Converters package may generate temporary files, which are not supported by encryption at rest.
These types of data can each be encrypted separately. You can configure encryption for databases individually, or at the cluster level. Encryption at rest is off by default. To use encryption at rest, you need to configure and enable encryption for your database(s), configuration files, and/or log files.
To access unencrypted forest data MarkLogic normally uses memory-mapped files. When files are encrypted, MarkLogic instead decrypts them to anonymous memory. As a result, encrypted MarkLogic forests use more anonymous memory and less file-mapped memory than unencrypted forests.
Encryption at rest provides data confidentiality, but not authentication of identity or access control (permissions). See Authenticating Users and Protecting Documents for information about authentication and other forms of security in MarkLogic Server.
If you cannot access your PKCS #11 secured wallet (or external KMS if you are using one), or lose your encryption keys, you will not be able to decrypt any of your encrypted data. There is no mechanism to recover the encrypted data. We recommend that you backup your encryption keys in a secure location. See Backup and Restore for more details.
A keystore is a secure location where the actual encryption keys used to encrypt data are stored. The keystore for encryption at rest is a key management system (KMS). This keystore can be either the MarkLogic embedded PKCS #11 secured wallet, an external KMS that conforms to the KMIP-standard interface, or the native AWS KMS (Amazon Web Services Key Management System). The embedded keystore is installed by default when you install MarkLogic 9.0-x or later.
The MarkLogic embedded wallet uses a standard PKCS #11 protocol, using the PKCS #11 APIs. The wallet or another KMS, must be available during the MarkLogic startup process (or be bootstrapped from MarkLogic during start-up). You can also use any KMIP-compliant external keystore with MarkLogic or the native AWS KMS.
To configure an external KMS you will need the following information for your cluster:
If you are using the native AWS KMS, you will not need the Client certificate or the Server certificate. You will need the other information.
If you plan to use an external key management system, configure the external KMS first, and then turn on encryption in the MarkLogic server.
For details, see Configuring an External Keystore.
The following section provides an overview of the encryption key hierarchy used by MarkLogic encryption at rest to secure data. Keys in the encryption hierarchy wrap (or encrypt) those keys below them in the hierarchy. Three possible configurations of the encryption key hierarchy are described. The first is an idealized key hierarchy that provides a generic example. The second is an embedded KMS (the PKCS #11 secured wallet) configuration, and the third shows an external keystore management system (KMS) configuration.
You do not need to completely understand the details of the key hierarchy to use the encryption feature, but this section will help to understand the general concepts involved.
The keystore contains the Master Key Encryption Key (MKEK). The keystore generates the Cluster Key Encryption Key (CKEK), which is enveloped (encrypted) with or derived from the Master Key Encryption Key. Both the Master Key Encryption Key and the Cluster Key Encryption Key reside in the keystore (key management system or KMS). These keys never leave the keystore and MarkLogic Server has no knowledge or control over these keys. The keys are referenced from the keystore by their key IDs.
The KMS can be either the internal keystore provided by MarkLogic or an external KMIP-compliant KMS; the same mechanism is used by both types of keystores. The configuration happens at the cluster level because there is one keystore configuration per cluster. The encryption feature is fully compliant with the KMIP standard and the Amazon KMS.
The external KMS provides even higher security. The key IDs are provided by the KMS and returned through a TLS tunnel after the MarkLogic-generated keys have been sent to the KMS and wrapped (encrypted). The actual encryption keys never leave the KMS.
There are multiple levels to the key hierarchy, each level wrapping (encrypting) the level below it. The KMS generates the Cluster Level Data Encryption Keys for data (CDKEK), configuration files (CCKEK), and log files (CLKEK). The corresponding key (CDKEK, CCKEK, or CLKEY) is used to encrypt (wrap) all the Object Encryption Keys (OKEY) generated by MarkLogic Server for each file, so that an encryption key protects each file, no matter what category (data, configuration files, logs).
The Object Encryption Keys (OKEY) are randomly generated per file (for stands, journals, config files, and log files, etc.) wrapped (encrypted) with the corresponding keys (CDKEK, CCKEK, or CLKEK). So an encryption key protects each file within a category (data, configuration files, logs).
For example, the Master Key Encryption Key (MKEK) wraps (encrypts) the Cluster Key Encryption Keys (CKEK), which in turn wraps (encrypts) the Data Key Encryption Key (CDKEK). The Data Key Encryption Key encrypts the Object Encryption Key (OKEY) for a file such as a stand. The keys at the bottom of the diagram are encrypted as headers in each file, wrapped (encrypted) with each of the keys above them in the hierarchy. Each of the three categories of objects (data, configuration files, and logs) has its own key encryption hierarchy.
Database backups are encrypted using a generated backup key (BKEK). This key is then encrypted with the cluster key (CDKEK). See Backup and Restore for more information about backups.
When you use the embedded PKCS #11 secured wallet provided with MarkLogic Server, the recommended key hierarchy would be similar to this illustration.
MarkLogic Server generates the Data Key Encryption Key (CDKEK), the Configuration Key Encryption key (CCKEK) and the Logs Key Encryption Key (CLKEK). The Data Key Encryption Key is then used to wrap the OKEYs for the database objects (journals, data files, etc.). These keys are stored in the wallet (internal KMS). The key IDs are generated in the MarkLogic Server for encryption and decryption by the KMS (the PKCS #11 secured wallet in this case). The configuration happens at the cluster level because there is one keystore per cluster.
The individual Object Encryption Keys (OKEYs) are then randomly generated and used to directly encrypt individual files (journals, config files, and log files, etc.). These keys (the OKEYs) are wrapped (encrypted) with the corresponding KEK for data, config, and logs. A unique key protects (encrypts) each file. The keys at the object levels are wrapped (encrypted by the keys above them) for each category.
For example, the Data Key Encryption Key (CDKEK) wraps (encrypts) the Object Encryption Key (OKEY) for a file such as a journal. The keys at the bottom of the diagram are encrypted (wrapped) by all the keys above them in the hierarchy, and then placed in the header for each file. In the case of the embedded KMS, there is only one CDKEK for the entire cluster - all databases in the cluster will use that key. When using the embedded KMS, it is not possible to use per database keys for encryption.
Database backups are encrypted using the locally generated backup key (BKEK) that is used to encrypt all of the files in the backup. The BKEK is then encrypted with the cluster data key (CDKEK) and then encrypted with the cluster key (CKEK). Additionally you could encrypt this key with the BDKEY and a passphrase. See Backup and Restore for more information about backups.
The external KMS provides even higher security, along with additional key management features. When you use an external key management system (KMS or keystore), the recommended key hierarchy deployment might look like this illustration.
The keystore contains the Master Key Encryption Key (MKEK). The KMS generates or derives the Cluster Key Encryption Key (CKEK), which is enveloped (encrypted) with the Master Key Encryption Key. Both the Master Key Encryption Key and the Cluster Key Encryption Key reside in the KMS keystore. These keys never leave the keystore. MarkLogic Server has no knowledge or control over these keys. The keys are referenced from the keystore by their key IDs. The actual encryption keys never leave the KMS.
There are multiple levels to the key hierarchy in this deployment, each level wrapping (encrypting) the level below it. The KMS generates the cluster level encryption keys for data (CDKEK), configuration files (CCKEK), and log files (CLKEK). The corresponding KEK is used is used to encrypt (wrap) all the Object Encryption Keys (OKEY) generated by MarkLogic Server for each file, so that a unique key protects each file, no matter what category (data, configuration files, logs). A unique key protects each file within a category (data, configuration files, logs).
The corresponding KEK (for data, config, or logs) is used to encrypt (wrap) all the Object Encryption Keys (OKEY) generated by MarkLogic Server for each file, so that an encryption key protects each file, no matter what category (data, configuration files, logs).
For example, the Master Key Encryption Key (MKEK) wraps (encrypts) the Cluster Key Encryption Keys (CKEK), which in turn wraps (encrypts) the Data Key Encryption Key (CDKEK), then wraps (encrypts) the Object Encryption Key (OKEY) for a file such as a stand. The keys at the bottom of the diagram are encrypted (wrapped) by all the keys above them in the hierarchy, and then placed in the header for each file.
Database backups are encrypted using the BKEK, the locally generated backup KEK, the BKEK is encrypted with the CDKEK. Then the CDKEY may be encrypted or derived from the cluster key (CKEK). This last step is outside of the control of MarkLogic. You can also use a password or passphrase to encrypt and secure your backup. See Backup and Restore for more information about backups and the use of a passphrase to secure your backup.
If you plan to use an external key management system, configure the external KMS first, and then turn on encryption in the MarkLogic server.
This section describes a scenario using encryption at rest to encrypt a database. This example is for informational purposes only. It is not meant to demonstrate the correct way to set up and use encryption at rest, as your situation is likely to be unique. However, it demonstrates how encryption at rest works and may give you ideas for how to configure your own encryption at rest security model.
Description:To set up encryption at rest for this scenario, you will need Admin privileges. You will need access to both MarkLogic Admin Interface and Query Console.
To run through the example, perform the steps in each of the following sections:
Install MarkLogic 9.0-1 or later. Encryption at rest options are not available in earlier versions of MarkLogic Server. You must explicitly select which data (databases, configuration files, log files, or entire clusters) you want to have encrypted. This example shows how to set up encryption for a single database.
The Security database or other databases used by MarkLogic will not be encrypted by default. Existing data can be encrypted by forcing a merge or a reindex of the database.
See Configuring Encryption at Rest for more details.
For this example, we will use the Admin UI to set up encryption for the Documents database.
on
from the drop-down menu. (The other options are default-cluster
and off
.)If you select default-cluster
, encryption for that database will default to whatever encryption option has been set for the cluster as a whole. If the cluster is set to encrypt data, this database will be encrypted. If encryption has not been turned on for the cluster, this database will not be encrypted if default-cluster
is selected. See Cluster Encryption Options for details.
As you access data in your database, it will be encrypted when it is written back to disk. You can view the encryption progress on the Database Status page by looking at the Size and Encrypted Size numbers.
To encrypt the existing data in your database, you will need to re-index your database. On the Database Configuration page, click the reindex button at the top of the page (below the OK button), and then click ok. You can also force a merge of the database to encrypt the data.
Encryption of large databases will take some time initially. Updates and changes to the database will be fairly transparent to the user after initial encryption. The Size and Encrypted Size numbers will be equal when the encryption process is complete.
Using Query Console, you can run a simple query to verify that the Documents database has encryption turned on.
You can also check the Size and Encrypted Size numbers on the Database Status page. These numbers will be equal when the encryption process is complete and the entire database is encrypted.
off
from the drop-down menu. To verify that encryption is turned off, run this query in Query Console:
To decrypt the existing data in your database, you will need to re-index your database. On the Database Configuration page, click the reindex button and then click ok.
You can also decrypt the data by forcing a merge on the database to decrypt its contents. This process may take a while.
Install MarkLogic Server version 9.0-x or later. The encryption at rest feature and the PKCS #11 secured wallet are installed by default. You can configure encryption at rest for databases (data encryption), log files (log encryption) and configuration files (config encryption). The encryption feature will need to be configured and enabled for your data to be encrypted.
When you start up MarkLogic for the first time after installation, the keystore.xml
file will be loaded first. It contains the encryption key IDs. After loading the keystore.xml
configuration, MarkLogic validates connectivity to the KMS (local or external) and the validity of the keys stored in keystore.xml
. Once validated, encryption keys will be loaded and decrypted. Normal startup then continues. If configuration files are encrypted, the file layer will decrypt them as they are being loaded, making the encryption transparent to the cluster.
If a node in your cluster is offline for any reason, wait until the host comes back online to make any changes to your encryption at rest settings. Do not change your encryption settings while a host is offline.
This section contains the following topics:
You can configure encryption for each database on the Database Configuration page in the Admin UI. Encryption at rest can be separately enabled per database, or at the cluster level by setting the database encryption to default to the cluster encryption settings. The encryption options for databases are shown in the following table.
Encryption | Encryption Options | ||
---|---|---|---|
Default-Cluster | On | Off | |
Database encryption | encryption defaults to cluster setting | encryption enabled for database |
With encryption enabled, files are encrypted as they are ingested into the database, or when those files are written back to disk. If you want to encrypt existing data in a database either reindex the database or force a merge on the database. This will take a few minutes depending on the size of database. See Cluster Encryption Options
Large binary files are only encrypted during initial ingestion into the database. If you want to encrypt existing large binary files already loaded into MarkLogic Server prior to turning on encryption, you must reindex the database or force a merge.
on
from the drop-down menu. (The other options are default-cluster
and off
.)You can set cluster encryption options for configuration files and log files, and also set or override the encryption options for databases on the Cluster Configuration page.
Encryption at rest for configuration files and/or log files is done on the Cluster Configuration page in the Admin UI. Navigate to this page by choosing Clusters from the left tree menu, clicking the cluster name, and then clicking the Configure tab.
The encryption options are shown in the following table.
File Type | Cluster Encryption Settings | ||
---|---|---|---|
Default On | Default Off | Force | |
Configuration files | encrypt | do not encrypt | encrypt |
Log files | encrypt | do not encrypt | encrypt |
The keystore.xml
and hsm.cfg
files are never be encrypted because they are configuration for the Keystore. The servers.xml
file is not immediately encrypted until a server (apps server) is updated, a new server is created, or an existing server is deleted. This is because these actions trigger a restart of the MarkLogic server.
Cluster configuration settings for encryption at rest interact with the encryption settings for databases. You can separately configure encryption for each database on the Database Configuration page in the Admin UI or set database encryption to default to the cluster encryption settings.
The database encryption configuration settings take precedence unless the cluster Force Encryption option is set. If Force Encryption is on, configuration files and log files will be encrypted. Please check all database encryption settings to ensure that they are set correctly.
The following table shows the interaction between the cluster configuration options and the database configuration options. There are three possible database encryption settings and three possible cluster encryption settings. The cell where the row and column intersect shows the outcome of that configuration combination.
The Force Encryption option in the Cluster Encryption Settings will force encryption for all of the databases in the cluster. If the Cluster Encryption Setting is Force Encryption (or Default On), or the Database Encryption Setting is On, then the database will be encrypted.
You can either configure encryption for the embedded keystore (the PKCS #11 secured wallet) or for a external KMIP-compliant keystore using the Admin UI. Use the Edit Keystore Configuration page to configure encryption at rest for a cluster. Using this page you can configure data encryption, configuration file encryption, encryption of log files, or key synchronization.
Beneath these options on the Edit Keystore Configuration page, there are two tabs for specifying further options for either the Internal KMS or the External KMS. The following options are available for the Internal KMS:
Use the Edit Keystore Configuration page to configure encryption at rest for a cluster using an external keystore. Using this page you can configure data encryption, configuration file encryption, encryption of log files, or key synchronization.
Beneath these options on the Edit Keystore Configuration page, there are two tabs for specifying further options for either the Internal KMS or the External KMS. The following options are available for the Internal KMS:
Adding or changing any encryption information will require a restart of all of the hosts in the cluster.
You can change the password for the internal KMS using the Change Internal KMS Password screen. To change the internal KMS password do the following:
Synchronize the KMS keys with the enveloped keys on MarkLogic Server to ensure that MarkLogic uses the most current keys for encryption.
To synchronize the KMS keys, do the following:
The MarkLogic wallet (the embedded KMS) uses SoftHSM as its default hardware security module (HSM). Customers wishing to use another PKCS #11-compliant key storage device must follow these steps before starting MarkLogic with encryption for the first time.
MARKLOGIC_P11_DRIVER_PATH
(locally or in /etc/marklogic.conf
) to point to the library that you want to use.Follow these steps to change your KMS wallet location, or change the location of the backup for your KMS:
Use the options available in admin:cluster-set-keystore-wallet-location
(XQuery) or admin.clusterSetKeystoreWalletLocation
(SJS)
to change the location of the internal wallet.
The admin:cluster-set-keystore-wallet-location
(XQuery) or admin.clusterSetKeystoreWalletLocation
(SJS) function will also set the backup location for an embedded KMS.
MarkLogic does not support having the wallet location being on some shared NFS location at this time.
Instead of using the Admin UI, you can configure encryption for your MarkLogic instance using XQuery.
In Query Console, you can use admin:cluster-set-data-encryption (XQuery) or
admin.clusterSetDataEncryption (SJS) to turn on data encryption for the current database:
For example, to set the encryption for log files at cluster level:
To see whether encryption is turned on for log files, you can run this XQuery in the Query Console:
You can use REST Management APIs to work with encryption at rest.
GET /manage/v2/databases/{id|name}/properties
This command gets the current properties of the Documents database, including the encryption status and encryption key ID in JSON format:
$ curl -GET --anyauth -u admin:admin \ -H "Accept:application/json,Content-Type:application/json" \ http://localhost:8002/manage/v2/databases/Documents/properties
{"database-name":"Documents", "forest":["Documents"], "security-database":"Security", "schema-database":"Schemas", "triggers-database":"Triggers", "enabled":true, "data-encryption":"off", "encryption-key-id":"",
The same command in XML format:
$ curl -GET --anyauth -u admin:admin \ -H "Accept:application/xml,Content-Type:application/xml" \ http://localhost:8002/manage/v2/databases/Documents/properties
<database-properties xmlns="http://marklogic.com/manage"> <database-name>Documents</database-name> <forests> <forest>Documents</forest> </forests> <security-database>Security</security-database> <schema-database>Schemas</schema-database> <triggers-database>Triggers</triggers-database> <enabled>true</enabled> <data-encryption>on</data-encryption> <encryption-key-id/> ... </database-properties>
GET /manage/v2/security/properties
This command returns the current encryption status, along with other properties including encryption key ID, for localhost in JSON format:
$ curl -GET --anyauth -u admin:admin \ -H "Accept:application/json,Content-Type:application/json" \ http://localhost:8002/manage/v2/security/properties
{"keystore":{"data-encryption":"default-off", "data-encryption-key-id":"091fd9a0-f090-4c7e-91ca-fedfe21dbfef", "config-encryption":"off", "config-encryption-key-id":"", "logs-encryption":"off", "logs-encryption-key-id":"", "host-name":"LOCALHOST", "port":9056}}
Here is the same version of the command, this time returning XML:
$ curl -GET --anyauth -u admin:admin \ -H "Accept:application/xml,Content-Type:application/xml" \ http://localhost:8002/manage/v2/security/properties
<security-properties xsi:schemaLocation="http://marklogic.com/manage/security/properties manage-security-properties.xsd" xmlns="http://marklogic.com/manage/security/properties" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> <keystore> <data-encryption>default-off</data-encryption> <data-encryption-key-id>8d0b07d8-b655-4408-affd-e49a2ece0af3 </data-encryption-key-id> <config-encryption>off</config-encryption> <config-encryption-key-id/> <logs-encryption>off</logs-encryption> <logs-encryption-key-id/> <host-name>LOCALHOST</host-name> <port>9056</port> </keystore> </security-properties>
POST:/manage/v2/security/properties
This command sets the protected path for //d
with read
permissions for manage-user
:
$ curl -POST --anyauth -u admin:admin \ -d @file.xml -H "Content-Type:application/xml" \ http://localhost:8002/manage/v2/protected-paths
Here is the payload (file.xml):
<protected-path-properties xmlns="http://marklogic.com/manage/protected-path/properties"> <path-expression>//d</path-expression> <path-namspaces/> <permissions> <permission> <role-name>manage-user</role-name> <capability>read</capability> </permission> </permissions> </protected-path-properties>
Here is the same operation in JSON:
curl -X POST --anyauth -u admin:admin \ -d @file.json -H "Content-Type:application/json" \ http://localhost:8002/manage/v2/protected-paths
Here is the payload (file.json):
{ "path-expression": "//e", "path-namespace": [], "permission": [{ "role-name": ["manage-user"], "capability": "read" }] }
PUT /manage/v2/databases/{id|name}/properties
This command will turn on encryption for the Documents database:
$ curl -X PUT --anyauth -u admin:admin -d '{"data-encryption":"on"}' \ -H "Content-Type:application/json" \ http://localhost:8002/manage/v2/databases/Documents/properties
To export the embedded KMS (the PKCS #11 secured wallet) using REST, you can use this form with REST and XML:
POST manage/v2/security? operation=export-wallet&filename=/my/test..wallet&password=test
As a curl command (using MANAGEADMIN="admin" and MANAGEPASS="admin") it would look like this:
curl -v -X POST --anyauth --user $MANAGEADMIN:$MANAGEPASS \ --header "Content-Type:application/xml" \ -d@data/security/export-wallet.xml \ http://$host:8002/manage/v2/security
<export-wallet-operation xmlns="http://marklogic.com/manage/security"> <operation>export-wallet</operation> <filename>/tmp/mywallet.txt</filename> <password>mypassword</nassword> </export-wallet-operation>
Or you can use this form for REST using a JSON format:
POST manage/v2/security {"operation":"export-wallet","filename":"/my/test.wallet","password":"test"}
As a curl command (using MANAGEADMIN="admin" and MANAGEPASS="admin") it would look like this:
curl -v -X POST --anyauth --user $MANAGEADMIN:$MANAGEPASS \ --header "Content-Type:application/json" \ -d@data/security/export-wallet.json \ http://$host:8002/manage/v2/security
{ "operation":"export-wallet", "filename":"/tmp/mywallet.tmp", "password":"mypassword" }
The export wallet operation saves the wallet to a directory on the server on which MarkLogic is running. Similarly, the import wallet operation imports from the filesystem on which MarkLogic is running.
To import the embedded KMS (the PKCS #11 secured wallet) using REST, you can use this form with a JSON payload:
POST manage/v2/security {"operation":"import-wallet","filename":"/my/test.wallet","password":"test"}
As a curl command (using MANAGEADMIN="admin" and MANAGEPASS="admin") it would look like this:
curl -v -X POST --anyauth --user $MANAGEADMIN:$MANAGEPASS \ --header "Content-Type:application/json" \ -d@data/security/import-wallet.json \ http://$host:8002/manage/v2/security
{ "operation":"import-wallet", "filename":"/tmp/mywallet.tmp", "password":"mypassword" }
Or you can use this form with an XML payload:
POST manage/v2/security? operation=import-wallet&filename=/my/test.wallet&password=test
As a curl command (using MANAGEADMIN="admin" and MANAGEPASS="admin") it would look like this:
curl -v -X POST --anyauth --user $MANAGEADMIN:$MANAGEPASS \ --header "Content-Type:application/xml" \ -d@data/security/import-wallet.xml \ http://$host:8002/manage/v2/security
<import-wallet-operation xmlns="http://marklogic.com/manage/security"> <operation>import-wallet</operation> <filename>/tmp/mywallet.txt</filename> <password>mypassword</password> </import-wallet-operation>
MarkLogic will only import keys generated by the embedded MarkLogic KMS.
Encryption key management for the embedded KMS (the PKCS #11 secured wallet) is handled automatically by MarkLogic. Keys are never purged from the wallet, which is encrypted by a MarkLogic-generated key activated by a passphrase. The administrator's password is used as the initial passphrase.
By default the keystore passphrase is set to the admin password. We strongly recommend that you set a new, different passphrase before turning on encryption. Using a separate passphrase for admin and the keystore helps support the strong security principle called Separation of Duties.
This passphrase can be changed using either the XQuery (xdmp:keystore-set-kms-passphrase
) or JavaScript (xdmp.keystoreSetKmsPassphrase
) built-ins. As part of key management, you may want to export, import, or rotate encryption keys. MarkLogic provides built-in functions for exporting and importing encryption keys, and manually rotating encryption keys. If you require additional key management functionality, you may want to consider an external key management system. See Configuring an External Keystore for more information.
If you believe that an encryption key has been compromised, you should force a merge or start a re-index of your data to change/update the encryption keys. See Key Rotation for more about updating encryption keys.
This section includes the following topics:
For the internal wallet, key encryption keys (KEK) can be manually rotated. Keys can be manually rotated at regular intervals or if an encryption key has been compromised. This type of key rotation can be triggered on individual encryption categories (configuration, data, logs) using MarkLogic built-in functions.
There are two steps to key rotation. First, rotating the KEK keys (using AES 256 symmetric encryption) used to envelope the object file encryption keys, and second, re-encrypting the object file encryption keys (also using AES 256 symmetric encryption).
After calling the built-in function to rotate encryption keys, all new data will be written to disk using the new key encryption key. Old data will be migrated as it is re-written to disk. If you wish to force re-encryption using the new key, you can either force a merge or re-index the forest.
At the cluster level, you can manually rotate the data keys, configuration keys, and the logs keys (CDKEK, CCKEK, CLKEK) using these APIs:
admin:cluster-rotate-config-encryption-key-id,
admin:cluster-rotate-data-encryption-key-id,
admin:cluster-rotate-logs-encryption-key-id
admin.clusterRotateDataEncryptionKeyId,
admin.clusterRotateLogsEncryptionKeyIdThese key rotation functions are only available for the MarkLogic internal KMS (the PKCS #11 secured wallet) and not for any keys that are managed by an external KMS.
When you are using an external KMS, MarkLogic does not have access to the envelope key, it only has access to the key ID, and asks for the KMS to open the envelope.
The intermediate fast rotation keys enable immediate envelope key rotation with a minimum of I/O. File level keys can be rotated at any time by forcing a merge. Log rotation and configuration file updates use new keys. Old logs, backups, and configuration files are not re-encrypted.
The internal KMS (the PKCS #11 secured wallet) follows these steps for fast key rotation:
An external KMS, follows these steps for fast key rotation:
Expired keys can be used for decryption, but not encryption. Expired keys may be needed for decrypting backups.
The ability to export and import key encryption keys (KEK) from the PKCS #11 secured wallet (the embedded KMS) is useful when you want to clone a cluster. Exporting a key encryption key (KEK) is restricted to cluster-level keys (CDKEK, CCKEK, CLKEK) and requires a passphrase and a filepath. The data will be exported (encrypted with the passphrase) into a file at the location specified by the filepath.
To export a keystore from the embedded KMS:
To import a keystore into the embedded KMS:
Key encryption keys can only be imported from MarkLogic exported files. Imported keys can only be used for decryption. The import requires the passphrase that was provided at the time of the export.
If a duplicate key ID is supplied during the import, the imported key ID will be ignored. Duplicate key IDs can be caused by importing the keystore twice.
An external key management system (KMS) or keystore offers additional security for your encryption keys, along with key management capabilities like automatic key rotation, key revocation, and key deletion. If you want the ability to perform these tasks, you will need an external KMS. MarkLogic Encryption at Rest supports KMIP 1.2 compliant KMS servers and Amazon's KMS.
The use of an external Key Management System (KMS) or keystore with encryption at rest, requires an Advanced Security License, in addition to the regular MarkLogic license.
When using an external KMS, usually there is a security administrator role separate from the MarkLogic administrator. The security administrator would be the role setting up and configuring the external keystore. The MarkLogic administrator can also perform this task, but for greater security it is recommended that the separate security administrator configure the KMS.
Having a separate security administrator follows an important security principle called Separation of Duties and is recommended by security experts.
This section covers setting up MarkLogic encryption for use with an external key management system from the MarkLogic Admin UI on the MarkLogic host. You don't need to have MarkLogic encryption turned on for your cluster while you are setting up and configuring the external key management system.
If you plan to use an external key management system, we recommend that you configure the external keystore first, and then turn on encryption in the MarkLogic server.
The installation process for the external keystore will vary depending on the type of external KMS you plan to use. A security administrator must configure the external keystore using the administration set up tools that come with the external KMS. This section provides a high-level overview of the process from the MarkLogic Server point of view.
Amazon Web Services (AWS) provides a key management system (KMS) that you can use with MarkLogic encryption at rest to encrypt your data. The AWS KMS is supported for customers running their cluster on AWS. You must set up your AWS KMS encryption keys and configure the encryption key IDs in your MarkLogic server before using the AWS KMS.
To set up the AWS key management system, first set up your AWS instance. See Getting Started with MarkLogic Server on AWS and Overview of MarkLogic Server on AWS in the MarkLogic Server on Amazon Web Services (AWS) Guide for details.
The AWS KMS keys must be configured in MarkLogic before using encryption.
You cannot use the master key and roles from the MarkLogic KMS to access the AWS KMS, so you will need to have a Key Administrator specify access to the AWS KMS keys on a per-key basis tied to the user's IAM role. The Key Administrator can specify access using the Encryption Keys section of the IAM AWS management console. See the next section (AWS KMS on EC2) for details and the AWS documentation regarding key policies for more information.
If an encryption key stored in the AWS KMS is disabled for any reason, it cannot be used for encryption or decryption, and MarkLogic loses access to any data encrypted with the disabled key. Deleting a key will lead to permanent data loss as deleted keys can never be recovered. Any keys created in the AWS KMS are cluster management keys and should never be deleted. See https://docs.aws.amazon.com/kms/latest/developerguide/enabling-keys.html for more information.
If your cluster is running on AWS, the IAM role associated with the EC2 instance running MarkLogic is used to access the AWS KMS on behalf of MarkLogic. The hostname and port number will be automatically entered in the correct fields in the Keystore tab of the Admin UI.
The key policy is tied to the the user's IAM role. To set up your IAM role and privileges, see Creating an IAM Role in the MarkLogic Server on Amazon Web Services (AWS) Guide.
Once you have set up your MarkLogic Server (and IAM roles if necessary), follow these steps:
To separate the encryption keys for data, configuration, and log files, we recommend that you create three separate encryption keys. Give each type of key a descriptive name (for example ML_data_key) for the type of content it will be used to encrypt.
We recommend that you create three separate encryption key IDs (one for data, one for configuration, and one for logs). Give each a descriptive name in order to help distinguish between them.
For more about IAM roles and privileges, see Creating an IAM Role in the MarkLogic Server on Amazon Web Services (AWS) Guide. To learn more about using MarkLogic with Amazon Web Services, see the MarkLogic Server on Amazon Web Services (AWS) Guide.
Starting with MarkLogic 9.0-8, Amazon AWS S3 support with encryption is built into the MarkLogic server as an available file system or a storage location for backup/restore. When MarkLogic server writes or updates objects on AWS S3, it can use the AWS KMS server side encryption to protect data. You can choose the encryption method by GUI or API.
To use the AWS KMS key to encrypt data that will be stored on AWS S3, specify which key to be used to encrypt. You can do this using the Admin UI or by using the admin:group-set-s3-server-side-encryption-kms-key API. To find the S3 encryption key (if it has already been set) use the admin:group-set-s3-server-side-encryption-kms-key API.
To set the AWS KMS in the MarkLogic Admin UI, navigate to Groups Configuration page. Scroll down to the S3 protocol configuration field. Select https
as the s3 protocol and aws:kms
as the s3 server side encryption. Paste the s3 server side encryption kms key into the field.
Configure the external KMS keys as shown in the previous section.
Elastic Block Storage Volume is a durable, block-level storage device that you can attach to a single EC2 instance. Encryption on EBS offers a simple encryption solution for your EBS volumes without the need to build, maintain, and secure your own key management infrastructure. AWS EBS volumes support encryption with a custom key.
Starting in MarkLogic 9.0-8, this capability is supported by MarkLogic for AWS. Users can turn on encryption on EBS volumes on their cluster and also optionally specify a custom key for volumes. This can be done using MarkLogic CloudFormation templates and Managed Cluster Feature. See The Managed Cluster Feature and Deploying MarkLogic on EC2 Using CloudFormation in the MarkLogic Server on Amazon Web Services (AWS) Guide.
If a cluster is created by the MarkLogic CloudFormation template, a same encryption key will be used to encrypt all EBS volumes in the cluster. If encryption option is specified, all volumes attached to an instance will apply the same setting. EBS Encryption is only supported by some EC2 instance types, mostly the new generation. The key that is used to encrypt the volume must be in the same region.
KMS keys are never transmitted outside of the AWS regions in which they were created.
Microsoft Azure Key Vault can encrypt your data in MarkLogic. Azure Key Vault is supported for customers running their cluster on Microsoft Azure. You must set up your Azure Key Vault, create the encryption keys in Key Vault, and configure the encryption key IDs in your MarkLogic server before using the keys to encrypt data in MarkLogic.
To set up the Microsoft Azure Key Vault, first set up your Azure instance. See Getting Started with MarkLogic Server on Azure and Overview of MarkLogic Server on Azure for details. Keys are governed by access policies created by the Key Administrator. See the next section (Microsoft Azure Key Vault) for details and the for more information.
If an encryption key stored in the Azure Key Vault is disabled, it cannot be used for encryption or decryption, and MarkLogic loses access to any data encrypted with the disabled key. Deleting a key will lead to permanent data loss as deleted keys can never be recovered.
To set up Microsoft Azure Key Vault, you will create a virtual machine (VM) on Azure. Then create a Key Vault, set up your access policy, and create your encryption keys in the Key Vault.
On the Azure Home page, click virtual machines and click Add to create a new VM. Enter information into the fields for the basic setup.
Under the Management tab, set Identity to On.
On the Review tab, enter your prefered email address and phone number. Review your information and click Create. This process may take a bit of time. Once the virtual machine has been created, you can configure the Key Vault.
To create an Azure Key Vault, navigate to Key Vaults under Home (use Search to find Key Vaults).
Create a new Key Vault with name/resource group/location and a new access policy with keys permissions (decrypt and encrypt) and principle (your newly created VM).
Under Settings navigate to Keys, and generate new keys for data/config/logs encryption. Use these keys IDs to configure MarkLogic encryption.
Install MarkLogic on the Azure virtual machine. See Set up a Simple Deployment in the MarkLogic Server on Microsoft® Azure® Guide for details. Once MarkLogic is installed on Azure, start MarkLogic and navigate to the Admin UI (port 8001).
You may need to stop the firewall from the command line (sudo service firewalld stop
).
In the MarkLogic Admin UI, click on Clusters in the left navigation bar, and then click the Keystore tab. Select external as the kms type. Click the External KMS tab.
Enter the following information to identify the Azure Key Vault and the required encryption key identifiers. Add the appropriate encryption key ID to each field.
Click OK to configure encryption.
We recommend that you create three separate encryption key IDs (one for data, one for configuration, and one for logs). Give each a descriptive name in order to help distinguish between them.
For more about roles and privileges, see the MarkLogic Server on Microsoft® Azure® Guide.
To configure the external key management system using the MarkLogic Admin UI on the MarkLogic host, you will need the following information for your external KMS:
The TLS certificates, used to secure the communication with the KMS, must be stored locally on each host in the MarkLogic data directory (/var/opt/MarkLogic
). By default, the files are expected to be located in the MarkLogic data directory and must have the following names:
kmip-CA.pem
- The root/certificate of the CA that signed the certificate request for MarkLogic.kmip-cert.pem
- The certificate that was issued to MarkLogic and one that was signed by the CA.kmip-key.pem
- The private key that was generated for MarkLogic and is associated with the Certificate issued to MarkLogic (kmip-cert
). (Optional for some KMS servers.)These certificates are the Certificate Authority (CA) for the root of the certificate chain for the kmip-cert.pem
. A certificate could be a self-signed root used by an enterprise or an external CA. Copy these files into the MarkLogic data directory (/var/opt/MarkLogic
). The location and name of these files can be changed by calling the admin functions. See Admin APIs for Encryption at Rest for details.
These settings are cluster wide, so each individual host must have a local copy at the location specified.
In most cases, an external KMS is configured by security administrator, a separate role from the MarkLogic admin role. However, in some cases the security administrator may also be the MarkLogic admin role.
If you don't already have the external KMS configured and running, set up the external KMS using the appliance's interface before turning on MarkLogic encryption. The steps in the process for setting up the external KMS will depend on the type of KMIP-compliant external KMS you are using.
Only KMIP v1.2 is supported. Using KMIP v1.1 will result in an error stating that the server cannot validate the KMS KMIP protocol.
The security administrator can enable encryption for user data, configuration files, and/or logs, either per cluster or per database. You must use the administration tools that come with the external KMS to set up the external keystore.
The external key management system (KMS) must be available during the MarkLogic startup process. Access to the external KMS must be granted to all nodes in the cluster.
Before you set up encryption at rest, be sure that your cluster has upgraded to MarkLogic 9. If the cluster has not been upgraded, the encryption feature will not be available.
Adding the encryption information will require a restart of all of the hosts in your cluster.
When using an external KMS, key encryption keys (KEK) might be rotated according to the policy set in the KMS. Each time that the keys are rotated in an external KMS, you will have to update the new KEK IDs (UUIDs - i.e. key encryption keys - KEKs) to MarkLogic. Data will then start to be encrypted with new KEK ID, as described in Key Rotation.
Encryption at rest may be configured using REST, XQuery, or JavaScript APIs. See APIs for Encryption at Rest for details.
Transitioning from the internal PKCS #11 secured wallet to an external KMS will re-encrypt of all configuration files and forest labels. Re-encryption will happen the next time a file is written to disk. If a you want to force re-encryption of all data, start a re-index of the database.
Customer-provided cluster KEK IDs will be validated against the KMS for encryption/decryption. If any KEK ID validation fails or MarkLogic cannot connect to the KMS, there will be no changes to the configuration files.
Even after you have migrated to an external KMS, the PKCS #11 secured wallet will retain and manage any encryption keys that were generated before the migration to the external keystore.
To migrate from the PKCS #11 secured wallet to an external keystore (KMS) do the following:
If you are using MarkLogic 9.0-6 or later, and for some reason you want to stop encrypting your data with your external KMS, you can switch encryption to use the internal PKCS #11 wallet by setting the KMS type configuration back to the internal KMS.
The following procedure is recommended for a user to perform in order to switch from external KMS to internal KMS and stop access to the external KMS:
To switch encryption from the external KMS to the internal PKCS #11 secured wallet (KMS) in the Admin UI, set the kms-type to internal
on the Edit Keystore Configuration page.
Or you can use admin:cluster-set-keystore-kms-type
(XQuery) or admin.clusterSetKeystoreKmsType
(SJS) to change the KMS for encryption.
After these steps, it should be safe for the system to lose access to the originally configured external KMS.
Moving from an external KMS to the internal KMS will downgrade your overall security, as the external KMS is more secure than the internal PKCS #11 secured wallet.
MarkLogic encryption at rest enables you to specify multiple hosts, multiple ports, and multiple KMIP credentials to connect to more than one KMIP server. The information to connect to these servers is specified in the fields on the external Key Management Service (KMS) section of the Edit Keystore Configuration page.
The information must be validated at configuration time. For each host specified, there must exist a PEM-encoded Cerficate Authority file and a PEM-encoded KMIP certificate file accessible to each node of the MarkLogic server. The PEM files are checked using the user-specified path or default location for the first host. For subsequent hosts, the file names are expected to be accessible through the original file name prepended by the host's index in the configuration sequence.
For example, if the configured host names are kms1.marklogic.com and kms2.marklogic.com. The configured port is 9010. The specified CA file is at path/kmip-CA.pem. The specified certificate file is at /path/kmip-cert.pem. The configuration must be validated through the following:
/path/kmip-CA.pem
, /path/1-CA.pem
, /path/kmip-cert.pem
, and /path/1-cert.pem
all exist.kms1.marklogic.com
at port 9010
.kms2.marklogic.com
at port 9010
.If the first specified KMIP host stops responding, MarkLogic will try to connect to each of the other hosts on the user-specified list in turn until it successfully connects.
If MarkLogic is unable to connect with a valid KMIP server after multiple attempts, it will report exception.
This section covers additional tasks you may want to perform once you have configured encryption.
Individual backup files are encrypted with the cluster data encryption key (CDKEK). Backups are forest driven, so data from an encrypted forest will also be encrypted in backups. Configuration files included in a backup will be encrypted if the cluster is enabled for configuration file encryption. This encryption works with full backups, incremental backups, and journal archiving.
If any forest in the backup has encryption enabled, then the entire backup will be encrypted.
The encryption keys residing in the PKCS #11 secured wallet (the embedded KMS) will be exported as part of a full backup by default. This is true whether encryption is configured to use the internal KMS or an external KMS. Full backups will include this exported copy of the keystore, encrypted using the embedded KMS passphrase, unless you specify otherwise. See .
If you cannot access your PKCS #11 secured wallet (or external KMS if you are using one), or lose your encryption keys, you will not be able to decrypt any of your encrypted data (including backups). There is no workaround to recover the encrypted data. We recommend that you backup your encryption keys in a secure location.
The built-in function admin:cluster-set-keystore-passphrase (XQuery) or
admin.clusterSetKeystorePassphrase (SJS) can be used to change the KMS passphrase. When you first set up encryption, we strongly recommend that you change the KMS passphrase to something other than the admin passphrase. This is to ensure that you utilize the Separation of Duties security principle as much as possible.
By default the keystore passphrase is automatically set to the admin password. We strongly recommend that you set a new, different passphrase before turning on encryption.
During an internal keystore backup/restore, data is added to the embedded PKCS #11 secured wallet; no keys are deleted. The encrypted file containing the keys is named kms.exp
. The exported keystore is not imported during a restore from a backup. If you need to restore the keys, use the xdmp:keystore-import (XQuery) or xdmp.keystoreImport (SJS) function. The keystore passphrase will be required to decrypt the exported keystore file when restoring backups on another MarkLogic instance.
To change the keystore passphrase, the current password or passphrase is required.
To restore an encrypted backup to the same cluster:
To restore an encrypted backup to a different cluster:
xdmp:keystore-import
(XQuery) or xdmp.keystoreImport
(SJS) function to import the keystore. The function requires the keystore passphrase to decrypt the keystore. xdmp:keystore-import(
"strong passphrase", "/backups/MarkLogic.wallet.bak")
xdmp.keystoreImport(
'strong passphrase', '/backups/MarkLogic.wallet.bak');
The import process will reject duplicate keys and log a warning that includes the ID of the rejected keys. Imported keys can only be used for decryption.
As long as the current database being restored is encrypted, the restored database will also be encrypted.
Using this process you can move your encrypted backups from one system to another and restore them, as long as you have the passphrase and import the keystore into the new system before restoring the backup. See Backup and Restore Overview in the Administrator's Guide for more information about backup and restore procedures.
If you lose the cluster configuration information, you must first manually restore the keystore before an encrypted backup can be restored.
To export your keystore, use the xdmp:keystore-export (XQuery) or xdmp.keystoreExport (SJS)
function.
xdmp:keystore-export(
"strong passphrase", "/backups/MarkLogic.wallet.bak")
xdmp.keystoreExport(
'strong passphrase', '/backups/MarkLogic.wallet.bak');
This function exports all of the encryption keys stored in the MarkLogic embedded KMS (the PKCS #11 secured wallet) and stores them at the location provided to the function.
By default the MarkLogic embedded KMS (the PKCS #11 secured wallet) is automatically included in a backup. You can exclude the embedded wallet using the options in
admin:cluster-set-keystore-backup-option (XQuery) or
admin.clusterSetKeystoreBackupOption (SJS). The include
or exclude
options enable you to choose whether to have the embedded KMS included as part of backups.
Setting the option to exclude
prevents the embedded KMS from being included in the backup.
If you set the backup option to exclude
and turn off the automatic inclusion of the keystore, you are responsible for saving keystore (the embedded KMS) to a secure location. If you cannot access your PKCS #11 secured wallet (or external KMS if you are using one), or lose your encryption keys, you will not be able to decrypt any of your encrypted data (including backups).
MarkLogic encryption at rest includes the ability to use a secondary backup key encryption key (BDKEK) for encrypting backups when encryption is configured with an external KMS. Using this BDKEK you can restore your backup to a new system, one that might not have access to the CDKEK and/or CCKEK.
For example, with this XQuery statement you can backup your Documents database using the BDKEK:
xdmp:database-backup(xdmp:database-forests(xdmp:database("Documents"))"/backups/Data", fn:true(), "/backups/JournalArchiving", 15,"bf44aab-3f7a-41d2-a6a5-fc41a0e5e0cf")
Or you could use server-side JavaScript:
xdmp.databaseBackup(xdmp.databaseForests(xdmp.database("Documents")),"/backups/Data", fn:true(), "/backups/JournalArchiving", 15,"bf44aab-3f7a-41d2-a6a5-fc41a0e5e0cf");
In these examples bf44aab-3f7a-41d2-a6a5-fc41a0e5e0cf
is the secondary backup key (BDKEK).
The built-ins xdmp:database-backup (XQuery) or xdmp.databaseBackup (SJS) and xdmp:database-incremental-backup (XQuery) or xdmp.databaseIncrementalBackup (SJS) have an optional argument to take advantage of the BDKEK from the external KMS. The REST API can also take advantage of a secondary backup key as part of the backup operations.
MarkLogic also provides the ability to encrypt backups with a backup passphrase. The xdmp:database-backup (XQuery) or xdmp.databaseBackup (SJS) and
xdmp:database-incremental-backup (XQuery) or xdmp.databaseIncrementalBackup (SJS) APIs take an optional argument for the passphrase ($backup-passphrase
).
Similarly, the built-in xdmp:database-restore (XQuery) or xdmp.databaseRestore (SJS) for restoring a database accepts an optional parameter for the backup passphrase ($backup-passphrase
). Using a passphrase, a user can restore into any system without requiring import of the original keys or connection to an external KMS.
MarkLogic encryption at rest includes the mlecat
command line tool, which can be used to view encrypted log files outside of the server. The mlecat
tool can be used successfully in either of these conditions:
mlecat
tool is given access to the MarkLogic data directory and the .pem
files.mlecat
with -p
option.The mlecat
tool should be run by a user with sufficient OS privileges to access the PKCS#11 wallet (located by default at /var/opt/MarkLogic
). It is suggested that the user be a member of group running MarkLogic (by default daemon
).
If you want to decrypt log files without having access to your KMS, you must set a logs-encryption-passphrase
. To set this passphrase, use the admin:cluster-set-keystore-logs-encryption-passphrase (XQuery) or
admin.clusterSetKeystoreLogsEncryptionPassphrase (SJS) function.
For example:
Log file encryption must be enabled for this passphrase to be used.
For every OS you must add MARKLOGIC_INSTALL_DIR
and MARKLOGIC_INSTALL_DIR/bin
to your PATH
. For example,
PATH=$MARKLOGIC_INSTALL_DIR:$MARKLOGIC_INSTALL_DIR/bin:$PATH
To see the command line options for the mlecat
tool, invoke mlecat
(or mlecat.bat
) with no arguments.
mlecat ==> mlecat [option] filepath(s) option: -i iDIR, iDir is MarkLogic's Install directory, alternatively the environment variable MARKLOGIC_INSTALL_DIR can be used to set this value. -d dDIR, dDIR is MarkLogic's Data directory, alternatively the environmental variable MARKLOGIC_DATA_DIR can be used to set this value. -p PASS, PASS is the logs encryption passphrase (if you are using one); alternatively the environmental variable MARKLOGIC_KMS_PASSPHRASE can be used to provide this value. [-f] filepath(s), one or more file paths (-f can be specified before each file for explicit file list)
mlecat -p admin /var/opt/MarkLogic/Logs/ErrorLog.txt
In order to run this tool the environment must be set to include MarkLogic's libraries.
LD_LIBRARY_PATH
must include $MARKLOGIC_INSTALL_DIR/lib
PATH
must include %MARKLOGIC_INSTALL_DIR%
For example, to run on Linux, you could could set the path like this:
MARKLOGIC_DATA_DIR=/var/opt/MarkLogic MARKLOGIC_INSTALL_DIR=/opt/MarkLogic LD_LIBRARY_PATH=$MARKLOGIC_INSTALL_DIR/lib:$LD_LIBRARY_PATH PATH=$PATH:$MARKLOGIC_INSTALL_DIR/bin export MARKLOGIC_INSTALL_DIR MARKLOGIC_DATA_DIR PATH LD_LIBRARY_PATH
Defaults for the MarkLogic data and install directories are shown in the following
Windows users will use mlecat.bat
, instead instead of mlecat
.
For more about setting environment variables on various platforms, see the information about installation and data directories as part of Installing MarkLogic in the Installation Guide.
Unless you have suffered a complete loss of your host, disaster recovery should work just fine with encryption at rest. See High Availability and Disaster Recovery in the Concepts Guide for information about setting up shared disk failover and steps for disaster recovery.
If you have experienced a complete loss of your host, you will need to do the following:
xdmp.keystoreImport (SJS) ). See Export and Import Encryption Keys for details.The encryption at rest feature includes APIs for working with encryption, using either the default keystore (the interal PKCS #11 secured wallet) or a KMIP-compliant external KMS.
These functions will work with both the internal PKCS #11 secured wallet, or a external KMIP-compliant keystore. Using these functions you can encrypt data and check the status of encryption in your clusters using either JavaScript or XQuery.
The Server-Side JavaScript built-ins are:
The Server-Side XQuery built-ins are:
The xdmp:http-options function now accepts a credential-id when used with XQuery. The schema looks like this:
<xs:complexType name="options"> <xs:sequence> <xs:element ref="timeout" minOccurs="0"/> <xs:element ref="data" minOccurs="0"/> <xs:element ref="headers" minOccurs="0"/> <xs:element ref="credential-id" minOccurs="0"/> <xs:element ref="authentication" minOccurs="0"/> <xs:element ref="client-cert" minOccurs="0"/> <xs:element ref="client-key" minOccurs="0"/> <xs:element ref="pass-phrase" minOccurs="0"/> </xs:sequence> </xs:complexType>
These functions are used to set the mode and descriptions for the host, set the keystore host name and the keystore host port. You can also set the keystore data key ID, config key ID, or logs key ID, along with setting the keystore serve certificate and enabling encryption.
These server-side Javascript functions work with either the PKCS #11 secured wallet or a third-party KMIP-compliant keystore:
admin.clusterGetKeystoreLogsEncryption
admin.clusterGetKeystoreWallet
admin.clusterGeternalDataEncryptionKeyId
These server-side XQuery functions will work with either the PKCS #11 secured wallet or a third-party KMIP-compliant keystore:
admin:cluster-get-keystore-backup-option
The admin.clusterRotateXXXXEncryptionKeyId
or admin:cluster-rotate-
xxxx-encryption-key-id
APIs are only for use with the embedded KMS provided by MarkLogic (the PKCS #11 secured wallet). Using these functions with an external KMS will cause an error.
These next two APIs are used in transitioning from an internal keystore (the PKCS #11 secured wallet) to an external KMIP-compliant keystore. If these functions are set to external, MarkLogic Server will first look for the external keystore to verify the keys.
These Javascript and XQuery functions are designed to work with a external KMIP-compliant keystore.
admin.clusterGetLogsEncryptionId
admin.clusterSetLogsEncryptionId
The functions designed to work with a external KMS will return an error if you try to use them with the PKCS #11 secured wallet (the default built-in KMS).
You can manage encryption using the REST Management APIs. Some of the tasks you can do with these APIs include:
The REST Management APIs that are used to query and manage the cluster security properties include encryption information for database, cluster, and forest.
Below is a XML payload example for the security endpoint:
<security-properties xmlns="http://marklogic.com/manage/security/properties" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://marklogic.com/manage/security/properties manage-security-properties.xsd"> <keystore> <data-encryption>default-off</data-encryption> <config-encryption>off</config-encryption> <logs-encryption>off</logs-encryption> <kms-type>internal</kms-type> <host-name>localhost</host-name> <port>9056</port> <data-encryption-key-id>92ed7360-458a-427e-abad-c6595b192cb7</data-encryption-key-id> <config-encryption-key-id>8b9a9bdb-7b0e-41eb-9aa6-ed6e8cb23ad5</config-encryption-key-id> <logs-encryption-key-id>01c50d02-b43f-46bc-bbe5-6d4111d1180b</logs-encryption-key-id> </keystore> </security-properties>
And here is a JSON payload example for the security endpoint:
{ "keystore": { "data-encryption": "default-off", "config-encryption": "off", "logs-encryption": "off", "kms-type": "internal", "host-name": "localhost", "port": 9056, "data-encryption-key-id": "92ed7360-458a-427e-abad-c6595b192cb7", "config-encryption-key-id": "8b9a9bdb-7b0e-41eb-9aa6-ed6e8cb23ad5", "logs-encryption-key-id": "01c50d02-b43f-46bc-bbe5-6d4111d1180b" } }
These operations are available for encryption key rotation:
curl -v -X POST --anyauth --user admin:admin \ --header "Content-Type:application/json" -d \ '{"operation":"rotate-config-encryption-key"}' \ http://localhost:8002/manage/v2/security curl -v -X POST --anyauth --user admin:admin \ --header "Content-Type:application/json" -d \ '{"operation":"rotate-data-encryption-key"}' \ http://localhost:8002/manage/v2/security curl -v -X POST --anyauth --user admin:admin \ --header "Content-Type:application/json" -d \ '{"operation":"rotate-logs-encryption-key"}' \ http://localhost:8002/manage/v2/security
In most cases the encryption at rest feature will be transparent to the user, that is data on disk will be encrypted, decrypted during use (by users with the appropriate security permissions), and re-encrypted when the data is written back to disk.
Encryption at rest is a feature introduced in MarkLogic 9. Clusters running older versions need to be completely upgraded to MarkLogic 9 before using this feature. See Rolling Upgrades in the Administrator's Guide for more about rolling upgrades.
During upgrades, the default passphrase for the upgraded system is not set. You will need to reset the default passphrase after an upgrade.
The telemetry feature is not available for use until the cluster is upgraded to MarkLogic 9.0-1 or later. See Telemetry in the Monitoring MarkLogic Guide for more about telemetry.