Adaptive Memory Algorithms for Optic Sorts and Joins
MarkLogic 11 adds the ability for large sorts and joins in the Optic engine to overflow to disk. This provides improved support for large analytic queries, reporting, and exports of large result sets. Prior to MarkLogic 11, these queries may have been canceled due to a lack of available RAM. In MarkLogic 11, the Optic engine has access to new algorithms for sorts and joins that limit the amount of RAM used and that overflow to a temporary location on disk when there is too much data to fit in RAM, allowing the query to complete.
Other than configuring a temporary directory if the default is not acceptable, there is nothing users need to do to take advantage of this new capability. The Optic query optimizer now has access to these algorithms and will select them when sufficiently large operations need to be performed or if memory is sufficiently constrained.
Temporary Files
The Optic engine encrypts all temporary files with continuously changing, random AES 256 keys. By default, the temporary files for large sorts and joins are written to the Temp/Optic directory under the default MarkLogic data directory. This will typically be /var/opt/MarkLogic/Temp/Optic.
For most use cases, there will be no reason to change this temporary directory. However, if there are cases where many large analytic queries are often overflowing to disk and the system could benefit from faster disks than are currently in use for the default data directory, in MarkLogic 11, the location of the Temp directory can be configured via the "temporary-directory" group setting (see /REST/PUT/manage/v2/groups/[id-or-name]/properties).
Important
Because the temporary directory is a group-level setting, if it is set to a path other than the default, all hosts in the group must make a file system available to MarkLogic at that path. If MarkLogic cannot access the configured temporary directory, it will fall back to writing to the default location.
If, under special circumstances, the temporary directory needs to be overridden at the host level instead, the MARKLOGIC_TEMP_DIR environment variable can be set for the MarkLogic process. See the Best Practice Editing MarkLogic Server Environment Variables knowledge base article for more information about setting environment variables.
Diagnostics
In MarkLogic 11, xdmp.queryMeters() now reports per request the number of bytes read from temporary files and the number of bytes written to temporary files via temporaryFileReadBytes and temporaryFileWriteBytes respectively.
In MarkLogic 11, xdmp.serverStatus() now reports, on a per app server basis, the cumulative number of bytes read from temporary files and the cumulative number of bytes written to temporary files via temporaryFileReadBytes and temporaryFileWriteBytes respectively.
Additional Information
REST API to get group properties : /REST/GET/manage/v2/groups/[id-or-name]/properties
REST API to set group properties: /REST/PUT/manage/v2/groups/[id-or-name]/properties
See Getting Started with Optic for more detail about working with the Optic API.