Query Performance and Tuning Guide — Chapter 8

XDBC Invoke Requests

You can enable request monitoring at both the endpoint level and at the server level for XDBC invoke request. For a specific endpoint, you must add a monitoring section to the XDBC Server Endpoint Declaration file and specify which metrics will be logged. To configure monitoring on a global level, you must add a default.api file in the modules root directory for the XDBC Server. For the Task Server, as there is no port number, the output request log file will be logged into a new type of log file called TaskServer_RequestLog.txt

XDBC Eval Requests

For XDBC eval requests, request monitoring is only available at the server level, as there are no real endpoints. To seperate the monitoring configuration between the invoke and eval request, and to add more control over the monitoring of the eval request, you can add an eval.api file to the module root in addition to the default.api file. The eval.api file has the same format as the default.api file, which contains only a monitoring section, but the settings in eval.api override those in default.api.

The Endpoint Declaration File

The name, parameters, and return value for each endpoint are declared in the *.api file. The *.api file contains a JSON data structure with the following properties:

When monitoring a module that is not defined as an endpoint, none of the properties defined in the preceding table are needed

The following is a list of meters that can be logged with the parameters that control them:

The following parameters may be included in a *.api file:

Constraints on Meters

To control the number of meters that are logged, you can put the following constraints on meters:

The declaration format of a constraint is:

meter_name : {"operator":value, "operator":value, ...}

For example:

"constraints": {
  "tripleCacheHits" : { "ge":1 }
}

In this example, tripleCacheHits is logged only if the the value of tripleCacheHits is >= 1.

Meters with zero or empty values are not normally logged. This is done to minimize the size of the Request Log file. To log a zero or empty value, use the following code:

"constraints": {
  "meter_name" : { "ge":0 }
}

The default constraint value on any meter is:

gt:0

Controlling Request Logging With Thresholds

You can add thresholds to specify that a request is logged only when the threshold conditions are satisfied. To declare a threshold, create a thresholds section in your *.api file as follows:

{
  "monitoring":{
    "thresholds" : {
      "elapsedTime": { "gt": 1.0 }
    }
   general:
     enabled: true
  }
}

The following operators are allowed in thresholds:

To disable threshold checks on custom meters so the meters can always be logged, set the boolean flag excludeCustom to true in the thresholds section, as follows:

{
  "monitoring":{
    "thresholds" : {
      "excludeCustom": true,
      "elapsedTime": { "gt": 1.0 }
    }
   general:
     enabled: true
  }
}

In the following example, the general meters for the request are logged only when the total runtime of the request is greater than one second:

{
  "monitoring":{
    "thresholds" : {
      "elapsedTime": { "gt": 1.0 }
    }
    "general" : {
      "enabled": true
    }
  }
}

Enabling Request Monitoring

Request monitoring is enabled by default on the following default MarkLogic application servers:

• The default App-Services application server, normally Port 8000:
  • For all requests related to the Query Console.
  • For requests with runtime exceeding one second that are not related to the Query Console. This mainly covers the REST Client API.
• The default Manage application server, normally Port 8002, for requests running longer than one second. This covers the Configuration Manager and Monitoring Dashboard.

If request monitoring is not already enabled, you can enable request monitoring by calling any server-side JavaScript (*.sjs) or XQuery (*.xqy) functions in files that reside in the modules directory as declared on the HTTP server via the Admin interface, and you can create other *.api files in the same directory or, if using the file system, in the same subdirectory as the JavaScript or XQuery file being called. For more information on configuring HTTP servers and the modules directory, see the HTTP Servers chapter in the Administrator's Guide.

The following example enables request monitoring:

{
"monitoring":{
  "general":{
    "enabled":true
  },
  "custom":true
}
}

The following example logs tripleCacheHits if the the value of tripleCacheHits is >= 0:

"constraints": {
     "tripleCacheHits" : { "ge":0 }
    }

The following code fragment logs up to 10 documents, 10 fragments, and 4 hosts:

"documents": 10,
"fragments" : 10,
"hosts" : 4

The following example is called countdocs.api. The function getCount is defined in a file called countdocs.sjs that resides in the same directory. The general object under the monitoring section of the file has enable and custom set to true; this enables request logging.

{
  "functionName":"getCount",
  "params":[
    {
      "name":"collection",
      "datatype":"string",
      "multiple":false,
      "nullable":false
    },
    {
      "name":"method",
      "datatype":"int",
      "multiple":false,
      "nullable":false
    }

  ],
  "return":{
    "datatype":"string",
    "nullable":false,
    "multiple":false
  },

  "monitoring":{
    "general": {
      "enabled": true,
      "constraints": {
        "tripleCacheHits" : { "ge":1 }
      }
    },
    "custom": true ,
    "documents": 10,
    "fragments" : 0,
    "hosts" : 4
  }
}

The Default Declaration File

You can configure request monitoring globally for all requests on an App Server by adding a default.api file where the modules root is configured for the App Server (either to the modules database or to the file system). The default.api file only contains a monitoring section.

The following is a sample default.api file:

{
"monitoring":{
    "general":  {
      "enabled": true,
      "constraints": {
        "tripleCacheHits" : { "ge":0 }
      }
    },
    "custom": true,
    "documents": 10,
    "fragments" : 10,
    "hosts" : 5  }
}

When you make a request to an endpoint and do not specify an App Server Endpoint Declaration file, the default.api file in the module database or file directory is used if it exists. If a module has a *.api file associated with it, the monitoring settings in the *.api file for that module are used instead of those in the default.api file.

If you want to save the default.api file, use JavaScript:

declareUpdate();
xdmp.documentInsert("/default.api",{ 
 "monitoring":{
  "general": {
   "enabled": true
   },
  "custom": true,
  "documents": 10,
  "fragments" : 10,
  "hosts" : 5 }
}); 

Request Logs

The logs for MarkLogic Server containing information about the requests you have chosen to log are stored in the Logs directory under the MarkLogic Server data directory (typically c:\Program Files\MarkLogic\Data\Logs on Windows, /var/opt/MarkLogic/Logs on UNIX-based systems) in the file port_no_RequestLog.txt. Each RequestLog.txt file will contain the meters from multiple monitored endpoints that:

• are configured on an App Server with some monitoring switched on.
• have calls to xdmp:request-log-put in their module.

The collected data is logged in JSON format at the rate of one line per request information. Even if monitoring is completely switched off on an endpoint, calls to xdmp:request-log-put(key,value) during a request will result in data being logged for all the (key,value) pairs that have been stored during the request.

The following is an example of a request log where different meters are logged for two endpoints:

{"time":"2018-11-09T14:07:14-08:00", "url":"/booleanApiDecl.sjs?booleanArg=true", "user":"admin", "result":10, "elapsedTime":0.007145, "requests":1, "dbProgramCacheMisses":1, "dbMainModuleSequenceCacheMisses":1, "dbLibraryModuleCacheMisses":0}

{"time":"2018-11-09T14:10:18-08:00", "url":"/booleanApiDecl.xqy?booleanArg=true", "user":"admin", "result":10, "elapsedTime":0.001603, "requests":1, "dbProgramCacheMisses":1, "dbMainModuleSequenceCacheMisses":1, "dbLibraryModuleCacheMisses":0}