MarkLogic Server 11.0 Product Documentation

Monitoring MarkLogic Guide — Chapter 4

Understanding Telemetry

If telemetry is enabled, MarkLogic Server registers with a well-known endpoint. Data is collected from each host in a cluster: log records from the Error log (ErrorLog.txt which contains system logs), monitoring history and usage data (XML documents from the Meters database), application and host config files from Data/*.xml, (including license key feature and entitlement info) and Telemetry Usage data. See Telemetry Usage Feature for more information. Personally Identifiable Information (PII) and Application data have been physically split from System data (into application-specific log files), and are not collected. Only System data is collected for Telemetry. Telemetry data is sent to secured cloud storage in a protected location. See Viewing Monitoring History for more about monitoring history and the Meters database.

The type and granularity of data sent via telemetry is configurable. You can:

• Enable/disable log data: to collect log data; you can select values for the level of logging from fine through debug to emergency levels
• Enable/disable metering history data: to collect metering history data at the level you choose
• Enable/disable host/cluster configuration data: to collect any changes to configuration files
• Enable/disable usage data: to collect Telemetry usage data

The MarkLogic telemetry feature enables you to send relevant historical data to MarkLogic Support to facilitate troubleshooting and resolution of issues.

Configure Telemetry in the Admin UI

To configure telemetry for your MarkLogic cluster, see the following sections:

• Enable Telemetry on the Group Configuration Page
• Telemetry on the Support Page

Example --Telemetry

For this example, perform these steps:

1. Configure telemetry - see Enable Telemetry on the Group Configuration Page for details.
2. Let your system run for 10-15 minutes to generate data and then change the log level or make another configuration change to your system.
3. To view the output, you can see the staged messages being collected to be uploaded in /var/opt/MarkLogic/Stage (Config, Logs, Meters, Usage). See View Staged Telemetry Files.

   Checking the locally staged files will not work if you have encryption at rest enabled. Encryption settings apply to all staged files, if enabled See Encryption at Rest in the Security Guide for more information.

[localhost Logs]$ tail -f ErrorLog.txt |grep -i telemetry
2021-08-25 16:36:47.001 Info: Uploaded 47 records, 1 MB of Config Files to Telemetry
2021-08-25 16:36:47.972 Info: Uploaded 4 records, 1 MB of Meters Data to Telemetry
2021-08-25 16:36:48.794 Info: Uploaded 64 records, 1 MB of Error Logs to Telemetry
2021-08-25 16:37:03.169 Info: Uploaded 1 records, 1 MB of Usage Data to Telemetry

xdmp:logfile-scan("/var/opt/MarkLogic/Logs/ErrorLog.txt")

xdmp:logfile-scan("/var/opt/MarkLogic/Stage/Logs/ErrorLog.txt")

Telemetry on the Support Page

To verify the MarkLogic telemetry data being collected or see a sample of what is being sent, check the Telemetry section on the Support page. To navigate to the Support page in the Admin UI, click the Support tab.

The top part of this page is used to create Support Request information, and is independent of Telemetry data files, which are sent continuously once enabled. The Support Request data is a snapshot of the system information that can be sent to MarkLogic secure storage on demand.

In the Telemetry section, you can verify the information that telemetry has collected to send on a scheduled basis, and check the current telemetry status for the different types of data. To change the configuration, click the hyperlink next to each telemetry option. You can also:

• Enable or disable collection of different types of telemetry on the Group Settings page by clicking the link next to each type.
• Check the levels for the types of data to collect for telemetry: log messages, metering data, and configuration information. You can set these on the Group Settings page. See Enable Telemetry on the Group Configuration Page for details.

After you have configured your telemetry settings on the Group Setting page, the information will be displayed on this page, and collected and sent automatically in the background. You can manually upload support request data to MarkLogic Support by selecting the radio button next to upload to MarkLogic Secure Storage and clicking the ok button. The Support Request confirmation screen will be displayed when the upload is complete.

The confirmation includes the cluster ID for the Support Request information and a timestamp. Include this information in your email to MarkLogic Support. Support will also need to know the time of the incident or when the issue was first noticed. See Upload a Support Request to Support for more information.

Telemetry data is collected only after you enable the telemetry settings. We recommend that you enable telemetry as a normal practice, so that you have the system information available as part of a Support request should you have an issue.

Configure Telemetry With XQuery

You can also configure telemetry using the Telemetry APIs. See APIs for Telemetry for the complete list. Here are a few examples of configuring telemetry using XQuery. You can run these examples in the Query Console. To set the telemetry log level with XQuery:

xquery version "1.0-ml";
import module namespace admin = "http://marklogic.com/xdmp/admin"
  at "/MarkLogic/admin.xqy";

let $config := admin:get-configuration()
let $groupid := admin:group-get-id($config, "Default")
let $value := "finest"
let $tlogconfig := admin:group-set-telemetry-log-level($config, $groupid, $value)
return
admin:save-configuration($tlogconfig)

To check the telemetry log configuration, use this query:

xquery version "1.0-ml";
import module namespace admin = "http://marklogic.com/xdmp/admin"
  at "/MarkLogic/admin.xqy";
  
let $config := admin:get-configuration()
let $groupid := admin:group-get-id($config, "Default")
return
admin:group-get-telemetry-log-level($config, $groupid)
=>
finest

To see the type of metering data that is being collected by telemetry, you can use the admin:group-get-telemetry-metering function.

xquery version "1.0-ml";
import module namespace admin = "http://marklogic.com/xdmp/admin"
  at "/MarkLogic/admin.xqy";

let $config := admin:get-configuration()
return
admin:group-get-telemetry-metering($config,
  admin:group-get-id($config, "Default"))
=>
daily

To see whether telemetry usage data is being collected by telemetry, you can use the admin:group-get-telemetry-usage function.

xquery version "1.0-ml";
import module namespace admin = "http://marklogic.com/xdmp/admin" 
at "/MarkLogic/admin.xqy";
let $config := admin:get-configuration()
let $groupid := admin:group-get-id($config, "Default")
return admin:group-get-telemetry-usage($config, $groupid)
=>
enabled

To see the type of proxy server URL that is being used by telemetry, you can use the admin:group-get-telemetry-proxy function.

xquery version "1.0-ml";
import module namespace admin = "http://marklogic.com/xdmp/admin"
  at "/MarkLogic/admin.xqy";

let $config := admin:get-configuration()
return
admin:group-get-telemetry-proxy($config,
  admin:group-get-id($config, "Default"))
=>
"https://proxy.marklogic.com:8080"

Baseline System Information

Baseline system information is sent whenever telemetry is on. The system baseline data, which is sent regardless of what type of telemetry is enabled, includes the following information: system data (hostid, clusterid, license) and Standard HTTP headers added by the system and internet routers (IP address, content type, and so on). Data containing Personally Identifiable Information (PII) is not collected.

If telemetry metering is enabled, metering data will be collected as part of the baseline information that is sent.

Upload a Support Request to Support

When you contact Support (https://help.marklogic.com/) you may be asked to create a Support Request (Support Dump) for your MarkLogic installation to help diagnose and solve the issue. The Support Request can be configured to send information about the host, or the whole cluster. You can select the level of detail to provide and which version (current or all) of the status and/or system logs to send. The Support Request is a snapshot of a point in time in your MarkLogic environment, and includes status information, log files, configuration files, and other information. Depending on what options you select, the Support Request will collect the latest, or all versions of the selected options (status, logs, config files). See Telemetry on the Support Page for option details. These files are read from disk and can be reviewed by opening the data directory located at /var/opt/MarkLogic or C:\Program Files\MarkLogic\Data.

Telemetry data files, in addition to the Support Request data mentioned in the previous paragraph, will help MarkLogic Support determine root causes for issues. Telemetry, if enabled, sends a packet of current information on a scheduled basis about a MarkLogic cluster (node) to the secured cloud storage. MarkLogic Support can then retrieve historic information about the MarkLogic environment from the day telemetry was enabled. This helps Support to identify at what point in time the reported issue started and determine the possible root cause. The Support Request snapshot and telemetry stream are independent of one another. However, since telemetry data captures information about your system as snapshots of current data on a regular basis, enabling telemetry early before issues arise, is the key to leveraging this feature.

When providing information for a Support request, only provide required information to MarkLogic Support and which is cleared of confidential or other sensitive information. MarkLogic does not require Protected Health Information (PHI), Payment Card Industry (PCI) information, or Personally Identifiable Information (PII) to provide Support Services. Therefore, do not forward any of such types of information to MarkLogic in connection with a Support request. At all times, information provided to MarkLogic in the course of Support will be handled in accordance with the Privacy Policy available here: http://www.marklogic.com/privacy-policy/. Similarly, any information collected via telemetric functionality enabled by users of MarkLogic Server will be handled in accordance with the above-referenced Privacy Policy.

To create and upload a Support Request, navigate to the Support tab. When selecting the Support information, click the upload to MarkLogic Secure Storage radio button. The contents will be uploaded to a secure server where MarkLogic Support can access the information.

You will also be asked to send an email to Support with your cluster ID and the approximate time and date of the Support Request. This information is available to you as part of the Support Request confirmation screen. Support will also need to know the time the incident first occurred or first was noticed. See Telemetry on the Support Page for an example of the Support Request Saved confirmation screen, which includes the cluster ID, along with time and date information. You can also find the cluster ID through the Admin UI by clicking on Clusters on the left tree menu. The cluster ID is listed on the Cluster Summary View page.

Telemetry Usage Feature

In MarkLogic 10.0-8, the Telemetry Usage Feature is added. It collects and sends usage data to the secured cloud storage from MarkLogic clusters. Telemetry Usage is enabled by default. The upload of the usage data is scheduled every time after the server starts/restarts, and every 7 days since. This feature does not collect any PII info. Telemetry usage data provides essential information for MarkLogic Global Customer Success team to better provide guidance and suggestions for customers, and ensure business compliance. This feature can be switched off through Admin UI, Admin APIs, and REST APIs.

Example of usage data collected by Telemetry:

<usage-files xmlns="http://marklogic.com/xdmp/telemetry"> 
  <telemetry-usage xmlns="http://marklogic.com/xdmp/telemetry-usage">
   <current-time>2021-08-25T17:32:21.099407-07:00</current-time>   
   <license-key/>   
   <cluster-id>6970588360785956700</cluster-id> 
   <host-id>18163410317407796614</host-id>   
   <group-id>6124767890639452752</group-id> 
   <group-name>Default</group-name>  
   <cpus>2</cpus>  
   <cores>12</cores>   
   <core-threads>24</core-threads>  
   <architecture>x86_64</architecture>   
   <platform>linux</platform>   
   <edition>Essential Enterprise</edition>   
   <version>10.0-8</version>   
   <software-version>10000800</software-version>   
   <effective-version>10000800</effective-version>  
   <os-version>Linux 4.18.19-100.fc27.x86_64 (Fedora release 27 (Twenty Seven))</os-version>   
   <host-size>3</host-size>   
   <host-large-data-size>0</host-large-data-size>   
   <memory-size>65536</memory-size> 
  </telemetry-usage>
</usage-files>

APIs for Telemetry

These functions are available for managing telemetry functionality, both Admin APIs and REST Management APIs.

Interactions With Other MarkLogic Features

The following section describes possible feature interactions between telemetry and other MarkLogic features.