Skip to main content

What's New in MarkLogic 11

HTTP Compression and Chunking

MarkLogic 11 adds out of the box support for automatic HTTP compression and chunking.

HTTP Compression

Prior to MarkLogic 11, HTTP compression on standard endpoints (such as /v1/rows) was not available. HTTP compression could be added to custom REST extensions via the use of xdmp.gzip to compress the response. However, this required custom code and lacked the ability to stream the response. For example, a large JSON document with an array with a million rows would have to be compressed in memory prior to returning the response.

MarkLogic 11 now supports compressing of data at the HTTP layer. This means that if clients specify the Accept-Encoding: gzip HTTP header, HTTP compression will be done for all out of the box endpoints. And, custom endpoints will not need to use custom code to compress data. Custom endpoints can leverage streaming results, and the HTTP layer will compress the results on the fly.

Notice

Custom endpoints that looked for the Accept-Encoding: gzip HTTP header and performed compression using custom code must remove that logic. If the logic is not removed, the results will be compressed twice, and not be usable by the client. 

Custom HTTP endpoints can override the client request header and turn on or off the HTTP compression via a new built-in function xdmp.setResponseCompression(). This function also allows the compression level to be set.

Since most browsers send the Accept-Encoding: gzip HTTP header by default, all traffic to the MarkLogic built-in applications, such as the Admin UI and Query Console, will now be compressed.

HTTP Chunking

MarkLogic 11 adds the ability to use HTTP chunked transfer encoding. This adds the ability to report errors during the streaming of large results over HTTP.

HTTP chunking is activated in one of two ways: 

  1. The client includes the TE:trailers request header

  2. Custom HTTP endpoints use the built-in function xdmp.setResponseChunkedEncoding(). Custom trailers can be added using xdmp.addResponseTrailer() function

If HTTP chunking has been activated by one of the above mechanisms, MarkLogic will send the following trailers (including the header):

  • ML-CONTENT-SHA256 - The SHA256 of the content that was sent (this can be used to check the all of the data that was sent was received)

  • ML-ERROR-CODE - The error code if an error occurred, or "NONE" if there wasn't an error

  • ML-ERROR-MESSAGE - The error message if an error occurred, or "NONE" if there wasn't an error

  • Any custom trailers added via xdmp.addResponseTrailer()

In this section: