HTTP Compression and Chunking
MarkLogic 11 adds out of the box support for automatic HTTP compression and chunking.
HTTP Compression
Prior to MarkLogic 11, HTTP compression on standard endpoints (such as /v1/rows) was not available. HTTP compression could be added to custom REST extensions via the use of xdmp.gzip to compress the response. However, this required custom code and lacked the ability to stream the response. For example, a large JSON document with an array with a million rows would have to be compressed in memory prior to returning the response.
MarkLogic 11 now supports compressing of data at the HTTP layer. This means that if clients specify the Accept-Encoding: gzip
HTTP header, HTTP compression will be done for all out of the box endpoints. And, custom endpoints will not need to use custom code to compress data. Custom endpoints can leverage streaming results, and the HTTP layer will compress the results on the fly.
Notice
Custom endpoints that looked for the Accept-Encoding: gzip
HTTP header and performed compression using custom code must remove that logic. If the logic is not removed, the results will be compressed twice, and not be usable by the client.
Custom HTTP endpoints can override the client request header and turn on or off the HTTP compression via a new built-in function xdmp.setResponseCompression(). This function also allows the compression level to be set.
Since most browsers send the Accept-Encoding: gzip
HTTP header by default, all traffic to the MarkLogic built-in applications, such as the Admin UI and Query Console, will now be compressed.
HTTP Chunking
MarkLogic 11 adds the ability to use HTTP chunked transfer encoding. This adds the ability to report errors during the streaming of large results over HTTP.
HTTP chunking is activated in one of two ways:
The client includes the
TE:trailers
request headerCustom HTTP endpoints use the built-in function xdmp.setResponseChunkedEncoding(). Custom trailers can be added using xdmp.addResponseTrailer() function
If HTTP chunking has been activated by one of the above mechanisms, MarkLogic will send the following trailers (including the header):
ML-CONTENT-SHA256
- The SHA256 of the content that was sent (this can be used to check the all of the data that was sent was received)ML-ERROR-CODE
- The error code if an error occurred, or "NONE" if there wasn't an errorML-ERROR-MESSAGE
- The error message if an error occurred, or "NONE" if there wasn't an errorAny custom trailers added via xdmp.addResponseTrailer()