What's New in MarkLogic 12

MarkLogic 12.0 EA1 supports Best Match 25 (BM25) for relevance scoring of search results.

The BM25 method of scoring documents is widely used because of its effectiveness in ranking documents based on relevance to a query. With the rise of Generative AI (GenAI) and Retrieval Augmented Generation (RAG) workflows, BM25 has come into play for retrieving relevant documents to supply as context to a Large Language Model (LLM). LLM-generated answers see a significant quality boost when ranked documents become the sources of knowledge.

The MarkLogic Server core text search has always returned results based on relevance. The default MarkLogic Server relevance scoring method, logTF-IDF, implicitly weights document length by counting unique terms. The new BM25 scoring method explicitly does this by adding a tunable parameter to directly increase or decrease the weight of a document's length on the score.

Consider the following documents' term frequencies and lengths retrieved by cts.wordQuery("trojan"):

Compare the ranking differences between traditional logTF-IDF, which uses term frequency alone, and BM25, which heavily penalizes /doc4.json for being significantly longer than the average document length of 1396:

The BM25 scoring method accounts for the fact that a short document with a high term frequency is more likely to be about that term than a long document with the same term frequency, which is more likely to simply be using that term in passing.

Use one of these code samples to enable the BM25 scoring method for core text search queries:

CTS

cts.search(
  cts.wordQuery("trojan"), 
  [
    "score-bm25",
    "bm25-length-weight=0.25"
  ]
)

Optic

const op = require('/MarkLogic/optic');
op.fromSearch(
  cts.wordQuery("trojan"), 
  null, 
  null, 
  {
    "scoreMethod" : "bm25",
    "bm25LengthWeight": 0.25
  }
)

Search/REST Search API

import module namespace search = "http://marklogic.com/appservices/search"
    at "/MarkLogic/appservices/search/search.xqy";
search:search("trojan",
  <options
      xmlns="http://marklogic.com/appservices/search">
    <search-option>score-bm25</search-option>
    <search-option>bm25-length-weight=0.25</search-option>
  </options>
)

JSearch

import jsearch from '/MarkLogic/jsearch.mjs';
jsearch.documents()
  .where(cts.wordQuery("trojan"))
  .withOptions(
    {
      search: [
        'score-bm25', 
        'bm25-length-weight=0.25'
      ]
    }
  )
  .result()

Using core text search within MarkLogic Server has always been the key to finding the data that you need. BM25 is an extra knob to further tune your search results.