Configure a Mastering Step Using QuickStart

In Smart Mastering, you must define two categories of rules: matching and merging.

  • The matching rules determine if two records are candidates for merging, based on the degree of similarity between them and the weight of the comparison.
    • Match options define how the records are compared.
    • Match thresholds define the limits and the actions to take when those limits are exceeded.
  • The merging rules specify how the candidates are handled, based on thresholds.
    • Merge options define how the properties of the candidate records are combined.
    • Merge strategies are sets of merge options that you can name and reuse.
    • Merge collections are sets of records that have the same collection tags.

These rules are defined in the mastering step.

For more information on Smart Mastering, see the Smart Mastering Framework documentation.

Before you begin

You need:

Procedure

  1. Navigate to the flow definition of the flow you want.

    QuickStart Flows - Manage Flows table - Click flow name

    1. In QuickStart's navigation bar, click Flows.
    2. In the Manage Flows table, search for the row containing the flow.
      Tip: To make your search easier, you can sort the table by one of the columns.
    3. Click the flow's name.
  2. In the flow sequence, click the summary box of the mastering step to configure.


    The step detail panel is displayed below the flow sequence panel.

Matching

  1. In the step detail panel at the bottom, click the Matching tab.


  2. Configure the Match Options.
    • To add a new option, click the Add button.
    • To edit an option, click the vertical ellipsis (⋮) and choose Edit Settings.
    • To delete an option, click the vertical ellipsis (⋮) and choose Delete.
    Match Options fields
    Field Description
    Type The matching type.
    • Exact. Determines if the values of the specified property in two or more records are exactly the same.
    • Synonym. Determines if the values of the specified property in two or more records are synonyms, according to the specified thesaurus.
    • Double Metaphone. Determines if the values of the specified property in two or more records sound similar, based on the Double Metaphone algorithm. For example, "Smith" might sound like "Schmidt".
    • Zip. Determines if the zip/postal code in two or more records match.
    • Reduce. Reduces the significance of certain matches. For example, even if the addresses and last names of two records match, the similarity might not necessarily indicate that the two records refer to the same person, because they might be two members of the same family.
    • Custom. Runs a function in your custom module to compare the values of a specified property in two or more records.

    The set of fields (properties) in the dialog varies, depending on the match type.

    Click a match type to view its properties:

    Exact match properties

    Synonym properties

    Double Metaphone properties

    Zip properties

    Reduce properties

    Custom match properties
    Property to Match The property whose values to compare.
    Properties to Match One or more properties whose values to compare.
    Weight A factor that signifies the relative importance of the rule.
    Weight A positive integer that denotes how much to reduce the weight of a match.
    Thesaurus URI

    The location of the thesaurus that is stored in a MarkLogic Server database and used to determine synonyms. See also: Managing Thesaurus Documents

    Filter

    A node in the thesaurus to use as a filter. For example, <thsr:qualifier>birds</thsr:qualifier>.

    See the $filter parameter in thsr:expand.
    5-vs-9 Match Weight

    The weight to use if, given one 9-digit zip code and one 5-digit zip code, the first five digits of the zip codes match. Applicable only to US zip codes.

    9-vs-5 Match Weight

    The weight to use if, given two 9-digit zip codes, the first five digits match and the last four do not. Applicable only to US zip codes.

    Note: To add a weight when all nine digits match, use the Exact matching type to compare the zip codes as strings. If some of your data might have non-standard formats (e.g., using whitespace or em dash instead of the typical hyphen), remember to map your zip code fields to a standard format in your entity.
    Dictionary URI

    The location of the phonetic dictionary that is stored in a database and used when comparing words phonetically. See also: Custom Dictionaries

    Distance Threshold

    The threshold below which the phonetic difference (distance) between two strings is considered insignificant; i.e., the strings are similar to each other. See also: spell functions

    Collation

    The URI to the collation to use. A collation specifies the order for sorting strings. See also: Encodings and Collations

    URI The location of the custom module.
    Function The name of the custom function within the custom module.
    Namespace The namespace of the library module where the custom function is. Blank, if the custom function is JavaScript code.
  3. Configure the Match Thresholds.
    • To add a new threshold, click the Add button.
    • To edit a threshold, click the vertical ellipsis (⋮) and choose Edit Settings.
    • To delete a threshold, click the vertical ellipsis (⋮) and choose Delete.
    Match Thresholds fields

    Match threshold properties
    Field Description
    NameThe threshold rule name.
    Weight ThresholdThe threshold with which to compare the total weight of the matches.
    Action What to do if the total weight exceeds the Weight threshold.
    • Merge. Automatically merges the candidate records, according to the merging rules.
    • Notify. Sends a notification for a human to review the match and decide on the action to take.
    • Custom. Performs actions defined in a custom module.
    URI(Displayed if the action is Custom.) The location of the custom module.
    Function(Displayed if the action is Custom.) The name of the custom function within the custom module.
    Namespace(Displayed if the action is Custom.) The namespace of the library module where the custom function is. Blank, if the custom function is JavaScript code.

Merging

  1. In the step detail panel at the bottom, click the Merging tab.


  2. Configure the Merge Options.
    • To add a new option, click the Add button.
    • To edit an option, click the vertical ellipsis (⋮) and choose Edit Settings.
    • To delete an option, click the vertical ellipsis (⋮) and choose Delete.
    Merge Options fields
    Field Description
    Type The merge type.
    • Standard. Merging uses its own settings, instead of a defined strategy.
    • Strategy. Merging is done according to a defined strategy.
    • Custom. Merging is done using a custom function in a custom module.

    Click a merge type to view its properties:

    Standard merge properties

    Strategy merge properties

    Custom merge properties
    Property to MergeThe name of the property to merge.
    Max ValuesThe maximum number of values to allow in the merged property. Default is 99.
    Max SourcesThe maximum number of data sources from which to get values to merge.
    Source WeightsThe list of data sources and the weights assigned to them.
    Length WeightThe weight assigned to the length of a string.
    Strategy NameThe predefined strategy or set of settings to use in merging.
    URIThe location of the custom module.
    FunctionThe name of the custom function within the custom module.
    NamespaceThe namespace of the library module where the custom function is. Blank, if the custom function is JavaScript code.
  3. Configure the Merge Strategies.
    • To add a new strategy, click the Add button.
    • To edit a strategy, click the vertical ellipsis (⋮) and choose Edit Settings.
    • To delete a strategy, click the vertical ellipsis (⋮) and choose Delete.
    Merge Strategies fields

    Merge Strategy properties
    Field Description
    Default?Yes to make this strategy the default.
    NameThe name for the strategy.
    Max ValuesThe maximum number of values to allow in the merged property. Default is 99.
    Max SourcesThe maximum number of data sources from which to get values to merge.
    Source WeightsThe list of data sources and the weights assigned to them.
    Length WeightThe weight assigned to the length of a string.
  4. (Optional) Set Timestamp Path to the path to a timestamp field within the record.

    This field is used to determine which values to include in the merged property, based on their recency, up to the maximum number specified in the Max Values field in Merge Options (Standard) or in Merge Strategies.

    For example, if Max Values is set to 3, and five records meet the matching criteria, only the most recent three values will be included in the resulting record, and the two oldest values are ignored.

    Note: Namespaces used in the path must be defined within the record.
  5. Configure the Merge Collections.
    • To add a new collection tag, click the Add button.
    • To edit an collection tag, click the vertical ellipsis (⋮) and choose Edit Settings.
    • To delete an collection tag, click the vertical ellipsis (⋮) and choose Delete.
    Merge Collections fields

    Merge Strategy properties
    Field Description
    Event The event that triggers some action on a collection.
    • On Merge. When a merge occurs (automatically or manually) because of a match.
    • On No Match. If no match is found in the entire source database/file.
    • On Notification. When a notification is sent or logged.
    • On Archive. When a record is archived.
    Collections to AddOne or more collection tags to add to the default union of tags.
    Collections to RemoveOne or more collection tags to remove from the default union of tags.
    Collections to SetOne or more collection tags to replace the default union of tags.