Flow Configuration Structure (QuickStart Format)
Information in a flow configuration file (QuickStart format) with inline steps.
Components of the Flow Configuration Structure
Flow configuration files are found in your-project-root/flows.
Flow Settings
{
"name" : "MyFlow",
"description" : "This flow contains examples of steps plus additional settings.",
"batchSize" : 100,
"threadCount" : 4,
"stopOnError" : false,
"options" : {
"sourceQuery" : null,
"provenanceGranularityLevel" : "fine",
},
"disableJobOutput" : "false",
"permissions" : "data-hub-common,read,data-hub-common,update",
"steps" : {
"1" : { ... },
"2" : { ... },
"3" : { ... },
"4" : { ... }
}
}
Field | Description |
---|---|
name | The human-friendly name of the flow. |
description | A description of the flow. |
batchSize | The number of documents to process per batch. A smaller batch size provides finer granularity in the jobs reporting. However, a smaller batch file also costs more because of the processing overhead. The recommended batch size for merging is 1. |
threadCount | The number of threads to use when running a flow. The default is 4. |
stopOnError |
If true and an error is encountered, the flow run ends, the rest of the source data is ignored, and the remaining steps are not performed.
Information about the failure is logged in the job document.
The default is false .
|
options | Key-value pairs to pass as parameters to custom modules in every step in the flow. |
options » sourceCollection | The collection tag to use to search for the records to process in this step. |
options » sourceQuery |
The CTS query to use to select the source data to process.
To filter by a collection tag, use Learn more: CTS Query. |
options » provenanceGranularityLevel |
The granularity of the provenance tracking information: coarse (default) to store document-level provenance information only, fine to store document-level and property-level provenance information, or off to disable provenance tracking in future job runs.
Applies only to mapping, matching, merging, mastering, and custom steps.
Learn more: About Provenance and Lineage.
|
disableJobOutput |
If true , the job document is not created.
The default is false .
|
permissions |
The permissions required to access the documents created by the flow.
The string must be in the format |
steps | The list of steps to run within the flow. Each step in the flow has a sequence number and a customized copy of the step definition. The step definition differs based on the step type (ingestion, mapping, matching, merging, mastering, or custom). |
Common Step Settings
The following settings are common among all step types.
"..." : {
"name" : "...",
"description" : "This is my step.",
"stepDefinitionName" : "default-...",
"stepDefinitionType" : "...",
"batchSize" : 100,
"threadCount" : 4,
"customHook" : {
"module" : "/custom-modules/your-step-type/your-hook-directory/your-hook-module-name.sjs",
"parameters" : {},
"user" : "flow-operator",
"runBefore" : false
},
"options" : {
"stepUpdate" : false,
"acceptsBatch" : false,
...
"collections" : [ "default-..." ],
"additionalCollections" : [],
...
},
...
},
Field | Description |
---|---|
name | The name of the step instance. |
description | A description of the step. |
stepDefinitionName |
The name of a step definition, which could be one of the default step definitions or a custom one.
Custom step definitions can be created using QuickStart or using the Gradle task hubCreateStepDefinition.
Tip: If you are customizing a default step type (ingestion, mapping, or mastering), leave the value as
default-ingestion , default-mapping , or default-mastering . |
stepDefinitionType | The type of the step definition: INGESTION, MAPPING, MATCHING, MERGING, MASTERING, or CUSTOM. |
batchSize |
The number of documents to process per batch.
A smaller batch size provides finer granularity in the jobs reporting. However, a smaller batch file also costs more because of the processing overhead.
The recommended batch size for merging is 1.
If not defined or if set to 0 or null in the step settings, the value in the flow settings is used. |
threadCount |
The number of threads to use when running a flow.
The default is 4.
If not defined or if set to 0 or null in the step settings, the value in the flow settings is used. |
customHook |
A step add-on that performs additional processes in its own transaction before or after the core step transaction. Results are saved within a transaction.
Learn more: Creating a Custom Hook Module and Adding a Custom Hook to a Step Manually. Important: Custom hooks are deprecated.
|
customHook » module | The path to your custom hook module. |
customHook » parameters | Parameters, as key-value pairs, to pass to your custom hook module. |
customHook » user |
The user account to use to run the module. The default is the user running the flow; e.g., flow-operator .
|
customHook » runBefore |
For a pre-step hook, set to true . For a post-step hook, set to false .
|
options » stepUpdate |
If true , custom modules can make changes directly to records in the database (inserting, deleting, or locking); otherwise, custom modules can make changes indirectly by passing content objects to Data Hub APIs.
Direct changes to the database are rarely needed.
For a combined mastering step, the default is true . For all other types of steps, the default is false .
|
options » acceptsBatch |
If true , all the records in the batch are processed within a single step run; otherwise, the step is restarted and run for each record in the batch.
The default is true .
|
options » collections | The list of collection tags to assign to the resulting records. The default list is comprised of a collection with the same name as the step. |
options » additionalCollections | The list of collection tags to assign to the resulting records, in addition to the default collections. |
Ingestion Step Settings
"1" : {
"name" : "MyIngestionStep",
"description" : "This is my ingestion step.",
"stepDefinitionName" : "default-ingestion",
"stepDefinitionType" : "INGESTION",
"batchSize" : 100,
"threadCount" : 4,
"customHook" : {
"module" : "/custom-modules/your-step-type/your-hook-directory/your-hook-module-name.sjs",
"parameters" : {},
"user" : "flow-operator",
"runBefore" : false
},
"options" : {
"stepUpdate" : false,
"acceptsBatch" : false,
"targetDatabase" : "data-hub-STAGING",
"collections" : [ "default-ingestion" ],
"additionalCollections" : [],
"outputFormat" : "json"
},
"fileLocations" : {
"inputFilePath" : "path/to/folder",
"inputFileType" : "json",
"outputURIPrefix" : "output/URI,'substitute/URI'"
}
},
Field | Description |
---|---|
options » targetDatabase | The STAGING database where you want to store the ingested data. The default is data-hub-STAGING . |
options » collections | The list of collection tags to assign to the resulting records. The default list is comprised of a collection with the same name as the step. |
options » additionalCollections | The list of collection tags to assign to the resulting records, in addition to the default collections. |
options » outputFormat | The format of the processed record: Text, JSON, XML, or Binary. The default is JSON. |
fileLocations » inputFilePath | The location of your source files. |
fileLocations » inputFileType |
The format of your source files: Text, JSON, XML, Binary, or Delimited Text.
The default is JSON.
Note: If you select Text, the source file's content must be in either JSON or XML format.
|
fileLocations » outputURIPrefix |
A comma-separated list of replacements used to customize the URIs of the ingested records.
The list is comprised of regular expression patterns and their replacement strings in the format For example, if the original URI is in the form "/foo/bar/filename", you can customize it to be "/mydir/filename" using the following comma-separated list:
Java's regular expression language is supported. If Source File Type is set to |
Common Step Settings |
Mapping Step Settings
"2" : {
"name" : "MyMappingStep",
"description" : "This is my mapping step.",
"stepDefinitionName" : "default-mapping",
"stepDefinitionType" : "MAPPING",
"batchSize" : 100,
"threadCount" : 4,
"customHook" : {
"module" : "/custom-modules/your-step-type/your-hook-directory/your-hook-module-name.sjs",
"parameters" : {},
"user" : "flow-operator",
"runBefore" : false
},
"options" : {
"stepUpdate" : false,
"acceptsBatch" : false,
"sourceDatabase" : "data-hub-STAGING",
"sourceCollection" : "MyIngestionStep",
"sourceQuery" : "cts.collectionQuery('my-custom-query')",
"constrainSourceQueryToJob" : false,
"targetEntityType" : "MyEntity",
"validateEntity" : false,
"targetDatabase" : "data-hub-FINAL",
"collections" : [ "default-mapping" ],
"additionalCollections" : [],
"outputFormat" : "json",
"provenanceGranularityLevel" : "fine",
"mapping" : {
"name" : "mapping-name",
"version" : "1"
}
}
},
Field | Description |
---|---|
options » sourceDatabase |
The database from which to take the input data.
Choose the STAGING database where you stored ingested data. The default is data-hub-STAGING .
|
options » sourceCollection | The collection tag to use to search for the records to process in this step. |
options » sourceQuery |
The CTS query to use to select the source data to process.
To filter by a collection tag, use Learn more: CTS Query. |
options » constrainSourceQueryToJob |
If true , the query is applied to the documents that were created or modified in the same job that executes the step. Otherwise, the query disregards the job ID.
For example, if sourceQuery is cts.collectionQuery('example') and constrainSourceQueryToJob is true , the query searches for documents that are in the example collection and were created or modified in the current job.
The default is false .
|
options » targetEntityType | The entity to map against the source data. Required only if the flow includes a mapping step. |
options » validateEntity |
Indicates whether to validate the mapped entity instance against the schema document based on the entity model, and what action to take.
Set to false to skip validation. Set to accept to write the mapped entity instance to the database regardless of the validation result. Set to reject to skip writing the mapped entity instance to the database if the validation failed.
The default is false .
Learn more: about-mapping.html#about-mapping__validation-of-mapped-expressions.
|
options » targetDatabase | The FINAL database where you want to store mapped data. The default is data-hub-FINAL . |
options » outputFormat | The format of the processed record: Text, JSON, XML, or Binary. The default is JSON. |
options » provenanceGranularityLevel |
The granularity of the provenance tracking information: coarse (default) to store document-level provenance information only, fine to store document-level and property-level provenance information, or off to disable provenance tracking in future job runs.
Applies only to mapping, matching, merging, mastering, and custom steps.
Learn more: About Provenance and Lineage.
|
options » mapping |
How to map the properties of the targetEntityType to the fields of the source data.
|
options » mapping » name | The name of your mapping that is defined in your-project-root/mappings/your-mapping-name/mapping.version.json. |
options » mapping » version | The version of the mapping to use. Your mapping must be defined in your-project-root/mappings/your-mapping-name/mapping.version.json. |
Common Step Settings |
Matching Step Settings
"3" : {
"name" : "MyMatchingStep",
"description" : "This is my matching step.",
"stepDefinitionName" : "default-matching",
"stepDefinitionType" : "MATCHING",
"batchSize" : 100,
"threadCount" : 4,
"customHook" : {
"module" : "/custom-modules/your-step-type/your-hook-directory/your-hook-module-name.sjs",
"parameters" : {},
"user" : "flow-operator",
"runBefore" : false
},
"options" : {
"stepUpdate" : false,
"acceptsBatch" : false,
"sourceDatabase" : "data-hub-FINAL",
"sourceCollection" : "MyMappingStep",
"sourceQuery" : "cts.collectionQuery('my-custom-query')",
"constrainSourceQueryToJob" : false,
"targetEntityType" : "MyEntity",
"targetDatabase" : "data-hub-FINAL",
"collections" : [ "MyMatchingStep", "MyPersonEntity" ],
"additionalCollections" : [],
"provenanceGranularityLevel" : "fine",
"matchOptions" : { ... }
}
},
Field | Description |
---|---|
options » sourceDatabase |
The database from which to take the input data.
Choose the FINAL database where you stored mapped data. The default is data-hub-FINAL .
|
options » sourceCollection | The collection tag to use to search for the records to process in this step. |
options » sourceQuery |
The CTS query to use to select the source data to process.
To filter by a collection tag, use Learn more: CTS Query. |
options » constrainSourceQueryToJob |
If true , the query is applied to the documents that were created or modified in the same job that executes the step. Otherwise, the query disregards the job ID.
For example, if sourceQuery is cts.collectionQuery('example') and constrainSourceQueryToJob is true , the query searches for documents that are in the example collection and were created or modified in the current job.
The default is false .
|
options » targetEntityType | The entity to map against the source data. Required only if the flow includes a mapping step. |
options » targetDatabase |
The same database you selected in Source Database. The default is data-hub-FINAL .
Important: For split mastering (matching step and merging step), both the source database and the target database for both steps must be the same.
|
options » provenanceGranularityLevel |
The granularity of the provenance tracking information: coarse (default) to store document-level provenance information only, fine to store document-level and property-level provenance information, or off to disable provenance tracking in future job runs.
Applies only to mapping, matching, merging, mastering, and custom steps.
Learn more: About Provenance and Lineage.
|
Matching Options
"matchOptions" : {
"dataFormat" : "json",
"propertyDefs" : {
"property" : [
{
"name" : "ssn",
"namespace" : "",
"localname" : "IdentificationID"
},
]
},
"algorithms" : {
"algorithm" : [
{
"name" : "std-reduce",
"function" : "standard-reduction",
"namespace" : "",
"at" : ""
},
]
},
"collections" : {
"content" : [ "my-content-collection" ]
},
"scoring" : {
"add" : [
{
"propertyName" : "ssn",
"weight" : "50"
},
],
"expand" : [
{
"propertyName" : "first-name",
"algorithmRef" : "thesaurus",
"weight" : "6",
"thesaurus" : "/mdm/config/thesauri/first-name-synonyms.xml"
},
{
"propertyName" : "last-name",
"algorithmRef" : "dbl-metaphone",
"weight" : "8",
"dictionary" : "name-dictionary.xml",
"distanceThreshold" : "50"
}
],
"reduce" : [
{
"algorithmRef" : "std-reduce",
"weight" : "4",
"allMatch" : { "property" : ["last-name", "addr1"] }
}
]
},
"actions" : {
"action" : [
{
"name" : "my-custom-action",
"function" : "custom-action",
"namespace" : "http://marklogic.com/smart-mastering/action",
"at" : "/custom-action.xqy"
}
]
},
"thresholds" : {
"threshold" : [
{ "above" : "30", "label" : "Possible Match" },
{ "above" : "50", "label" : "Likely Match", "action" : "notify" },
{ "above" : "68", "label" : "Definitive Match", "action" : "merge" },
{ "above" : "75", "label" : "Custom Match", "action" : "my-custom-action" }
]
},
"tuning" : {
"maxScan" : 200
}
},
Field | Description |
---|---|
matchOptions | The settings used to find potential matches. Learn more: Smart Mastering Core - Matching Options |
dataFormat | The format of your source records: Text, JSON, XML, or Binary. The default is JSON. |
propertyDefs | Definitions of properties to compare. |
propertyDefs » property » name | The alias for this property definition. |
propertyDefs » property » namespace | (Optional) The namespace that encompasses the XML element or JSON property (record field) to compare. |
propertyDefs » property » localname | The name of the XML element or JSON property (record field) to compare. |
algorithms | Definitions of algorithms that compare values. Each algorithm corresponds to a match type (Exact, Synonym, Double Metaphone, Reduce, Zip, and Custom). The default algorithm is that of the Exact match type, which determines whether two values are equal. |
algorithms » algorithm » name | The alias for this algorithm definition. |
algorithms » algorithm » function | The function to run if this algorithm definition is selected. |
algorithms » algorithm » namespace | (Optional) The namespace of the module that contains the function. |
algorithms » algorithm » at | The path to the module that contains the function. |
collections | A set of collections that overrides the default collection used to determine the scope of the dataset being compared. If multiple content elements are specified, the dataset is restricted to an intersection of those collections. |
collections » content | One or more collections used to determine the dataset to be compared. |
scoring | Rules (add, expand, reduce) that define how the comparison is scored based on assigned weights. The maximum possible score is the sum of the weights of all of the weight attributes. The match process uses the simple scoring option, with the property weight controlling how much influence each should have. Learn more: Relevance Scores. |
scoring » add | Properties whose values are simply compared between the records and, if the values match exactly, the assigned weight is added to the score. |
scoring » add » propertyName | The alias of a property definition under the matchOptions/propertyDefs node of this step. |
scoring » add » weight | The weight added to the score if the property values of two records match exactly. |
scoring » expand | Properties whose values are compared using a different algorithm to determine a match. For example, the property values can be considered a positive match if one is a synonym of the other or if both values phonetically sound alike. If so, the assigned weight is added to the score. |
scoring » expand » propertyName | The alias of a property definition under the matchOptions/propertyDefs node of this step. |
scoring » expand » algorithmRef | The alias of an algorithm definition under the matchOptions/algorithms node of this step. |
scoring » expand » weight | The weight added to the score if the property values of two records are considered a match based on the selected algorithm. |
scoring » expand » thesaurus | The location of the thesaurus that is stored in a MarkLogic Server database and used to determine synonyms. Learn more: Managing Thesaurus Documents |
scoring » expand » dictionary | The location of the phonetic dictionary that is stored in a database and used when comparing words phonetically. Learn more: Custom Dictionaries |
scoring » expand » distanceThreshold | The threshold below which the phonetic difference (distance) between two strings is considered insignificant; i.e., the strings are similar to each other. Learn more: spell functions |
scoring » reduce | Combinations of properties whose matching values might be a false match. For example, two members of the same family with the same last names and addresses might be misinterpreted as being the same person. In this case, the score is reduced by the assigned weight to give the match less importance. |
scoring » reduce » algorithmRef | The alias of an algorithm definition under the matchOptions/algorithms node of this step. |
scoring » reduce » weight | A positive integer that denotes how much to reduce the weight of a match. |
scoring » reduce » allMatch | The combination of properties that might falsely indicate a match if the values of these properties are equal between two records. |
actions | Custom actions that can be performed when a threshold is reached. Learn more: Custom Match Actions |
actions » action | The custom action to perform when a threshold is reached. |
actions » action » name | The alias for this action definition. |
actions » action » function | The function to run if this action definition is selected. |
actions » action » namespace | (Optional) The namespace of the module that contains the function. |
actions » action » at | The path to the module that contains the function. |
thresholds | Score thresholds that trigger an action. |
thresholds » threshold | A score threshold definition, including the action to perform if the threshold is exceeded. |
thresholds » threshold » above | The score threshold. If the match score exceeds this value, the action is performed. |
thresholds » threshold » label | The alias for this threshold definition. |
thresholds » threshold » action |
The action to perform if the score is above the threshold. Possible values:
|
tuning » maxScan | The maximum number of highest scoring potential matches that will be considered for merging. |
Merging Step Settings
"4" : {
"name" : "MyMergingStep",
"description" : "This is my merging step.",
"stepDefinitionName" : "default-merging",
"stepDefinitionType" : "MERGING",
"batchSize" : 100,
"threadCount" : 4,
"customHook" : {
"module" : "/custom-modules/your-step-type/your-hook-directory/your-hook-module-name.sjs",
"parameters" : {},
"user" : "flow-operator",
"runBefore" : false
},
"options" : {
"stepUpdate" : false,
"acceptsBatch" : false,
"sourceDatabase" : "data-hub-FINAL",
"sourceCollection" : "MyMatchingStep",
"sourceQuery" : "cts.collectionQuery('my-custom-query')",
"constrainSourceQueryToJob" : false,
"targetEntityType" : "MyEntity",
"targetDatabase" : "data-hub-FINAL",
"collections" : [ "MyMergingStep", "MyPersonEntity" ],
"additionalCollections" : [],
"outputFormat" : "json",
"provenanceGranularityLevel" : "fine",
"mergeOptions" : { ... }
}
},
Field | Description |
---|---|
options » sourceDatabase |
The database from which to take the input data.
Choose the same source database that you selected in the matching step. The default is data-hub-FINAL .
|
options » sourceCollection | The collection tag to use to search for the records to process in this step. |
options » sourceQuery |
The CTS query to use to select the source data to process.
To filter by a collection tag, use Learn more: CTS Query. |
options » constrainSourceQueryToJob |
If true , the query is applied to the documents that were created or modified in the same job that executes the step. Otherwise, the query disregards the job ID.
For example, if sourceQuery is cts.collectionQuery('example') and constrainSourceQueryToJob is true , the query searches for documents that are in the example collection and were created or modified in the current job.
The default is false .
|
options » targetEntityType | The entity to map against the source data. Required only if the flow includes a mapping step. |
options » targetDatabase |
The same database you selected in Source Database. The default is data-hub-FINAL .
Important: For split mastering (matching step and merging step), both the source database and the target database for both steps must be the same.
|
options » outputFormat | The format of the processed record: Text, JSON, XML, or Binary. The default is JSON. |
options » provenanceGranularityLevel |
The granularity of the provenance tracking information: coarse (default) to store document-level provenance information only, fine to store document-level and property-level provenance information, or off to disable provenance tracking in future job runs.
Applies only to mapping, matching, merging, mastering, and custom steps.
Learn more: About Provenance and Lineage.
|
Merging Options
"mergeOptions" : {
"matchOptions" : "mlw-match",
"propertyDefs" : {
"properties" : [
{
"name" : "ssn",
"localname" : "IdentificationID",
"namespace" : ""
},
{
"name" : "shallow",
"path" : "/es:envelope/es:headers/shallow"
}
],
"namespaces" : {
"has" : "has",
"m" : "http://marklogic.com/smart-mastering/merging",
"es" : "http://marklogic.com/entity-services"
}
},
"algorithms" : {
"stdAlgorithm" : {
"timestamp" : { "path" : "/es:envelope/es:headers/sm:sources/sm:source/sm:dateTime" },
"namespaces" : {
"sm" : "http://marklogic.com/smart-mastering",
"es" : "http://marklogic.com/entity-services"
}
},
"custom" : [
{
"name" : "customMerge",
"function" : "doCustomMerge",
"namespace" : "http://marklogic.com/smart-mastering/merging",
"at" : "/custom-merge-xqy.xqy"
}
],
"collections" : {
"onMerge" : {
"function" : "collections",
"namespace" : "test/merge-collection-algorithm",
"at" : "/test/suites/customizing-collections/lib/merged-collections.xqy"
},
"onArchive" : {
"remove" : { "collection" : ["Entity"] },
"add" : { "collection" : ["custom-archived"] }
},
"onNoMatch" : {
"function" : "noMatchCollections",
"namespace" : "",
"at" : "/test/suites/customizing-collections/lib/noMatchCollections.sjs"
},
"onNotification" : {
"set" : { "collection" : ["notification"] }
}
}
},
"mergeStrategies" : [
{
"name" : "crm-source-weight",
"algorithmRef" : "standard",
"sourceWeights" : [
{
"source" : {
"name" : "CRM",
"weight" : "10"
}
}
]
},
{
"name" : "length-weight",
"algorithmRef" : "standard",
"maxValues" : "1",
"length" : { "weight" : "10" }
}
],
"merging" : [
{
"propertyName" : "ssn",
"maxValues" : "1",
"maxSources" : "1",
"strategy" : "crm-source-weight"
},
{
"propertyName" : "name",
"maxValues" : "1",
"doubleMetaphone" : {
"distanceThreshold" : "50"
},
"synonymsSupport" : "true",
"thesaurus" : "/mdm/config/thesauri/first-name-synonyms.xml",
"length" : { "weight" : "8" }
},
{
"propertyName" : "dob",
"maxValues" : "1",
"algorithmRef" : "standard",
"sourceWeights" : {
"source" : {
"name" : "better-source",
"weight" : "4"
}
}
},
{
"default" : "true",
"strategy" : "crm-source-weight"
}
],
"tripleMerge" : {
"function" : "custom-trips",
"namespace" : "http://marklogic.com/smart-mastering/merging",
"at" : "/custom-triple-merge.xqy",
"some-param" : 3
}
}
Field | Description |
---|---|
mergeOptions | The settings used to merge records that match. Learn more: Smart Mastering Core - Merging Options. |
matchOptions | The name of a set of match options that were previously stored in the server. Learn more: Saving Options. |
propertyDefs | Definitions of properties to merge. |
propertyDefs » properties » name | The alias for this property definition. |
propertyDefs » properties » localname | The name of the XML element or JSON property (record field) to merge. |
propertyDefs » properties » namespace |
(Optional) The namespace that encompasses the XML element or JSON property (record field) to merge. |
propertyDefs » properties » path |
Path leading to the headers or instance sections of records, where the merge properties are defined.
|
propertyDefs » namespaces | Key-value pairs that assign aliases to namespaces. The key is the alias and the value is the full namespace. |
algorithms | Definitions of algorithms that merge values. |
algorithms » stdAlgorithm | The standard algorithm that implements the default merge behavior. |
algorithms » stdAlgorithm » timestamp | The path to a timestamp field within the record. This field is used to determine which values to include in the merged property, based on their recency, up to the maximum number specified in the Max Values field in Merge Options (Standard) or in Merge Strategies. Namespaces used in the path must be defined within the record. |
algorithms » stdAlgorithm » namespaces | (Optional) Key-value pairs that assign aliases to namespaces. The key is the alias and the value is the full namespace. |
algorithms » custom | Definitions of custom algorithms that merge values. |
algorithms » custom » name | The alias for this custom algorithm definition. |
| The custom merge function to run. |
algorithms » custom » namespace | (Optional) The namespace of the module that contains the function. |
algorithms » custom » at | The path to the module that contains the function. |
algorithms » collections | Rules that specify how collection tags are managed when an event occurs. |
algorithms » collections » onMerge |
How collection tags are applied to the new record that was created when matching records are merged.
The default set of collection tags is comprised of:
|
algorithms » collections » onMerge » function | The function that manages collection tags if the event occurs. |
algorithms » collections » onMerge » namespace | (Optional) The namespace of the module that contains the function. |
algorithms » collections » onMerge » at | The path to the module that contains the function. |
algorithms » collections » onArchive |
How collection tags are applied to the original records after their content have been merged into a new record.
The default set of collection tags is comprised of:
|
algorithms » collections » onArchive » remove | One or more collection tags to remove from the default union of tags. |
algorithms » collections » onArchive » add | One or more collection tags to add to the default union of tags. |
algorithms » collections » onNoMatch |
How collection tags are applied to records that were not merged because no matches were found or because the total matching scores did not exceed the defined thresholds.
The default set of collection tags is comprised of:
|
algorithms » collections » onNoMatch » function | The function that manages collection tags if the event occurs. |
algorithms » collections » onNoMatch » namespace | (Optional) The namespace of the module that contains the function. |
algorithms » collections » onNoMatch » at | The path to the module that contains the function. |
algorithms » collections » onNotification |
How collection tags are applied to notification records. The default set of collection tags is comprised of mdm-notification only.
|
algorithms » collections » onNotification » set | One or more collection tags to replace the default union of tags. |
mergeStrategies | Predefined configurations for merging. |
mergeStrategies » name | The name for the strategy. |
mergeStrategies » algorithmRef | The alias of an algorithm definition under the mergeOptions/algorithms node of this step. |
mergeStrategies » sourceWeights |
The list of data sources and the weights assigned to them.
If the set of matching records come from more sources than maxSources , the source weights are used to determine which records are included in the merge.
Example: If maxSources is set to 1, only records from the highest weighted source are included in the merge.
|
mergeStrategies » sourceWeights » source » name | The name of the source, exactly as shown in the envelopes of the records under headers » sources. |
mergeStrategies » sourceWeights » source » weight | The weight used to decide the priority of a source when merging. |
mergeStrategies » maxValues | The maximum number of values to allow in the merged property. The default is 99. |
mergeStrategies » length » weight | The weight assigned to the length of a string. |
merging | Rules that specify how to merge records that match. |
merging » propertyName | The alias of a property definition under the mergeOptions/propertyDefs node of this step. |
merging » maxValues | The maximum number of values to allow in the merged property. The default is 99. |
merging » maxSources |
The maximum number of data sources from which to get values to merge.
For example, to copy values from a single source, set maxSources to 1.
|
merging » strategy | The alias of a strategy definition under the mergeOptions/mergeStrategies node of this step. |
merging » doubleMetaphone | If this setting is present, the Double Metaphone algorithm is used to determine the values to merge. |
merging » doubleMetaphone » distanceThreshold | The threshold below which the phonetic difference (distance) between two strings is considered insignificant; i.e., the strings are similar to each other. Learn more: spell functions |
merging » synonymsSupport |
If true , synonyms are included in the list of values to merge. Synonyms are determined using the specified thesaurus.
|
merging » thesaurus | The location of the thesaurus that is stored in a MarkLogic Server database and used to determine synonyms. Learn more: Managing Thesaurus Documents |
merging » length | The weight assigned to the length of a string. |
merging » algorithmRef | The alias of an algorithm definition under the mergeOptions/algorithms node of this step. |
merging » sourceWeights |
The list of data sources and the weights assigned to them.
If the set of matching records come from more sources than maxSources , the source weights are used to determine which records are included in the merge.
Example: If maxSources is set to 1, only records from the highest weighted source are included in the merge.
|
merging » sourceWeights » source » name | The name of the source, exactly as shown in the envelopes of the records under headers » sources. |
merging » sourceWeights » source » weight | The weight used to decide the priority of a source when merging. |
merging » default |
If true , the specified strategy is the default.
Important: If this setting is present, do not include a propertyName setting.
|
merging » strategy | The alias of a strategy definition under the mergeOptions/mergeStrategies node of this step. |
tripleMerge | Definition of an algorithm that merges triples. |
tripleMerge » function | The function that merges triples. |
tripleMerge » namespace | The namespace of the module that contains the function. |
tripleMerge » at | The path to the module that contains the function. |
tripleMerge » some-param | Parameters, as key-value pairs, to pass to your triple merge function. |
Mastering Step Settings
"5" : {
"name" : "MyMasteringStep",
"description" : "This is my mastering step.",
"stepDefinitionName" : "default-mastering",
"stepDefinitionType" : "MASTERING",
"batchSize" : 100,
"threadCount" : "1",
"customHook" : {
"module" : "/custom-modules/your-step-type/your-hook-directory/your-hook-module-name.sjs",
"parameters" : {},
"user" : "flow-operator",
"runBefore" : false
},
"options" : {
"stepUpdate" : false,
"acceptsBatch" : false,
"sourceDatabase" : "data-hub-FINAL",
"sourceCollection" : "MyMappingStep",
"sourceQuery" : "cts.collectionQuery('my-custom-query')",
"constrainSourceQueryToJob" : false,
"targetEntityType" : "MyEntity",
"targetDatabase" : "data-hub-FINAL",
"collections" : [ "default-mastering, mastered" ],
"additionalCollections" : [],
"outputFormat" : "json",
"provenanceGranularityLevel" : "fine",
"matchOptions" : { ... },
"mergeOptions" : { ... }
}
},
Field | Description |
---|---|
options » sourceDatabase |
The database from which to take the input data.
Choose the FINAL database where you stored mapped data. The default is data-hub-FINAL .
|
options » sourceCollection | The collection tag to use to search for the records to process in this step. |
options » sourceQuery |
The CTS query to use to select the source data to process.
To filter by a collection tag, use Learn more: CTS Query. |
options » constrainSourceQueryToJob |
If true , the query is applied to the documents that were created or modified in the same job that executes the step. Otherwise, the query disregards the job ID.
For example, if sourceQuery is cts.collectionQuery('example') and constrainSourceQueryToJob is true , the query searches for documents that are in the example collection and were created or modified in the current job.
The default is false .
|
options » targetEntityType | The entity to map against the source data. Required only if the flow includes a mapping step. |
options » targetDatabase |
The FINAL database where you want to store mastered data. The default is data-hub-FINAL .
Important: For combined mastering (mastering step), the source database and the target database should be the same. If duplicates are found, the original records are archived and the merged version is added to the same database. If you want the target database to be different, you can create a custom step with a custom module to override the default behavior of the mastering step.
|
options » provenanceGranularityLevel |
The granularity of the provenance tracking information: coarse (default) to store document-level provenance information only, fine to store document-level and property-level provenance information, or off to disable provenance tracking in future job runs.
Applies only to mapping, matching, merging, mastering, and custom steps.
Learn more: About Provenance and Lineage.
|
Custom Step Settings
"9" : {
"name" : "MyCustomOtherStep",
"description" : "This is my custom-other step.",
"stepDefinitionName" : "custom-step-def",
"stepDefinitionType" : "CUSTOM",
"batchSize" : 100,
"threadCount" : 4,
"customHook" : {
"module" : "/custom-modules/your-step-type/your-hook-directory/your-hook-module-name.sjs",
"parameters" : {},
"user" : "flow-operator",
"runBefore" : false
},
"options" : {
"stepUpdate" : false,
"acceptsBatch" : false,
"sourceDatabase" : "data-hub-STAGING",
"sourceCollection" : "my-collection-tag",
"sourceQuery" : "cts.collectionQuery('my-custom-query')",
"constrainSourceQueryToJob" : false,
"targetEntityType" : "MyEntity",
"targetDatabase" : "data-hub-FINAL",
"collections" : [ "my-collection-tag" ],
"additionalCollections" : [],
"outputFormat" : "json"
}
}
Field | Description |
---|---|
options » sourceDatabase |
The database from which to take the input data.
|
options » sourceCollection | The collection tag to use to search for the records to process in this step. |
options » sourceQuery |
The CTS query to use to select the source data to process.
To filter by a collection tag, use Learn more: CTS Query. |
options » constrainSourceQueryToJob |
If true , the query is applied to the documents that were created or modified in the same job that executes the step. Otherwise, the query disregards the job ID.
For example, if sourceQuery is cts.collectionQuery('example') and constrainSourceQueryToJob is true , the query searches for documents that are in the example collection and were created or modified in the current job.
The default is false .
|
options » targetEntityType | The entity to map against the source data. Required only if the flow includes a mapping step. |
options » targetDatabase |
The database where you want to store the processed data.
Important:
For combined mastering (mastering step), the source database and the target database should be the same. If duplicates are found, the original records are archived and the merged version is added to the same database. If you want the target database to be different, you can create a custom step with a custom module to override the default behavior of the mastering step. For split mastering (matching step and merging step), both the source database and the target database for both steps must be the same. |
options » outputFormat | The format of the processed record: Text, JSON, XML, or Binary. The default is JSON. |
Common Step Settings |