Create a Flow Using Gradle

Overview

The Gradle task hubCreateFlow can create a flow configuration file in the QuickStart format or in the Hub Central format.

Learn more about the differences between the two formats: Learn more: About Flow and Step Configuration Structures

Before you begin

You need:

Procedure

  1. Open a command-line window, and go to your project root directory.
  2. At your project root, run the Gradle task hubCreateFlow.
    ./gradlew hubCreateFlow -PflowName=YourFlowName -PwithInlineSteps=true -igradlew.bat hubCreateFlow -PflowName=YourFlowName -PwithInlineSteps=true -i
    flowName
    (Required) The name of the flow to create.
    withInlineSteps
    • To create a flow in the QuickStart format, set to true. The flow configuration includes examples of the different step types.
    • To create a flow in the Hub Central format, set to false. The flow configuration includes only references to the steps.

      Example of a step reference:

         "stepId" : "yourstepname-yoursteptype"

    The default is false (Hub Central format).

Results

View an example.
   {
    "name": "your-flow-name",
    "description": "This is the default flow containing all of the default steps",
    "batchSize": 100,
    "threadCount": 4,
    "options": {
      "sourceQuery": null
    },
    "steps": {
      "1": {
        "name": "ingestion-step",
        "description": "This is the default ingestion step",
        "stepDefinitionName": "default-ingestion",
        "stepDefinitionType": "INGESTION",
        "customHook": {},
        "fileLocations": {
          "inputFilePath": "path/to/folder",
          "outputURIReplacement": "output/URI,'substitute/URI'",
          "inputFileType": "json"
        },
        "options": {
          "targetDatabase": "data-hub-STAGING",
          "sourceQuery": null,
          "outputFormat": "json",
          "collections": [
            "default-ingestion"
          ]
        }
      },
      "2": {
        "name": "mapping-step",
        "description": "This is the default mapping step",
        "stepDefinitionName": "default-mapping",
        "stepDefinitionType": "MAPPING",
        "options": {
          "sourceDatabase": "data-hub-STAGING",
          "targetDatabase": "data-hub-FINAL",
          "sourceQuery": "cts.collectionQuery('default-ingestion')",
          "collections": [
            "default-mapping",
            "mdm-content"
          ],
          "targetEntity": "entity-name",
          "mapping": {
            "name": "mapping-name",
            "version": 1
          }
        }
      },
      "3": {
        "name": "mastering-step",
        "description": "This is the default mastering step",
        "stepDefinitionName": "default-mastering",
        "stepDefinitionType": "MASTERING",
        "options": {
          "sourceDatabase": "data-hub-FINAL",
          "targetDatabase": "data-hub-FINAL",
          "targetEntity": "entity-name",
          "sourceQuery": "cts.andQuery([cts.collectionQuery('default-mapping'),cts.collectionQuery('mdm-content')])",
          "collections": [
            "default-mastering, mastered"
          ],
          "mergeOptions" : {
            "matchOptions" : "",
            "propertyDefs" : {
              "properties" : [ ],
              "namespaces" : { }
            },
            "algorithms" : {
              "stdAlgorithm" : {
                "timestamp" : { }
              },
              "custom" : [ ],
              "collections" : { }
            },
            "mergeStrategies" : [ ],
            "merging" : [ ]
          },
          "matchOptions" : {
            "dataFormat": "json",
            "propertyDefs": {
              "property": []
            },
            "algorithms": {
              "algorithm": []
            },
            "collections": {
              "content": []
            },
            "scoring": {
              "add": [],
              "expand": [],
              "reduce": []
            },
            "actions": {
              "action": []
            },
            "thresholds": {
              "threshold": []
            },
            "tuning": {
              "maxScan": 200
            }
          }
        }
      }
    }
  }