Create Steps Using Gradle - QS Format

Overview

A flow must have at least one step.

  • For Ingestion, Mapping, Matching, Merging, and Mastering steps: Create a flow using Gradle with the -PwithInlineSteps=true option. The resulting flow configuration file includes an example step for each predefined type of step. Simply customize these example steps.
  • For Custom steps: You can create the flow separately, run the Gradle task hubCreateStepDefinition, and manually copy and insert the step definition into the flow configuration file.

The options for custom step types are different between QuickStart and Gradle:

QuickStart
Custom Step Type field selections		 Gradle
hubCreateStepDefinition -PstepDefType options
Ingestion ingestion
  • Mapping
  • Mastering
  • Other
 custom
Note: Steps that are configured directly in the flow configuration file, as well as steps that use the step definition created by hubCreateStepDefinition, are equivalent to the custom steps generated by QuickStart.

Before you begin

You need:

About this task

Depending on the type of the step to create, this topic includes the following procedures:

Before creating a step using Gradle, you need:

  • Ingestion / Mapping / Matching / Merging / Mastering

    When you created the flow using the Gradle task hubCreateFlow, examples for each default step type are already embedded in the flow configuration file. Simply edit those examples or use them as templates to add more steps.

  • Custom-Ingestion / Custom-Other

    To create Custom steps in Gradle:

    1. Using Gradle, create the step definition.
      ./gradlew hubCreateStepDefinition -PstepDefName=yourstepname -PstepDefType=[ingestion|custom] -Pformat=[sjs|xqy] -igradlew.bat hubCreateStepDefinition -PstepDefName=yourstepname -PstepDefType=[ingestion|custom] -Pformat=[sjs|xqy] -i
      stepDefName
      (Required) The name of the custom step definition to create.
      stepDefType
      The type of the custom step definition to create:
      • ingestion (To create a Custom-Ingestion step.)
      • custom (To create a Custom-Mapping, Custom-Mastering, or Custom-Other step.)

      The default is custom.

      format
      The format of the module to associate with the new step definition: xqy for XQuery or sjs for JavaScript. The default is sjs.

      A module is created under your-project-root/src/main/ml-modules and is associated with the step definition to perform the processes required for the step; for example, you can create a module to wrap each document in your own custom envelope.

      • If -Pformat=sjs or if the option is not specified, only one file is created:
        • main.sjs, which is the JavaScript module that you must customize.
      • If -Pformat=xqy, two files are created:
        • lib.xqy, which is the XQuery module that you must customize.
        • main.sjs, which acts as a wrapper around lib.xqy.

      In the resulting step definition, the modulePath setting points to the custom module file.

    2. Manually add the step to the flow.
      1. In a text editor, open the step definition file.
      2. Likewise, open the flow configuration file.

        You can find your flow configuration file in your-project-root/flows.

      3. In the steps node, add the step as a key-value pair.
        • For the key, enter a string containing a number which represents the order of the step in the sequence.
          Note: The steps can be listed in any order, as long as the keys are unique within the steps node of the flow. Duplicate keys can produce unexpected results. The key number must be greater than 0.
        • For the value, copy and paste the entire content of the step definition file.
      4. Edit the step in the flow configuration file.
        • Rename the name setting to stepDefinitionName.
        • Rename the type setting to stepDefinitionType.
        • Create a new name setting and assign it a name for the step.
        • (Optional) Delete the following settings:
          • language
          • version
          • modulePath

          The values for these settings are retrieved from the step definition.

    3. Modify your custom module.

    Learn more: Custom Step Settings

What to do next

  1. (Optional) To perform other tasks outside the Data Hub space, you can add interceptors and custom hooks to the step.
  2. Run the flow.