Custom Processing with Hub Central

Overview

A typical Data Hub data flow involves the following operations:
  1. Load/Ingest your raw data into MarkLogic Server.
  2. Create an entity model to standardize your data fields.
  3. Map the fields in your raw data to the fields of the entity model.
  4. (Optional) Match and merge duplicates.

Security

You must be assigned the following security roles:

  • To view, create, edit, or delete a step: Hub Central Developer or Hub Central Curator
  • To add a step to a flow: Hub Central Developer or Hub Central Curator
  • To run a step: Hub Central Operator or Hub Central Curator

Or any role that inherits the required role. See Users and Roles.

Create a Custom Step

To create a custom step,

  1. Create a custom step in HC format using Gradle.
  2. Customize the step by editing the associated step file. Learn more about flow and step configuration structures in the HC format.
  3. For a DHS environment with Hub Central,
    1. Deploy to DHS.
    2. In Hub Central, manage and run your steps.

Managing Custom Steps

To edit or delete a custom step,

  1. To edit the custom step, manually edit the associated step file in your local project files.
  2. To delete the custom step,
    • In the flow configuration file, remove the reference to the custom step.
    • (Optional) Delete the associated step definition file in your local project files.
  3. For a DHS environment with Hub Central,
    1. Deploy to DHS.
    2. In Hub Central, manage and run your steps.

Learn more about flow and step configuration structures in the HC format.