Custom Processing with Hub Central
Overview
A typical Data Hub data flow involves the following operations:
- Load/Ingest your raw data into MarkLogic Server.
- Create an entity model to standardize your data fields.
- Map the fields in your raw data to the fields of the entity model.
- (Optional) Match and merge duplicates.
Security
You must be assigned the following security roles:
- To view, create, edit, or delete a step: Hub Central Developer or Hub Central Curator
- To add a step to a flow: Hub Central Developer or Hub Central Curator
- To run a step: Hub Central Operator or Hub Central Curator
Or any role that inherits the required role. See Users and Roles.
Create a Custom Step
To create a custom step,
- Create a custom step in HC format using Gradle.
- Customize the step by editing the associated step file. Learn more about flow and step configuration structures in the HC format.
- For a DHS environment with Hub Central,
Managing Custom Steps
To edit or delete a custom step,
- To edit the custom step, manually edit the associated step file in your local project files.
- To delete the custom step,
- In the flow configuration file, remove the reference to the custom step.
- (Optional) Delete the associated step definition file in your local project files.
- For a DHS environment with Hub Central,
Learn more about flow and step configuration structures in the HC format.