Edit a Loading Step Using Hub Central
Before you begin
You need:
- Security role: Hub Central Developer, Hub Central Curator, or any role that inherits any of these. Learn more: Users and Roles
Procedure
- Go to the Load area of Hub Central.
- Open the step to edit its properties.
- Configure the step's basic settings.
Name Description Name The name of the step instance. Note: The step name is used as part of the names of related assets, as a collection name to tag processed documents, and as metadata in provenance and lineage logs. Therefore, it cannot be changed after the step is created.Description (Optional) A description of the step. Source Format The format of your source files: Text, JSON, XML, Binary, or Delimited Text. The default is JSON. Note: If you select Text, the source file's content must be in either JSON or XML format.Field Separator (Displayed if Source Format is Delimited Text.) The character used separate fields in a delimited text file. Target Format The format of the processed record: Text, JSON, XML, or Binary. The default is JSON. Target URI Prefix The prefix to add to the URI of each ingested document. - Go to the Advanced tab.
- Configure the step's advanced settings.
Name Description Target Database The STAGING database where you want to store the ingested data. The default is data-hub-STAGING
.Target Collections Comma-separated string containing additional collection tags to apply to the processed records. The Default Collections tags are automatically applied. Target Permissions The permissions required to access the documents created by the step. The string must be in the format
role,capability,role,capability,...
, wherecapability
can beread
,insert
,update
, orexecute
.Provenance Granularity The granularity of the provenance tracking information: coarse
(default) to store document-level provenance information only,fine
to store document-level and property-level provenance information, oroff
to disable provenance tracking in future job runs. Applies only to mapping, matching, merging, mastering, and custom steps. Learn more: About Provenance and Lineage.Batch Size The number of documents to process per batch. A smaller batch size provides finer granularity in the jobs reporting. However, a smaller batch file also costs more because of the processing overhead. The recommended batch size for merging is 1. Header Content A JSON object that represents additional metadata to add to the header section of the envelope of each record. For the Loading step, the following JSON is added as the default:
{ sources: ["myLoadingStepName"], createdOn: "currentDateTime", createdBy: "currentUser" }
Interceptors An array of JSON objects specifying the custom modules that perform additional processes on a batch after the core step processes are completed and before the results are saved in the database. Syntax:
[ { "path": "/uri/of/custom/module/in/modules/database/a.sjs", "vars": { "myParameter": "myParameterValue" } "when": "beforeContentPersisted", } ]
- path
- The URI of the interceptor in the MODULES database.
- vars
- (Optional) A JSON object containing parameters to pass to the interceptor.
- when
- Currently, only
beforeContentPersisted
is supported.
Custom Hook A step add-on that performs additional processes in its own transaction before or after the core step transaction. Results are saved within a transaction. Syntax:
{ "module" : "/uri/of/custom/module/in/modules/database/a.sjs", "parameters" : { "myParameter" : "myParameterValue" }, "user" : "account-required-to-run-module", "runBefore" : false }
- module
- The URI of your custom hook module in the MODULES database.
- parameters
- (Optional) A JSON object containing parameters to pass to your custom hook module.
- user
- The user account to use to run the module. Typically, a user with the security role data-hub-operator.
- runBefore
- For a pre-step hook, set to
true
. For a post-step hook, set tofalse
.
Learn more: Creating a Custom Hook Module and Adding a Custom Hook to a Step Manually.
Important: Custom hooks are deprecated. - Add the step to a new flow or an existing one.