Create a Loading Step Using Hub Central

Before you begin

You need:

  • Security role: Hub Central Developer, Hub Central Curator, or any role that inherits any of these. Learn more: Users and Roles

Procedure

  1. Go to the Load area of Hub Central.
    Learn how.
    1. Go to your Hub Central endpoint.
      Note: Disregard this step if you are working from an on-prem environment. See Step 1b.

      Learn more:

    2. In the icon bar, click the Load icon ().
      Hub Central - icon bar - Load
  2. Click Add New.

    Hub Central - Loading step - grid view - Add New

    Hub Central - Loading step - table view - Add New
  3. Configure the step's basic settings.

    Hub Central - Loading step - properties
    Name Description
    Name The name of the step instance.
    Note: The step name is used as part of the names of related assets, as a collection name to tag processed documents, and as metadata in provenance and lineage logs. Therefore, it cannot be changed after the step is created.
    Description(Optional) A description of the step.
    Source Format The format of your source files: Text, JSON, XML, Binary, or Delimited Text. The default is JSON.
    Note: If you select Text, the source file's content must be in either JSON or XML format.
    Field Separator (Displayed if Source Format is Delimited Text.) The character used separate fields in a delimited text file.
    Target Format The format of the processed record: Text, JSON, XML, or Binary. The default is JSON.
    Target URI Prefix The prefix to add to the URI of each ingested document.
  4. Go to the Advanced tab.
  5. Configure the step's advanced settings.

    Hub Central - Loading step - advanced settings
    Name Description
    Target DatabaseThe STAGING database where you want to store the ingested data. The default is data-hub-STAGING.
    Target Collections Comma-separated string containing additional collection tags to apply to the processed records. The Default Collections tags are automatically applied.
    Target Permissions The permissions required to access the documents created by the step.

    The string must be in the format role,capability,role,capability,..., where capability can be read, insert, update, or execute.

    Provenance Granularity The granularity of the provenance tracking information: coarse (default) to store document-level provenance information only, fine to store document-level and property-level provenance information, or off to disable provenance tracking in future job runs. Applies only to mapping, matching, merging, mastering, and custom steps. Learn more: About Provenance and Lineage.
    Batch Size The number of documents to process per batch. A smaller batch size provides finer granularity in the jobs reporting. However, a smaller batch file also costs more because of the processing overhead. The recommended batch size for merging is 1.
    Header Content A JSON object that represents additional metadata to add to the header section of the envelope of each record.

    For the Loading step, the following JSON is added as the default:

       {
    sources: ["myLoadingStepName"],
    createdOn: "currentDateTime",
    createdBy: "currentUser"
  }
    Interceptors An array of JSON objects specifying the custom modules that perform additional processes on a batch after the core step processes are completed and before the results are saved in the database.

    Syntax:

       [
    {
      "path": "/uri/of/custom/module/in/modules/database/a.sjs",
      "vars": {
        "myParameter": "myParameterValue"
      }
      "when": "beforeContentPersisted",
    }
  ]
    path
    The URI of the interceptor in the MODULES database.
    vars
    (Optional) A JSON object containing parameters to pass to the interceptor.
    when
    Currently, only beforeContentPersisted is supported.
    Example
    Custom Hook A step add-on that performs additional processes in its own transaction before or after the core step transaction. Results are saved within a transaction.

    Syntax:

       {
    "module" : "/uri/of/custom/module/in/modules/database/a.sjs",
    "parameters" : {
      "myParameter" : "myParameterValue"
    },
    "user" : "account-required-to-run-module",
    "runBefore" : false
  }
    module
    The URI of your custom hook module in the MODULES database.
    parameters
    (Optional) A JSON object containing parameters to pass to your custom hook module.
    user
    The user account to use to run the module. Typically, a user with the security role data-hub-operator.
    runBefore
    For a pre-step hook, set to true. For a post-step hook, set to false.

    Example

    Learn more: Creating a Custom Hook Module and Adding a Custom Hook to a Step Manually.

  6. Add the step to a new flow or an existing one.

    Hub Central - Loading step - grid view - add step to a new flow / an existing flow
    1. Hover over the step tile.
    2. Click Add step to a new flow or select an existing flow under Add step to an existing flow.

    Hub Central - Loading step - table view - add step to a new flow / an existing flow
    1. Click the add-to-flow icon (a) for the step to add.
    2. Click Add step to a new flow or select an existing flow under Add step to an existing flow.
    Tip: You can add the step to multiple flows.

What to do next

In the Run area, expand the flow and run the step.