Run a Flow Using the REST Client API

When running a flow, Data Hub performs the following:

  1. Calls the collector to get the list of IDs of the records to process.
  2. Splits the list into batches.
  3. Calls a custom REST extension that invokes the main.[xqy|sjs] for your flow.

MarkLogic Data Hub provides a REST Client API extension runFlow which allows you to specify:

  • The individual records to process.
  • The single step to run.
ml:runFlow

The REST Client API extension that runs a single step within the flow to process the specified records.

POST /v1/resources/ml:runFlow?rs:job-id=YourJobID&rs:flow-name=YourFlowName&rs:step=1&database=YourSourceDatabase&rs:target-database=YourTargetDatabase&rs:options={ "uris" : [ "comma-separated","uris","of","records","to","process" ] }
rs:job-id
A unique job ID to associate with the flow run. This option can be used if the flow run is part of a larger process (e.g., a process orchestrated by NiFi with its own job/process ID). Must not be the same as an existing Data Hub job ID. If not provided, a unique Data Hub job ID will be assigned.
rs:flow-name
The name of the flow.
rs:step
The sequence number of the step to execute. To run multiple specific steps, use your orchestration tool to send one runFlow request for each step.
database
The database containing the source data; e.g., data-hub-STAGING.
rs:target-database
The database where you want to store the processed data; e.g., data-hub-FINAL.
rs:options
A JSON object containing additional options to pass to the flow.
  • To specify the list of records to process, add the key uris whose value is an array of the URIs of the records to process.