Run a Flow Using the REST Client API
When running a flow, Data Hub performs the following:
- Calls the collector to get the list of IDs of the records to process.
- Splits the list into batches.
- Calls a custom REST extension that invokes the main.[xqy|sjs] for your flow.
MarkLogic Data Hub provides a REST Client API extension runFlow which allows you to specify:
- The individual records to process.
- The single step to run.
ml:runFlow
The REST Client API extension that runs a single step within the flow to process the specified records.
POST /v1/resources/ml:runFlow?rs:job-id=YourJobID&rs:flow-name=YourFlowName&rs:step=1&database=YourSourceDatabase&rs:target-database=YourTargetDatabase&rs:options={ "uris" : [ "comma-separated","uris","of","records","to","process" ] }
- rs:job-id
- A unique job ID to associate with the flow run. This option can be used if the flow run is part of a larger process (e.g., a process orchestrated by NiFi with its own job/process ID). Must not be the same as an existing Data Hub job ID. If not provided, a unique Data Hub job ID will be assigned.
- rs:flow-name
- The name of the flow.
- rs:step
- The sequence number of the step to execute. To run multiple specific steps, use your orchestration tool to send one runFlow request for each step.
- database
- The database containing the source data; e.g., data-hub-STAGING.
- rs:target-database
- The database where you want to store the processed data; e.g., data-hub-FINAL.
- rs:options
- A JSON object containing additional options to pass to the flow.
- To specify the list of records to process, add the key uris whose value is an array of the URIs of the records to process.