You can use Gradle to run flows outside a GUI.
Tip: To avoid depending on Gradle and project files, especially in production environments, run your flows using
the Data Hub Client JAR.
Important: When you load/ingest delimited text files into MarkLogic Server, URIs are randomly generated each time you load/ingest. To generate the same URIs each time you load/ingest delimited text files, use
MLCP.
Before you begin
You need:
Procedure
- Open a command-line window, and go to your project root directory.
- At your project root, run the Gradle task hubRunFlow.
./gradlew hubRunFlow -PflowName=YourFlowName -PentityName=YourEntityName -PbatchSize=100 -PthreadCount=4 -PshowOptions=[true|false] -PfailHard=[true|false] -Psteps="1,2" -PjobId="abc123" [ -Poptions="{ customkey: customvalue, ... }" | -PoptionsFile=/path/to.json ] -igradlew.bat hubRunFlow -PflowName=YourFlowName -PentityName=YourEntityName -PbatchSize=100 -PthreadCount=4 -PshowOptions=[true|false] -PfailHard=[true|false] -Psteps="1,2" -PjobId="abc123" [ -Poptions="{ customkey: customvalue, ... }" | -PoptionsFile=/path/to.json ] -i
- flowName
- (Required) The name of the harmonize flow to run.
- entityName
- (Required if the flow includes a mapping step) The name of the entity used with the mapping step.
- batchSize
- The maximum number of items to process in a batch. The default is 100.
- threadCount
- The number of threads to run. The default is 4.
- showOptions
- If
true
, options that were passed to the command are printed out. The default is false
.
- failHard
- If
true
, the flow's execution is ended immediately if a step fails. The default is false
.
- steps
- The comma-separated numbers of the steps to run. If not provided, the entire flow is run.
- jobId
- A unique job ID to associate with the flow run. This option can be used if the flow run is part of a larger process (e.g., a process orchestrated by NiFi with its own job/process ID). Must not be the same as an existing Data Hub job ID. If not provided, a unique Data Hub job ID will be assigned.
- options
- A JSON structure containing key-value pairs to be passed as custom parameters to your step modules.
- optionsFile
- The path to a JSON file containing key-value pairs to be passed as custom parameters to your step modules.