Upgrade to MarkLogic Data Hub 6.0

The notes and steps in this tab are for the following upgrade paths:

• DHF 4.3 » Data Hub 6.0

Procedure

1. Update the /your-project-root/build.gradle file.

   • In the plugins section, set com.marklogic.ml-data-hub to the new Data Hub version.

        plugins {
           // Gradle Properties plugin
           id 'net.saliman.properties' version '1.4.6'
     
           // Data Hub plugin
           id 'com.marklogic.ml-data-hub' version 'VERSION_NUMBER'
       }
     

     net.saliman.properties	Gradle Properties plugin	Allows you to create different environments for your Gradle deployment and set up a gradle-env.properties file, where env is the environment name. When running a Gradle task, you can specify the target environment with the environmentName option. For more information, see https://github.com/stevesaliman/gradle-properties-plugin.
     com.marklogic.ml-data-hub	Data Hub plugin	Extends the ml-gradle plugin with Data Hub-specific commands.

   • In the dependencies section, change compile to implementation.
     Note: This step is only required if one of the following applies to your project:

     • you are upgrading from a project using Gradle 6.x or earlier to 7.0 or later.
     • the build.gradle file already uses compile for dependencies.

     You must use implementation for dependencies because compile is not supported in Gradle versions 7.0 and later.

        dependencies {
           implementation "com.marklogic:marklogic-data-hub:VERSION_NUMBER"
       }
     

2. Update the /your-project-root/gradle/wrapper/gradle-wrapper.properties file.
   Note: This step is only required if you are upgrading from a project using Gradle 6.4 or earlier to 6.4 or later.

   Set distributionUrl to https\://services.gradle.org/distributions/gradle-6.4-bin.zip.

   distributionUrl=https\://services.gradle.org/distributions/gradle-6.4-bin.zip

3. At your project root, run the Gradle task hubUpdate.

   Running the hubUpdate task with the -i option (info mode) displays specifically what the task does, including configuration settings that changed.

   Unix systemsWindows
   ./gradlew hubUpdate -igradlew.bat hubUpdate -i
   Tip:

   • The hubUpdate command creates a flow artifact (dh_Upgrade_{entityName}Flow.flow.json) for every entity in the plugins/entities directory where entityName is the name of the directory.
   • hubUpdate adds all the 4.x flows inside an entity directory (either ingestion or harmonize) as steps to the flow created above.
   • The artifacts appear in the user project and users should deploy the artifacts. See Step 6.

   hubUpdate automatically moves the following artifacts to the new project.

   From old project	To Data Hub 6.0 project
   your-project-root/plugins/entities/entity1/entity1.entity.json ... your-project-root/plugins/entities/entityN/entityN.entity.json The input and harmonize folders remain in the same plugins/entities/entity* folders.	your-project-root/entities/entity1.entity.json ... your-project-root/entities/entityN.entity.json
   your-project-root/plugins/mappings (the entire directory)	your-project-root/mappings

   Note: Data Hub 6.0.0 removes LegacyFlowRunner. To run 4.3.2 legacy flows, the user must upgrade their flows.

   hubUpgradeLegacyFlows upgrades all 4.3.2. legacy flows in the plugins directory.

   Unix systemsWindows
   ./gradlew hubUpgradeLegacyFlows -igradlew.bat hubUpgradeLegacyFlows -i

   When using these tasks, consider:

   • If the user only provides a particular kind of legacyFlow, then the task will search in the user-provided directory for the flows and upgrade them. If it does not find any flows, it will not upgrade anything. See Migrating 4.x Flows.
   • To only upgrade a particular subset of the legacy flows, add values for the following parameters: [PlegacyEntities={entity-dirctory-name} -PlegacyFlowTypes={input|harmonize} -PlegacyFlowNames=flowName1,flowName2]. See Migrating 4.x Flows.
   • If the user provides a combination of flow options, the task will upgrade the flows according to the flowType the user previously set in a user-provided directory.
4. Update the gradle.properties file.
   • Delete the mlDHFVersion line.
   • (Recommended) To use the new Data Hub 5.2 default permissions for modules (data-hub-module-reader,read,data-hub-module-reader,execute,data-hub-module-writer,update,rest-extension-user,execute), delete mlModulePermissions.
5. If you have existing Data Hub 4.x flows,

   1. Migrating 4x Flows.
   2. Add them to new 5.x or 6.x flows.
6. Run the Gradle task mlDeploy.
   Unix systemsWindows
   ./gradlew mlDeploy -igradlew.bat mlDeploy -i