Upgrade to MarkLogic Data Hub 5.5

Prerequisites

You need:

Java JRE (OpenJDK) 8
MarkLogic Server (See Version Compatibility.)
Gradle 6.4 up to the latest 6.x release

Data Hub 5.3.0 was released only as an internal beta. These upgrade instructions intentionally skip 5.3.0.

Important: Remember to archive your old project files before performing an upgrade.

Important: If you are deploying to a Data Hub Service environment, contact Support.

The notes and steps in this tab are for the following upgrade paths:

Important: Upgrading to this release would trigger a reindexing of the STAGING and FINAL databases. Learn more about how reindexing works and its impact on performance..

Important:

If you are upgrading from version 5.1.x or an earlier version to this release, users might not have permission to read or query Template Driven Extraction (TDE) documents.

To read or query TDE documents, users must be assigned the data-hub-operator role or any role that inherits the required role. See Users and Roles.

Important: Upgrading to this release would require Data Hub REST extension names to be written in camel case format. For details, see Data Hub Extensions to the REST Client API.

Data Hub 5.4 » 5.5
Data Hub 5.2 » 5.5
Data Hub 5.1 » 5.5
Data Hub 5.0 » 5.5

Procedure

Update the /your-project-root/build.gradle file.

In the plugins section, set com.marklogic.ml-data-hub to the new Data Hub version.

   plugins {
      // Gradle Properties plugin
      id 'net.saliman.properties' version '1.4.6'

      // Data Hub plugin
      id 'com.marklogic.ml-data-hub' version 'VERSION_NUMBER'
  }


`net.saliman.properties`	Gradle Properties plugin	Allows you to create different environments for your Gradle deployment and set up a gradle-`env`.properties file, where `env` is the environment name. When running a Gradle task, you can specify the target environment with the `environmentName` option. For more information, see https://github.com/stevesaliman/gradle-properties-plugin.
`com.marklogic.ml-data-hub`	Data Hub plugin	Extends the ml-gradle plugin with Data Hub-specific commands.

Update the /your-project-root/gradle/wrapper/gradle-wrapper.properties file.
Set distributionUrl to https\://services.gradle.org/distributions/gradle-6.4-bin.zip.
```
distributionUrl=https\://services.gradle.org/distributions/gradle-6.4-bin.zip
```
At your project root, run the Gradle task hubUpdate.
Running the hubUpdate task with the -i option (info mode) displays specifically what the task does, including configuration settings that changed.

./gradlew hubUpdate -igradlew.bat hubUpdate -i
Update the gradle.properties file.
- Delete the mlDHFVersion line.
- (Recommended) To use the new Data Hub 5.2 default permissions for modules (data-hub-module-reader,read,data-hub-module-reader,execute,data-hub-module-writer,update,rest-extension-user,execute), delete mlModulePermissions.
Run the Gradle task mlDeploy.
./gradlew mlDeploy -igradlew.bat mlDeploy -i
(Optional) If you intend to use Hub Central, convert your artifacts to the Hub Central format. If you do not plan to use Hub Central, the conversion is not required.

The notes and steps in this tab are for the following upgrade paths:

Important: Upgrading to this release would trigger a reindexing of the STAGING and FINAL databases. Learn more about how reindexing works and its impact on performance..

Important:

If you are upgrading from version 5.1.x or an earlier version to this release, users might not have permission to read or query Template Driven Extraction (TDE) documents.

To read or query TDE documents, users must be assigned the data-hub-operator role or any role that inherits the required role. See Users and Roles.

Important: Upgrading to this release would require Data Hub REST extension names to be written in camel case format. For details, see Data Hub Extensions to the REST Client API.

DHF 4.3 » Data Hub 5.5

Procedure

Update the /your-project-root/build.gradle file.

In the plugins section, set com.marklogic.ml-data-hub to the new Data Hub version.

   plugins {
      // Gradle Properties plugin
      id 'net.saliman.properties' version '1.4.6'

      // Data Hub plugin
      id 'com.marklogic.ml-data-hub' version 'VERSION_NUMBER'
  }


`net.saliman.properties`	Gradle Properties plugin	Allows you to create different environments for your Gradle deployment and set up a gradle-`env`.properties file, where `env` is the environment name. When running a Gradle task, you can specify the target environment with the `environmentName` option. For more information, see https://github.com/stevesaliman/gradle-properties-plugin.
`com.marklogic.ml-data-hub`	Data Hub plugin	Extends the ml-gradle plugin with Data Hub-specific commands.

Update the /your-project-root/gradle/wrapper/gradle-wrapper.properties file.
Set distributionUrl to https\://services.gradle.org/distributions/gradle-6.4-bin.zip.
```
distributionUrl=https\://services.gradle.org/distributions/gradle-6.4-bin.zip
```

At your project root, run the Gradle task hubUpdate.

Running the hubUpdate task with the -i option (info mode) displays specifically what the task does, including configuration settings that changed.

./gradlew hubUpdate -igradlew.bat hubUpdate -i

hubUpdate automatically moves the following artifacts to the new project.

From old project To Data Hub 5.5 project

From old project	To Data Hub 5.5 project
`your-project-root/plugins/entities/entity1/entity1.entity.json ... your-project-root/plugins/entities/entityN/entityN.entity.json` The input and harmonize folders remain in the same plugins/entities/entity* folders.	`your-project-root/entities/entity1.entity.json ... your-project-root/entities/entityN.entity.json`
your-project-root/plugins/mappings (the entire directory)	your-project-root/mappings

   your-project-root/plugins/entities/entity1/entity1.entity.json
  ...
  your-project-root/plugins/entities/entityN/entityN.entity.json

The input and harmonize folders remain in the same plugins/entities/entity* folders.

   your-project-root/entities/entity1.entity.json
  ...
  your-project-root/entities/entityN.entity.json

your-project-root/plugins/mappings (the entire directory) your-project-root/mappings

Update the gradle.properties file.
- Delete the mlDHFVersion line.
- (Recommended) To use the new Data Hub 5.2 default permissions for modules (data-hub-module-reader,read,data-hub-module-reader,execute,data-hub-module-writer,update,rest-extension-user,execute), delete mlModulePermissions.
If you have existing Data Hub 4.x flows,
1. Migrate 4x flows to 5.x steps.
2. Add them to new 5.x flows.
Run the Gradle task mlDeploy.
./gradlew mlDeploy -igradlew.bat mlDeploy -i

The notes and steps in this tab are for the following upgrade paths:

Important: Upgrading to this release would trigger a reindexing of the STAGING and FINAL databases. Learn more about how reindexing works and its impact on performance..

Important:

If you are upgrading from version 5.1.x or an earlier version to this release, users might not have permission to read or query Template Driven Extraction (TDE) documents.

To read or query TDE documents, users must be assigned the data-hub-operator role or any role that inherits the required role. See Users and Roles.

Important: Upgrading to this release would require Data Hub REST extension names to be written in camel case format. For details, see Data Hub Extensions to the REST Client API.

DHF 4.2 » Data Hub 5.5
DHF 4.1 » Data Hub 5.5
DHF 4.0 » Data Hub 5.5

Procedure

Upgrade to DHF 4.3 to adopt the new roles.
Upgrade from DHF 4.3 to Data Hub 5.5.

The notes and steps in this tab are for the following upgrade paths:

Important: Upgrading to this release would trigger a reindexing of the STAGING and FINAL databases. Learn more about how reindexing works and its impact on performance..

Important:

If you are upgrading from version 5.1.x or an earlier version to this release, users might not have permission to read or query Template Driven Extraction (TDE) documents.

To read or query TDE documents, users must be assigned the data-hub-operator role or any role that inherits the required role. See Users and Roles.

Important: Upgrading to this release would require Data Hub REST extension names to be written in camel case format. For details, see Data Hub Extensions to the REST Client API.

DHF 3.x and earlier » Data Hub 5.5

Procedure

Upgrade to DHF 4.3.

Significant changes were made in Data Hub 4.x releases, including changes in the project directory structure, security roles, and databases. These changes require that you manually update configuration files, run the Data Hub 4.3.2 versions of the Gradle tasks to correctly reconfigure your project and environment, and perform tests before proceeding to the Data Hub upgrade. For details, see the Upgrade Notes and Additional Upgrade Notes sections in Upgrade to DHF 4.3.

The data that you ingested and processed in Data Hub Framework 2.x, 3.x, or 4.x is compatible with Data Hub 5.x. Therefore, you can install 5.x directly, instead of upgrading; however, you must recreate your 2.x/3.x/4.x flows as steps in 5.x, and you might need to update your custom code.
Upgrade from DHF 4.3 to Data Hub 5.5.