Convert Your Artifacts from Quickstart to Hub Central

The format of entity models, flows, and steps required by Hub Central is different from the format required by QuickStart.

  • In Hub Central, entity models allow properties of entity types to be used as facets when filtering your data.
  • In QuickStart, you run a flow with steps in it, and the steps are embedded in the flow configuration. In Hub Central, you run individual steps, and each step is in its own file, which the flow configuration refers to.

If you have existing Data Hub 5.0–5.2 models, flows, and steps that you intend to use in Hub Central in DHS/an on-premises or local environment, you must convert them to the Hub Central format. After the conversion, QuickStart will no longer recognize those files. If you do not plan to use Hub Central, the conversion is not required.

As of Data Hub 5.4, a new set of match/merge formats have been created which opens up new capabilities and eliminates redundant configuration that is completed by Modeling with Entity Services. This new set of match/merge formats must be associated with an Entity Type and are required for compatibility with the Hub Central User Interface, but will not be compatible with Data Hub QuickStart.

Important: If you have 4.x flows that you intend to use in Hub Central, you must do the following before converting to the Hub Central format.
  1. Migrating 4x Flows.
  2. Add them to new 5.x or 6.x flows.
Important: You must upgrade your Data Hub project to 5.4.0 or later before converting your artifacts from QuickStart to Hub Central format.
Note: The legacy QuickStart format is converted to an XML equivalent when the custom function is defined in XQuery. This is behavior is maintained for backward compatibility. Once converted to the Hub Central format, the options will not be converted to XML prior to being passed to the XQuery function. They will be passed as an object-node() in the same structure stored in the project and database. JavaScript functions will continue to receive the options as a JSON object but reflecting the Hub Central format.

The Data Hub Gradle Plugin and the Data Hub Client JAR can handle both formats.

Before you begin

You need:

Procedure

  1. Convert your entity models, flows, steps, and other artifacts.

    At your project root, run the Gradle task hubConvertForHubCentral.

    ./gradlew hubConvertForHubCentral -Pconfirm=true -igradlew.bat hubConvertForHubCentral -Pconfirm=true -i
    confirm
    (Required) Confirmation to convert your artifacts.

    Learn more: Conversion Tasks

hubConvertForHubCentral converts the mapping configurations (your-project-root/mappings/*) into mapping steps (your-project-root/steps/mapping/*).
  • In the QuickStart format, mapping properties are stored in mapping configuration files, separate from the mapping step.
  • In the Hub Central format, all mapping properties are included in the mapping step.

After running hubConvertForHubCentral, the old mapping configuration files might still exist in other environments.

  1. (Optional) Delete your legacy mappings in each environment where you intend to use Hub Central.

    At your project root, run the Gradle task hubDeleteLegacyMappings.

    ./gradlew hubDeleteLegacyMappings -PenvironmentName=myEnvName -Pconfirm=true -igradlew.bat hubDeleteLegacyMappings -PenvironmentName=myEnvName -Pconfirm=true -i
    environmentName
    (Required) The name of your environment.
    confirm
    (Required) Confirmation to convert your artifacts.

    Learn more: Conversion Tasks

    hubDeleteLegacyMappings deletes the old mapping configuration files.

  2. Manually clean up your entity models.
    Important: Learn about the structured and relationship types of entity properties in Hub Central: Modeling in Hub Central.
    1. Go to your-project-root/entities.
    2. Delete the duplicate definitions of your structured types.

      For each *.entity.json file:

      1. Open the file.
      2. Each child under the definitions section is an entity type definition.
        • The first child's key matches the value of $.info.title in the file you are inspecting and the name of the file. No action is required for this child.
        • If any child has the same name as another *.entity.json file, verify that the entity type definition is the same in both files. If so, delete the *.entity.json file with the same name as the child.
      Example

      Suppose you have the following Customer.entity.json file:

         {
    "info" : {
      "title" : "Customer",
      "version" : "0.0.1",
      "baseUri" : "http://example.org/"
    },
    "definitions" : {
      "Customer" : {
        ...
      },
      "Address" : {
        ...
      }
    }
  }

      If the file Address.entity.json exists and the definition of Address is the same in both files, delete the file Address.entity.json.

    3. After removing all duplicate definitions, update the links of your relationship types.

      For each *.entity.json file:

      1. Open the file.
      2. For each $ref reference that points to an external entity type, replace the value (#/definitions/entityTypeName) with a valid Internationalized Resource Identifier (IRI) (baseURI/entityTypeName-version/entityTypeName).
      Example

      Suppose your entity type Customer has a property that is a reference to another entity type Order, which is defined in another file. In the Customer.entity.json file, replace the following reference:

         "$ref" : "#/definitions/Order"

      With the following:

         "$ref" : "http://example.org/Order-0.0.1/Order"
    Tip: Alternatively, you can clean up your entity models in Hub Central.
    Learn how.
    1. Deploy to DHS.
    2. Go to the Model area of Hub Central.
    3. Delete the duplicate definitions of structured types that are already defined within an entity type definition.
    4. In each entity type, replace properties that point to other entity types.
      1. Delete the property that should be an external reference.
      2. Recreate the property as a Relationship type.
      3. Select the target entity type.
    5. At your project root, run the Gradle task hubPullChanges to download your changes to your local project.
      ./gradlew hubPullChanges -igradlew.bat hubPullChanges -i
  3. Update your custom modules.
    If your custom modules read or modify flow or step configurations, remember to update them to accommodate the new configuration structures.

    Learn more: About Flow and Step Configuration Structures

  4. Deploy converted artifacts to an on-premises or local environment.

    At your project root, run the Gradle task hubDeploy, hubDeployAsDeveloper, or hubDeployAsSecurityAdmin.

    Tip: If you do not have any custom roles to deploy, you can run the Gradle task hubDeployAsDeveloper.
  5. Download the marklogic-data-hub-central-VERSION_NUMBER.war file.
  6. Run the Hub Central .war.
    • To use the default port number for the internal web server (port 8080): 
      java -jar marklogic-data-hub-central-VERSION_NUMBER.war
    • To use a few configured properties set using the command line; e.g., port 8020 used by the STAGING app server: 
      java -jar marklogic-data-hub-central-VERSION_NUMBER.war --mlStagingPort=8020
    • To use many configured properties placed in a single external configuration file; e.g., gradle.properties file (single-environment gradle properties): 
      java -jar marklogic-data-hub-central-VERSION_NUMBER.war --spring.config.additional-location=file:/path/to/project/gradle.properties
    • To use many configured properties placed in multiple external configuration files; e.g., gradle.properties and gradle-prod.properties files (multiple-environment gradle properties): 
      java -jar marklogic-data-hub-central-VERSION_NUMBER.war --spring.config.additional-location=file:/path/to/project/gradle.properties,file:/path/to/project/gradle-prod.properties
      Important: The gradle.properties file should be set before the gradle-env.properties file. This ensures that environment-specific properties have priority over the other properties.

    For the full list of configurable properties, see Data Hub Properties.

    Note: If you are using Windows and a firewall alert appears, click Allow access.