Convert Your Artifacts from Quickstart to Hub Central

The format of entity models, flows, and steps required by Hub Central is different from the format required by QuickStart.

  • In Hub Central, entity models allow properties of entity types to be used as facets when filtering your data.
  • In QuickStart, you run a flow with steps in it, and the steps are embedded in the flow configuration. In Hub Central, you run individual steps, and each step is in its own file, which the flow configuration refers to.

If you have existing Data Hub 5.0–5.2 models, flows, and steps that you intend to use in Hub Central in DHS/an on-premises or local environment, you must convert them to the Hub Central format. After the conversion, QuickStart will no longer recognize those files. If you do not plan to use Hub Central, the conversion is not required.

Important: If you have 4.x flows that you intend to use in Hub Central, you must do the following before converting to the Hub Central format.
  1. Migrate 4x flows to 5.x steps.
  2. Add them to new 5.x flows.
Important: You must upgrade your Data Hub project to 5.4.0 or later before converting your artifacts from QuickStart to Hub Central format.

The Data Hub Gradle Plugin and the Data Hub Client JAR can handle both formats.

Before you begin

You need:

Procedure

  1. Convert your entity models, flows, steps, and other artifacts.

    At your project root, run the Gradle task hubConvertForHubCentral.

    ./gradlew hubConvertForHubCentral -Pconfirm=true -igradlew.bat hubConvertForHubCentral -Pconfirm=true -i
    confirm
    (Required) Confirmation to convert your artifacts.

    Learn more: Conversion Tasks

hubConvertForHubCentral converts the mapping configurations (your-project-root/mappings/*) into mapping steps (your-project-root/steps/mapping/*).
  • In the QuickStart format, mapping properties are stored in mapping configuration files, separate from the mapping step.
  • In the Hub Central format, all mapping properties are included in the mapping step.

After running hubConvertForHubCentral, the old mapping configuration files might still exist in other environments.

  1. (Optional) Delete your legacy mappings in each environment where you intend to use Hub Central.

    At your project root, run the Gradle task hubDeleteLegacyMappings.

    ./gradlew hubDeleteLegacyMappings -PenvironmentName=myEnvName -Pconfirm=true -igradlew.bat hubDeleteLegacyMappings -PenvironmentName=myEnvName -Pconfirm=true -i
    environmentName
    (Required) The name of your environment.
    confirm
    (Required) Confirmation to convert your artifacts.

    Learn more: Conversion Tasks

    hubDeleteLegacyMappings deletes the old mapping configuration files.

  2. Manually clean up your entity models.
    Important: Learn about the structured and relationship types of entity properties in Hub Central: Modeling in Hub Central.
    1. Go to your-project-root/entities.
    2. Delete the duplicate definitions of your structured types.

      For each *.entity.json file:

      1. Open the file.
      2. Each child under the definitions section is an entity type definition.
        • The first child's key matches the value of $.info.title in the file you are inspecting and the name of the file. No action is required for this child.
        • If any child has the same name as another *.entity.json file, verify that the entity type definition is the same in both files. If so, delete the *.entity.json file with the same name as the child.
      Example

      Suppose you have the following Customer.entity.json file:

         {
    "info" : {
      "title" : "Customer",
      "version" : "0.0.1",
      "baseUri" : "http://example.org/"
    },
    "definitions" : {
      "Customer" : {
        ...
      },
      "Address" : {
        ...
      }
    }
  }

      If the file Address.entity.json exists and the definition of Address is the same in both files, delete the file Address.entity.json.

    3. After removing all duplicate definitions, update the links of your relationship types.

      For each *.entity.json file:

      1. Open the file.
      2. For each $ref reference that points to an external entity type, replace the value (#/definitions/entityTypeName) with a valid Internationalized Resource Identifier (IRI) (baseURI/entityTypeName-version/entityTypeName).
      Example

      Suppose your entity type Customer has a property that is a reference to another entity type Order, which is defined in another file. In the Customer.entity.json file, replace the following reference:

         "$ref" : "#/definitions/Order"

      With the following:

         "$ref" : "http://example.org/Order-0.0.1/Order"
    Tip: Alternatively, you can clean up your entity models in Hub Central.
    Learn how.
    1. Deploy to DHS.
    2. Go to the Model area of Hub Central.
    3. Delete the duplicate definitions of structured types that are already defined within an entity type definition.
    4. In each entity type, replace properties that point to other entity types.
      1. Delete the property that should be an external reference.
      2. Recreate the property as a Relationship type.
      3. Select the target entity type.
    5. At your project root, run the Gradle task hubPullChanges to download your changes to your local project.
      ./gradlew hubPullChanges -igradlew.bat hubPullChanges -i
  3. Update your custom modules.
    If your custom modules read or modify flow or step configurations, remember to update them to accommodate the new configuration structures.

    Learn more: About Flow and Step Configuration Structures

  4. Choose one of the following:
  5. Download the marklogic-data-hub-central-5.4.4.war file.
  6. Run the Hub Central .war.
    • To use the default port number for the internal web server (port 8080): 
      java -jar marklogic-data-hub-central-5.4.4.war
    • To use a few configured properties set using the command line; e.g., port 8020 used by the STAGING app server: 
      java -jar marklogic-data-hub-central-5.4.4.war --mlStagingPort=8020
    • To use many configured properties placed in a single external configuration file; e.g., gradle.properties file (single-environment gradle properties): 
      java -jar marklogic-data-hub-central-5.4.4.war --spring.config.additional-location=file:/path/to/project/gradle.properties
    • To use many configured properties placed in multiple external configuration files; e.g., gradle.properties and gradle-prod.properties files (multiple-environment gradle properties): 
      java -jar marklogic-data-hub-central-5.4.4.war --spring.config.additional-location=file:/path/to/project/gradle.properties,file:/path/to/project/gradle-prod.properties
      Important: The gradle.properties file should be set before the gradle-env.properties file. This ensures that environment-specific properties have priority over the other properties.

    For the full list of configurable properties, see Data Hub Properties.

    Note: If you are using Windows and a firewall alert appears, click Allow access.
    Note: If you are using both QuickStart and Hub Central, there will be a port conflict on port 8080.