Manifest

The Global Risks Index (GRIx) is an integrated framework designed to assess and manage risks across the water-health-food-energy nexus. It encompasses a wide array of risk categories, including financial, political, systemic, health, and climate risks. GRIx utilizes the Microsoft Common Data Model (CDM) to ensure seamless data integration, interoperability, and scalability. A pivotal component of CDM, the Manifest object, serves as the organizational backbone, enabling GRIx to manage complex datasets and their interrelationships effectively.

---

Understanding the Manifest Object

Definition and Purpose

In GRIx, a Manifest object represents a comprehensive directory that organizes and manages related data files and schema descriptions within a data lake. It acts as an entry point, delineating the structure and interconnections of various entities essential for risk assessment and management. Each Manifest document (*.manifest.cdm.json) encapsulates a collection of entities, their schemas, data partitions, relationships, and submanifests, thereby forming a cohesive solution tailored to specific risk domains or analytical purposes.

Structure of a Manifest Document

A typical Manifest document in GRIx encompasses the following key components:

1. Version Information

2. Imports

3. Manifest Definition

4. Entity Lists

5. Relationship Lists

6. Submanifest Lists

Each component plays a crucial role in defining how data is organized, accessed, and interrelated within the GRIx ecosystem.

---

Imports in a Manifest

Imports in a Manifest allow GRIx to reference and utilize definitions from other CDM documents, promoting reusability and modularity. By importing foundational and common definitions, the Manifest ensures consistency and adherence to standardized schemas across various risk domains.

Example Import Statement:

{
  "imports": [
    {
      "corpusPath": "cdm:/commonDefinitions.cdm.json"
    },
    {
      "corpusPath": "cdm:/foundations.cdm.json"
    },
    {
      "corpusPath": "cdm:/grixRiskEntities.cdm.json"
    }
  ]
}

In this example, the Manifest imports common definitions, foundational schemas, and GRIx-specific risk entities, establishing a robust base for subsequent data definitions and relationships.

---

Shared Concepts Inside the Manifest

The Manifest object and the entities it contains share several properties and functionalities that ensure uniformity and interoperability within GRIx:

• Entity Names and Paths: Standardized naming conventions and corpus paths facilitate easy reference and management of entities.

• Data Partitions: Organized collections of data files associated with each entity, enabling efficient data retrieval and analysis.

• Traits and Annotations: Metadata annotations that define behaviors, formats, and relationships, enhancing the semantic richness of the data.

• Version Control: Embedded version information ensures backward compatibility and smooth schema evolution.

---

Entity Lists in a Manifest

Entity lists within a Manifest represent collections of related entities, organized to serve specific analytical or operational purposes within GRIx. These entities can either be owned by the Manifest or referenced from external Manifests.

Local Entity Declaration

Entities that are owned by the Manifest are fully managed within its scope. The Manifest specifies the location of the entity's schema document, the data partitions containing the entity's records, and any partition patterns used to discover new data files.

Properties of Local Entity Declaration:

Property / Method	Description	SDK Version
entityName	The name of the owned entity (e.g., RiskEntity).	1.0
entityPath	A corpus path to the entity definition within its containing document (e.g., local:/RiskData/RiskEntity.cdm.json/RiskEntity). This should be the resolved form of the entity definition.	1.0
dataPartitions	Collection of dataPartition objects, each describing the location, format, and details of a data file for the entity.	1.0
dataPartitionPatterns	Collection of dataPartitionPatterns objects used to discover and describe new partition files based on defined patterns.	1.0
incrementalPartitions	Collection of incremental dataPartition objects for entities that receive periodic updates.	1.6
incrementalPartitionPatterns	Collection of incremental dataPartitionPatterns objects for discovering and managing incremental data files.	1.6

Example Local Entity Declaration:

{
  "entityName": "RiskEntity",
  "entityPath": "RiskData/RiskEntity.cdm.json/RiskEntity",
  "dataPartitions": [
    {
      "location": "RiskData/2023/Q1/risk_records.csv",
      "exhibitsTraits": [
        {
          "traitReference": "is.partition.format.CSV",
          "arguments": [
            {
              "name": "columnHeaders",
              "value": "true"
            },
            {
              "name": "delimiter",
              "value": ","
            }
          ]
        }
      ]
    }
  ],
  "dataPartitionPatterns": [
    {
      "name": "QuarterlyRiskDataPattern",
      "rootLocation": "RiskData/",
      "regularExpression": "2023/Q(\\d)/risk_records\\.csv$",
      "parameters": [
        "quarter"
      ]
    }
  ]
}

Referenced Entity Declaration

Entities that the Manifest references are managed externally. The Manifest lists these entities without owning their data partitions, allowing GRIx to borrow entities from other solutions or Manifests.

Properties of Referenced Entity Declaration:

Property / Method	Description
entityName	The name of the referenced entity (e.g., Person).
entityPath	A corpus path to the entity declaration within the referenced Manifest document (e.g., remote:/OtherSolution/default.manifest.cdm.json/Person).

Example Referenced Entity Declaration:

{
  "entityName": "Person",
  "entityPath": "remote:/OtherSolution/default.manifest.cdm.json/Person"
}

In this example, the Person entity is borrowed from an external Manifest, allowing GRIx to utilize its schema and data without direct ownership.

---

The SubManifest Object

SubManifests allow GRIx to create hierarchical and modular structures within its data lake. By nesting SubManifests within a primary Manifest, GRIx can organize related solutions or extensions, facilitating scalability and manageability.

Properties of SubManifest:

Property / Method	Description
definition	The corpus path to the SubManifest's *.manifest.cdm.json document (e.g., waterProject/WaterManifest.cdm.json).

Example SubManifest Declaration:

{
  "subManifests": [
    {
      "name": "WaterHealthFoodEnergyNexus",
      "definition": "NexusSolutions/WaterHealthFoodEnergy.manifest.cdm.json"
    }
  ]
}

In this example, the WaterHealthFoodEnergyNexus SubManifest encapsulates entities and data pertinent to the interconnected domains of water, health, food, and energy, maintaining a logical separation within the broader GRIx framework.

---

Entity Relationships

Understanding and managing entity-to-entity relationships is crucial for comprehensive risk analysis in GRIx. These relationships define how different entities interact and influence each other, enabling holistic risk assessments across multiple domains.

Relationship Properties:

Property	Description
fromEntity	A corpus path to the referencing entity (e.g., RiskData/RiskEntity.cdm.json/RiskEntity).
fromEntityAttribute	The attribute in the referencing entity that acts as a foreign key (e.g., personId).
toEntity	A corpus path to the referenced entity (e.g., PersonData/Person.cdm.json/Person).
toEntityAttribute	The attribute in the referenced entity that acts as a primary key (e.g., id).
name	The name of the relationship (e.g., RiskEntity_PersonRelationship).
exhibitsTraits	Collection of traits that define additional semantics or behaviors for the relationship.

Example Relationship Declaration:

{
  "relationships": [
    {
      "fromEntity": "RiskData/RiskEntity.cdm.json/RiskEntity",
      "fromEntityAttribute": "personId",
      "toEntity": "PersonData/Person.cdm.json/Person",
      "toEntityAttribute": "id",
      "name": "RiskEntity_PersonRelationship",
      "exhibitsTraits": [
        {
          "traitReference": "means.referencing.attribute",
          "arguments": [
            {
              "name": "relationshipName",
              "value": "RiskOwner"
            }
          ]
        }
      ]
    }
  ]
}

In this example, the RiskEntity references the Person entity through the personId attribute, establishing a relationship that identifies the owner of a specific risk.

---

File Status Check and Modification Times

The Manifest object in GRIx maintains metadata about file statuses and modification times to ensure data integrity and facilitate efficient data management.

Key Properties:

Property / Method	Description	SDK Version
LastFileStatusCheckTime	The last time the Manifest checked the status of its files and child objects.	1.0
LastFileModifiedTime	The last time any file within the Manifest was modified.	1.0
LastChildFileModifiedTime	The latest modification time reported by any child object within the Manifest.	1.0

File Status Check Methods:

Object	Method	Description	SDK Version
Manifest	fileStatusCheck	Checks the Manifest, all entity declarations, and all SubManifests.	1.0
Local entity	fileStatusCheck	Checks the schema documents and all data partitions and patterns for the entity.	1.0
Referenced entity	fileStatusCheck	Checks the status of the remote Manifest document.	1.0
dataPartition	fileStatusCheck	Checks the file indicated by the data partition.	1.0
dataPartitionPattern	fileStatusCheck	Evaluates the data partition pattern search and creates new data partitions if applicable.	1.0
incrementalPartition	fileStatusCheck	Checks the file indicated by the incremental partition.	1.6
incrementalPartitionPattern	fileStatusCheck	Evaluates the incremental data partition pattern search and creates new incremental partitions if applicable.	1.6
SubManifest	fileStatusCheck	Checks the files of SubManifests.	1.0

Example Usage:

{
  "LastFileStatusCheckTime": "2024-04-20T10:00:00Z",
  "LastFileModifiedTime": "2024-04-18T15:30:00Z",
  "LastChildFileModifiedTime": "2024-04-19T12:45:00Z"
}

In this example, the Manifest records the last status check and modification times, aiding in monitoring data freshness and integrity.

---

Manifest Example Document

The following example demonstrates a comprehensive Manifest document tailored for GRIx, encompassing various entities across the water-health-food-energy nexus and multiple risk categories.

Example Manifest Document:

{
  "manifestName": "GRIxSolution",
  "jsonSchemaSemanticVersion": "0.9.0",
  "imports": [
    {
      "corpusPath": "cdm:/commonDefinitions.cdm.json"
    },
    {
      "corpusPath": "cdm:/foundations.cdm.json"
    },
    {
      "corpusPath": "cdm:/grixRiskEntities.cdm.json"
    },
    {
      "corpusPath": "cdm:/hazardDefinitions.cdm.json"
    },
    {
      "corpusPath": "cdm:/vulnerabilityDefinitions.cdm.json"
    },
    {
      "corpusPath": "cdm:/exposureDefinitions.cdm.json"
    },
    {
      "corpusPath": "cdm:/riskDefinitions.cdm.json"
    }
  ],
  "lastFileStatusCheckTime": "2024-04-25T08:00:00Z",
  "lastFileModifiedTime": "2024-04-24T14:20:00Z",
  "lastChildFileModifiedTime": "2024-04-25T07:50:00Z",
  "entities": [
    {
      "entityName": "WaterQualityRisk",
      "entityPath": "WaterHealthFoodEnergyNexus/Water_Risks/WaterQualityRisk.cdm.json/WaterQualityRisk",
      "dataPartitions": [
        {
          "location": "WaterHealthFoodEnergyNexus/Water_Risks/2024/Q1/water_quality_risks.csv",
          "exhibitsTraits": [
            {
              "traitReference": "is.partition.format.CSV",
              "arguments": [
                {
                  "name": "columnHeaders",
                  "value": "true"
                },
                {
                  "name": "delimiter",
                  "value": ","
                }
              ]
            }
          ]
        }
      ],
      "dataPartitionPatterns": [
        {
          "name": "QuarterlyWaterQualityRiskPattern",
          "rootLocation": "WaterHealthFoodEnergyNexus/Water_Risks/",
          "regularExpression": "2024/Q(\\d)/water_quality_risks\\.csv$",
          "parameters": [
            "quarter"
          ]
        }
      ]
    },
    {
      "entityName": "DiseaseOutbreakRisk",
      "entityPath": "WaterHealthFoodEnergyNexus/Health_Risks/DiseaseOutbreakRisk.cdm.json/DiseaseOutbreakRisk",
      "dataPartitions": [
        {
          "location": "WaterHealthFoodEnergyNexus/Health_Risks/2024/Mar/disease_outbreak_risks.csv",
          "exhibitsTraits": [
            {
              "traitReference": "is.partition.format.CSV",
              "arguments": [
                {
                  "name": "columnHeaders",
                  "value": "true"
                },
                {
                  "name": "delimiter",
                  "value": ","
                }
              ]
            }
          ]
        }
      ],
      "dataPartitionPatterns": [
        {
          "name": "MonthlyDiseaseOutbreakRiskPattern",
          "rootLocation": "WaterHealthFoodEnergyNexus/Health_Risks/",
          "regularExpression": "2024/(\\w{3})/disease_outbreak_risks\\.csv$",
          "parameters": [
            "month"
          ]
        }
      ]
    },
    {
      "entityName": "FoodSecurityRisk",
      "entityPath": "WaterHealthFoodEnergyNexus/Food_Risks/FoodSecurityRisk.cdm.json/FoodSecurityRisk",
      "dataPartitions": [
        {
          "location": "WaterHealthFoodEnergyNexus/Food_Risks/2024/Q2/food_security_risks.csv",
          "exhibitsTraits": [
            {
              "traitReference": "is.partition.format.CSV",
              "arguments": [
                {
                  "name": "columnHeaders",
                  "value": "true"
                },
                {
                  "name": "delimiter",
                  "value": ","
                }
              ]
            }
          ]
        }
      ],
      "dataPartitionPatterns": [
        {
          "name": "QuarterlyFoodSecurityRiskPattern",
          "rootLocation": "WaterHealthFoodEnergyNexus/Food_Risks/",
          "regularExpression": "2024/Q(\\d)/food_security_risks\\.csv$",
          "parameters": [
            "quarter"
          ]
        }
      ]
    },
    {
      "entityName": "EnergySupplyRisk",
      "entityPath": "WaterHealthFoodEnergyNexus/Energy_Risks/EnergySupplyRisk.cdm.json/EnergySupplyRisk",
      "dataPartitions": [
        {
          "location": "WaterHealthFoodEnergyNexus/Energy_Risks/2024/Q3/energy_supply_risks.csv",
          "exhibitsTraits": [
            {
              "traitReference": "is.partition.format.CSV",
              "arguments": [
                {
                  "name": "columnHeaders",
                  "value": "true"
                },
                {
                  "name": "delimiter",
                  "value": ","
                }
              ]
            }
          ]
        }
      ],
      "dataPartitionPatterns": [
        {
          "name": "QuarterlyEnergySupplyRiskPattern",
          "rootLocation": "WaterHealthFoodEnergyNexus/Energy_Risks/",
          "regularExpression": "2024/Q(\\d)/energy_supply_risks\\.csv$",
          "parameters": [
            "quarter"
          ]
        }
      ]
    },
    {
      "entityName": "FinancialRisk",
      "entityPath": "FinancialRisks/FinancialRisk.cdm.json/FinancialRisk",
      "dataPartitions": [
        {
          "location": "FinancialRisks/2024/Q1/financial_risks.csv",
          "exhibitsTraits": [
            {
              "traitReference": "is.partition.format.CSV",
              "arguments": [
                {
                  "name": "columnHeaders",
                  "value": "true"
                },
                {
                  "name": "delimiter",
                  "value": ","
                }
              ]
            }
          ]
        }
      ],
      "dataPartitionPatterns": [
        {
          "name": "QuarterlyFinancialRiskPattern",
          "rootLocation": "FinancialRisks/",
          "regularExpression": "2024/Q(\\d)/financial_risks\\.csv$",
          "parameters": [
            "quarter"
          ]
        }
      ]
    }
  ],
  "relationships": [
    {
      "fromEntity": "WaterHealthFoodEnergyNexus/Water_Risks/WaterQualityRisk.cdm.json/WaterQualityRisk",
      "fromEntityAttribute": "locationId",
      "toEntity": "GeographyData/Geography.cdm.json/Geography",
      "toEntityAttribute": "id",
      "name": "WaterQualityRisk_GeographyRelationship",
      "exhibitsTraits": [
        {
          "traitReference": "means.referencing.attribute",
          "arguments": [
            {
              "name": "relationshipName",
              "value": "LocatedIn"
            }
          ]
        }
      ]
    },
    {
      "fromEntity": "WaterHealthFoodEnergyNexus/Health_Risks/DiseaseOutbreakRisk.cdm.json/DiseaseOutbreakRisk",
      "fromEntityAttribute": "healthFacilityId",
      "toEntity": "HealthFacilities/HealthFacility.cdm.json/HealthFacility",
      "toEntityAttribute": "id",
      "name": "DiseaseOutbreakRisk_HealthFacilityRelationship",
      "exhibitsTraits": [
        {
          "traitReference": "means.referencing.attribute",
          "arguments": [
            {
              "name": "relationshipName",
              "value": "HostedAt"
            }
          ]
        }
      ]
    },
    {
      "fromEntity": "FinancialRisks/FinancialRisk.cdm.json/FinancialRisk",
      "fromEntityAttribute": "sectorId",
      "toEntity": "Sectors/Sector.cdm.json/Sector",
      "toEntityAttribute": "id",
      "name": "FinancialRisk_SectorRelationship",
      "exhibitsTraits": [
        {
          "traitReference": "means.referencing.attribute",
          "arguments": [
            {
              "name": "relationshipName",
              "value": "Affects"
            }
          ]
        }
      ]
    }
  ],
  "subManifests": [
    {
      "name": "NexusSolutions",
      "definition": "NexusSolutions/NexusSolutions.manifest.cdm.json"
    },
    {
      "name": "FinancialSolutions",
      "definition": "FinancialSolutions/FinancialSolutions.manifest.cdm.json"
    }
  ]
}

Explanation of the Example:

• Imports: The Manifest imports common definitions, foundational schemas, GRIx-specific risk entities, and definitions for hazards, vulnerabilities, exposure, and risks.

• Entities: Defines various risk entities across the water-health-food-energy nexus and financial risks, each with their respective data partitions and partition patterns.

• Relationships: Establishes connections between risk entities and other entities like Geography, HealthFacility, and Sector.

• SubManifests: Incorporates SubManifests for modular solutions, allowing for scalable and organized data management within GRIx.

---

The Manifest object is a cornerstone of the GRIx framework, enabling structured and efficient management of complex risk data across multiple domains. By leveraging CDM's Manifest capabilities, GRIx ensures that data related to hazards, vulnerabilities, exposure, and risks is systematically organized, easily accessible, and interconnected. This structured approach facilitates comprehensive risk assessments, advanced modeling, and informed decision-making within the intricate water-health-food-energy nexus and beyond.

By adhering to the principles and best practices outlined in this document, organizations can effectively implement the Manifest object within GRIx, ensuring robust data governance, scalability, and interoperability in their global risk management endeavors.

---

Further Reading and Resources

• Common Data Model Repository on GitHub
• Microsoft Common Data Model Documentation
• Azure Data Factory Documentation
• Azure Purview Documentation
• Power BI Documentation
• Azure Machine Learning Documentation
• Qiskit Documentation
• SHAP Documentation
• LIME Documentation