> For the complete documentation index, see [llms.txt](https://docs.therisk.global/organization/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.therisk.global/organization/cooperation/nexus-guilds/future-of-web/xi.-quality-system.md).

# XI. Quality System

Evidence Integrity, Measurement Rigor, and “No-Game” Durability for Web Intelligence

**Where this fits:** all six sectors ([The Model](/organization/model.md)) · [Labs](/organization/programme/labs.md), [Risk Academy](/organization/programme/academy.md) · communities of practice by sector, supplying the questions of the Labs pillar; National Working Groups ([NWG](/organization/governance-in-operation/nwg.md)) · community of practice.

**In brief.** Any comparative benchmark must meet elevated anti-gaming and disclosure minima. Read against the international frameworks it serves ([UN General Assembly 2015](https://www.undrr.org/publication/sendai-framework-disaster-risk-reduction-2015-2030); [FAO 2014](https://www.fao.org/4/bl496e/bl496e.pdf); [One Health High-Level Expert Panel (OHHLEP) et al. 2022](https://doi.org/10.1371/journal.ppat.1010537)) and the [Nexus Standards](/organization/institutions/nexus-standards.md) that govern its records ([NS-0](https://nexustandards.com/standards/ns-0/), [NS-13](https://nexustandards.com/standards/ns-13/)). "An experiment is exploratory: its results are labelled as such and become evidence only after the Foundry rebuilds and the Universe tests them." (Labs pillar boundary). Ordinary participation has no fee. Paid work, internships and fellowships are separate arrangements, and unqualified contributors never take safety-critical tasks.

{% hint style="info" %}
**Read with the current model (2026-10-21).**

* Everything this page describes is a design in formation. No national, regional or global consortium is recorded as constituted or active; the current service level is R0, information and enquiries ([Status and service levels](/organization/governance-in-operation/status-and-service-levels.md)). Where the text below reads as operation, read it as design.
* Recognition, scoring and validation record what was shown under stated conditions. None is a certification, licence, rating or approval, and no country or institution is ranked.
  {% endhint %}

### Part 11 — Quality System: Benchmarks, Anti-Gaming, and Longitudinal Comparability

#### 1. Status, Objective, and Design Doctrine

1.1 **Status.** This Part establishes the Guild’s quality system for methods, datasets, benchmarks, results publications, and enterprise-grade evidence outputs.

1.2 **Objective.** The objective is to ensure that Guild outputs are:\
1.2.1 **benchmarked** (comparable, repeatable, and interpretable),\
1.2.2 **tamper-resistant** (hard to game; easy to audit),\
1.2.3 **drift-aware** (explicitly monitored over time), and\
1.2.4 **correctionable** (errors are corrected by record; no silent edits).

1.3 **Design doctrine.** The quality system is built to survive:\
1.3.1 adversarial actors (gaming, poisoning, benchmarking arbitrage),\
1.3.2 vendor and sponsor influence pressure,\
1.3.3 regulatory and reputational incentives to overclaim, and\
1.3.4 natural drift in web ecosystems, tooling, and measurement methods.

1.4 **Non-equivalence.** Quality markings, benchmark rankings, and deployability levels do not constitute certification, endorsement, procurement recommendations, or compliance determinations.

***

#### 2. Quality Ladders and Minimum Disclosures

2.1 **Evidence sufficiency ladder (E0–E4).** Every claim-bearing output must declare its evidence level and the evidence it depends on.\
2.1.1 **E0** — opinion/hypothesis without formal evidence;\
2.1.2 **E1** — single-source or limited observation;\
2.1.3 **E2** — multi-source observation with basic validation;\
2.1.4 **E3** — reproducible evidence with structured tests and error bounds;\
2.1.5 **E4** — high-confidence evidence with broad coverage, bias controls, and replayable lineage.

2.2 **Reproducibility ladder (RS0–RS4).** Every benchmark, dataset, method, or scoring output must declare its reproducibility level.\
2.2.1 **RS0** — not replayable;\
2.2.2 **RS1** — partially replayable (key steps missing);\
2.2.3 **RS2** — replayable with significant effort (manual dependencies);\
2.2.4 **RS3** — replayable using published runbooks and pinned environments;\
2.2.5 **RS4** — fully replayable with automated pipelines, provenance, and deterministic outputs within stated tolerances.

2.3 **Dataset quality ladder (DQ0–DQ4).** All datasets must declare DQ level.\
2.3.1 **DQ0** — ad hoc sample; unknown coverage;\
2.3.2 **DQ1** — documented source; limited coverage; minimal cleaning;\
2.3.3 **DQ2** — documented sampling; basic bias controls; refresh posture stated;\
2.3.4 **DQ3** — robust provenance; drift monitoring; systematic refresh and deprecation;\
2.3.5 **DQ4** — benchmark-grade datasets with audit-ready lineage, bias controls, and controlled evolution.

2.4 **Minimum disclosure package (mandatory).** Any published benchmark result, dataset, or scorecard must disclose:\
2.4.1 sampling method and frame;\
2.4.2 measurement boundaries and exclusions;\
2.4.3 uncertainty and error posture;\
2.4.4 known failure modes;\
2.4.5 drift indicators and last recalibration;\
2.4.6 handling class and reliance bounds;\
2.4.7 correction path and supersession pointers.

***

#### 3. Benchmark Constitution

3.1 **Benchmark definition.** A benchmark is a defined measurement procedure producing outputs that are comparable over time and across subjects under declared assumptions.

3.2 **Benchmark components.** Every benchmark must include:\
3.2.1 a formal specification (what is measured; how);\
3.2.2 a test harness (implementation that executes the benchmark);\
3.2.3 scoring rules (how results are computed);\
3.2.4 validity conditions (when results are admissible);\
3.2.5 exclusion rules (what must not be measured or must be redacted);\
3.2.6 an anti-gaming posture (tamper indicators and defenses);\
3.2.7 drift monitoring and recalibration rules;\
3.2.8 an appeals and dispute pathway.

3.3 **Benchmark classes.** Benchmarks may be:\
3.3.1 **descriptive** (measure and describe reality),\
3.3.2 **comparative** (rank or compare), or\
3.3.3 **diagnostic** (identify likely causes or contributing factors).\
Any comparative benchmark must meet elevated anti-gaming and disclosure minima.

3.4 **Benchmark reliability statement.** Each benchmark must publish a reliability statement specifying expected false positive and false negative behavior, known blind spots, and “do not use for” scenarios.

***

#### 4. Measurement Error Budgets and Uncertainty Discipline

4.1 **Error budget requirement.** Every benchmark must define and publish an error budget:\
4.1.1 acceptable measurement error bounds;\
4.1.2 expected noise sources (network variance, caching, geo effects, CDNs, bot countermeasures, consent banners);\
4.1.3 model inference uncertainty where AI classifiers are used;\
4.1.4 tolerance bands for replayability;\
4.1.5 confidence intervals or equivalent uncertainty indicators.

4.2 **Uncertainty must travel.** Uncertainty bounds must follow results into every derivative output, including scorecards, dashboards, and enterprise evidence packs.

4.3 **No precision theater.**\
4.3.1 Results must not be presented with more precision than the method can support.\
4.3.2 Rank ordering must be avoided where confidence intervals overlap materially, unless explicitly labeled as “non-separable.”

***

#### 5. Anti-Gaming and Tamper Resistance

5.1 **Adversarial posture.** The Guild assumes that benchmarks and scorecards will be targeted for manipulation.

5.2 **Anti-gaming controls (minimum set).** For benchmark-grade releases, at least the following must be implemented and documented:\
5.2.1 **sampling hardening** (diverse vantage points; stratification; rotation);\
5.2.2 **measurement randomization** where appropriate (timing jitter; route diversity);\
5.2.3 **tamper indicators** (unexpected uniformity, sudden discontinuities, unnatural compliance spikes);\
5.2.4 **cross-source corroboration** (multiple independent signals);\
5.2.5 **drift alarms** (threshold-based and statistical);\
5.2.6 **replay verification** (periodic re-runs for comparability);\
5.2.7 **appeals channel** to detect systematic artifacts and false positives.

5.3 **Benchmark tampering register.** A controlled internal register must track:\
5.3.1 suspected manipulation events;\
5.3.2 affected benchmarks and time windows;\
5.3.3 mitigation actions;\
5.3.4 whether public correction notices were issued.

5.4 **No “security through obscurity” as a primary control.** Methods must be publishable at the level required for reproducibility, while sensitive exploit-enabling detail is constrained via handling.

***

#### 6. Drift Monitoring, Recalibration, and Version Discipline

6.1 **Drift is expected.** The web changes continuously; benchmarks must treat drift as a first-class phenomenon.

6.2 **Drift categories.**\
6.2.1 ecosystem drift (infrastructure concentration, protocol changes);\
6.2.2 measurement drift (tooling updates, vantage changes);\
6.2.3 adversarial drift (countermeasures, deception);\
6.2.4 regulatory drift (consent requirements, content rules).

6.3 **Recalibration rules.** Benchmarks must define:\
6.3.1 triggers for recalibration;\
6.3.2 how recalibration affects comparability;\
6.3.3 whether historic results are restated or merely annotated;\
6.3.4 how supersession is recorded.

6.4 **No silent edits.** Any recalibration that changes outputs or interpretations must be recorded, published at the appropriate handling level, and linked to prior versions.

***

#### 7. Appeals, Disputes, and Correctionability

7.1 **Appeals channel.** Subjects of measurement may request review when they believe a benchmark result is wrong or misleading.

7.2 **Dispute admissibility.** Disputes must include:\
7.2.1 the claimed error;\
7.2.2 evidence or reproduction attempts;\
7.2.3 relevant time windows;\
7.2.4 any changes made by the subject that could explain differences.

7.3 **Resolution outcomes.**\
7.3.1 confirm result;\
7.3.2 correct result;\
7.3.3 annotate with limitations;\
7.3.4 deprecate benchmark method;\
7.3.5 suspend publication pending investigation.

7.4 **Correction clocks.** Corrections follow published clocks, with emergency tiers for high-risk errors.

***

#### 8. Longitudinal Comparability and “No Reliance Traps”

8.1 **Longitudinal comparability objective.** Results must remain interpretable across time without creating false narratives due to unannounced method changes.

8.2 **Comparability mechanisms.**\
8.2.1 stable reference sets (anchor samples);\
8.2.2 back-testing and bridge studies when methods change;\
8.2.3 explicit regime-change flags;\
8.2.4 “apples-to-apples” disclaimers when comparability is broken.

8.3 **No reliance traps.**\
8.3.1 Historical results must remain accessible with clear supersession pointers.\
8.3.2 Deprecated methods must be labeled as such, with safe citation guidance.\
8.3.3 Users must be warned against using out-of-date outputs for consequential decisions.

***

#### 9. Quality Governance: Roles, Reviews, and Independence

9.1 **Quality authority.** The Integrity Steward (or designated Quality Steward) holds stop-the-line authority for quality failures.

9.2 **Independent review requirement.** Benchmark-grade releases require review by parties not responsible for producing the benchmark.

9.3 **Rotation and concentration controls.**\
9.3.1 rotating reviewers and maintainers for high-impact benchmarks;\
9.3.2 influence caps and COI recusal for sponsor-linked contributors;\
9.3.3 publication of governance minima at the appropriate handling level.

9.4 **Audit posture.** For D3/D4 outputs, the Guild must maintain an audit trail sufficient to:\
9.4.1 reproduce published results within stated tolerances;\
9.4.2 show who approved the release and under what gates;\
9.4.3 demonstrate that correction processes are functioning.

***

#### 10. Quality Markings and Publication Rules

10.1 **Permitted markings.** Outputs may carry markings such as:\
10.1.1 Guild-Reviewed;\
10.1.2 Lab-Validated;\
10.1.3 Benchmark-Ready;\
10.1.4 Release-Ready (bounded);\
10.1.5 Enterprise-Deployable (governed).\
Each marking must be backed by recorded criteria.

10.2 **Prohibited markings.** Outputs must not use:\
10.2.1 “certified,” “approved,” “compliant,” “safe,” “secure,” or equivalent unqualified claims;\
10.2.2 “recommended vendor,” “approved list,” or procurement-adjacent phrasing.

10.3 **Public communications integrity.** Any public summary must:\
10.3.1 carry uncertainty and limitation statements;\
10.3.2 avoid sensationalism;\
10.3.3 avoid actionable exploit enablement;\
10.3.4 avoid claims exceeding E/RS/DQ declarations.

## Sources

1. UN General Assembly (2015). Sendai Framework for Disaster Risk Reduction 2015–2030. A/RES/69/283, 3 June 2015. <https://www.undrr.org/publication/sendai-framework-disaster-risk-reduction-2015-2030>
2. FAO (2014). The Water-Energy-Food Nexus: A new approach in support of food security and sustainable agriculture. Rome: Food and Agriculture Organization of the United Nations. <https://www.fao.org/4/bl496e/bl496e.pdf>
3. One Health High-Level Expert Panel (OHHLEP), Adisasmito, W. B. et al. (2022). One Health: A new definition for a sustainable and healthy future. PLoS Pathogens 18(6): e1010537. doi:10.1371/journal.ppat.1010537. <https://doi.org/10.1371/journal.ppat.1010537>
4. Nexus Standards. NS-0: Constitution and principles. Status: PROPOSED; UNI `NXS-STD-ST-000004`. <https://nexustandards.com/standards/ns-0/>
5. Nexus Standards. NS-13: Open standards development process. Status: PROPOSED; UNI `NXS-STD-ST-000008`. <https://nexustandards.com/standards/ns-13/>
6. GCRI. The Labs pillar home. <https://therisk.global/nexus-labs/>
7. GCRI. Working groups (NWG). <https://therisk.global/consortium/national/nwg/>
8. This documentation: [Governance in operation](/organization/governance-in-operation.md); [The Model](/organization/model.md); [historical labels](/organization/reference.md).


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.therisk.global/organization/cooperation/nexus-guilds/future-of-web/xi.-quality-system.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.