Skip to content

Managing Feature Flags

This guide explains how to add, modify, and use Boolean feature flags in SyRF, including non-production runtime overrides shared by browser and API consumers.

Overview

The feature flags system uses env-mapping.yaml as the single source of truth for all environment configuration, including feature flags. The generator reads from this unified schema and produces:

  • TypeScript interfaces (FeatureFlags, FeatureFlagsExternalConfig)
  • Default values (FEATURE_FLAG_DEFAULTS)
  • NgRx selectors for each flag
  • Config parser (parseFeatureFlags()) for runtime parsing
  • Updates to the web chart's Helm values.yaml for deployment configuration

The NGINX envsubst template (env.template.generated.json), appConfig.default.generated.json, the config TypeScript files and _env-blocks.tpl are written by a separate generator, pnpm run generate:env-blocks (from src/charts/syrf-common). generate:runtime-catalog writes RuntimeFeatureFlagCatalog.generated.cs. See Work with generated files.

Key benefits:

  • Adding a new feature flag requires only editing env-mapping.yaml and running the generator
  • Uses unified SYRF__ prefix for all environment variables (not legacy SPA__)
  • The API receives the same deployed defaults as the web app, so cross-stack gates such as bulkPdfUpload stay consistent
  • Non-production administrators can temporarily override catalogued Boolean feature flags where runtime changes are supported, without replacing the deployed defaults
┌─────────────────────────────────────────────────────────────┐
│           src/charts/syrf-common/env-mapping.yaml            │
│                  (Single Source of Truth)                    │
│                                                              │
│  featureFlags:                                               │
│    services: [api]  # Shared web/API deployed defaults       │
│    envVars:                                                  │
│      - name: SYRF__FeatureFlags__MyNewFeature                │
│        valuePath: featureFlags.myNewFeature                  │
│        default: "false"                                      │
│        web: { tsName: myNewFeature, tsType: boolean }        │
│        featureFlag: true                                     │
│        displayName: "My new feature"                         │
│        category: uiFeatures                                  │
│        description: "Enable my new feature"                  │
└─────────────────────────────────┬────────────────────────────┘
                                  │
                        pnpm run generate:flags
                                  │
                      ┌───────────┴────────────┐
                      ▼                        ▼
           ┌─────────────────────┐  ┌─────────────────────┐
           │  TypeScript Files   │  │   Helm values.yaml  │
           │                     │  │  (web .chart)       │
           │  - types.ts         │  │  featureFlags:      │
           │  - constants.ts     │  │    myNewFeature:    │
           │  - selectors.ts     │  │      false          │
           │  - parser.ts        │  │                     │
           └─────────────────────┘  └─────────────────────┘

File Locations

File Purpose
src/charts/syrf-common/env-mapping.yaml Schema definition (edit this!)
src/services/web/scripts/generate-feature-flags.ts Generator script
src/services/web/src/app/generated/feature-flags.types.ts Generated TypeScript interfaces (types only, no JS output)
src/services/web/src/app/generated/feature-flags.constants.ts Generated runtime constants (as const)
src/services/web/src/app/generated/feature-flags.selectors.ts Generated NgRx selectors
src/services/web/src/app/generated/feature-flags.parser.ts Generated config parser
src/services/web/src/assets/data/env.template.generated.json envsubst template, written by generate:env-blocks (not generate:flags)
src/services/web/.chart/values.yaml Auto-updated Helm values

Flag lifecycle requirements

A feature flag switches a feature on and off. It does not migrate data. Chris's rule (2026-10-17):

"Flags are used to turn features on and off, not necessarily to auto migrate databases back and forth. So whilst the flag is present and flippable both the database and app code should work well with either on or off state and should be happy going back and forth. That should be an implementation requirement when adding the feature flag in the first place. (The extra infrastructure for and to support a specific flag can be removed at a later stage when the flag is retired.)"

So while a flag exists, the code and the database must work with it on or off, and survive being switched back and forth any number of times, including with data created in the other state. The author of the flag builds and tests that when the flag is added; it is not left to be discovered during a rollback.

This section is the canonical text. The agent checklist in .claude/rules/feature-flags.md summarises it, and ADR-023 records the migration side. If they differ, this section wins.

Environments, deployments, migrations and flags

Four terms have been getting confused. Use them in this sense in pull requests, runbooks and ADRs.

  • Environment: local, preview, staging or production. Each has its own database and its own flag values.
  • Deployment: puts a build into an environment. A promotion deploys staging's build to production; a rollback deploys the previous build. Flag values are configuration: changing one is a separate decision from deploying a build, even when a deployment-managed flag needs a rollout to change.
  • Migration: part of a build. It runs before that build is deployed (ADR-023). While the previous build may still be serving, a migration only adds things, and it must suit every flag state of both builds. Flags never run, skip or reverse a migration, and no migration reads a flag.
  • Flag: a switch that a running build reads, per environment. The only allowed link with migrations is one-way: a feature may refuse to switch on, failing closed, until its migrations have run (its readiness check).

For example, deletionLifecycle needs the deletion indexes, which are migrations 0005 to 0008 (ADR-024). With the flag on but those migrations missing, the search-delete routes answer a typed 503 instead of recording a deletion that nothing could advance. That is the allowed direction: the missing migration keeps the feature closed. Turning the flag on or off never applies or reverses those migrations, and applying them does not turn the flag on.

Requirements when a flag is added

  1. Both states work. On is the new behaviour. Off is the previous behaviour, or the feature shown as unavailable; never broken. Off is not only a rollback state: for a flag that production has not turned on, it is production's behaviour, which is what keeps main promotable.
  2. Round trips are safe, any number of times. Data created while the flag is on is handled correctly while it is off (kept, ignored, or finished by work that is not flag-gated), and the reverse. Work already accepted keeps converging after the flag goes off: a kill switch stops new exposure, it does not strand what was already started.
  3. Flags read by several services. The browser, the API and Project Management may read the same flag, and a rollout reaches them at different times. Every combination of per-service values that a rollout can produce is safe, or the feature refuses to start until the values agree. The dependency catalog (see Non-Production Runtime Overrides) is one way to enforce this.
  4. Tests. An on, then off, then on test, with data created in each state.
  5. Retirement. The dead path is deleted. Anything that existed only for the flag (code, collections, indexes, data) is removed in a later clean-up, by a migration if indexes or data are involved, once no deployable build and no flag state needs it. Until then it stays, so that switching the flag back is always possible. pnpm feature:settle does the mechanical part (see Settling a feature).

The pull request template has a checkbox for requirements 1, 2 and 4.

Adding a New Feature Flag

Adding a flag includes meeting the flag lifecycle requirements: both states work, round trips are safe and the on, off, on test exists. The steps below cover only the wiring.

Step 1: Scaffold the feature and its flag

Two commands, from the repository root:

pnpm feature:new <feature-id> --type <Release|Operational|KillSwitch|Experiment|Permission> --owner "<team>" \
  [--permission <SignedIn|Project...Policy|Application...Policy>] [--flag <key>] [--hosts api,project-management] [--title "Sentence case label"] [--description "..."] [--category <category>]
pnpm run validate:generated

--permission is required with --type Release (see Feature fields). For example, pnpm feature:new demo-x --type Release --owner "API maintainers" --permission ProjectEditMembershipsPolicy --flag demoX. The command:

  1. Adds one entry to the featureFlags section of src/charts/syrf-common/env-mapping.yaml: SYRF__FeatureFlags__<Key>, Helm value featureFlags.<key> (default "false"), the web name and displayName, and runtimePolicy with the next free order and activationRequirement: immediate. --flag defaults to the feature ID in camelCase, --title to the ID in sentence case, and --category to uiFeatures.
  2. Adds one schema 2 feature to src/charts/syrf-common/feature-definitions.yaml: lifecycle: Developing, grain: Global, the flag in flags, a single-flag capability and the type's defaults (see Feature fields). A Release gets topicIds: [<feature-id>]: its user guide is Required, so validate:registry reports "guidance missing" for that topic until content exists. Every other type is exempt with a reason and has no topics.
  3. With --hosts api,project-management, also adds the Project Management half in backendFeatureFlags and a deploymentManaged pin, because Project Management never reads runtime overrides (see Carrying a web flag to Project Management).
  4. Runs the generators that read those two files (pnpm generate runs all of them, in order) and the registry validator, then prints the remaining steps. If a generator throws, it restores every file it could have changed (both sources, the checksum manifest and those generators' outputs) and reports that nothing changed.

It refuses, and writes nothing, when the feature ID, the flag key, its environment variable, its web name or its Helm value already exists, or when the key is a removed flag (a removed key is never reused). It writes no consumer code, content, What's New entry, walkthrough or cluster-gitops value. Those are the remaining steps it prints:

  • Consumer code: the web reads select<Key> (Step 2 below); .NET reads RuntimeFeatureFlagKeys.<Key>, which is generated. A .NET consumer also needs its property in FeatureFlags.cs and its entry in the consumer manifest.
  • Catalog digest: a new catalogued flag changes the catalog digest that every stored runtime policy identity carries. Update PinnedCatalogDigest in RuntimeCatalogDigestPinTests to the value the failing test prints. It is the only pin on the catalog's membership.
  • Content and What's New (a Release): a non-Draft guide page that declares the topic in its front matter (syrf.topics, see Guide topics), and a runtime: entry for the feature in docs/content/whats-new.yaml, before the feature leaves Developing (an announced release Apply refuses without it: WhatsNewEntryMissing). See Runtime What's New entries.
  • Walkthrough: optional; a walkthroughs.yaml entry plus data-tour-id anchors and communication.walkthroughId. See Walkthroughs.
  • cluster-gitops: featureFlags.<key> per environment (Step 3 below).

This is what feature:new writes, for reference or for a flag that needs a different section:

featureFlags:
  services: [api]  # Shared web/API deployed defaults
  displayName: "Feature Flags"
  envVars:
    # ... existing flags ...

    - name: SYRF__FeatureFlags__MyNewFeature
      runtimePolicy:
        key: myNewFeature
        order: 54
        category: uiFeatures
        description: "Enable my new feature"
        activationRequirement: immediate
      valuePath: featureFlags.myNewFeature
      default: "false"
      web:
        tsName: myNewFeature
        tsType: boolean
      featureFlag: true
      displayName: "My new feature"
      category: uiFeatures
      description: "Enable my new feature"

displayName is required for every featureFlag: true entry: write a short sentence-case label. The generator emits it as FEATURE_FLAG_DISPLAY_NAMES in feature-flags.constants.ts, and the admin runtime feature flags page (/admin/features/flags) shows it as the row name. Generation fails with the list of offending entries if any feature flag lacks a displayName.

runtimePolicy is the runtime catalog entry: stable key and order, category, description, activation requirement and dependencies. The runtime key must match web.tsName when that browser alias is present. Application-settings-only catalog entries belong in featureFlags.catalogOnlyFlags and require configurationReason; they do not create new Helm values. Dependency entries must name known runtime or deployment keys, and cycles are rejected. Catalog membership does not confer runtime eligibility; do not add arbitrary Boolean operational or security settings.

After a hand edit of env-mapping.yaml or feature-definitions.yaml, run pnpm generate and then pnpm run validate:generated. pnpm generate runs, in order: generate:env-blocks (env template, appConfig defaults, config TS files, _env-blocks.tpl), generate:flags (feature-flag TypeScript and the web .chart/values.yaml), generate:runtime-catalog (RuntimeFeatureFlagCatalog.generated.cs and RuntimeFeatureFlagKeys.generated.cs), generate:feature-definitions, the Dockerfiles, and the NSwag client only when swagger.json changed. Then it runs the registry validator. pnpm generate <name> runs only the named generators. Do not hand-edit the generated outputs or add definitions or key constants to the handwritten RuntimeFeatureFlagCatalog.cs facade.

Step 2: Use the Flag in Components

Import and use the generated selector:

import { selectMyNewFeature } from '@app/generated/feature-flags.selectors';

@Component({
  selector: 'app-my-component',
  template: `
    @if (myNewFeatureEnabled()) {
      <div>New feature content!</div>
    }
  `
})
export class MyComponent {
  private store = inject(Store);

  // Using signals (recommended)
  myNewFeatureEnabled = this.store.selectSignal(selectMyNewFeature);

  // Or using observables
  myNewFeature$ = this.store.select(selectMyNewFeature);
}

Step 3: Configure for Deployment

The Helm values.yaml is auto-updated. To enable in a specific environment, update cluster-gitops:

# cluster-gitops/syrf/environments/staging/web/values.yaml
featureFlags:
  myNewFeature: true  # Enable in staging

Env var precedence and boolean defaults

The chart renders one env entry per name. Two rules keep that true:

  • Explicit env: overrides generated flag env vars. If an environment's raw env: map (in cluster-gitops) sets a name that the generated block also emits, for example SYRF__FeatureFlags__BulkStudyUpdateRequireCurrentExecutionVersion: "true", the generated entry is skipped and only the explicit value is rendered. This is done in syrf-common.deployment.dotnet (syrf-common.env.withoutExplicit), so it applies to api, project-management, quartz and identity. Prefer the flag's Helm value (featureFlags.<name>) over a raw env: entry; the override exists so a duplicate key can never be rendered, not as the normal way to set a flag.
  • An explicit false is honoured when the default is "true". Helm's default treats false as empty, so a plain | default "true" can never render false. For any envVars entry whose default: "true", the generator emits include "syrf-common.boolOrDefault", which keeps an explicit bool and falls back to the default only when the value is unset. Do not hand-write | default "true" on a boolean in a chart template.

A chart value that has no env-mapping.yaml entry renders nothing. Do not add one to a service chart's values.yaml as documentation; an application setting that is read from appsettings (for example FeatureFlags:MaxInProgressSessions) is not configured by the chart.

Carrying a web flag to Project Management

The featureFlags section renders only into the API. When Project Management also reads a web flag, declare the same variable and valuePath in the backendFeatureFlags section too (and keep the web: block only on the featureFlags entry), as bulkPdfUpload and reviewEligibilityPolicy do. One featureFlags.<name> Helm value then drives the browser, the API and Project Management. A flag that is read by Project Management but declared only in featureFlags is silently off there (#4292).

Project Management does not read the API's runtime override store, so a runtime override changes the API and the browser but never Project Management. A flag read by both services must therefore be deployment-managed: the API pins its effective value to the deployed value, rejects a runtime change with 400 Bad Request and shows it as deployment-managed on the Feature flags admin page. An override saved before a flag was pinned has no effect and can still be cleared. reviewEligibilityPolicy is deployment-managed for this reason (it keeps its pageReload activation, which now only describes how a changed deployed value reaches an open browser). Change it in GitOps for every consuming service. To pin another flag, add it with its constraint and refusal text to deploymentManaged in src/charts/syrf-common/feature-definitions.yaml and regenerate (see Feature registry).

Carrying a web flag to Identity

Do not add identity to the featureFlags section's services: that would render every catalogued web flag into the Identity pod. When Identity needs one web flag, add a narrow Identity-only section that maps the same Helm value, as identityThemeFlags does for themeToggle (account portal slice 3, shared theme preference):

identityThemeFlags:
  services: [identity]
  envVars:
    - name: SYRF__FeatureFlags__ThemeToggle
      valuePath: featureFlags.themeToggle
      default: "false"

Identity binds it on IdentityFeatureFlags at start-up and never reads the API's runtime override store, so the Identity half is deployment-only: a runtime override on the Feature flags admin page changes the app, not Identity. Set the value in the Identity service's environment values in cluster-gitops as well as the web and API values.

Reading a flag in .NET

API and Project Management code reads a runtime flag through one interface, IFeatureDecisions (SyRF.Mongo.Common.RuntimeFeatureFlags, spec 6R.3 slice R3c). Inject it and use the decision's value:

public sealed class ExampleController(IFeatureDecisions decisions) : ControllerBase
{
    [HttpPost]
    public IActionResult Start() =>
        decisions.Decide(RuntimeFeatureFlagKeys.ProportionalStudyAllocation).Value ? Ok() : NotFound();
}
  • Never pass a default. Value is always a decision you may act on. When the policy cannot decide (not yet read, stale, unavailable, no reviewed contract), it is the safe value from the catalog's consumer contract, which is false for every reviewed contract today; Reason and ErrorCode say why. The safe value belongs to the catalog, never to the caller.
  • Decisions are synchronous and never throw for any reader state, so they fit route selection, authorization handlers and admission checks.
  • Several keys at once: DecideAll(keys) answers every key from one capture of each policy source, so a capability that needs all (or any) of several flags never mixes two revisions. AllEnabled and AnyEnabled answer AllOf and AnyOf.
  • FeatureFlags is deployed configuration, bound at start-up. Read it for start-up or domain settings, not for a decision that a runtime override or stop must change.

Which path decides a key depends only on the host's configuration, exactly as before R3c:

Host configuration Keys Decided by
RuntimeFeatureFlags:SharedReaderMode=Membership (API) disableMembership The legacy-format shared reader (60 s maximum age)
SharedReaderMode=StatisticsServing (PM) statistics serving and its three consumer selectors The legacy-format shared reader, with this host's FeatureFlags as fallbacks while it is current
RuntimeFeatureFlags:Mode=Shared keys with a consumer contract The bound authority (contract maximum age; stops force the key off; production confines runtime policy to eligible keys)
Anything else every other key The host's compatibility values: the API's runtime provider (legacy overrides, deployment pins, AuthorityPreserved), or Project Management's FeatureFlags

Start-up refuses Mode=Shared together with an enrolled shared reader, so no key has two paths. Production with ProductionPolicyMode=Legacy resolves to Legacy mode. The reasons are Evaluated, Stopped, DeploymentControlled, Cached, Legacy, AuthorityPreserved, Stale, Unavailable and Unreviewed. PolicyVersion is an opaque string, comparable only between decisions from the same source in one process.

Do not read IRuntimeFeatureFlagProvider, SharedRuntimeFeatureFlagReader or RuntimePolicyAuthorityEvaluator directly: the consumer inventory guard fails CI on a new direct read. In tests, mock IFeatureDecisions (Decide returns a FeatureDecision), or build RuntimeFeatureDecisions over a real reader or evaluator when the test is about the path. The vocabulary follows OpenFeature, but SyRF does not use the OpenFeature SDK: it would lose the catalog-owned safe value, synchronous decisions and single-capture multi-key reads (spec 6R.3). A later adapter could expose IFeatureDecisions as an OpenFeature provider.

Flag decision (R3c): not flagged. Parity tests in SyRF.Mongo.Common.Tests, SyRF.API.Endpoint.Tests and SyRF.ProjectManagement.Endpoint.Tests prove the same value as the replaced call sites for every catalog key and every reader state; a flag would keep both paths alive.

A flag with a dependency and a deployment-managed halt: notificationEmail

notificationEmail (feature notification-email, #3942, ADR-026 decision 13) shows three patterns together:

  • A declared dependency on a pinned flag. It depends on notificationInbox, which is deployment-managed (deploymentManaged). Because notificationInbox is a catalogued key, the dependency is declared runtimeManageable: true (a runtimeManageable: false dependency is only for a non-catalog backend key such as deletionLifecycle, which has a value resolver; a catalogued key there stops the API at start-up). The pin still refuses a runtime change to notificationInbox itself.
  • Both flags from one capture. Its single gate, INotificationEmailAdmission, reads DecideAll([notificationEmail, notificationInbox]).AllEnabled, so the two are never read from different revisions. The route gate (FeatureGated) decides notificationEmail alone; the in-action check adds the dependency.
  • A deployment-managed halt beside the flag. NotificationEmailDelivery:Paused (chart value notificationEmailDelivery.paused, default true; unset also means paused) stops sending without touching the flag or any data: paused, the worker keeps discovering saved items but claims and sends nothing; lifting it sends each waiting item once with its send-time checks. So switching notificationEmail on sends nothing until an environment's values lift the pause. Lifting it is an owner decision per environment (G-NOTIF); outside production the email transport must be Mailpit (emailTransport.provider: smtp), and production's is SES. Rendered with dig so an explicit false survives (a plain default would turn it back into true); notification_email_delivery_pause_test.yaml in the API chart pins all four cases.

Off, the preference routes answer 404, /inbox/email-settings falls through to not-found and the worker reads no preference or ledger row; every row is kept, so on, off, on loses nothing (NotificationEmailWorkerTests).

Daily digests (#3943) add no flag: NotificationDigestWorker asks the same INotificationEmailAdmission (which also requires the digest window migration 0032) and honours the same pause. Off, a digest cycle reads nothing; paused, it prepares, claims and sends nothing. Windows and held ledger entries are kept, so on, off, on sends each prepared digest once (NotificationDigestWorkerTests).

Settling a feature

A Released feature settles when it has been on everywhere for longer than its settle period and nobody needs "off" any more: its flag and the "off" branches go, and the feature becomes a permanent part of the product. The clean-up PR is titled "remove flag <key> for <feature-id>".

pnpm feature:settle <feature-id>
pnpm run validate:generated   # after deleting the references it prints

feature:settle does the mechanical part:

  1. Removes every env-mapping.yaml declaration of the feature's flags (each section's entry for the same SYRF__FeatureFlags__<Key>, and a section left empty), and any catalogOnlyFlags entry.
  2. Sets the feature's registry entry to lifecycle: Settled. Its flags and capability stay as history.
  3. Adds the keys to the removed flags (removedFlags under featureFlags in env-mapping.yaml, generated as RuntimeFeatureFlagKeys.RemovedFlags). Only catalogued keys are evaluated, so a stored runtime override for a removed key is ignored; RuntimeFeatureFlagCatalogTests checks that for every removed flag. A removed key is never reused: the generator and feature:new refuse it.
  4. Drops the flag's deploymentManaged pin, if any.
  5. Regenerates, then prints every reference still to delete: the consumer-manifest reads, and a search of tracked files (generated files excluded) for the key, its configuration property, its select<Key> selector and its environment variable. If a generator throws, it restores every file it could have changed and reports that nothing changed.

It refuses, and writes nothing, a feature that is not Released, and a flag that a flag group still names (group IDs are versioned contracts, so remove the group in its own reviewed change first).

In the same PR, delete the "off" branches and their tests, the FeatureFlags.cs properties and the consumer-manifest entries, and update the catalog digest pin as for a new flag. The registry validator refuses a Settled feature whose key is still catalogued or is not a removed flag.

Ready to settle

Nobody types a review-by date (D21). Each environment's admin area works out which features are ready from its own records, and nothing time-based ever fails a build. A feature is listed as Ready to settle when it is a Release that is not Settled or Withdrawn, no emergency stop is in force on its flags, and it has been live for everyone here for longer than its settle period (registry settleDays, 30 days for a Release unless the registry sets it).

Two sources (D43 = B). The read takes "went live for everyone" from both:

  • Release records. The latest GeneralRelease record of this binding that is confirmed, or imported with Date precision. It counts from the later of its confirmation and its last Restored event, and gives no date while its last stop event (Paused or EmergencyStopped) has no restore after it. Month and Unknown imports give no date: settling needs one.
  • The flag change history. The current policy lineage's audit receipts (runtimeFeatureFlagAudit), walked back from the current revision: the date is the commit of the earliest revision from which every revision up to now had all the feature's flags on for everyone (saved value on, no stop, no targeting rules). A feature switched on by hand, with no release record, therefore still appears. If it was already on when the lineage began, the date is when the lineage began; if more than 500 changes have happened since, the date is the oldest one read. Both are later than the truth, never earlier. In production, flags the deployment controls are not recorded in the policy, so the history says nothing about them and the release record decides.

The later date wins. When the sources disagree, the later date is used, so a feature is never called ready sooner than both allow, and the feature page notes the disagreement with both dates (dates more than an hour apart, or a day against a Date-precision import, which stores the start of its day; a confirmation always lands seconds after its commit). If the release record says live but the flags are not on for everyone now, the feature is not listed and the page says so.

Where it shows (6G AD3, behind RuntimeFeatureReleases:Enabled): an Overview attention item per ready feature ("live for everyone for N days — ready to settle") and a "N features ready to settle" line on the summary card; a "Ready to settle" lifecycle filter, status chip and banner on Features; and on the feature's Overview tab the lifecycle step, a "Settling this feature" card with the settle period, the live-since and settle dates, the source and any disagreement note. The Create clean-up issue link is built in the browser: a GitHub new-issue URL (https://github.com/camaradesuk/syrf/issues/new?title=…&body=…, under 8 KB, no credential) prefilled with the environment, the feature, its flags and owner, the dates, "set lifecycle: Released if it is not yet", pnpm feature:settle <id> and the ordering-safety checklist below. Settling stays a reviewed pull request.

The read. GET /api/feature-releases/ready-to-settle (InspectRuntimeFeaturePolicy or ManageFeatureReleases, Shared mode, 404 while RuntimeFeatureReleases:Enabled is off) returns, per candidate, featureId, flags, lifecycle, status (ready, notYet, stopped or notLive), settleDays, liveSinceUtc (the later date), releaseRecordSinceUtc, flagHistorySinceUtc, generalReleaseId, disagreement, releaseRecordNote, flagHistoryCoverage, readyOnUtc and stopped, with the page's asOfUtc from the API's clock. It is read-only and uses existing indexes, so it needs no migration. One request makes, after the binding check, at most four bounded queries: the policy document by _id; the binding's released GeneralRelease records through {environmentId, status} (0012, at most 1,000); their stop and restore events through {releaseId, recordedAtUtc} (0014, at most 1,000); and the lineage's audit receipts newest first through {storeEpoch, revision} (0010, at most 500). More records or events than the bound fail the read (503) rather than list from a partial read. Times: release confirmations are MongoDB time and audit receipts the writer's clock; "now" is the API's clock. Host clocks keep NTP time, which is irrelevant at the scale of days.

Ordering safety. Settling makes "on" unconditional. Before merging, confirm that every environment serves "on" for each flag: the deployed value in cluster-gitops and any runtime override on that environment's Feature flags admin page. Then deploying the clean-up changes nothing anywhere. After it deploys, remove the featureFlags.<key> values from cluster-gitops; a removed key renders nothing, so this order is safe (production values need Chris).

Non-Production Runtime Overrides

The deployed GitOps value remains the default and recovery fallback. In staging and preview environments, a real SyRF administrator can set or clear an override through the API:

PUT /api/runtime-feature-flags/bulkPdfUpload
Content-Type: application/json

{
  "value": true,
  "expectedRevision": 12,
  "reason": "Exercise bulk PDF upload in preview 2943"
}

Use "value": null to clear an override and return to the deployed value. GET /api/runtime-feature-flags returns the full authoritative snapshot, revision, deployed and override values, and activation metadata. Mutation is disabled in production.

The Feature groups section offers bulk PDF upload, reviewer workspace and supported statistics application-use controls, with separate configured and effective On/Off/Mixed states. Group membership comes from the API snapshot, which serves the flagGroups of the feature registry. Enable/Disable group writes all members and its audit entry atomically at the current revision; additional dependencies still require confirmation. Individual flags remain available below. Statistics calculation/maintenance, project readiness and explicit initial builds remain in the statistics operations workflow. A failed or timed-out response may have committed: the UI refreshes without replaying it.

Use precomputed screening statistics (statistics-consumers-v2) includes serving, pages, Project Overview, screening history and SignalR statistics refreshes. Enabling requires deployed serving, maintenance and screening; live refreshes also need SignalR connectivity. Existing per-project enrollment, freshness and fallback still apply. Disabling stops these consumers while calculation/maintenance stays on. The exports flag has no implemented export/report consumer, so it is excluded. Stage Overview and stage history remain individual because staging has annotation maintenance disabled. Reviewer history remains individual because it requires membership-screening maintenance, which is also disabled on staging. Maintenance and the seven metric-family gates remain deployment-managed across write hosts and are not group members. The old statistics-screening-v1 and statistics-consumers-v1 requests remain accepted with their original members for already-open clients; new snapshots advertise only v2.

The Bulk PDF upload and processing activity preset (bulk-pdf-upload-v1) changes bulkPdfUpload and studyManagementProcessing together. The Processing activity page also shows other study jobs; use the individual flags to control these separately. Deletion handling in API and Project Management, notifier cleanup authority and a ready PDF processing pipeline remain deployment prerequisites. Missing deletion handling blocks the whole group change. Without notifier authority, a saved upload override remains effectively off in the browser. Follow the notifier authority activation runbook before enabling uploads. Disabling this preset hides upload and Processing entry points; it does not cancel existing jobs or disable cleanup authority. The notifier cleanup authority (bulkPdfNotifierAuthority.enabled, deployment-managed) follows the same lifecycle rule: off pauses clean-up and refuses deletes with nothing lost, and switching it back on is checked against the recorded receipt's binding to its environment root and cleanup authority (ADR-017 amendment 2026-10-17).

Default is the environment's deployed baseline, a saved change (override) replaces it, and the pill shows the effective value after dependency normalization and domain constraints. These overrides are environment-wide, not project enrollment or permission grants. Statistics maintenance and family gates are shown as deployment-managed instead of offering working-looking runtime buttons. An old override can still be cleared to the deployed default. Global statistics serving cannot be enabled unless deployed serving is already enabled. Use the statistics pilot administration workflow for project readiness, builds and consumer presets.

If a change needs other runtime-manageable flags, the first PUT returns 422 Unprocessable Entity with code: "featureFlagDependenciesRequired", the dependency set, and canApplyAutomatically: true. After showing that set to the administrator, repeat the PUT with the exact "confirmedDependencies" key/value set and the latest snapshot revision. Keep the legacy "includeDependencies" field false: an older API replica will then reject the retry instead of applying dependencies it cannot bind to that exact plan. The current server rejects the retry if another API generation resolves a different dependency plan, rather than applying changes the administrator did not review. It then applies the requested flag and every confirmed dependent change atomically in one revision and audit entry.

The snapshot's effective values are authoritative even when a flag has no override. Loaded flags with unsatisfied prerequisites are normalized Off in both API and browser. If a later mutation would reactivate another dormant configured flag without confirming it, that flag is pinned Off by an audited override and must be enabled explicitly in a separate reviewed change.

Deployment-managed dependencies are different: their 422 response has canApplyAutomatically: false and cannot be bypassed with includeDependencies. For example, bulkPdfUpload requires DeletionLifecycle=true in both API and Project Management. Roll out and verify that shared GitOps value across every consuming service before enabling the runtime override. During a rolling deployment, wait until all relevant workloads are healthy on the new generation; the API's local readiness check is deliberately only a fail-closed final guard, not proof that a different service has completed its rollout.

Each change atomically updates the normal current-state override document and appends a separate audit entry. Authorization always checks the real signed-in user's administrator group, even while that administrator is impersonating someone else. The audit entry records both the real and impersonated identities plus the required reason.

After a successful mutation, the API broadcasts only the new revision over SignalR. Browsers then fetch a complete snapshot; they also poll every five minutes as a slow fallback. The API refreshes its cached snapshot periodically so backend gates converge across pods.

There is no SignalR backplane, so a broadcast reaches only the connections of the pod that sends it. In Legacy mode only the writing pod broadcasts, after a legacy override change. In RuntimeFeatureFlags:Mode=Shared (6B.4 G1) every API pod hints its own connections when its authority reader observes a new policy identity (RuntimePolicyBrowserHint, called from RuntimePolicyAuthorityRefreshService after each 8–12 s refresh). That is the only hint path. The writing pod does not hint at commit time, because its browsers' refetch is served from this pod's reader, which would still hold the previous revision. Instead a successful commit, emergency stop or ReleaseStop restore wakes that pod's own refresh (IRuntimePolicyRefreshNudge), and the hint follows as soon as the reader has the new identity. If that early read still sees the old identity, nothing is marked as hinted and the next regular pass hints. Nudges coalesce (several during one pass cause exactly one more pass), the refresh never runs twice at once, and after a nudged pass the next wait is a full 8–12 s delay. The message is the existing RuntimeFeatureFlagsRevision invalidation; browsers refetch the snapshot and never use the hinted value. A pod hints each store epoch and revision at most once and never for a successful start-up read (if the start-up read fails, the first identity it reads later is hinted); a new store epoch after a re-initialisation or rebind is hinted even when its revision is lower. A hint failure, or a hub that does not answer within 2 s, is logged and changes nothing else: the refresh and acknowledgement carry on, and the commit receipt is never affected (the nudge neither blocks nor throws). Tabs without a live hub connection still rely on focus, online, reconnect and the five-minute poll. Flag decision: not separately flagged. The hint runs only under the existing Mode=Shared switch, which no shared environment runs, fires once per observed revision per pod and writes nothing, so switching Shared mode off and on again is safe.

bulkPdfUpload is applied to both the Angular gate and the API endpoint gate from that same snapshot.

Runtime-manageable Boolean flags use the same endpoint. Most changes apply immediately after the next snapshot refresh. The snapshot marks integratedPdfViewer, signalRActive, zonelessChangeDetection, errorTracking and logRocketEnabled with activationRequirement: "pageReload" because their viewer, transport, change-detection, diagnostic, telemetry or session-recording lifecycles are bound during page initialization. Their overrides are persisted immediately and become active when the browser is reloaded; the initial snapshot is applied before those lifecycle services start. reviewEligibilityPolicy is also marked pageReload (review configuration screens read the eligibility mode when they load) but it is not runtime-manageable: it is deployment-managed because Project Management reads it too (see Carrying a web flag to Project Management). It keeps its ReloadRequired(...) catalog entry because a changed deployed value still reaches an open browser only on reload. The authoritative list is ReloadRequiredFlags in src/services/api/SyRF.API.Endpoint.Tests/RuntimeFeatureFlagCatalogTests.cs, which pins every ReloadRequired(...) entry in the catalog. GitOps remains the deployed default and production source of truth.

See the administration delivery slices for current propagation boundaries, proposed grouped controls and retirement review criteria.

This MVP deliberately does not include percentage or audience targeting, typed non-Boolean configuration, event sourcing, projections, replay, or a richer audit-history UI.

Production runtime policy, release records, targeting and announcements are planned, not implemented: see the runtime feature releases plan and ADR-025 (Proposed). This guide describes current behaviour only.

The Features and releases area

/admin/features (6G AD1) gathers the runtime admin pages behind one side menu. Each page and tab is guarded by its OD-2 capability, read from GET /api/account/capabilities (claim administrator where PM authority is not enforced), and the API still decides every request:

Page What it shows Capability
Overview "Needs your attention" (features ready to settle, with a Create clean-up issue link; servers that have not confirmed the latest settings; features under an emergency stop) and summary cards. With nothing to report it shows one line: Nothing needs your attention. A failed Ready to settle read is named ("Couldn't check features ready to settle"); while releases are switched off the settle items are simply absent. The Releases card links to the Releases page; release attention items are a follow-up. inspectRuntimeFeaturePolicy
Features The registry (FEATURE_DEFINITIONS) with type, owner, lifecycle, live state here and links to the same feature in the other environments (D24). The lifecycle filter adds Ready to settle (see Ready to settle), a ready feature shows that status and its days live, and Needs action includes it. Read-only. inspectRuntimeFeaturePolicy
Feature /admin/features/<featureId>/{overview,targeting,settings,history} Registry facts, live flag values and stop state; for a Release, the "Settling this feature" card (status, dates, source, disagreement note, and when ready the steps and the clean-up issue link); Targeting shows and changes the rules of a Project or Personal feature (see The Targeting page; a Global feature reads "Global: not targetable"); Settings shows "set in code" from the registry and a read-only "set here" until 6F G; History lists the feature's releases here (the Releases table filtered to the feature, first page, no admission check). inspectRuntimeFeaturePolicy
Flags The Feature flags page below, unchanged. /admin/feature-flags redirects here. manageRuntimeFeatureFlags
Releases /admin/features/releases and /admin/features/releases/<releaseId> This environment's release records and one page per release (6G AD2, read-only; see The Releases screen). manageFeatureReleases
Environment The fleet and flag delivery panels below, plus the environment binding. The compatibility report is read only by the deployment's machine client, so it is not shown. inspectRuntimeFeaturePolicy
Content (6G AD4) Content preview: pick a feature and an audience (reviewer, project administrator, signed-out visitor) and see what that audience would get once the release is confirmed: the What's New entry (rendered by the same card people see), each help section and the walkthrough steps. Read-only; content is written in the repository. Offered only while RuntimeFeatureReleases:Enabled is on; while it is off the menu has no Content item and /admin/features/content goes to not-found. inspectRuntimeFeaturePolicy (the API also accepts manageFeatureReleases)

Not flagged: administrator-only navigation over existing reads; every release action stays refused while RuntimeFeatureReleases:Enabled is off.

Gradual rollout and Preview and check (6F W3). Two pages under a feature's Targeting tab, offered there only when the tab's own read of the key's targeting (GET /api/runtime-feature-flags/{key}/targeting) has answered: a Gradual rollout link in the rollout step (for an administrator who may change the policy) and a Check a project (or Check a person) link in "Who gets it":

  • /admin/features/<featureId>/targeting/rollout (changeRuntimeFeaturePolicy): the saved share, state and group; an amount entry from 0.01% to 100% in steps of 0.01% (quick choices or Custom), stored as basis points (0.01% is 1, 100% is 10,000); Pause / Resume and Turn rollout off / Turn rollout on. Nothing applies on click: each change goes into a draft, …/rollout/preview previews it with POST …/{key}/targeting/preview (a lower amount lists the projects that lose it, by name where the server gives one, before anything is saved, with the server's rollout notes), and the save sends PUT …/{key}/targeting with the preview digest and a required reason. The key's exclusions and allows are sent back exactly as read, because a targeting save replaces every rule of the key. A paused rollout holds its amount, so the amount entry is locked until it is resumed; Pause wording follows D32 C (only projects that existed when it was paused keep getting it). A key with no rollout can start one over projects (Project features only: rollouts over people are not supported yet).
  • /admin/features/<featureId>/targeting/check (inspectRuntimeFeaturePolicy), and the right-hand panel of the rollout preview: Check a project (a person, optionally in a project, on a Personal feature) calls POST …/{key}/simulate and shows the answer and the step that decided it, under the policy in force and, on the preview, with the draft side by side, then every step in a table. Owner decision D59 = B adds lookups: the project field searches by name or ID as you type (GET …/targeting/projects, debounced 300 ms, from two characters; each option shows the name, then "Owner … · n members · n stages · n studies · Created d MMM yyyy · short ID" as #4519's lookup does, leaving out what the server could not give and naming an owner the server could not name from the user list, else "unknown person" with a short ID; an archived project says so), and on a Personal feature the person field searches the Admin Console user list by name or e-mail address (GET /api/investigators, ListUsers, read into the store at sign-in; without ListUsers the field says to paste the ID). A pasted ID still works. An entry that is neither chosen nor an ID is refused in the browser. A refusal or failure is headlined in plain words (Chris, 2026-10-26), with the server's own message and reason code under a collapsed "Details", as on the Targeting page: an ID that matches nothing or a deleted one reads "No project with that ID exists." (or "No person with that ID exists.") and "You entered ", with no answer beside it; an unknown reason reads "The check could not be completed.". The rollout preview's and save's refusals follow the same pattern. Owner rule (Chris, #4519): the pages refer to projects and people by name, never by a whole GUID. A chosen entry shows its name in the field; a pasted ID is named through the search's exact-ID match or the user list (the answer never waits for it: the caption shows the short ID until the name arrives); the result's caption, the step table and the lowering preview show names, with a short ID (first eight characters) only as quiet secondary text, and a project the server cannot name reads "Unnamed project" with its short ID.

While RuntimeFeatureTargeting:Enabled is off the targeting read answers 404, so both routes go to not-found and the Targeting tab offers neither link. Flag decision: no new flag; the pages sit behind the existing targeting switch and the capability guards, and the API decides every request.

Content preview rules

GET /api/feature-releases/content-preview lists the features with a What's New entry or article in the current verified server bundle, or a walkthrough in the registry; GET /api/feature-releases/content-preview/{featureId}?audience=reviewer|projectAdministrator|signedOut answers for one feature and audience (400 for another audience, 404 for an unknown feature). Both are read-only, need InspectRuntimeFeaturePolicy or ManageFeatureReleases, answer 404 feature_availability_disabled before authorization while RuntimeFeatureReleases:Enabled is off, and read only the registry and the cached bundle. The answer applies the disclosure rules with the release taken as confirmed and the feature as on:

  • Who it is for is the registry audience (6R.A): signedIn and projectMembers reach reviewers and project administrators, projectAdministrators only project administrators, applicationAdministrators (or no audience) neither. Signed out reaches nothing personal.
  • What's New, for a feature that announces: a reached audience sees every stage its entry covers, with "New to you" when the feature notifies its release audience; everyone, signed in or not, sees what the public projection (6G.6) shows: a Global feature's entry at general release when its topic has a public guide page.
  • Help: public classes for everyone, Draft for nobody, restricted classes (Private pilot, Retired) only for a reached signed-in audience and never signed out. The preview shows the topic's section of the page (from its heading to the next heading of the same level), or the whole page for a whole-page topic.
  • Walkthrough: only signed in, for an announcing feature that reaches the audience. The steps come from the web build's walkthroughs.generated.ts.
  • A quiet or Withdrawn feature has no entry and no walkthrough. Left-out content carries only its IDs and a reason, never its text.

The Releases screen

Where to see releases. Admin console → Features and releases → Releases (/admin/features/releases), or a feature's History tab for that feature only. The page reads the release read routes (6G AD2a, #4476); 6G AD2b (#4518) is the page itself. It is read-only: preparing, applying, pausing and stopping from here come with the Apply page in a later PR, and emergency stops and restores stay on the Flags page.

  • List. Newest first, 50 a page (Show older releases follows nextBefore), with tabs Needs action (Prepared, waiting for servers, servers need action, paused or stopped, or an announcement row Failed), All and Live (Active). Each row shows the feature, its type, owner and stage; the status (Prepared, Ready to apply, Can't apply yet, Applying, Confirmed live, Waiting for servers, Servers need action, Paused, or Stopped while an emergency stop is in force on the feature's flag); the milestones; and the announcement state per channel: What's New (MaterialiseWhatsNew), in-app notice (NotifyInApp) and walkthrough. Outbox states read Pending, Sent, Cancelled by stop (CapabilityStopped) or Failed; rows written before D60 A (26 Oct 2026) can also read Sent to test sink (staging) or Not sent: delivery off here (DeliveryDisabled, previews and local stacks), which stay as they were. With the stored cancel reason or error class shown verbatim beside the label. A Prepared release shows the registry's plan (Queued on apply or None planned); an applied release without a row says Not queued (a quiet Apply, or a feature that does not announce that way). Walkthroughs have no outbox row, so they read None planned or Not tracked yet; an Operational feature reads "Not announced: operational features are invisible to users".
  • Admission check. For each Prepared release on the page (at most 10, one at a time) the screen runs the side-effect-free Apply preview (POST /api/feature-releases/{releaseId}/apply/preview, ManageFeatureReleases) with the feature's default announcements, against the current policy identity, and shows Ready to apply or Can't apply yet with the refusal. The release page shows the refusal's reason code verbatim and the server's text for it (RuntimeReleaseAdmission.Message; a 422 refusal's own message when the server sends one). Nothing is applied or recorded; the preview's reason reads "Admission check from the admin Releases screen (Apply preview only; nothing is applied)".
  • Release page. Status, the progress strip (Prepared, Applied, Confirmed live), milestones as exact times in Europe/London with the zone (requested, scheduled, apply requested, confirmed, recorded, how it was confirmed, precision), the release key, ID, stage, record version and pinned help bundle, the announcement rows (state, attempts, completed time, cancel reason, last error class, and "Delivered to: Test sink" only on a staging row the retired test sink completed before D60 A), and the event timeline newest first with each event's request reason in quotes, verbatim. An imported historical release has no record (its detail GET is 404), so its page shows the timeline only.
  • States. Loading; empty ("No releases in yet"); switched off (404 feature_availability_disabled: "Releases are switched off in "); refused (401/403: "You can't see releases here", with the server's title, such as "Release history requires Shared API authority mode.", quoted verbatim); not found; unavailable (503, with Try again).

Flag decision: not separately flagged. Every read sits behind RuntimeFeatureReleases:Enabled (off everywhere), and the page only reads.

Using the Feature flags admin page

The page at /admin/features/flags (redesigned in #4385; /admin/feature-flags redirects there) is a summary card above one list of flags.

Summary card. One line says whether the servers have the current settings: All servers up to date (only when the server reports the fleet Active), Updating: 2 of 4 servers have the latest settings while hosts adopt it, Only 2 of 4 servers have the latest settings after the one-minute confirmation window, or Server status unavailable. A saved policy is not adoption, so a save never reads "up to date" until the fleet does. The line re-checks every 10 seconds while servers are updating, for up to two minutes. Legacy mode says server status is not tracked; a read-only environment says so here once instead of on every row. In Shared mode a second line gives the emergency-stop state of the membership control (Membership disable: stopped, not stopped, updating… or stop state unknown), read from authority.stoppedKeys. Emergency stop… or Restore… opens a dialog (both appear when the state is unknown or updating). Details is collapsed by default. It holds the mode, policy revision, the last saved revision while the page catches up, the stop-state explanation, the fleet host table (each host state in words, such as Up to date or Catching up), flag delivery status and the explanatory notes. Its diagnostics load only when it is opened. A revision the page has saved stays the floor until a snapshot reaches it, even if its receipt is dismissed or replaced, so a lagging snapshot never reads "not stopped" straight after a stop. Policy identities compare GUIDs and digests case-insensitively, as the snapshot guard does.

List. Search matches the key, name, description and category. Show filters to All, Changed from default (a saved override) or Can't change here. Each row has the name, a one-line description, an On/Off pill (marked changed, stopped or updating only when it is; a row at its default shows just the value), and a Change menu (Turn on, Turn off, Use default (On/Off); the current choice is disabled). Selecting the name expands the row to show the full description, key, default, saved change, reload requirement and dependencies. A deployment-managed flag shows a lock and its reason instead of a menu. In Legacy mode its menu offers only Use default, to clear an old override. Feature groups is a collapsed section below the list; each group has Turn all on and Turn all off in Legacy mode and shows why it is locked otherwise.

Change dialog (Shared). The dialog opens straight away and asks the server for a preview. It then states the effect in one sentence ("Off → On for everyone in this environment"), lists any other flags the preview changes, and offers an optional comment. The preview digest covers the audit reason, so editing the comment requests a fresh preview half a second after typing stops; the primary button is enabled only while the preview on screen matches the comment. Confirming commits exactly that previewed request and digest. Cancel, Escape or closing the dialog commits nothing. A dependency refusal, at preview or at commit, shows the needed flags in the same dialog; Review combined change previews again with the confirmed set and still needs confirmation. A revision conflict closes the dialog and reloads current values. When saved, the dialog closes, a short message says "Servers update within about a minute", and the row shows the new value marked updating until the API's reader catches up (the page re-reads quietly every 3 seconds, up to five times).

Outcome unknown. If the connection drops before the server answers, the dialog says it is not sure the change was saved and offers Check outcome and Retry unchanged request (the same mutation ID, so it cannot apply twice). Nothing is committed automatically. If the dialog is closed, a banner keeps both actions and the mutation ID on the page, and other changes stay blocked until the outcome is known. The stop has the same banner with Retry unchanged stop.

Reload recovery. Before a change, restore or stop is sent, the page saves a small record in the tab's session storage: the kind, key, mutation ID, the policy identity the request expects and (for a change or restore) the identity its preview proposed. It holds no value, reason, comment or preview digest, and it is removed once the outcome is known. If the page reloads first, it reads the current policy, then looks up the original mutation ID; both are reads, and nothing is previewed, committed or stopped again. A committed request shows as saved. If it was not committed and the policy has since moved on, an information banner says it wasn't saved and can't be saved now (every commit checks the identity it expects). Otherwise the banner offers Check again and Discard; ordinary changes are paused until one of them settles it, and the stop stays available while a change is unresolved. While the page's own earlier stop is unresolved, Emergency stop… is unavailable but stays focusable, and its tooltip says to check the earlier stop first. When a recovered restore is saved while a remembered stop is still unresolved, the page checks that stop again (reads only), so its own receipt settles it. A blocked session storage leaves the page as it was without this record. This path is not flagged: it is administrator-only, reads only, and does nothing unless a record exists.

Emergency stop and restore. The stop dialog says what the stop does and needs a reason. Stop re-reads the current policy and then sends the stop, so the dialog is the confirmation. Restore previews the release on the server, shows the value the control goes back to, and needs a reason before it can be confirmed; it is blocked while a stop request is unresolved.

Legacy mode uses the same list and a simple confirmation dialog with an optional comment, because the legacy PUT has no preview. Dependencies the change needs are listed in that dialog. A revision conflict reloads current values and requires a fresh action rather than silently overwriting another administrator.

Schema Properties Reference

Property Required Description
name Yes Environment variable name (SYRF__FeatureFlags__*)
valuePath Yes Path in Helm values.yaml (e.g., featureFlags.myNewFeature)
default Yes Default value as string ("true" or "false")
web.tsName Yes camelCase name used in TypeScript (e.g., myNewFeature)
web.tsType Yes TypeScript type (always boolean for feature flags)
featureFlag Yes Must be true to mark as feature flag
displayName Yes Sentence-case label shown on the admin runtime feature flags page (generation fails without it)
description Yes Human-readable description (used in JSDoc comments)
category Yes Grouping for organization (see categories below)
adminOnly No If true, flag requires admin UI to be visible

Categories

Use existing categories for consistency:

Category Purpose
projectSettings Project configuration features
uiFeatures General UI features
robFeatures Risk of Bias related features
dataFeatures Data import/export features
development Development/debugging features
monitoring Observability features (Sentry, LogRocket)

Admin-Only Flags

Flags marked adminOnly: true are automatically gated by the admin UI state:

- name: SYRF__FeatureFlags__ExampleAdminAction
  valuePath: featureFlags.exampleAdminAction
  default: "false"
  web:
    tsName: exampleAdminAction
    tsType: boolean
  featureFlag: true
  category: robFeatures
  description: "Example admin action (illustrative, not a deployed flag)"
  adminOnly: true

Generated selector automatically combines with admin state:

export const selectExampleAdminAction = createSelector(
  selectConfigState,
  selectShowAdminUiOptions,
  (configState: AppConfig, showAdminUi: boolean): boolean =>
    configState.exampleAdminAction && showAdminUi,
);

Generator Commands

cd src/services/web

# Generate all files
pnpm run generate:flags

# Validate schema without writing (CI/pre-commit friendly)
pnpm run validate:flags

# Preview changes without writing files
pnpm run generate:flags -- --dry-run

# Show verbose output including Helm env vars
pnpm run generate:flags -- --verbose

Pre-commit Hooks

Two pre-commit hooks validate generated code when relevant files change:

1. validate-env-blocks (syrf-common)

Triggered by changes to:

  • src/charts/syrf-common/env-mapping.yaml
  • src/charts/syrf-common/templates/_env-blocks.tpl

2. validate-feature-flags (web)

Triggered by changes to:

  • src/charts/syrf-common/env-mapping.yaml
  • src/services/web/src/app/generated/feature-flags.*.ts
  • src/services/web/src/assets/data/env.template.generated.json
  • src/services/web/.chart/values.yaml

If validation fails (files are stale), the commit is blocked until you run the appropriate generator.

CI Validation

The validate-generated-code job in pr-tests.yml runs both validators:

  1. Detects if schema or generated files changed
  2. Runs pnpm run validate:env-blocks in syrf-common
  3. Runs pnpm run validate:flags in web
  4. Reports status in PR test summary

Why validate-only (not auto-generate)? Generated TypeScript could break the build. Developers should run the generator, review changes, test the build passes, then commit.

Note: The web chart's deployment.yaml env vars are generated by syrf-common/scripts/generate-env-blocks.ts, not the web generator. The web chart uses shared templates via {{ include "syrf-common.env.webConfig" . }} and {{ include "syrf-common.env.featureFlags" . }}.

The generators are idempotent - running them multiple times produces identical output. This ensures consistent validation in CI and pre-commit hooks.

How It Works at Runtime

  1. Build time: Helm templates reference values.yaml paths
  2. Container startup: NGINX envsubst replaces ${SPA__*} placeholders in env.template.json (the image build copies env.template.generated.json in under that name)
  3. App bootstrap: main.ts fetches config JSON files
  4. Config parsing: parseFeatureFlags() converts string values to booleans
  5. State management: NgRx store holds parsed config
  6. Component access: Selectors provide typed access to flags
Helm values.yaml → deployment.yaml env vars
                          ↓
                   Docker container
                          ↓
           NGINX envsubst → appConfig.env.json
                          ↓
                main.ts fetch()
                          ↓
           parseFeatureFlags() → AppConfig
                          ↓
              NgRx Store (config state)
                          ↓
              selectMyFeature selector
                          ↓
               Component signal/observable

Architecture: AppConfigService Integration

The AppConfigService cleanly separates generated code (feature flags) from manual code (derived values):

// app-config.service.ts

// AppConfig = FeatureFlags (generated) + AppSettings (manual)
export interface AppConfig extends FeatureFlags, AppSettings {}

// ExternalConfig = FeatureFlagsExternalConfig (generated) + ExternalConfigSettings (manual)
export interface ExternalConfig extends FeatureFlagsExternalConfig, ExternalConfigSettings {}

static loadConfig(externalConfig: ExternalConfig): void {
  // Generated: parses all feature flags using schema defaults
  const featureFlags = parseFeatureFlags(externalConfig);

  // Manual: complex derived values (URLs, version strings, etc.)
  const appSettings = buildAppSettings(externalConfig);

  this.staticConfig = { ...featureFlags, ...appSettings };
}

Why Some Values Aren't Generated

Non-feature-flag configuration (AppSettings) requires manual logic that can't be generated:

Property Reason
displayVersion Environment-specific formatting (30+ lines of logic)
auth0Domain Computed from {tenant}.{region}.auth0.com
apiUrl Derived from apiOrigin + '/api'
protectedUrls String split + array append logic
randomId Runtime-generated browser session ID

File Structure

src/app/
├── generated/                      # AUTO-GENERATED (don't edit!)
│   ├── feature-flags.types.ts      # Interfaces only (no JS output)
│   ├── feature-flags.constants.ts  # FEATURE_FLAG_DEFAULTS, env vars, helm paths
│   ├── feature-flags.selectors.ts  # NgRx selectors
│   └── feature-flags.parser.ts     # parseFeatureFlags()
│
└── core/services/config/
    ├── app-config.service.ts       # Main service (uses generated code)
    └── app-settings.interface.ts   # Non-feature-flag types (manual)

Note: types.ts contains only TypeScript interfaces (erased at compile time). All runtime values (as const) are in constants.ts for clear separation.

Troubleshooting

Generated files out of sync

pnpm run generate:flags
git diff  # Review changes
pnpm run build  # Ensure TypeScript compiles

Validation fails in CI

The validate-generated-code job reports which files are stale:

# Run locally to see the same output
cd src/services/web
pnpm run validate:flags

This checks if values.yaml and the TypeScript files match the schema.

Pre-commit hook fails

# Run the generator to update files
pnpm run generate:flags

# Check what changed
git diff

# If TypeScript changed, test the build
pnpm run build

# Then stage the generated files
git add src/app/generated/ .chart/values.yaml

Flag not working in deployment

  1. Check Helm values in cluster-gitops
  2. Verify deployment has correct env var: kubectl exec ... -- env | grep SYRF__FeatureFlags
  3. Check browser dev tools → Network → fetch for config JSON
  4. Verify flag value in parsed config (browser console)

TypeScript build fails after generation

Generated code might reference types that don't exist yet. Check:

  1. Is selectConfigState exported from the config selectors?
  2. Is selectShowAdminUiOptions available for admin-only flags?
  3. Are NgRx imports correct in the generated selectors file?

Shared reader: first API/PM migration slice

The shared reader is an opt-in non-production migration seam. Its independent deployment setting is RuntimeFeatureFlags:SharedReaderMode (SYRF__RuntimeFeatureFlags__SharedReaderMode in the host environment). Configure it through the existing environment/GitOps configuration; it is deliberately not a runtime flag that could disable its own provider. The default is Disabled, preserving existing behavior.

Mode API Project Management
Disabled Existing runtime provider and behavior Existing startup/domain behavior
Shadow Observe persisted global overrides without changing consumers Observe the same persisted overrides without changing domain behavior
Membership Use the shared snapshot for the existing disableMembership action-selector gate Continue shadow observation; no membership or reviewer domain control is changed
StatisticsServing Invalid for this host Use the shared materializedProjectStatisticsServing value as an additional stop for precomputed statistics serving

StatisticsServing also uses shared overrides for the Pages, Stage Overview, SignalR and Exports consumer selectors. These are reversible consumer requests, not evidence that any project is Fresh or allowlisted. disableMembership gates the auditable disable endpoint; re-enabling an already-disabled member stays available with the flag off. It does not remove membership authorization or rewrite persisted membership state. Each route decision captures one immutable observation. Already admitted requests complete under the existing application protocol. All other API consumers continue using their existing provider. In StatisticsServing mode PM keeps writes, family gates, project allowlists and reviewer-capacity controls deployment-owned. Shared serving and consumer values can stop precomputed reads, but cannot enable them when deployment serving or writes are off. The durable statistics control and freshness checks still apply. An unavailable or stale observation stops new materialized reads in PM; it does not stop maintenance or undo an admitted operation. This makes the existing Use precomputed statistics admin switch meaningful in both API and PM on an enrolled non-production deployment. Calculate and maintain statistics is still a separate, deployment-bound control; its coordinated runtime transition remains follow-up work.

Both hosts reconcile the existing global Mongo override document every 15 seconds with majority reads and a five-second read deadline. There are no change streams in this first slice. Complete frozen snapshots replace previous snapshots, so resetting a flag removes the old override. Missing/malformed documents, read failure, revision regression, conflicting content at an equal revision and observations older than 60 seconds produce an unknown/unavailable decision. The enrolled API route fails closed in those states; the last observation remains available for diagnosis. A successful later reconciliation recovers. Cold start with no persisted state does not synthesize an authoritative revision zero: first initialize the legacy state through an ordinary reviewed flag edit while in Shadow mode, then compare before selecting Membership.

The admin flag page exposes GET /api/runtime-feature-flags/provider-status to real administrators only. It distinguishes the independently fetched saved revision from the local observed revision, mode, state, observation time and nullable evaluated value. Current means a recent successful local observation, not fleet convergence. PM also exposes a limited read-only GET /api/runtime-feature-flags/reader-status containing only local mode, observation state/revision/time and the effective statistics-serving decision; the same Boolean policy is already public in the API snapshot. The admin page displays that result separately from the API result. PM retains the runtime-feature-flag-reader diagnostic health entry. Neither endpoint is a fleet acknowledgment. There is no fleet registry, and the UI explicitly reports fleet state as unknown.

The legacy-global-overrides digest covers only sorted override content. It excludes deployed defaults, catalog metadata and environment identity. Matching digests do not establish equal effective values when hosts have different defaults. PM health includes its selected consumer's deployed default and evaluated value. This is not the approved-baseline/epoch authority described by the broader architecture; a document carrying the future storeEpoch field is rejected by this legacy reader until its migration is implemented. Production and unknown/conflicting deployment identities reject opt-in modes at startup.

Before rollback to Disabled, compare the saved revision, the legacy provider's observed value and the shared reader's value for disableMembership; wait for legacy convergence or write a reviewed forward revision first. Do not use rollback to bypass an unavailable authority or restore a stopped capability. For PM StatisticsServing, first make the deployment fallback agree with the reviewed runtime serving decision. Otherwise reverting the reader to Disabled can reopen precomputed serving despite an Off override. No deployment setting or runtime policy is changed merely by merging this implementation.

Consumer inventory guard

pnpm run validate:generated also checks the C# typed consumer inventory in consumer-manifest.json. New or changed common flag seams and reads through typed/inferred aliases require a reviewed classification, owner, scope and activation boundary. Changes to _flags.SomeGate in an already classified file are detected. This guard is conservative static evidence, not proof of every dynamic configuration lookup or external worker; those remain explicit audit work before migration. Update the manifest after reviewing a change, never regenerate it in CI to accept a new read automatically. In PR Tests, the Validate Generated Code lane runs for any change to the manifest, the validate:generated scripts or a .cs file under src/libs/ or src/services/, so a changed flag read cannot merge unclassified.

Since R3c the guard also enforces reading flags through IFeatureDecisions. A production file that names IRuntimeFeatureFlagProvider, SharedRuntimeFeatureFlagReader or RuntimePolicyAuthorityEvaluator in code, or an existing-runtime consumer that reads FeatureFlags, fails unless the manifest's legacySeamAllowList names it with a kind and a reason. The kinds are implementation (the decision paths and their registration), control-plane (diagnostics and reports), deployed-configuration (a start-up value read beside runtime decisions) and pending-migration. An entry that no longer needs to be there fails as stale, so the list only shrinks. The pending migrations are the five domain callers of IRuntimeFeatureFlagProvider.IsEnabled (ReviewController, StageAllocationReadController, StageWorkloadSharesController, AnnotationFoldTargets, SubmitAnnotationSessionService) and the four statistics history controllers. The history controllers read FeatureFlags as the runtime provider mutates it: moving them to IFeatureDecisions would decide materializedProjectStatisticsPages from the bounded authority in Shared mode, which changes their staleness behaviour, so it needs its own review.

The legacy store never interprets or overwrites a document containing storeEpoch, including a malformed epoch (owner decision D7=A, spec 6C.5). A host in Legacy mode (or production with ProductionPolicyMode=Legacy) that finds one starts normally and serves the deployment-controlled values for every key: the authority's saved overrides and stops are preserved but not applied. From that moment the process is a read-only legacy reader for its lifetime: snapshots carry policyState: DeploymentControlledAuthorityPreserved and mutationAllowed: false, legacy changes answer 409 runtime_policy_authority_preserved without writing, provider status says the runtime policy is preserved but not applied, authority-status reports reason LegacyModeAuthorityPreserved, and the process marks its own syrf-api writer-registry entry runtimePolicyReadOnly so it does not block a later lineage freeze. The legacy-format shared reader (API membership, PM statistics) likewise serves deployed defaults in state AuthorityPreserved. Returning the hosts to Shared resumes the unchanged authority at the same identity (runbook 10 of Operate runtime feature flags). Older images that predate this behaviour still refuse to start against an imported authority, and their registry entries are always counted by the freeze. Stop/freeze older writers before lineage import.

Runtime-policy reader/writer boundary guard

Runtime-policy code has two sides. The reader side (snapshots, evaluation, catalog, host acknowledgements and the host's own writer-registry entry) lives in SyRF.Mongo.Common/RuntimeFeatureFlags and RuntimeFeatureReleases; every service ships it, and PM publishes its own heartbeat. The writer, administration and release side (the mutation store, flag store, lineage import and its entry point, release intent, the environment-report and workload stores, the writer freeze) lives in the API-only project src/services/api/SyRF.API.RuntimeAdministration, which only SyRF.API.Endpoint references in production. Its types keep their SyRF.Mongo.Common.* namespaces. Logic both sides need (receipt validation and digests, effective values, deployment-constraint and observation validation) is reader-side, and the module reaches the reader's internals through InternalsVisibleTo.

RuntimePolicyBoundaryArchitectureTests in SyRF.ProjectManagement.Endpoint.Tests fails if:

  • PM's assembly closure contains the module, or any type the module declares;
  • a project other than those in its ModuleReferencers list references the module (the API, plus two test projects that use the writers for setup);
  • a PM production source, or a reader-side type, names a writer-side type (every type the module declares). The ReaderReachIns exception list is empty;
  • a reader-side type writes MongoDB documents, indexes or collections (host acknowledgements excepted).

The source scans ignore comments but not string literals, and assume file-scoped namespaces (a test checks that).

To read a failure, look at the named consumer, project or assembly. Route the consumer through a reader-side type, moving shared logic into a reader-side file, or move writer code into the module. A new test project that needs the writers goes into ModuleReferencers with its reason. Design: spec 6R.3 (reader/writer module split).

Reviewed runtime-policy lineage import

The step-by-step operator procedure for this command, with preflight checks and refusals, is runbook 2 of Operate runtime feature flags.

PR #4347 provides an explicit operator command in the API image for the Increment B legacy-lineage transition. It is separate from schema migrations and normal startup. It does not activate runtime modes, enroll consumers, establish fleet readiness, or authorize a production operation. Coordinate its use with an authority-capable reader/writer rollout; the legacy store and shared legacy reader refuse epoch-bearing state. Do not use it as a standalone live-environment rollout.

The preview requires an existing legacy policy, a verified environment binding, and a reviewed baseline JSON file with these fields:

  • values: a Boolean value for every key in the generated shared runtime catalog; unknown keys and missing keys are rejected. Reconcile all consuming services' rendered/deployed inputs before approving this artifact. The command never chooses the API's current settings as the environment baseline.
  • approval: the reviewed baseline decision/reference.
  • provenance: the source configuration/report identities used to reconcile it. These are operator review evidence, not an assertion that a compatibility collector or fleet acknowledgement already exists.

Use the existing Mongo configuration and RuntimeEnvironment, plus the binding expectations used by the migration tooling: SYRF_MIGRATION_BINDING_EXPECTED_ENVIRONMENT, ..._DATABASE, and ..._CLUSTER. The expected cluster must come from a verified binding/status result. Never substitute a guessed target.

To produce a read-only preview, run the API image's normal entry point with:

SYRF_RUNTIME_POLICY_MODE=preview
SYRF_RUNTIME_POLICY_INPUT=/reviewed/baseline.json
SYRF_RUNTIME_POLICY_OUTPUT=/reviewed/lineage-preview.json

The output contains the source revision/digests and audit references, accepted baseline and target identity, new store epoch, mutation ID and preview digest. Review and retain this exact artifact. It uses Canonical Extended JSON to preserve BSON integer and identifier types. Editing or reformatting through a tool that changes BSON types invalidates it; generate and review a new preview instead.

Before apply, freeze deployments/autoscaling and stop all legacy policy writers. Hold that operational freeze throughout the command. The command also requires an existing, well-formed syrf-api registry with zero registered instances; missing registry data and stale/live registrations refuse. There is no bypass switch. This registry check is evidence of stopped instances, not a Kubernetes deployment lock.

Apply the exact reviewed preview with:

SYRF_RUNTIME_POLICY_MODE=apply
SYRF_RUNTIME_POLICY_INPUT=/reviewed/lineage-preview.json
SYRF_RUNTIME_POLICY_OUTPUT=/reviewed/lineage-receipt.json
SYRF_RUNTIME_POLICY_ACTOR=<operator identity>
SYRF_RUNTIME_POLICY_WRITER_FREEZE_ACK=<reviewed maintenance/freeze reference>

Apply requires migration-owned audit indexes 0009 and 0010 with their expected definitions; it never creates indexes. It verifies the binding and source again, preserves the numeric revision and exact overrides, and commits the authority document and receipt together with snapshot reads and majority write concern. The receipt references the old audit rows; those rows remain untouched. Missing state, a changed source, an unknown legacy field or another lineage refuses rather than silently initializing/resetting.

Input and output paths must differ, and output must not already exist. If apply loses its response or cannot write its output artifact, retry the identical reviewed preview with a new output path. The persisted mutation receipt resolves the original result without another policy write. A different preview using the same mutation ID is refused. A receipt proves that import committed, not that any consumer has accepted the lineage or that the fleet is ready. Restore/rebind and later baseline changes require their own reviewed transitions; this command never rotates an existing lineage.

First-time initialisation (no legacy document)

Where no global flag document exists (production: legacy runtime mutation is disabled there, so there is nothing to import), the same command takes SYRF_RUNTIME_POLICY_SOURCE=initialize (owner decision D8=A, slice 6C.6). Unset or import, it is the import above, unchanged. Initialise creates revision 0 from the deployed values: the FeatureFlags configuration the command itself renders, for every catalog key, with no overrides and no stops, bound to the verified binding. Its input carries only approval and provenance. The preview is deterministic (the store epoch and mutation ID are derived from the reviewed content), and apply needs the reviewed digest, a new receipt path, the same deployed values and migration catalogue as the preview, the writer freeze, and the receipt kind deployedValueInitialization. It refuses any existing global document (legacy or authority), an unverified binding, registered writers, and migrations that are not Ready (the API image's project-management catalogue, checked as SYRF_MIGRATION_MODE=verify does). An exact replay returns the original receipt. Readers accept the new receipt kind only for revision 0 with no overrides and no stops. Steps, expected output and every refusal: runbook 2, Initialise.

Read-only authority observation

PR #4348 adds an optional observer for the imported authority in API and Project Management. The default RuntimeFeatureFlags:Mode=Legacy registers no refresh worker and makes no authority-source queries. Shadow is read-only and requires matching non-production identities plus all three pinned application configuration values: RuntimeFeatureFlags:Binding:Environment, ...:Database, and ...:Cluster. Use the verified binding/status evidence for these values. These are deployment/application settings, not live feature-flag controls; no Helm value is introduced by this slice. #4352 adds the explicit non-production Shared consumer mode described below; production Shared mode remains refused.

Each process reconciles every 15 seconds with a five-second read deadline. It reads the current envelope, binding and current mutation receipt coherently, validates the receipt and all digests, then atomically publishes an immutable snapshot. Legacy or missing state is never Ready. After initialization, a missing/legacy document, changed environment/binding/store epoch, revision rollback or equal-revision/different identity quarantines that reader instance. There is no automatic rebind or quarantine reset; complete the reviewed recovery procedure before restarting. Ordinary read failure immediately removes decision availability. Snapshots expire after 60 seconds measured with a monotonic clock, so moving wall time backwards cannot extend trust. An operation can retain one immutable valid capture while later operations see the refreshed observation.

GET /api/runtime-feature-flags/authority-status is authenticated and requires the current InspectRuntimeFeaturePolicy decision (legacy claims only in the existing Off/Shadow authorization modes). It returns local-process state, identity, timestamps and reason, with revisions as decimal strings. It returns no policy values, targeting lists or fleet-readiness claim, and does not query authority on demand. A valid local snapshot does not mean every expected host has acknowledged it.

PR #4349 extends receipt validation to ordinary global authority mutations as well as lineage import. Existing feature consumers remain on their existing paths; this observer does not enroll them or make a live lineage import operationally safe by itself. #4352 adds local Shared admission and #4358 adds host acknowledgements; scoped compatibility reporting remains subsequent work.

Idempotent authority mutation primitive

PR #4349 adds RuntimePolicyMutationStore in SyRF.Mongo.Common. It is not registered as a replacement for the legacy API store and exposes no additional public mutation endpoint. API wiring must preserve current permission checks and introduce the reviewed Shared-mode admission protocol; constructing this library service does not enroll consumers or enable production policy.

A request carries a mutation ID, complete expected policy identity, flag change and real/impersonated actor. The store requires a verified non-production binding, the existing migration-owned audit indexes, and a receipt-validated initialized authority. It reuses dependency planning and deployment compatibility checks, preserves the accepted baseline and unrelated overrides, and atomically commits the new envelope and immutable receipt with snapshot reads and majority writes. The deployment constraint guard is preserved and incremented. PR #4360 interprets receipt-verified schema-v1 deployment constraints: compatible changes proceed, while saved enabled keys outside the capability intersection refuse. Unknown or inconsistent constraints still refuse ordinary changes. Emergency stops remain independent of constraint evidence.

Concurrent editors at one identity produce one commit and one revision conflict. Retrying the identical mutation ID returns its original receipt, even after subsequent changes; reusing that ID with different payload or actor refuses. The receipt retains the complete original request and resulting envelope. FindReceiptAsync resolves the outcome without another write. Unknown commit outcomes are not converted into a success or a known-aborted conflict: retry the same request to resolve them. The read-only observer accepts both import and ordinary mutation receipts, and rejects inconsistent receipt data. Receipt success proves persistence only; fleet acceptance remains a separate coordinator responsibility.

Deployment constraint transactions

PR #4360 adds the bound ChangeDeploymentConstraintAsync shared-store primitive. A trusted coordinator can record one active rollout, extend its expiry or release it with an exact reviewed constraint digest. Every operation requires the complete current policy identity, a mutation ID, an unimpersonated actor and reason. Record validates the serving catalogs, target catalog and global schema-v1 enabled-key capability intersection against saved policy. A stopped feature retains its saved capability requirement.

Constraint operations increment the same constraintGuard as policy edits inside a snapshot/majority transaction. They leave policy revision, baseline, values and policy digest unchanged. This invalidates old previews without falsely indicating a policy change to hosts. An expired constraint keeps blocking incompatible changes; expiry is never a release. Ordinary extend/release requires the original coordinator. Break-glass is a distinct audited action whose caller must enforce its separate administrator permission. Receipts make identical retries idempotent, including a retry of record after release, and detect corrupted active constraint evidence. Emergency stop continues even when constraint evidence cannot be interpreted.

This is a transaction-layer increment, not a deployment endpoint. It inherits explicit non-production admission; Legacy remains default. No new collection, index, startup write, migration or environment setting is introduced. The next transport increment must authenticate environment-bound machine scopes, construct capability intersections from trusted reports and require collector evidence of rollout/rollback completion before normal release. The administrator break-glass capability, expired-constraint status in the fleet UI, and report/coordinator endpoints are still required. Never expose this primitive directly to policy editors.

Scoped compatibility reports

PR #4361 adds GET /api/environment/compatibility-reports/current, protected by the dedicated environment:report:read machine scope. Reporting defaults off. An enabled API requires RuntimeFeatureFlags:CompatibilityReport:Enabled=true, an explicit Provider (OpenIddict or Auth0), ClientId and Audience in that same section, plus the existing exact RuntimeFeatureFlags:Binding expectations. The report audience must match that provider's configured API validation audience, and the runtime/active environment must agree with the binding. This does not select Shared policy mode: Legacy can report that no accepted bound baseline exists without inventing one.

For staging Identity, the dedicated client is syrf-environment-report. Its only grant is client credentials and its only scope is environment:report:read; configure EnvironmentReport:ClientSecret through the normal secret owner to enable its registration. Missing, placeholder or weak secrets leave it disabled and revoke managed registrations holding that scope, using the same lifecycle as the statistics evidence client. No secret or environment value is supplied by this PR. Production Auth0 provisioning remains a separate approved tenant change; the API accepts only the configured client-credentials subject, client ID, audience and scope through the existing validated bearer handler. User/BFF principals, mixed scopes and other clients cannot read a report. A report token cannot authenticate on ordinary API routes, even if it also carries an end-user scope. Report permission grants no coordinator or policy-write permission.

Collection uses a snapshot transaction and verifies the current environment binding and policy receipt. It projects only policy identity, required global enabled keys, stops, accepted Boolean baseline and host self-observations. It excludes actor records, credentials, database names and targeting identities. Host self-observations never establish trusted workload inventory or readiness. A stable report is Complete only for a stated purpose whose required sources are all Current; see purposes and completeness. PRs #4362–#4365 supply constraint, workload, desired-state and migration evidence; #4366 revalidates the sources and returns Unstable if they keep changing. Never treat an HTTP 200 or the presence of a policy as deployment clearance.

The GET retains an immutable sanitized report in runtimeEnvironmentReport, so it is marked as a side effect for support impersonation. It requires existing migration-owned indexes 0019 and 0020; it never creates them. Reports expire for checks 15 minutes after collection and become TTL-eligible after 30 days (OD-20); top-level retention timestamps are BSON dates. Responses are private/no-store. Missing bindings, indexes, inconsistent authority or unavailable evidence return a bounded 503 without internal details. Turning reporting off denies collection and preserves previously retained reports until normal retention. The endpoint does not mutate policy, enroll consumers, activate a feature or promote a deployment.

Report purposes and completeness

PR #4405 (slice 6A.0) lets a report earn Complete. The caller states what the report is for with the optional query purpose:

purpose Use Extra required sources
policy-change Before a runtime policy edit none
promotion Before promoting a candidate build gitops (desired provenance)
legacy-return Before switching Runtime back to Legacy none; legacyParity must be present
absent Diagnosis only the report is never Complete (purpose and requiredSources are null)

Any other value, an empty value or a repeated purpose returns 400 ProblemDetails with code environment_report_purpose_invalid before anything is collected. Values are case-sensitive.

Every purpose requires policy, baseline, deploymentConstraint, hostAcknowledgements, workloads (the aggregate of workloads:api and workloads:project-management) and migrations. When the bound policy touches a statistics key, domain:statistics and domain:statistics-control are required too. A key touches statistics when its effective runtime value is on, or it has a saved override or an active stop. Statistics keys are the reviewed consumer contracts that Project Management also consumes (today materializedProjectStatisticsServing, Pages, StageOverview and SignalR). domain:statistics is always Missing, so a report whose policy touches statistics can never be Complete; that is intended, because statistics are not production-eligible. The report lists the set it applied in requiredSources.

The report is Complete only when all of these hold:

  • a supported purpose is given;
  • RuntimeFeatureFlags:FleetInventoryMode is Collector (same setting and Static default as fleet inspection). In Static mode a purposed report adds fleetInventoryMode:Static to unknownOrUnsupported and stays Incomplete, because configured counts are not observed inventory;
  • the sources were stable across captures (otherwise Unstable, as before);
  • every required source appears exactly once with state Current, and no required name is in unknownOrUnsupported;
  • the migrations prerequisite is Ready (NotReady or Unknown never completes a report);
  • the report has not expired;
  • for legacy-return, legacyParity is present (it is null without a bound policy).

Sources that are not required, such as gitops for policy-change or domain:statistics when the policy does not touch statistics, may be Missing or listed as unknown without blocking. A Current migrations source only means verification ran, so completeness also requires every migration this build knows to be applied with its checksum and declared indexes (Ready). Complete describes evidence, not permission: the offline checker still decides.

Host heartbeats no longer make reports Unstable. The stability comparison ignores each host's lastSeenAtUtc, and the hostAcknowledgements source digest is computed over the stored host rows without lastSeenAtUtc and authorityCheckedAtUtc. Instance UID, process start, image, catalog digest, acknowledged policy, readiness and state still count. The returned applied entries keep lastSeenAtUtc.

legacyParity (RT-15) is computed server-side for every baseline key that is production-eligible (today only disableMembership). Each entry records legacyValue (the deployed FeatureFlags value, as the evaluator computes it with stored policy ignored), runtimeValue (the evaluator's runtime decision: off while a stop is active, otherwise override or baseline), stopActive and agrees. An entry agrees only when no stop is active and the two values match; the section's agrees is true only when every entry agrees.

appliedMigrations lists the id, version and checksum of every migration the runner recorded as applied, in version order, read in the same verification step (at most 10,000 rows; more refuses collection). It never includes timestamps, actors or errors, and it is null unless the migrations source is Current. It takes part in the stability comparison, so the offline checker can confirm that a candidate's required migrations have been applied.

All four fields are additive and nullable; schemaVersion stays 1 and retained reports collected earlier read back with them null. The capabilities digest now ends purpose-completeness-v1:legacy-parity-v1:applied-migrations-v1. No flag is added: the route stays behind the default-off RuntimeFeatureFlags:CompatibilityReport:Enabled, and collection still writes only the retained report.

Checking a candidate offline

scripts/runtime-compatibility-check.py judges a candidate build against a report saved from GET /api/environment/compatibility-reports/current. It reads two local JSON files and prints one JSON object (verdict, exitCode, reasons[] of {code, detail}, sorted by code then detail). It is offline only: no network, credentials, database or cluster access, so it never collects or refreshes evidence and is not deployment clearance by itself.

python3 scripts/runtime-compatibility-check.py --report report.json --candidate candidate.json \
  --environment staging --binding-epoch <guid> --purpose policy-change|promotion|legacy-return [--now ISO8601]

The candidate uses schema syrf.runtime-candidate/v1 (catalogDigest, optional compatibleCatalogDigests, catalogKeys, policySchemaVersion, migrations[{id, version, checksum?, required}], images, targetMode Legacy|Runtime, and optional catalogGrains, a map of catalog key to contract grain). A report's policy.targetedKeys[{key, grain}] (policy schema 2, see Targeting) join the keys the candidate's catalog must contain, and when the candidate gives catalogGrains, a targeted key whose grain differs (or is missing) refuses targeted-key-grain-changed. A schema-1-only candidate (policySchemaVersion: 1) against a schema-2 report is already outside N-1..N (policy-schema-unsupported), so rolling back below a schema-2 binary while targeting exists never passes: remove every rule first (the document returns to schema 1). --now defaults to the current UTC time. Fixtures live in scripts/testdata/runtime-compatibility/; run python3 scripts/test-runtime-compatibility-check.py.

Exit Verdict Meaning
0 pass No evidence gap and no refusal
1 refuse Binding mismatch (checked first, short-circuits), or a replacement-build, legacy-parity or constraint refusal
2 insufficient-evidence Incomplete, unstable, unsupported or expired evidence (takes precedence over refusals)
3 unusable-input Unreadable or invalid files/arguments (short-circuits)

Reason codes: input-unreadable, input-invalid, candidate-schema-unsupported (3); binding-mismatch, catalog-key-missing, catalog-digest-incompatible, targeted-key-grain-changed, migration-not-applied, image-change-hosts-not-current, legacy-parity-stop-active, legacy-parity-differs, constraint-catalog-outside, constraint-key-outside, constraint-schema-outside (1); report-schema-unsupported, report-expired, report-unstable, report-incomplete, purpose-mismatch, required-source-unknown, required-sources-missing, policy-missing, migration-evidence-missing, legacy-parity-missing, policy-schema-unsupported (policy schema must be N or N-1), constraint-action-required, constraint-state-unknown, constraint-expired, required-source-undeclared (2). Timestamps must carry a UTC offset, and a missing argument is input-invalid (3), not argparse's usual exit 2.

The checker does not trust the report's own requiredSources alone: policy, baseline, deploymentConstraint, hostAcknowledgements, workloads and migrations must always be declared and Current. --now exists for fixtures; setting it earlier than the real time defeats the freshness and constraint-expiry checks, so never pass it for a real decision. A host counts as current when it acknowledges the report's policy identity; its readiness and heartbeat age are outside this checker (the server reports readiness as Unknown today, and fleet freshness belongs to the admission path).

Retrieve retained compatibility evidence

PR #4368 adds GET /api/environment/compatibility-reports/{reportId} for the same dedicated machine reader and exact environment binding as collection. It uses the existing default-off RuntimeFeatureFlags:CompatibilityReport:Enabled setting. Human Inspect access and a browsing UI remain follow-up work; this route grants no additional token scope or policy authority.

The response preserves the original report ID, source observations, completeness and timestamps. A report past its 15-minute validity may still be retrieved for diagnosis during its 30-day retention, but it is historical evidence, never a fresh deployment check. Retrieval does not recollect sources, extend expiry, write a replacement, create indexes or refresh TTL. Records past DeleteAfterUtc return 404 even before the TTL sweep removes them. Missing IDs and records belonging to another binding also return 404; unreadable binding or inconsistent stored metadata returns sanitized 503. Every response is private/no-store.

Switching reporting off denies retrieval while retaining saved records; switching it back on restores access under the same binding until retention expires. A different binding never gains access to the old reports.

Constraint evidence in compatibility reports

PR #4362 adds the optional deploymentConstraint projection and a deploymentConstraint source entry. Both are captured in the report's policy snapshot. A bound authority with no active constraint returns a null projection and a Current source with an explicit absence digest. Legacy/uninitialized policy keeps constraint evidence Missing; null alone never proves that a deployment is unconstrained.

An active document must match its immutable constraint receipt and the current environment/binding/store lineage. The report exposes rollout ID, constraint digest, serving/target catalog digests, global schema-v1 capability intersection, policy revision at record and timestamps. Coordinator/actor identity, reason and receipt internals are excluded. Policy revision and the source's shared-guard revision are decimal strings. The source revision advances on record, extend and release even though policy identity does not change. Every policy edit also increments this shared guard; the source revision can advance with an unchanged constraint digest.

An expired but unreleased constraint remains present with state: ActionRequired and reason: Expired. Collection never releases or extends it. Inconsistent receipts, unsupported schema/lineage, invalid guards or malformed projection evidence refuse collection. A verified constraint is one required source among several (see purposes and completeness); it is not permission to deploy. This is additive evidence on the existing default-off, report-only machine route. It introduces no new scope, index, migration, activation or deployment setting.

Workload collector publication

PR #4363 adds POST /api/environment/observations behind RuntimeFeatureFlags:WorkloadCollector:Enabled (default false). Enabling it requires explicit Provider, ClientId and Audience under that section plus the same verified runtime environment binding used by the report service. environment:observations:write is a separate machine scope: it cannot read reports, read or mutate policy, or use any other API route. Mixed-scope, session, user and impersonated credentials are refused. The OpenIddict client is syrf-environment-collector; only an explicit strong EnvironmentCollector:ClientSecret seeds it. No credential or collector is provisioned by this PR.

Each publication contains one complete API or project-management Deployment snapshot: environment/binding, observation ID, previous observation ID, Deployment UID/resource version, lossless generation strings, desired replica count, collection timestamp and up to 1,000 pods with UID, pinned image digest, readiness and termination status. The body is bounded to 512 KiB. The collector must retain its last accepted observation ID: new publications compare against it; exact retries reuse the same ID and payload. A timeout is unknown, so retry the same request. Conflicting replay or a superseded publication is refused, never silently rebased. Snapshots older than 30 seconds or in the future are refused; collector and API clocks must agree.

PR #4369 requires every pod to include explicit Boolean ready and terminating fields. Omission is unknown evidence, not false: an authorized request missing either field returns 400 before publication. Explicit false values remain valid. Existing observations written by the typed publisher already include both fields and retain their exact replay digest; malformed stored payloads missing either field are refused rather than silently interpreted. No stored observation is rewritten or deleted and no migration is needed. The existing default-off collector setting continues to control publication.

Migration 0018 owns the age index. Publication verifies it and the environment binding, then updates only runtimeWorkloadObservation. It neither reads policy nor removes host acknowledgements. Reports expose collector evidence separately from host self-reports with per-service Current/Missing/Stale/Unknown source states. A fresh observation does not mean every pod is ready; deployment generation, replica and readiness checks still belong to the compatibility checker. Completeness additionally needs Collector inventory mode and a purpose (see purposes and completeness);

4364–#4366 add desired GitOps, migration and cross-source evidence. Automatic host retirement and live Kubernetes collection are

follow-ups; no runtime mode, activation, migration execution or production access changes here.

Read-only workload capture

PR #4370 adds scripts/capture-runtime-workload.py, an explicitly invoked capture command that produces one JSON request for the existing observations endpoint. It does not publish, provision credentials, select a default cluster, change kube context, read Secret resources or activate a feature. The existing collector endpoint remains default-off. Use a separately approved read-only Kubernetes identity with get on the named Deployment and list on ReplicaSets and Pods in the selected namespace.

Example for an approved non-production context (values must match the reviewed environment binding):

python3 scripts/capture-runtime-workload.py \
  --context "$REVIEWED_CONTEXT" --namespace "$REVIEWED_NAMESPACE" \
  --service api --deployment syrf-api --container syrf-api \
  --environment "$BOUND_ENVIRONMENT_ID" --binding-epoch "$BOUND_EPOCH" \
  --expected-observation-id "$LAST_ACCEPTED_OBSERVATION_ID" \
  --output api-observation.json

Omit --expected-observation-id only for a known first publication. The service/deployment/container mapping is explicit operator input; the command never guesses it. The capture can be submitted immediately to POST /api/environment/observations by the separately authenticated publisher. Retain the exact output and observation ID for uncertain-outcome retries; do not change evidence under that ID. The command never overwrites an existing output file or retries publication. Publication still enforces binding, expected observation ID and 30-second freshness; a capture file is not a durable freshness guarantee.

Each capture reads Deployment, ReplicaSet and Pod JSON twice using only kubectl get. It follows controller UIDs from Deployment to ReplicaSet to Pod, including old rollout ReplicaSets; labels alone cannot enroll a foreign pod. It requires the exact named container, an explicit readiness condition and container Boolean, and a repository-qualified @sha256: image ID (optionally prefixed docker-pullable://). Digest-only or missing image IDs are refused rather than inferred from image tags. Missing observations, paginated lists, duplicate UIDs, malformed ownership, changed selected resource versions and clock regression refuse capture. A valid zero-replica observation is allowed but never establishes fleet readiness.

The observation time precedes the first read. All six reads and revalidation must finish within 20 seconds; each command has a five-second deadline. Responses larger than 8 MiB are refused, as is output above the endpoint's 512 KiB limit. Only the allowlisted observation fields are emitted; raw environment values, annotations, labels, container diagnostics and command stderr never enter the artifact. Matching reads establish observed agreement, not an atomic Kubernetes snapshot. The collector endpoint and later compatibility checks still must validate freshness, desired generation, replica count and readiness.

This slice supplies workload acquisition only. Scheduled publication/credential handling, desired Argo/GitOps acquisition, trusted retirement and rollout-completion checks remain ordered follow-ups. Tests use synthetic responses exclusively; no live cluster or environment is enrolled by this change.

Authenticated deployment coordinator

PR #4374 adds default-off machine-only PUT and DELETE /api/environment/deployment-constraints/{rolloutId}. Enable only with reviewed RuntimeFeatureFlags:DeploymentCoordinator settings (Enabled, Provider, ClientId, Audience) and the existing exact environment/database/cluster binding. The API authority must also be Shared with the same binding. This does not relax the existing production Shared-mode refusal. The separate scope is environment:deployments:coordinate; report, collector, human and session credentials cannot use it, and coordinator tokens cannot use unrelated routes or the administrator break-glass path.

Native Identity seeds syrf-environment-coordinator only when EnvironmentCoordinator:ClientSecret is supplied with at least 32 characters, using client credentials and this sole scope. An absent/short secret removes the client's grant through the existing automation-client reconciliation. The API also accepts the explicitly configured Auth0 machine provider; this code does not provision or change an Auth0 tenant.

PUT accepts Record or Extend; DELETE accepts Release. The body carries a mutation ID, complete expected policy identity (revision as a canonical decimal string), action, reason and operation-specific constraint fields. Coordinator clients and proxies must preserve the JSON body on DELETE as well as PUT. Always send action explicitly: the current non-nullable wire enum defaults an omitted action to Record. Record includes the immutable completion targets described below; Extend and Release use the reviewed constraint digest. Release additionally selects Target or Rollback and always calls the verified store path. The actor is derived from the configured authenticated provider/client, never from a request actor field. Keep that client identity stable for an active rollout: only its owning coordinator may extend or release it. Rollout IDs are one ASCII path segment, up to 128 letters/digits or -_.: characters.

Responses expose only mutation ID, rollout ID, action, resulting constraint digest (null after release) and original receipt time. Exact uncertain-outcome retries preserve the entire request. HTTP 409 refuses a changed policy/constraint or unproven completion; HTTP 503 or transport loss leaves the outcome unknown. Requests are bounded to 32 KiB and ten seconds; responses and authorization denials are no-store. No endpoint accepts BreakGlassRelease. Its separately authorized administrator adapter remains a follow-up.

The gate defaults off, preserves all saved constraint/audit data, and never runs a migration. Credential provisioning, workflow invocation and environment enrollment are separate approved operations; none occurs in this change. Tests use synthetic identities, a TestServer and isolated local test data only. The API chart does not expose a dedicated coordinator values block; reviewed chart/config wiring and environment enablement remain follow-ups. Adapter tests pin configuration and identity refusal, the server-derived owner and receipt mapping independently of controller fakes. Further transport improvements are explicit missing-action rejection and distinguishing corrupt stored-state FormatException from malformed request identity (the current controller maps either to 400); neither changes the store's fail-closed mutation behavior.

Verified deployment completion primitive

PR #4373 adds ChangeVerifiedDeploymentConstraintAsync for the later authenticated coordinator adapter. Recording through this entry point requires immutable target and rollback expectations: one catalog digest and one repository-qualified SHA-256 image per API/PM service. The target catalog must equal the constraint's target; the rollback catalog must be in its serving set. Extension preserves these expectations unchanged.

Normal release explicitly selects Target or Rollback. Inside the existing binding/policy/guard transaction, the store reads current collector observations and host acknowledgements, requires collector-backed Active adoption under the configured positive counts, and matches every observed pod image and current catalog to the selected recorded expectations. Missing, stale, mixed-image, non-ready or unacknowledged evidence refuses release without changing the guard, constraint or audit. These expectations also enforce verification when the older internal primitive is called. Exact successful mutation replay uses the immutable receipt even if observations have since expired or disappeared; it never performs the operation a second time.

Old internal constraints/receipts without completion expectations remain readable/replayable, but cannot be released through the verified coordinator entry point. Its break-glass action is refused: that operation must remain on the separate administrator capability path. No HTTP coordinator or break-glass adapter is exposed by this slice. The original internal primitive retains its documented caller-authentication requirement.

No new flag is introduced for this internal safety boundary. Existing default-off runtime/collector modes are unchanged, no collections or indexes are migrated, and no live constraint is recorded or released. This proves observed target/rollback completion; candidate compatibility, domain readiness and live coordinator credentials/workflow integration remain separate requirements.

Collector-backed fleet inventory

PR #4372 adds deployment configuration RuntimeFeatureFlags:FleetInventoryMode=Static|Collector. The default is Static, preserving the configured-count inspection path. Collector is an explicit opt-in for environments with the trusted workload publisher; unknown values refuse inspection. No environment is enrolled by this change. This configuration gates read-only fleet semantics, not policy decisions or migrations.

Collector mode reads authority, acknowledgements and workload observations in one bound database snapshot. Both API and PM must have fresh observations (strictly younger than 30 seconds), matching binding and the configured positive FleetExpectedReplicas counts. Deployment generation must be observed, every expected pod must be present, ready and non-terminating, and UIDs must be unique across services. Missing evidence, zero/surplus replicas, generation lag, stale/future timestamps and service disagreement never establish Active. The existing host policy/catalog checks, autoscaler Unknown rule and statistics-domain Unknown rule still apply.

Retirement uses MongoDB's clock only (PR #4407). Every heartbeat write stores serverSeenAtUtc, stamped by MongoDB itself ($$NOW), beside the host's own lastSeenAtUtc. The write keeps its filter, upsert, transaction and late-write protection. Evaluation reads MongoDB's current time ($$NOW) in the same snapshot transaction. A historical acknowledgement absent from the fresh service inventory is marked Retired in the returned view only when MongoDB has not seen it rewritten for the quiet period, RuntimeCollectorFleet.RetirementQuietPeriod (the 30-second freshness window plus 5 seconds, so 35 seconds). Hosts heartbeat every 8–12 seconds, so a quiet row belongs to a stopped pod. A row written more recently may be a pod that started after the collector's read: it leaves the fleet Unknown (CollectorHostObservationDisagrees). A host clock that runs behind or ahead cannot change this, because both times come from MongoDB. A row whose start time follows its heartbeat is also Unknown.

Rows written before serverSeenAtUtc existed, or by an image without it, carry only the host clock. Such a legacy row retires only once its lastSeenAtUtc is at least LegacyRetirementAge (15 minutes) behind MongoDB's time. The 15 minutes is an assumed bound on host clock skew, not a measured or verified one: a live legacy host whose clock runs more than 15 minutes behind MongoDB could be retired. The bound applies only to legacy rows. It stops mattering once every host runs an image that writes the server stamp, because every row then carries MongoDB's own time. The first rollout away from images that do not stamp the field can therefore take 15 minutes to reach Active. After that, a rollout reaches Active about 35 seconds after the old pods stop, once the new pods have acknowledged.

MongoDB's time is read as the first statement of any transaction that may retire rows (fleet inspection, production admission and deployment completion). A snapshot transaction sees rows as of its first operation, so a slow transaction cannot make a live row look older than it was. Emergency stops never read it.

All rows remain stored and visible, including retired rows. No TTL, deletion or migration is introduced. serverSeenAtUtc is an additive root field: older readers ignore it, and older rows parse without it. It is not part of the stored payload or the API response. Switching back to Static uses every retained row again. Static mode counts every retained acknowledgement, so it suits only hosts whose instance UIDs stay the same (for example the local runtime-fleet harness). In Kubernetes every replaced pod has a new UID, so after any rollout the old pods' rows keep a Static fleet out of Active (it reads Applying, then Degraded). Kubernetes environments therefore need Collector mode before fleet Active can be relied on. A pod present in the inventory but missing an acknowledgement still blocks Active.

The local runtime-fleet harness exercises this end to end against real API and project-management processes: bash e2e/run-local.sh --runtime-fleet --scenario collector-scale-up. It restarts the APIs in Collector mode, publishes a synthetic inventory through POST /api/environment/observations with a mock-OIDC machine token, and asserts Active with every listed host current, Unknown (CollectorHostObservationDisagrees) while an unlisted API is live, and Active again with that host Retired once it has been quiet for 35 seconds by MongoDB's clock. What it covers and what it does not: runtime fleet lane.

This confirms policy adoption against observed UID membership. It does not prove deployment target image compatibility, release a deployment constraint, or complete statistics domain readiness. Live collector setup, rollout target proof and coordinator integration remain required follow-ups.

Explicit workload publication

PR #4371 adds scripts/publish-runtime-workload.py to submit a saved capture once. Supply an already-issued machine token in SYRF_RUNTIME_COLLECTOR_TOKEN; it must have the separate environment:observations:write scope and match the API's configured client, provider, audience and environment binding. The command does not obtain tokens, provision credentials, schedule collection or activate the default-off collector endpoint. No new flag is needed for an explicitly invoked command; the endpoint's existing gate remains authoritative.

python3 scripts/publish-runtime-workload.py \
  --api-origin "$REVIEWED_API_ORIGIN" --environment "$BOUND_ENVIRONMENT_ID" \
  --binding-epoch "$BOUND_EPOCH" --service api --input api-observation.json

The API origin must be HTTPS (HTTP is accepted only for loopback test servers), with no user information, path, query or fragment. Redirects are refused. The command checks the saved capture's environment, binding epoch, service and observation ID before sending its exact bytes. The API still validates the complete contract, permissions, expected-observation CAS and initial freshness. No local age check prevents an exact retry of an already accepted observation after its initial admission window.

Exit 0 means a 200 receipt with the matching observation ID and a valid digest was received; stdout contains only those two receipt fields. Exit 1 means an HTTP 3xx/4xx refusal, except 408/425/429; exit 2 means local input refusal; exit 3 means the outcome is uncertain (408/425/429, transport loss, 5xx, or an invalid/missing receipt). Preserve the exact capture and replay it after investigating an uncertain outcome. The command never retries, rewrites the capture, prints credentials or copies remote error bodies into logs. A successful receipt confirms publication, not fleet readiness or rollout completion. Rotated/advanced observations can make an old retry conflict; do not replace evidence under an existing observation ID.

Environment proxy settings are explicitly ignored, so bearer tokens are sent directly to the reviewed origin. HTTP 408, 425 and 429 are uncertain outcomes (exit 3), as are transport failures and 5xx responses: retain and replay the exact capture. The ten-second urllib timeout is per socket operation, not a total elapsed deadline; an invocation under a scheduler needs its own process deadline. Terminating a publisher must also be treated as an unknown outcome.

Tests use mocks and a loopback HTTP stub only. Live publication, token provisioning, automatic scheduling, desired GitOps acquisition and trusted retirement/rollout checks remain separate follow-ups.

Desired deployment provenance

PR #4364 adds optional desired evidence to each collector observation: applicationUid, lossless resourceVersion, a 40-character gitOpsRevision, chartTag and 64-character renderedSettingsDigest. The collector must bind these to the same service and timestamp as its workload snapshot, using the earliest observation time when collecting multiple sources. This is a collector obligation: desired has no independent timestamp, so the API applies the workload timestamp and cannot attest the actual source-read time. It must derive rendered-settings digests from allowlisted non-secret configuration; the API accepts digests, never raw settings or secret values.

Reports include per-service GitOps sources and project the aggregate desired revision/chart tags only when both API and PM have Current provenance at the same GitOps revision. The aggregate settings digest is a canonical hash of sorted service/digest pairs; each original digest stays available in the workload projection. Different revisions yield Unknown, not a guessed winner. Missing or stale evidence leaves the desired snapshot null. Optional evidence can be added and removed without invalidating stored observations; exact retries of older requests without desired preserve their receipt digest.

This uses the existing default-off collector/report settings and scopes, with no new migration or rollout. Reports can be Complete only in Collector mode (see purposes and completeness); migration verification and cross-source stability checks are implemented in #4365–#4366.

PR #4365 supplies fresh migration evidence through the existing default-off compatibility-report route. The API supplies its PmMigrations.All catalogue. Each request uses the migration runner's read-only verification: every known migration must be applied with its expected checksum, and every declared index must match. The migrations prerequisite is Ready or NotReady; a missing catalogue or unreadable state produces Unknown. This does not use the five-minute schema health cache.

The migration source revision is the ordered catalogue digest; its evidence digest covers that catalogue and the verification result. Neither raw diagnostics nor migration actors/history are returned. Evidence is dated before verification starts and expires after 30 seconds, including time spent collecting the remaining report. Collection writes only the retained report, never migration state or indexes. A later request rechecks everything, so an index removed after a healthy result is detected immediately.

This proves readiness only against this binary's catalogue, not against a future candidate build. Reads of migration records and index definitions are not atomic with the report's policy snapshot. Reports are revalidated as described below; completeness follows purposes and completeness. The former crossSourceRevalidation missing-input reason is no longer emitted. No migration, live collector, deployment or activation is performed by this slice.

PR #4366 revalidates report sources before retention. It collects an independent snapshot, recollects, and compares the policy/baseline, constraint, workload and desired evidence, host acknowledgement digest, migration verification result, prerequisite states and unknown reasons. Per-read timestamps and generated report IDs are excluded from comparison; stored observation timestamps (other than host heartbeats, since

4405) and freshness states are included.

A mismatch starts another comparison using the latest snapshot. Three comparisons (at most four captures) are allowed under the existing request deadline. A stable result is then evaluated for its purpose (purposes and completeness). Continuing changes return Unstable with crossSourceChanged; read failure or cancellation cannot produce a stable result. Only the final report is retained with majority acknowledgement.

Each snapshot rechecks the original environment binding. Freshness is evaluated at the end of collection, so a source that ages out while other evidence is read cannot remain Current. This verifies agreement between observed source snapshots, not an atomic transaction spanning Mongo migration/index reads and external collectors. No report in this slice authorizes activation or deployment.

PR #4367 adds optional statisticsControl metadata from the durable global statistics singleton. The domain adapter reads at most two rows in the report's own snapshot and rejects duplicate controls, missing required fields, malformed data, unknown modes and negative epochs/versions. No repository initialization, index creation or cache is involved. The output includes mode, active-reviewer-tracking mode, whether a mode transition is pending, catalogue/storage/source/schema versions and lossless string write/mode/client-invalidation epochs. It excludes document IDs, actor/audit fields, transition tokens and project allowlists. Missing or invalid evidence is explicit in domain:statistics-control.

The control metadata participates in cross-source revalidation. domain:statistics remains unknown: this record alone cannot establish deployed maintenance flags, fleet protocol compatibility, inventory or per-project serving readiness, so a report whose policy touches statistics stays Incomplete. The existing default-off report setting and machine permission boundary apply; collection does not initialize or transition statistics state.

Reviewed consumer evaluation

PR #4350 adds runtimePolicy.consumerContract metadata to the canonical YAML and includes it in the catalog digest. The initial disableMembership (API admission) and materializedProjectStatisticsServing (API/PM read admission) contracts are extended by #4352 with the existing pages, Stage Overview and SignalR selectors. Other keys remain unreviewed; absence of metadata never means safe or production-eligible. See the audited contract evidence.

GET /api/runtime-feature-flags/authority-evaluation/{key} requires the current InspectRuntimeFeaturePolicy permission before catalog lookup or evaluation. In Shadow mode it evaluates one immutable, receipt-validated local capture, with its accepted baseline and current deployment prerequisites. All five contracts expire at 30 seconds measured monotonically and tolerate no failed reads. Legacy mode reports Disabled. Unknown keys return 404; known but unreviewed keys report UnreviewedConsumer; an unsupported host or unavailable/stale policy returns no value. The declared safe value remains separate from that unknown decision.

Responses include local-process scope, catalog classification and a decimal-string revision when a fresh decision is available. ProductionEligible describes the reviewed catalog candidate, not permission to activate it, production readiness, or fleet acknowledgement. No database query happens on demand. This inspection route does not replace any domain gate or enable Shared mode by itself. Fleet coordinator admission remains required before a rollout can be declared Active.

The added catalog metadata changes CatalogDigest. Previously generated lineage previews must be regenerated and reviewed against this catalog; an old digest is rejected rather than silently reinterpreted. No live lineage import has run in this implementation programme. This change runs no data migration and changes no Helm/browser defaults.

Authority emergency-stop primitive

PR #4351 extends the bound shared writer with EmergencyStop and ReleaseStop actions for the audited global controls with a false safe value (initially membership and serving; #4352 also audits the three statistics selectors). These are library operations, not new public endpoints or permission grants. Legacy remains the deployment default and production writes remain refused.

A stop records the key in killSwitches and advances policy identity without changing the accepted baseline or saved overrides. The bounded evaluator applies the safe value before dependency normalization; ordinary policy edits cannot clear the stop. Releasing it explicitly resumes the latest saved policy, subject to the existing deployment and dependency checks. This does not remove saved memberships, statistics, or history, and does not alter deployment-owned statistics maintenance gates.

Both actions require exact binding/lineage/revision, actor and reason, and atomically commit an immutable receipt with policy state. Identical request replay resolves the original receipt, including after later edits. Reusing its mutation ID for a different action refuses. Emergency stop bypasses deployment-constraint and new-audit-index readiness gates (plan §6.4/Q1 and §14.5); the built-in unique audit _id and shared policy transaction still serialize receipts. It does not bypass binding or authority integrity. ReleaseStop retains the normal readiness and constraint gates. No index is created or migration run by either action.

Older authority readers that cannot understand non-empty stops refuse that policy. Before any live enrollment, deployment compatibility checks, the preview/commit UI and production-grade fleet admission (independently observed inventory, not only the host acknowledgements described below) must be complete. No emergency stop has been applied to a real environment.

Targeting (policy schema 2)

Project allow and exclude rules are writable, and open and close routes, behind a switch. 6F PR A (#4463) taught every API and Project Management binary to read and evaluate targeting rules; 6F PR B (#4472) adds the preview and commit routes behind RuntimeFeatureTargeting:Enabled (default off in every environment) and the first targetable flag; 6F PR C (#4482) decides the pilot's route gate and project availability for the admitted project (see Route gates and availability). 6F PR F (#4488) adds project rollouts with stable cohorts (see Gradual rollout). 6F PR E (#4495) adds person, pair and "active members of a project" rules on Personal keys (see Personal and combined targeting). 6F PR H (#4496) adds per-rule switches (see Switching a rule off). 6F PR G (#4510) adds consent (join and leave a beta) and stricter-only environment feature settings (see Consent and environment settings). Rollouts over people are refused as notYetSupported or personalRuleOnProjectGrain. Plan: delivery plan §7.2.

The targetable flag. screeningKeywordHighlighting has a reviewed grain: Project consumer contract (API only, productionEligible: false, new-request-admission) and owns the registry feature keyword-highlighting (grain: Project, permission: ProjectDesignPolicy, whatsNew: quiet while it is a Developing pilot). The pilot choice is owner decision D33, answered A on 2026-10-24; retargeting means moving that contract and feature to another flag and regenerating. With no rule saved it decides exactly as before in Legacy and Shadow mode; in Shared mode it moves from the API's compatibility values to the bound authority, which gives the same value while the policy is fresh and the contract's safe value (off) when it is stale or unavailable. Production stays deployment-controlled (non-Global keys are never production-eligible). Every other key is Global and refuses rules (globalKey). The catalog generator accepts grain: Project | Personal on a contract but never with productionEligible: true; registry check 9 requires a feature's grain to equal its flags' contract grain, a non-Global feature's flags to all have contracts, and no deployment-managed flag to be targeted.

Switch on an isolated stack (never staging or production without an owner decision): set RuntimeFeatureTargeting:Enabled=true for API and Project Management alike (chart value runtimeFeatureTargeting.enabled, env SYRF__RuntimeFeatureTargeting__Enabled), with RuntimeFeatureFlags:Mode=Shared and a bound environment. Every fresh API and PM host must advertise policy schema 2 (policySchemaVersions: [1, 2] on its acknowledgement, which every binary since PR A does) before the first rule is accepted.

Preview, then save (capability ChangeRuntimeFeaturePolicy; both answer 404 and read nothing while the switch is off):

POST /api/runtime-feature-flags/screeningKeywordHighlighting/targeting/preview
{ "expectedIdentity": { …current policy identity… },
  "targeting": { "rules": [
    { "id": "<new guid>", "kind": "allow", "label": "Wave 1 pilot", "subjects": [ { "type": "project", "id": "<project guid>" } ] },
    { "id": "<new guid>", "kind": "exclude", "label": "Held for data review", "subjects": [ { "type": "project", "id": "<project guid>" } ] } ] } }

PUT /api/runtime-feature-flags/screeningKeywordHighlighting/targeting
{ "mutationId": "<new guid>", "expectedIdentity": { … }, "targeting": { "rules": [ …the previewed rules… ] },
  "previewDigest": "<from the preview>", "reason": "Pilot for the keyword-highlighting study" }

The rules are the key's complete replacement; "targeting": null (or no rules) clears them, and removing the last rule anywhere writes schema 1 again with the v1 digest. Clearing a key that has no stored rules is an idempotent no-op: the preview proposes the current identity, and the commit writes no revision and no receipt and returns the unchanged identity. The preview returns the rules before and after by kind, the projects that gain and lose the key (counts, and up to 50 with their names), dependency notes, schema admission (required, hosts supporting and total, Ready | Blocked) and the usual identities; it writes nothing. The commit goes through the policy writer exactly as a policy change does: binding re-check, receipt chain, replay by mutation ID (retry the same ID and body after an unknown outcome; another body under that ID is refused), the preview digest (domain syrf-runtime-targeting-preview-v1, over the key, expected identity and rules, the candidate document, the constraint guard and the deployed values), the deployment constraint, then schema-2 admission. Its receipt is an ordinary globalPolicyMutation with request.action = "Targeting" and request.targeting = { key, rules } (null to clear), so GET mutations/{mutationId} resolves it. Saved settings, stops and other keys' rules are untouched, and a later policy change or stop keeps every stored rule.

Refusals are HTTP 422 { code: "runtime_targeting_refused", reason, message } and write nothing:

Reason When
globalKey A rule on a key with no Project or Personal contract (clearing is allowed on any key, so rules a catalog change left incompatible can be removed)
deploymentManaged The key is pinned to its deployed value across API and PM
personalRuleOnProjectGrain A person or pair subject, or a rollout over people, on a Project key
notYetSupported A rollout over people on a Personal key
rolloutPaused A change to a paused rollout's amount, group or subject type (make it Active or turn it off first)
invalidRule, duplicateSubject, limitExceeded Schema rules 2 to 6 (the message is the exact schema message; enabled on a rollout is invalidRule with a message pointing to its Off state, and a requirement, stop or default rule is invalidRule with a message naming exclude, allow and rollout), a document over 512 KiB, a rollout change over more than 100,000 projects, or a rollout over projects on a Personal key
unknownSubject A named project does not exist or is deleted, or a person the key's stored rules do not already name is not an active Investigator (absent, deactivated, suspended while applicationSuspension is on, or pending deactivation); one generic reason
parentOff An allow, or a rollout that is not Off, on a key whose required Global parent is off: turn the parent on or target it first; it is never widened
schemaUnsupportedByFleet The first schema-2 write while any fresh host (or no host at all) does not advertise schema 2, or pausing a rollout while any fresh host (or no host at all) does not advertise rolloutPausedAt (6F.F2)

A deployment constraint that allows only policy schema 1 refuses the write (runtime_policy_constraint), and a coordinator may now record allowedCapabilities.policySchemaVersion: 2. Production refuses every targeting write, a clear included and on any key, eligible or not: Legacy production has no policy administration, and Runtime production answers runtime_policy_constraint ("This key is not eligible for production runtime policy.").

How the browser reads a targeted feature (6F W1)

The web app asks one service, FeatureGate (src/services/web/src/app/core/services/feature-gate.service.ts), whether a feature is on: isEnabled('<featureId>') returns a signal.

  • A Global feature reads the runtime snapshot flag, exactly as before.
  • A Project or Personal feature reads the enabled value that GET api/projects/{projectId}/features/availability returns for the current person and project. enabled is absent (null) when no targeting is stored for the feature, and then the gate falls back to the runtime snapshot flag, so with no stored targeting the browser behaves as it did before the gate.
  • The availability client keeps one answer for the current (person, project) and clears it when either changes. A late response for the previous project is dropped, and until the new project's answer arrives the gate reads the snapshot flag, never the previous project's targeted value (RT-32, web half).
  • A revoked allow, or a new exclusion, takes effect on the next availability fetch, which follows every accepted runtime revision (SignalR hint, fallback poll, focus or reconnection): the control then disappears without a reload.
  • The gate hides controls only. The server's permission and route checks still decide what a person may do.

The keyword-highlighting pilot (keyword-highlighting, flag screeningKeywordHighlighting) uses it for the keyword lists in project settings, the review page's keyword toggle and the PDF viewer's highlighting. A new web consumer of a Project or Personal feature should inject FeatureGate rather than select the flag.

Switching a rule off

Owner decision D28 = A (6F PR H, spec 6F.9). An exclude or allow rule can be saved with "enabled": false (absent in a request means true). The switch is an ordinary reviewed change: preview, reason, a new revision and a receipt, exactly as for any other rule edit; send the same rule id with enabled flipped to switch it back. A switched-off rule keeps its place, label and subjects, and:

  • the evaluator skips it (the "Preview and check" trace shows skipped for the exclusions or allows step, naming the rule), so a disabled allow decides as if it were absent and a disabled exclusion stops excluding;
  • it counts for neither the required-enabled set (a key with only disabled allows does not need the capability on every host) nor the preview's gains (re-enabling it previews the gain, disabling it previews the loss), and it does not trip the parentOff check;
  • the stored document always carries enabled on exclude and allow rules (true or false), as schema 2 already shows, so documents written before this change are unchanged and removing every rule still returns the document to schema 1; changing only enabled changes the preview digest and the policy digest.

A rollout has no second control: its switch is its Off state. Requirements, stops and the environment default have no switch; a commit that sends enabled on a rollout, or a rule of kind requirement, stop or default, is refused as invalidRule (an enabled on a rollout points to its Off state; the other kinds are told that only exclude, allow and rollout rules exist). When RuntimeFeatureTargeting:Enabled is off the existing rules still apply (exclusions and stops stay in force, allows and rollouts are dropped); a switched-off exclusion is skipped in that state too, so it never excludes.

The Targeting page (6F W2)

/admin/features/<featureId>/targeting (PR #4519, canvas board Targeting) shows one Project or Personal feature's rules in the order SyRF checks them: requirements, stops, exclusions, specific projects (or people), gradual rollout and the environment default. It needs inspectRuntimeFeaturePolicy; adding, editing, switching, removing and saving need changeRuntimeFeaturePolicy, and an inspector sees the same page without those controls. It reads the saved rules from

GET /api/runtime-feature-flags/{key}/targeting

which answers { key, grain, identity, rules: { exclude, allow, rollout }, projects, consent }: the identity the rules were read at (the next preview's expected identity), the key's grain, the stored rules exactly as written (switched-off rules included), and the owning feature's consent (code, environment setting and reason, effective). It is behind RuntimeFeatureTargeting:Enabled (off: a plain 404 before any capability check or read; the page then says targeting is switched off), needs ChangeRuntimeFeaturePolicy or InspectRuntimeFeaturePolicy and Shared mode, and writes nothing. A key this binary's catalog does not declare answers a 404 problem whose code is runtime_feature_flag_unknown (#4563), so the page can tell it from the switch: it then says Rules not found (the server does not know the feature's flag, for example while it runs an older version), never "switched off". Both capabilities, like ListUsers, require the application Administrator role (ApplicationAuthorityPolicy), so the person IDs this read returns to an inspector reach no one who cannot already list every user (pinned by RuntimeAdministrationCapabilityHttpTests). projects names the projects the rules name only for a caller who may change the policy (the preview already returns names under that capability); for an inspector the page has no names (see "Names, never bare IDs" below). This read returns people as IDs only; the page names them from the Admin Console user list the people lookup below reads.

  • Every change is a draft. Adding or editing a rule (a reason of at most 80 characters, shown with the rule; projects and people found by name, e-mail address or ID: see the lookups below), switching an exclusion or allow on or off (D28 A) and removing a rule change only the draft. The draft bar says what is not saved yet. Preview changes posts the whole draft to the targeting preview and shows the changes, the projects that gain and lose the feature (with a "because" taken from the draft), people by name and e-mail address, inactive people listed for removal, dependency and rollout notes, and schema admission (a Blocked fleet disables saving). Save commits the previewed rules with the preview digest and a reason (proposed from the draft and editable; the proposal follows the draft, for example when a pasted project's name arrives, until you type in the field, and from then on your text is kept).
  • Refusals and conflicts. A refused preview or save is headlined in plain words for each known reason (unknownSubject, duplicateSubject, limitExceeded, invalidRule, personalRuleOnProjectGrain, notYetSupported, rolloutPaused, parentOff, schemaUnsupportedByFleet, globalKey, deploymentManaged, runtime_policy_constraint; 401/403 as "You may not change these rules."). For unknownSubject the page looks up every project of the draft by its exact ID and names the ones that no longer exist ("Example project C no longer exists.", or "A project in this draft no longer exists." with the short ID when it has no name), and people the user list does not have. The server's message and reason stay available, verbatim, in a collapsed Details disclosure. Any 400 on a preview or save is a refusal too (headlined with the server's text, any GUID replaced by a name or short ID), and so is a 404 there: targeting was switched off since the rules were read ("Targeting has been switched off here since the rules were read, so they can't be changed now."). A refused save is never offered to be sent again unchanged (#4563). After a 404 the page also disables Save until you go back to the draft or the rules are read again, and a refused preview closes with "Your draft is kept, but nothing can be previewed or saved until targeting is switched on here again." instead of "Change the draft, then preview it again."; after a 400 Save stays available, so the draft or reason can be fixed. A revision conflict keeps the draft; Preview again reads the rules again and previews the same draft against the new identity. An unknown save outcome (a timeout, a 5xx, or a receipt that cannot be verified) can be checked (GET mutations/{mutationId}) or sent again with the same mutation ID and body.
  • Stops. The emergency stop stays an immediate action on the Flags page: the Stops step links there for a stoppable flag and says when a stop is in force. Where a per-project stop would be, the page says "Need it off for just one project? Add an exclusion for that project (saved with a reason). Use Stop for everyone only in an emergency." (D35 A).
  • Consent. With an effective consent other than none, "Choices projects have made" (or "people") shows the joined and left counts from GET features/{featureId}/choices (counts only, D29/D30); the exclusions step notes how many left.
  • Rollout. The rollout is shown read-only, with its pause time when paused (D32 C: only projects that existed then keep getting it). The step links to the W3 rollout page, and "Who gets it" links to "Check a project" (the simulator).
  • Names, never bare IDs (owner feedback on #4519: "Don't just refer to entities in UI by unfriendly Guid"). Rule rows and chips, the summary, the draft bar, the rule editor's chips and sentence, the preview's tables, its proposed save reason, refusals and errors name projects and people; rules still store IDs only. Projects take their names from the targeting read, previews and the project lookup; a project the page knows only by ID (a pasted ID) is looked up by its exact ID through the same lookup (one request per project, deduplicated, at most 50, four at a time). People are named from the Admin Console user list (ListUsers), as name with the e-mail address as secondary text. Where there is no name the page says so in plain words, with the first eight characters of the ID as de-emphasised secondary text for support: "Project no longer exists" (the read or a preview listed it without a name, or the exact-ID lookup found nothing: absent or deleted), "Unknown project" (no name available, for example for an inspector, or the lookup failed), "No longer has an account" (the user list is read and does not have them), "Unknown person" (no ListUsers, or the list is not read). A project owner the lookup could not name is named from the user list, else "unknown person ()". Server text shown in running text has any GUID replaced by the name, else the short ID.
  • Project lookup (owner feedback on #4519). The rule editor's project field searches as you type (after a 300 ms pause, from two characters) and offers up to 20 projects, each on two lines so two similarly named projects are not confused: the name (with an Archived tag for an archived project, offered only for its exact pasted ID), then "Owner · members · stages · studies · Created · …", leaving out any part the server could not give. Choosing one adds it as a chip (for "one person in one project" it fills the project); hovering over or focusing a chip shows the same line, which screen readers also hear. Pasting one or more project IDs and pressing Add still works, and a single pasted ID the lookup finds shows its name and context. The "excluded by another rule" warning is unchanged. The lookup is
GET /api/runtime-feature-flags/targeting/projects?query=<text or project ID>

which answers [{ projectId, name, active, ownerId, ownerName, memberCount, stageCount, studyCount, createdAtUtc }]: a project ID matches that project exactly (archived or not, so a pasted ID resolves), any other text matches active project names case-insensitively as literal text, names starting with it first, then names containing it, at most 20. Deleted projects never match. It is behind RuntimeFeatureTargeting:Enabled (off: a plain 404 before any capability check or read), needs ChangeRuntimeFeaturePolicy (the editor's audience; claim-administrator only where PM authority does not decide), refuses a query shorter than 2 or longer than 100 characters with a 400 before reading, and writes nothing. pmProject has no name index, so a name query scans the collection, bounded by a 5-second server time limit and the 20-result cap. Where each field comes from:

Field Source and cost
memberCount Active members, computed in the same pmProject projection: the schema-1 Memberships (schema-0 Registrations) whose status history ends Active, or with no history whose IsActive is not false, as Project.Memberships decides. No extra read.
stageCount The size of the project's Stages, in the same projection.
createdAtUtc createdAtUtc (6F.F2) when the project has it, else CreationDate (every older project; stored as a UTC instant); null when neither exists.
ownerId, ownerName The project's OwnerId, and the owner's name from one pmInvestigator read by _id for all the matches' owners (first name and surname, else the preferred name), under the same 5-second limit; ownerName is null when the record is missing or has no name, and every ownerName is null when that read fails or times out (logged as a warning; the matches are still returned, #4563); the editor then names the owner from the Admin Console user list, else shows "unknown person ()".
studyCount One aggregation over pmStudy for all the matches ($match on ProjectId $in the result IDs, $group by ProjectId): a covered scan of the ProjectId_1 index, about one index key per study of the listed projects, never one query per project. Same 5-second limit; if it fails or times out, every studyCount is null (logged as a warning) and the matches are still returned. Studies a search deletion moved out carry a holding project ID, so they are not counted. FEAT-024 materialized statistics are not used: they are dark and default-off, so most projects have no row.
- People lookup (owner feedback on #4519: "like the user impersonation lookup, where I can enter an email or a
person's name"). On a Personal feature the people options ("Turn it on for specific people", "Keep it off for specific
people") and the person field of "one person in one project" search as you type, the same way (after a 300 ms pause,
from two characters), and offer up to 20 people as name plus e-mail address. Choosing one adds that person's ID as a chip
(for "one person in one project" it fills the person); pasting one or more person IDs and pressing Add still works.
The chips and the one-sentence summary show the names the lookup knows; the rule itself stores IDs only, never names or
e-mail addresses. A person ID matches that person exactly; other text matches first names, surnames, preferred names,
full names or any of the person's e-mail addresses case-insensitively as literal text, those starting with it first,
then those containing it.

It reuses the search behind Admin Console > User Impersonation: the user list GET /api/investigators (ApplicationListUsersPolicy, decided as ListUsers), which the browser already reads at sign-in for anyone allowed to list users, searched in the browser. No new endpoint. ListUsers and ChangeRuntimeFeaturePolicy both require the application Administrator role (ApplicationAuthorityPolicy), a small trusted group who can already see every user, so everyone who may change targeting can use it. The separate impersonation candidates route (GET /api/impersonation/candidates) needs ImpersonateUsers, the support impersonator role, and is not used. Where the account may not list users (for example while impersonating), the lookup says so and person IDs are pasted as before; a failed read of the list says the search is unavailable. The list carries no deactivated flag, so the lookup can offer a deleted account, as the User Impersonation page does.

Flag decision: not a new flag. The page, its read and the project lookup sit behind RuntimeFeatureTargeting:Enabled (default off everywhere) like every targeting route, and the page itself is administrator-only navigation. The people lookup adds no route: it searches, inside that page, the user list administrators already read.

Preview and check (6F.5). POST /api/runtime-feature-flags/{key}/simulate explains why one subject gets a decision and what a draft would change. It needs InspectRuntimeFeaturePolicy or ChangeRuntimeFeaturePolicy and sits behind RuntimeFeatureTargeting:Enabled: off, it answers a plain 404 before any capability check or read. It is read-only (it grants and writes nothing, and a POST that only reads, so read-only impersonation admits it as it does the policy preview).

POST /api/runtime-feature-flags/screeningKeywordHighlighting/simulate
{ "subject": { "projectId": "<guid>", "personId": null },
  "draft": { "rules": [ …the rules to try; an empty list clears the key… ] } }   // "draft" is optional

The response is { key, grain, subject, current, draft, comparison }. current is the trace under the policy in force and draft (null without a draft) the trace with the draft's rules replacing this key's rules; each trace is { value, reason, decidedByStep, ruleId, steps: [{ step, name, outcome, detail, ruleId?, ruleLabel?, bucket? }] } with the seven steps in the displayed order (requirements, stops, exclusions, allows, rollout, default, baseline) and outcomes pass, off, on, skipped or notApplicable. comparison is { valueChanged, changedStep }: the first step whose outcome or rule differs. The admin "Check a project" panel (6F W3) shows "Now" and "With your draft" side by side. It is the same RuntimeTargetingEvaluator and DecideTargeted path a request uses with its trace switched on, so the trace cannot disagree with the decision.

The project and person are subjects of the simulation, never the caller: the context is marked FeatureContextOrigin.Simulation and the administrator's own identity and memberships are never read. Reads are bounded by grain: a Project key reads the project once, for its creation time (the person is ignored, never read or checked); a Personal key reads the person once and, when one is named, the project once for the membership (suspended or pending deactivation leaves no person; an unreadable project or an inactive membership is not a member).

Unknown IDs are refused (owner decision D59 = B). Only application administrators can simulate, and they can already list every project and person, so the route says when an ID matches nothing instead of answering as for a subject no rule names. After the switch and capability checks, a project the read confirms does not exist or is deleted answers 422 runtime_targeting_refused with reason unknownProject and message "No project with that ID. It does not exist or has been deleted."; on a Personal key a person the read confirms does not exist or whose account was deleted (deactivated) answers reason unknownPerson, "No person with that ID. No account has it, or the account was deleted." (the person is checked first). A suspended person, or one whose deletion is pending, exists and gets the untrusted-person answer. A read that does not answer refuses nothing and keeps the fail-closed answer. Real IDs get the same answer as before.

A draft is checked against the schema rules (shapes, duplicates, limits; 422) but not the save-time checks that need the database or grain (known projects, parent flags, deployment pins): preview a save for those. A Global key, production and a non-Shared host answer 422 globalKey or simulationUnavailable.

Switch off. Stored rules are kept. Allows (and rollouts) stop applying, while exclusions and stops still apply, so switching off never widens access; on again, the rules apply as saved. Decisions are a function of the stored policy and the switch alone, so on, off, on gives the same decisions with policy written in either state. This holds for the route gates and availability below too: with rules stored and the switch off, an exclusion still closes the pilot's route and availability for its project, while an allowed project falls back to the saved default.

Gradual rollout

6F PR F (spec 6F.8, RT-36 and RT-37). A key's one rollout rule turns it on for a stable share of projects (on a Project key; a rollout over people is refused: personalRuleOnProjectGrain on a Project key, notYetSupported on a Personal key, where the population preview over people has not been built):

{ "id": "<new guid>", "kind": "rollout", "label": "Gradual", "subjectType": "project", "group": "screening-keywords",
  "basisPoints": 2500, "state": "Active" }
  • Cohorts. Algorithm syrf-cohort-v1 (RuntimeCohort): the fields ["syrf-cohort-v1", environmentId, decimal(saltVersion), group, subjectType, subjectId], each as a big-endian 4-byte UTF-8 length then the UTF-8 bytes, are hashed with SHA-256; the bucket is the first 8 bytes as a big-endian unsigned integer, modulo 10,000; a project is in the rollout when its bucket is below basisPoints (decision reason Split, and the simulator's rollout step shows the bucket). environmentId comes from the environment binding, so each environment has its own cohort and a restored copy with a new binding epoch keeps it; saltVersion is 1. No random, clock, IP or e-mail input exists, and an architecture test pins that RuntimeCohort references none. Rules in the same group share one queue: a 10% feature's projects are inside a 20% feature's. Raising the amount only adds projects.
  • Vectors. src/charts/syrf-common/cohort-vectors.json (produced once by scripts/cohort-vectors.py, Python standard library) is reproduced by C# (RuntimeCohortVectorTests), TypeScript (scripts/cohort.test.ts, in pnpm run validate:generated) and Python (python3 scripts/test-cohort-vectors.py, in the script-tests workflow). The vectors are append-only: scripts/cohort.test.ts refuses a change to an existing vector against the git base; a new algorithm needs a new ID and a new file.
  • States. Active applies the share. Off (Turn off) removes the rollout's projects (named allows keep theirs) and keeps the amount and group, so Active again restores the identical cohort. Paused holds the amount (the writer refuses a change to its amount, group or subject type, rolloutPaused) and is exact (owner decision D32 = C, spec 6F.F2): only projects that existed when it was paused keep getting it. With the targeting switch off, rollouts (like allows) are not applied; exclusions and stops still are.
  • Exact Pause (6F.F2). When a rollout moves to Paused (from Active, Off or no rollout), the commit stamps pausedAtUtc on the rollout rule with MongoDB's time, read in the commit transaction (the writer's other server-time mechanism; the preview cannot know that time, so its proposed rule shows none). A label change while paused keeps it; a submitted pausedAtUtc is ignored. Making the rollout Active again, or turning it Off, clears it, and the field is written only while paused, so every other rollout keeps its stored shape and digest. While paused, a project inside the held share is in only if its createdAtUtc (reliable creation times) is absent, which every project created before creation times existed is, or earlier than pausedAtUtc; a project created at or after the pause is out (decision reason RolloutPaused, availability rolloutPaused) and resuming reaches it. Outside the share nothing changes. A rollout over people follows the same rule with the Investigator's createdAtUtc, once person rollouts are written. The request context takes the project's creation time from the project read it already makes (the enforced project policy's per-request read where there is one, otherwise one read per request deciding a Project key), and the person's from the read that trusts them; a project read that does not answer counts as created after any pause (fail closed). The simulator's rollout step shows pausedAtUtc and the creation time it compared (subjectCreatedAtUtc). A rollout paused before 6F.F2 has no pause time and keeps the old meaning (a new project inside the held share still joins) until it is resumed and paused again; the preview's rolloutNotes says so.
  • Pause and older binaries. A binary before 6F.F2 refuses pausedAtUtc (schema rule 3), so the writer stamps a pause only while every fresh API and PM host advertises the policy feature rolloutPausedAt on its acknowledgement (policyFeatures); otherwise the pause is refused with schemaUnsupportedByFleet and the preview's schemaAdmission is Blocked. Before rolling an environment back below this release, resume or turn off every paused rollout, or the older binary cannot read the policy. Production never stores targeting, so this concerns non-production environments only.
  • Preview of a change. When a change alters who is in the rollout (an amount raised or lowered, a group changed, turned off or on), the preview reads the project population in its own snapshot transaction: a covered scan of the pmProject _id index (IDs only, at most 100,000, else limitExceeded), each ID's bucket before and after, then _id $in reads that keep only the non-deleted projects whose membership differs. gaining and losing then list exactly the projects whose decision changes (counts, and up to 50 names the administrator can already list); lowering 25% to 5% lists exactly the projects with buckets 500 to 2,499 that no allow or default keeps on. The preview digest covers that list, and a commit recomputes it in its transaction, so the commit goes through only with the digest of a preview that named the same projects (a project created in the lowered range in between makes the preview stale). Pausing at the same amount reads nothing. While the stored or proposed rollout is paused with a pause time, the preview also reads the stored createdAtUtc of every project it decides (the population candidates and the projects the rules name), so resuming lists exactly the projects created during the pause that gain the key, and turning a paused rollout off never lists them as losing it. rolloutNotes says in plain words what the proposed state means (Pause: "only projects that existed when it was paused keep getting it"; Turn off; lowering).
  • Availability. A project outside an Active rollout that the default leaves off reports rolloutPending ("Coming to you"; decision reason RolloutPending); a Paused rollout's share cannot grow, so it is not pending. A project inside a paused rollout's held share but created after the pause reports rolloutPaused (decision reason RolloutPaused). A request with no project never matches a rollout and gets the context-free decision.

Personal and combined targeting

6F PR E (spec 6F.6, RT-33 to RT-35). A Personal key takes three subject types, in one canonical order: pair (a person in one project), person (the person wherever they are) and project (meaning the project's active members). The catalog has no Personal key yet: this is the code path a future reviewed grain: Personal contract uses.

{ "id": "<new guid>", "kind": "allow", "label": "Beta testers",
  "subjects": [ { "type": "person", "id": "<investigator guid>" },
                { "type": "pair", "projectId": "<project guid>", "personId": "<investigator guid>" },
                { "type": "project", "id": "<project guid>" } ] }
  • One person level (owner decision D31 = A). The person is the Investigator ID, which is what the API's user_id claim carries; there is no separate "user" level. Allows match pair, then person, then project (the order only picks the explanation). A pair or project allow needs the person to be an active member of that project, read live; an exclusion matches on its subject alone, so it never excludes less than it says.
  • Who the person is (RT-34). FeatureTargetContextFactory.ClaimedPerson accepts exactly one authenticated identity, exactly one user_id claim that parses to a non-empty GUID, and no machine credential (gty=client-credentials, or an environment:* scope). FeatureTargetSubjects then confirms an active Investigator through the authority snapshot reader: missing, deactivated, suspended or pending deactivation, another ID, an unreadable or malformed record, or a read that throws all leave no confirmed person. Fails closed (owner decision D44 = B): for a Personal key, a request whose person cannot be confirmed (two identities or two user_id claims, no or a malformed claim, a machine token, an unknown, deactivated or suspended Investigator, or a failed read) decides Off with the decision reason PersonUnconfirmed, never on by default: under default On, a person you excluded by name no longer gets the feature just because their identity could not be confirmed. The feature's own route answers the normal unrouted 404, and availability reports it unavailable / notReleased with enabled: false, so the feature is simply hidden (no new wire reason, nothing about the cause is disclosed). A stop still reports Stopped first; the rule wins over allows, rollouts and the default. A signed-in request with no user_id claim at all counts as unconfirmed on a Personal key. Global and Project keys never read a person and decide exactly as before; so does background work (ForOperation), which never carries a person. The rule applies only where the policy stores targeting for the key and only while RuntimeFeatureTargeting:Enabled is on; with the switch off nothing changes. A deployment pin that fixes a key's value still decides as the context-free value does. Under support impersonation, full or read-only, the impersonated person is decided once the real caller passes the same checks (6F.6-4).
  • Membership (RT-35). For a trusted person and an admitted project, one authority read of the project (uncached, linearizable): the membership must exist and be active in a project that is not deleted. Removing a member, or disabling the membership, changes availability (enabled: false) and the gated route on the very next request, with no policy change. The route then answers the project policy's own refusal (a 403 like any non-member's), because the policy no longer admits that person; a member whom no rule reaches gets the unrouted 404.
  • Bounded reads. The person read happens only when a requested key is Personal grain, and the membership read only for a trusted person with an admitted project; Global and Project requests read nothing new, and the reads happen only where the policy stores targeting for the key. The route gate's phase 2, availability (with or without a project), and the simulator all use the same FeatureTargetSubjects, so they trust a person on the same terms.
  • Saving (owner decision D34 = A). Rules hold opaque person IDs only (no names or e-mails). A person the key's stored rules do not already name must be an active Investigator (unknownSubject otherwise). A stored person who later becomes inactive stays in the rules, never matches, and is listed in the preview's inactivePeople for removal; rules are never rewritten automatically, and the person does not block an unrelated change to the key. A rollout over people is notYetSupported; a rollout over projects never fits a Personal key (invalidRule).
  • Preview. On a Personal key, gaining and losing list projects whose active members gain or lose the key (decided for an active member no person or pair rule names); gainingPeople and losingPeople list { personId, projectId } for the people that person rules (projectId null) and pair rules change, IDs only (a person's name needs ListUsers), counts and up to 50 each; inactivePeople lists the proposed rules' people who are no longer active. All three are empty on a Project key.
  • Not yet. A Personal feature cannot be released through a targeted release intent (PersonalAudienceNotSupported). RT-33's "left an offered feature" row is decided by consent (6F PR G): U6's stored Left is OptedOut (availability optedOut), see Consent and environment settings.
  • Switch. As for project rules: off, person, pair and project allows stop applying while person and project exclusions still apply; on, off, on gives the same decisions with rules written in either state.

Tests: RuntimeTargetingPersonalDecisionTests (the RT-33 table through IFeatureDecisions), RuntimePolicyPersonalTargetingMutationTests (saving, unknownSubject, inactive people) in SyRF.Mongo.Common.Tests; FeatureAvailabilityHttpTests.Personal.cs (RT-33, RT-34, RT-35 and impersonation over HTTP) and FeatureTargetPersonTests in SyRF.API.Endpoint.Tests.

6F PR G (spec 6F.7, owner decisions D29 = A and D30 = A). Behind RuntimeFeatureTargeting:Enabled (default off everywhere); no migration, no start-up work, and with no consent configured every decision, gate and availability body is exactly as before.

  • The setting is code's. A registry feature declares consent: none | offered | required (default none). none is an ordinary rollout that cannot be left. offered: the feature is on where targeting reaches the subject ("Turned on for you") and the subject may leave; leaving keeps it off even as the rollout grows. required: listed as "Open to join" and off until the subject joins. KillSwitch, Operational and Permission features, and every Global feature, are always none. A non-none feature needs release content: a non-Draft syrf.topics guide page for one of its topics, whose title is the beta's name (generated into the C# FeatureDefinition.Title only, never the public TypeScript). The registry validator (check 10) refuses each breach with an exact message.
  • Whose choice. A Project feature's choice belongs to the project and is made by its administrator (ProjectEditPolicy); a Personal feature's belongs to the person. The other subject's choice is ignored.
  • Evaluation. required and not joined is a requirement (step 1, reason ConsentRequired); a stored Left is an exclusion (step 3, after exclusion rules, reason OptedOut). Leaving a required feature therefore decides ConsentRequired, not OptedOut, by design: the Beta features page lists people who left a required beta under "Open to join" with Rejoin (availability consentRequired with choice: left). A stop dominates consent: a stopped key reports Stopped (availability stopped) even when consent is required and not given, never "Open to join". A Joined never turns a feature on by itself: it only lifts the required gate where targeting already reaches the subject. The context-free decision of a required feature is off (nobody has joined); the route gate's phase 1 still opens when the default is on, so a project that joined reaches the route. Consent applies with the switch on or off, so switching off can never widen access. A failed choice read keeps features with consent off (ConsentRequired) and makes availability Degraded.
  • "Open to join" only where a Join would count (#4572). A subject that has made no choice on a required feature gets ConsentRequired only where targeting would turn the feature on for it once joined (the decision core decides the later steps as if it had joined). Where targeting does not reach it, the decision is still Off and reports the step that keeps it off: RolloutPending / RolloutPaused for a rollout that has not reached it (availability rolloutPending / rolloutPaused, "Coming soon"), Excluded for an exclusion, and Evaluated (availability notReleased, not listed) when nothing turns it on. So availability never offers a Join that the consent route would not record. For a feature over several keys, availability reports consentRequired only when the capability would be on with the keys held off for consent turned on; otherwise another key's reason. A stop still reports Stopped, and a stored Left still reports ConsentRequired (Rejoin is always recorded). Only the reported reason changed; every value is as before.
  • Environment settings: stricter only. The effective consent is the stricter of code and environment (none < offered < required). An environment tightens with a reason through preview and commit, like targeting (ChangeRuntimeFeaturePolicy, Shared mode, receipt and replay, schema-2 admission):
POST /api/runtime-feature-flags/features/keyword-highlighting/settings/preview
{ "expectedIdentity": { … }, "setting": { "consent": "required", "reason": "Pilot terms not yet agreed" } }
PUT /api/runtime-feature-flags/features/keyword-highlighting/settings
{ "mutationId": "<new guid>", "expectedIdentity": { … }, "setting": { … }, "previewDigest": "<from preview>", "reason": "…" }

A setting not stricter than the code's refuses 422 consentNotStricter (loosening needs a code change); a feature that cannot take consent refuses consentNotAllowed, a Settled or Withdrawn one consentFeatureNotLive (it owns no keys, so the setting would do nothing), one without release content consentNeedsContent, a malformed one invalidSetting. setting: null clears it (back to the code's setting). The setting is stored in the policy document's featureSettings (schema 2). A deployment whose code tightens a setting applies at once with no write; the preview's codeSettingChanged is true when a stored environment setting no longer tightens anything ("code setting changed in this deployment"). Production refuses every setting write. Quieter announcements stay with D13 and 6G. - Joining and leaving. PUT /api/account/features/{featureId}/consent (Personal) and PUT /api/projects/{projectId}/features/{featureId}/consent (Project), body { "choice": "joined" | "left" }. Validation is by format only (a 400 never depends on data); otherwise the answer is always 204, and a row is written only when the caller has the feature's permission (and, for a project, passes ProjectEditPolicy; a reviewer's PUT writes nothing), the effective consent is not none, and, for joined, targeting reaches the subject or a choice already exists. Off, both answer a plain 404 before authorization. - Support impersonation (owner decision on #4523). The account route records a choice on the impersonated person's behalf under full impersonation: the older ("legacy") contract (M2b impersonationReadOnlyEnforcement off, which has no read-only mode, so it counts as full: Join, Leave and Rejoin are allowed and the row and log line carry both identities) or an M2b edit context. Read-only M2b impersonation is refused 403 impersonation-read-only before the body is read, by the M2b contract and again by the route's own RefusesReadOnlySupportImpersonation guard, and writes nothing. The project route refuses every impersonation, full or read-only, with 403 consent_requires_own_session before the body is read: a project's choice changes the project for every member. The other rules are unchanged: joined only where targeting reaches the impersonated person, and the 204 that never says whether anything was written. - Audit of who chose. Every row records the subject (subjectId), the identity the request ran as (actorId: the person, the impersonated person under impersonation, or the project's administrator) and, always written: realActorId (the signed-in person who made the request: under impersonation the administrator, from the M2b ActorInvestigatorId or the principal's own user_id on the legacy contract; otherwise the same as actorId), recordedUnderImpersonation (true only for a choice made under impersonation; a later choice of the person's own sets it back to false) and realActorRecordedAtUtc (MongoDB time, equal to updatedAtUtc in the same write). All three are additive: rows written before them have none and read exactly as before, and a build without them ignores them. A build without them still rewrites a row's choice and updatedAtUtc and leaves the three as they were, so they describe the current choice only while realActorRecordedAtUtc equals updatedAtUtc. No migration and no rewrite of stored rows. Each recorded choice is also logged (Information, FeatureConsentController) with the choice, feature, subject type and ID, ActorId, RealActorId and RecordedUnderImpersonation: IDs only, never a name or an email. Under M2b, the contract's own pmImpersonationAudit change-admitted record for the route is written first, as for every edit-mode change. - Store. runtimeFeatureConsent, _id = "{featureId}:{subjectType}:{subjectId}", read and written by _id only (see MongoDB reference). Reads ignore rows of another binding epoch. A request reads choices once (_id $in), only for features whose effective consent is not none, so Global and none features read nothing (IFeatureDecisions.ConsentPlan). Background work uses RuntimeFeatureConsentReads.ForOperationAsync (the project's choice only); no Project Management consumer of a Project key exists yet. Erasure on account deletion joins the D27 / D34 follow-up. - Availability. An entry for a feature with consent gains consent: { mode, choice, canChange } and title, and the reasons consentRequired (Open to join, only where joining would turn the feature on, #4572) and optedOut (Rejoin); rolloutPending comes from 6F PR F. Absent means no consent. canChange is the caller for a Personal feature (under support impersonation too, on the person's behalf) and a project administrator for a Project one, never under impersonation. A GET cannot tell read-only from edit mode, so canChange says who may choose; read-only impersonation still has the PUT refused, and the web disables the buttons from its own impersonation mode. Permission still comes first. - Counts. GET /api/runtime-feature-flags/features/{featureId}/choices (InspectRuntimeFeaturePolicy) answers { subjectType, joined, left } for the captured binding: counts only, never IDs or names, one anchored _id range.

Tests: RuntimeTargetingConsentTests, RuntimeFeatureConsentStoreTests (including the audit fields and their rollback behaviour) and the RT-33 U6 row in RuntimeTargetingPersonalDecisionTests (SyRF.Mongo.Common.Tests); FeatureAvailabilityHttpTests.Consent.cs, RuntimeTargetingAdministrationIntegrationTests.Consent.cs and the impersonation and catalogue tests (SyRF.API.Endpoint.Tests); registry check 10 in scripts/feature-registry.test.ts.

Moving a beta to general availability

A required step (owner decision on #4523, Leave wording option A). A beta's Leave confirmation tells the person that it "remains off for you until you rejoin or the feature moves from beta testing to generally available". That is true only because a generally available feature takes no consent: with effective consent none, no stored choice is read (IFeatureDecisions.ConsentPlan skips the feature), so a stored Left no longer keeps it off and the feature follows its release like any other. So when a feature with consent becomes generally available:

  1. In the registry PR that sets lifecycle: Released (the registry's "live for everyone" state; see Lifecycle), set consent: none or delete the line. Registry check 10 refuses a Released or Settled feature that keeps consent: offered or consent: required, in pnpm run validate:generated and in generation, with the message consent "<mode>" is not allowed for a <lifecycle> feature; a generally available feature is not a beta, so set consent: none ….
  2. In every environment that tightened it, clear the environment setting (setting: null through the settings preview and commit above), because the effective consent is the stricter of code and environment. The settings route still accepts a setting for a Released feature (only Settled and Withdrawn refuse consentFeatureNotLive), so this step is checked by review, not by the server. Production refuses every setting write, so it never has one.

Stored runtimeFeatureConsent rows are kept (no migration, no clean-up job) and are simply not read while consent is none; they join the D27 / D34 subject-deletion follow-up like every other choice. The Beta features pages stop listing the feature, because an entry carries consent only while effective consent is not none.

The Beta features pages (6F W4)

The consent UI. It reads only the availability client (FeatureAvailabilityService, the single discovery source) and the FeatureGate, and writes only through the generated FeatureConsentHttpService. It lists exactly the entries that carry consent (effective consent not none); with the switch off, or no feature with consent, it renders nothing new.

  • Where. My Account → Beta features (/manage/beta-features) for Personal betas, and a "Beta features" section on a project's Settings → General page for Project betas. The account link appears only while the person's answer lists a beta (or while the page is open); the project section only for a project administrator (edit in the permission report, the same ProjectEditPolicy the PUT checks) and only while the project's answer lists a Project-grain beta.
  • How an entry is grouped (shared/beta-features/beta-features.model.ts, classifyBeta), in this order: stopped shows Stopped (never "Open to join"), under "Using now" for a subject that is signed up (joined, or an offered beta not left), which can still leave, otherwise under "Coming soon" with no Join; a left choice (or optedOut) is "Open to join" with Rejoin, for both modes; consentRequired is "Open to join" with Join; rolloutPending and rolloutPaused are "Coming soon"; on (the gate) is "Using now" ("Turned on for you" for offered, "You're in" for required); off for another reason after joining (for example a person SyRF cannot confirm, D44 = B, reported as notReleased) is "Using now", Off, and can be left. Anything else (excluded, unreleased, blocked by a prerequisite, with no choice) is not listed.
  • Join and Leave. The page sends { choice: "joined" | "left" }, then calls FeatureAvailabilityService.reread() (which resolves once that fetch has settled) and reports what the new answer says: "You joined …", "Nothing changed for …" when the server wrote nothing (the PUT always answers 204), or, for a Leave, the owner's wording (option A on #4523): "You left the '' beta. It remains off for you until you rejoin or the feature moves from beta testing to generally available." (the project section: "This project left the '' beta. It remains off for everyone in the project until you rejoin or the feature moves from beta testing to generally available."). The second half is true because general availability drops consent (Moving a beta to general availability). The page records one choice at a time, so while one beta's choice is saved and re-read, Join and Leave are disabled on every card (the saving card's button says "Saving…"), never only on the saving card.
  • One "on" rule. Whether a beta is on for this context is betaIsOn (beta-features.model.ts): the FeatureGate for a single-flag feature (the answer's enabled, else the runtime snapshot flag), otherwise enabled, falling back to availability. The page, the project section's visibility and the My Account link (hasListedBetas) all use it, so the link and section never appear for a page that would list nothing, or hide while it would list a beta.
  • Project confirmation. In the project section every Join, Rejoin and Leave first opens a Material dialog (ProjectBetaConfirmDialogComponent, M3): the title names the beta ("Leave "…" for the whole project?"), the text says the change affects the entire project and every member and may change how the project behaves for them, and the buttons are Cancel and "Join / Rejoin / Leave for the whole project". Only the action button sends the PUT; Cancel, Escape or a backdrop click send nothing.
  • Impersonation (the tab's SupportImpersonationService state). On the account page, full impersonation (not-enforced, the M2b contract off, or edit; the older ("legacy") impersonation has no read-only mode and counts as full, an owner decision on #4523) keeps Join, Leave and Rejoin enabled: each button and each outcome says "(on behalf of )", with the name the impersonation banner shows ("the impersonated user" when the app has none), and the notice says the choice is made on that person's behalf and recorded as made by you while impersonating them. Read-only (read-only, and every state that is not clearly full: checking, entering or leaving edit mode, refused, unavailable) keeps them disabled, with a notice saying why and that edit mode in the banner turns them on. The project section stays disabled under any impersonation, with a short note. A 403 impersonation-read-only or consent_requires_own_session shows the matching notice. consent.canChange: false also disables them.
  • Notices. "New to you" uses the 6E/6G rule (announcement delivered and not seen), and a beta with a runtime What's New entry embeds the 6G syrf-whats-new-entry row, which records seen and hide.
  • Search and filters (account page): search matches the beta's name and its What's New text; the filter chips show one group at a time with counts.

Specs: beta-features.model.spec.ts, manage/beta-features/beta-features.component.spec.ts, project-betas.component.spec.ts (including the confirmation), manage.component.spec.ts (the link) and the reread cases in feature-availability.service.spec.ts. User guide: user-guide/account/beta-features.md.

Targeted route gates and availability

6F PR C (spec 6F.3 and 6F.4). Both act only where the policy stores targeting for the key a request needs (IFeatureDecisions.HasStoredTargeting: RuntimeFeatureFlags:Mode=Shared, a Project or Personal contract for this host, outside production, a fresh policy, and at least one stored rule), with RuntimeFeatureTargeting:Enabled on or off. With the switch on, allows apply; off, only exclusions and stops do. With no stored targeting (always the case until the switch has been on, because rules can only be written then), and in Legacy or Shadow mode or production, every gate and availability answer is byte-identical to main before PR C, the pilot included (pinned by a golden recorded against main's code, FeatureAvailability/Goldens/feature-targeting-switch-off.golden.txt; the stored-rules, switch-off answers are feature-targeting-switch-off-stored-rules.golden.txt).

  • Two-phase route gate. [FeatureGated] on a Global key is unchanged: one decision during action selection, before authorization, so an off route is a plain 404 for everyone. On a Project or Personal key (today the pilot's PUT api/projects/{projectId}/screening-keywords), phase 1 asks only MayBeOnForSomeTarget(key): the context-free value, or, where targeting is in force, whether requirements pass, no stop holds the key and an enabled allow (or an Active/Paused rollout) the grain allows exists; with the switch off it is exactly the context-free value. Nothing about any project is decided before authorization. Phase 2, FeatureTargetGateFilter (an action filter, ordered before [ApiController] model validation), acts when the key has stored targeting: it decides the key for the routed projectId that the project policy admitted and answers exactly an unrouted request's 404 (no body, no content type) when it is off, so a member of an excluded or untargeted project sees the route as absent. An absent or deleted project, and a caller the policy refuses, get the policy's own answer before phase 2 runs, exactly as any route of that policy; a non-member therefore gets the same refusal in a pilot project and any other, and learns nothing about the pilot. A test keeps every targeted gate on a {projectId} route behind a project policy.
  • Context. FeatureTargetContextFactory is the one place a request builds a FeatureTargetContext: the project only from the admitted route value (never a body or query), no target at all for a Global-only request, and no person or membership read for a Project key (every member gets one decision). A Personal key reads the trusted person and their membership (Personal and combined targeting), and a feature with consent its subject's stored choice (Consent and environment settings). Background or message work names its project with FeatureTargetContext.ForOperation(projectId), which never carries a person.
  • Project-aware availability. GET api/projects/{projectId}/features/availability decides Project and Personal features for that project in the same single DecideAll. Each entry gains enabled (the capability is on for this caller here and nothing holds it off), the browser's gate for a non-Global feature, separate from availability, which still needs a confirmed release: a pilot creates none, so a Developing pilot is enabled: true but unavailable/notReleased in its project. New reasons: excluded, prerequisiteMissing and (PR F) rolloutPending (an Active rollout has not reached this project yet: "Coming soon"). The route without a project omits Project-grain features (D25 A). The contextId domain is syrf-feature-availability-v2 and covers enabled. This applies whenever a permitted Project or Personal feature's key has stored targeting, switch on or off (so the response shape does not change when the switch is turned off); with no stored targeting, enabled is absent and the domain stays v1.

The policy document (runtimeFeatureFlagOverrides, _id: "global") has two schemas:

  • Schema 1 is today's document, byte-for-byte, with the syrf-runtime-global-policy-v1 digest. It is still written whenever there is no targeting and no feature setting, so removing every rule returns the document to a shape every older binary reads.
  • Schema 2 adds targeting (per catalog key, at most 20 rules: exclude and allow rules with enabled, a label and 1–200 project, person or pair subjects, and at most one rollout with subjectType, group, basisPoints and state Active|Paused|Off) and featureSettings (per registry feature, { consent, reason }). It needs at least one of them non-empty, allows at most 2,000 subjects, and digests under syrf-runtime-global-policy-v2 in one canonical order (keys ordinal, rules by kind then ID, subjects by type then IDs), so array order never changes the digest. Subject IDs are lowercase GUIDs only. RuntimePolicyAuthority.Parse refuses anything else with an exact message (RuntimeTargeting.Messages); a schema-1 document keeps its old messages.

Decisions follow one order, RuntimeTargetingEvaluator (displayed order = evaluation order): requirements (authority state, deployment configuration, a required parent off for the same subject), stops, exclusions, specific allows (pair, then person, then project), gradual rollout, the saved setting, the baseline. Steps before the allows can only keep a feature off; allows and rollouts can only turn it on. Grain is checked when deciding, not when reading: a Project key reads project subjects only (every member of a project gets one decision), a Personal key reads pair, person and project subjects (a project or pair allow means an active member), and a rule the grain forbids is ignored with reason TargetingIncompatible, so a catalog change never takes the policy down. Global keys ignore targeting and decide exactly as before (pinned against a frozen copy of main's results).

IFeatureDecisions.DecideAll(keys, in FeatureTargetContext) decides for one subject (ProjectId, the trusted Investigator as PersonId, PersonIsActiveMember); the older Decide and DecideAll(keys) use the empty context. Request contexts come from FeatureTargetContextFactory (above; the simulator builds a Simulation context for its named subject with the same FeatureTargetSubjects reads), and allows and rollouts apply only while the switch is on. A key whose value the deployment fixes (a pin forcing it on or off) decides exactly as the context-free values do on the targeted path too, so stored rules never override a deployment pin (#4469). The preview's gains and losses use the same deployment-aware decision (RuntimeTargetingDeployment.Decide). Consent and environment feature settings follow owner decisions D29 and D30 (Consent and environment settings), per-rule switches follow D28 (6F PR H), and the single "person" level follows owner decision D31 = A.

Compatibility: hosts advertise policySchemaVersions; a deployment constraint may allow policy schema 1 or 2 and refuses a schema-2 policy it does not allow; keys with an enabled allow or an Active or Paused rollout join the required-enabled set; the compatibility report gives the document's policy.schemaVersion and policy.targetedKeys. The anonymous browser snapshot keeps schemaVersion: 1 and never carries targeting (a targeted key shows its context-free decision). An older binary given a schema-2 document reports the authority Unavailable and serves safe values, which is why the writer refuses a first schema-2 write until every fresh host advertises 2.

MVP rehearsal (targeting-mvp scenario)

The runtime-fleet harness's targeting-mvp scenario (bash e2e/run-local.sh --runtime-fleet --scenario targeting-mvp, local stack only, run on Bramble; never part of all) walks the 6F MVP boundary through two API and two Project Management processes in Mode=Shared. It creates projects P and Q through the PM e2e setup API (the admin creates both; the reviewer is made a project Administrator of both; the standard user belongs to neither), then: the targeting routes answer 404 while the switch is unset; with RuntimeFeatureTargeting:Enabled=true on all four hosts (profile targeting, which also turns on RuntimeFeatureReleases:Enabled on the APIs because project availability sits behind it) it previews and saves an allow for P, checks the keyword route and availability for both members and the non-member on both APIs, runs the simulator for the policy in force and drafts, applies an exclusion with the default On, clears every rule (schema 1 again) and restarts all four hosts with the switch off (profile targeting-off) and on again. Fleet adoption is read from the harness key's fleet view: fleet readiness compares policy identities only, and every acknowledgement carries the one policy document's revision. When #4487 added the scenario the pilot's own /fleet always read Unknown (DomainReadinessNotEstablished); since 6F PR I it reaches Active too (see Confirming activation). The run evidence for #4487 is kept with the programme's handover notes.

Non-production bound consumer mode

PR #4352 connects RuntimeFeatureFlags:Mode=Shared to API membership-disable admission and API/PM statistics serving plus the existing pages, Stage Overview and SignalR selectors. It requires matching non-production environment identities and the three pinned binding expectations above. SharedReaderMode must be Disabled; legacy-format reader enrollment cannot be combined with the bound authority. No defaults, Helm deployment values or real environments are changed by this PR.

Every audited decision reads the current immutable authority, expires at 30 seconds, and refuses admission on the first failed read. The API disable route becomes 404 while stopped/unavailable, while the enable route retains its permission checks and can restore an already-disabled member. API/PM statistics stop materialized serving/selectors; maintenance, family gates, pilot allowlists and durable lifecycle controls remain separate. Accepted baseline and overrides survive on/stop/restore. The PM selectors use the same normalized evaluator as the API instead of losing their existing runtime overrides during cutover.

The API refresh worker reads a bounded compatibility projection of initialized authority rather than the legacy store. GET /api/runtime-feature-flags exposes that projection as read-only; no current authority returns 503. Its legacy numeric revision contract is unchanged in this slice: lossless browser identity and reconnect handling remain required follow-ons. Authenticated authority status/evaluation already expose full identity with decimal-string revisions. The legacy PUT/group API refuses writes in Shared mode even for administrators; the identity-aware permission/preview/receipt write path is still pending.

Only the audited keys bypass the API compatibility cache on each decision. Other API flag consumers keep their existing cached behavior and are not claimed as bounded runtime consumers. PM maintenance and non-enrolled selectors keep their deployment inputs. Legacy and Shadow modes keep their existing consumer paths. Shared mode is a per-host admission option, not a fleet readiness assertion: no Active status, production enrollment, coordinator clearance or rollout approval is implied. Host acknowledgements and the static expected inventory now exist (see the fleet section below); production-grade fleet admission, compatibility reporting and the production gate remain required.

Browser bootstrap and connection recovery

PR #4353 hands the exact successfully applied pre-bootstrap snapshot to RuntimeFeatureFlagsService. The service seeds its revision before subscribing to background fetches, so an older response from another API pod cannot overwrite the snapshot that selected the application's initial configuration. A successful bootstrap also resolves the initial-snapshot wait without requiring a second request. If bootstrap failed, the existing finite retry/fallback behavior remains.

The service refetches on successful first SignalR connection, explicit or automatic reconnection, browser online return and window focus return. The first-connection event is separate from the existing reconnect project-reload event; it does not force a project reload. Revision hints and the five-minute independent poll remain active, including when SignalR is disabled. A newer refresh trigger cancels the previous client fetch, and destruction removes the window listeners and subscriptions.

This repairs global snapshot recovery without changing permissions, saved review work, feature defaults or lifecycle teardown. It does not complete the browser contract: lossless revision transport, contextual identity/digest checks, authenticated/project-specific envelopes, and desired-versus-active lifecycle reporting remain required in later slices. No production or environment activation accompanies this change.

Browser safe fallback (browserRuntimeSafeFallback, 6B.4 G3)

If the bootstrap request for the runtime snapshot fails, the browser keeps the deployed values until a later fetch succeeds. Under an emergency stop that is permissive: E2E deploys disableMembership On, so a tab whose bootstrap failed shows "Disable membership" although the control is stopped. The server still refuses the action, so this is about visibility, not security (RT-14, browser half).

The deployment setting browserRuntimeSafeFallback changes that. When it is true and the bootstrap request fails (a network error, a non-OK status or an invalid snapshot), main.ts calls applyRuntimeFeatureFlagBootstrapFailure(). That applies RUNTIME_SAFE_VALUES before the app starts, so the store is seeded with them. The values hold until the first accepted snapshot, which then replaces them with the server state. A key the snapshot omits falls back to the deployed value, because the deployed values are captured before the safe values are applied (by design, but not covered by a test; the server always returns every catalogued key). RuntimeFeatureFlagsService.safeFallbackActive() is true until then, including while a rejected snapshot leaves the safe values in force. It is a diagnostic and test hook, not a UI contract: nothing renders it. A failed fetch after the first accepted snapshot keeps that snapshot, as before. Nothing is stored, so each page load decides again. The console warning says which fallback was used: using safe values … or using deployed defaults.

  • RUNTIME_SAFE_VALUES (src/app/generated/runtime-safe-values.generated.ts) is written by generate:runtime-catalog. It lists the safe value of every key that has a reviewed consumerContract with productionEligible: true, in catalog order. Today that is disableMembership: false. Statistics contracts are non-production and are not listed. generate-runtime-flag-catalog.test.ts pins the list.
  • The setting is SYRF__BrowserRuntimeSafeFallback in the webConfig section of env-mapping.yaml. The Helm value is runtimeFeatureFlags.browserSafeFallback in the web chart, default false, and the browser reads browserRuntimeSafeFallback from appConfig.env.json (AppSettings; only the string "true" turns it on). It is not a runtime catalog entry: it has no override, is not in the catalog digest and does not need catalogOnlyFlags. It reaches only the browser.

Flag decision: flagged, by this deployment-static setting, false in every environment until a reviewed rollout. It changes what every user sees when the bootstrap fails, so it needs a switch. A runtime flag cannot be that switch, because it would be read before any snapshot exists. Off is exactly the old behaviour. Nothing is persisted, so on, off and on again are all safe; runtime-feature-flags.service.spec.ts tests that sequence. The E2E case is runtime-fleet-browser-recovery.spec.ts on the runtime-fleet lane, which sets the setting per tab through an override of its E2E runtime overlay (appConfig.e2e.local.json).

Operation-specific administration capabilities

PR #4354 separates runtime administration at the existing application-authority gate. Provider status, authority status and authority evaluation request InspectRuntimeFeaturePolicy; individual and group policy mutations request ChangeRuntimeFeaturePolicy before reading policy or validating keys/bodies. EmergencyStopRuntimeFeatures and ManageFeatureReleases are distinct registered capabilities for the remaining stop and release routes. All four initially require the PM Administrator role (OD-2).

PR #4375 advances the policy version to application-authority.runtime-operations.2 and adds the Administrator-only BreakGlassRuntimeDeploymentConstraint capability. Existing capability ordinals and the legacy ManageRuntimeFeatureFlags name are preserved for older clients; the old aggregate is not a fallback grant for these routes. Self-capabilities includes the new names without changing its string DTO schema. Off and Shadow application-authority modes retain their existing claim behavior; Enforced uses current PM authority, denies on unavailability, and authorizes the impersonated target rather than the real actor. Successful mutations retain both identities in the audit record. Break-glass recovery is the explicit exception: impersonation is refused before capability evaluation.

Flag decision: use the existing application-authority rollout modes; no new switch or role migration is needed for an operation split whose initial role mapping is unchanged. Production mutation refusal, Shared-mode legacy-write refusal, runtime defaults and saved review data are unchanged. Public bound preview/commit, stop routes, release management and browser presentation of separate capabilities remain follow-up integration work; registering a capability does not expose a new operation or activate a feature.

Bound emergency-stop API

PR #4355 adds POST /api/runtime-feature-flags/{key}/emergency-stop for explicit non-production Shared API mode. It requires EmergencyStopRuntimeFeatures before policy access. Send mutationId, reason and expectedIdentity copied from the authority observation, including environment, binding/store epochs, catalog/baseline/policy digests and the decimal-string revision. The writer accepts only an audited global safe-false consumer; it preserves saved baseline and override values and writes the stop and immutable audit receipt atomically. Legacy and Shadow modes cannot write through this route.

A successful response contains mutationId and the committed identity; it means durably committed, not Active or fleet-converged. Consumers observe the stop through their existing bounded refresh path. A 409 runtime_policy_revision_conflict includes a decimal-string actualRevision: inspect the new identity before deciding on a new mutation. A 503 runtime_policy_commit_unknown does not prove failure: retry the same mutation ID and unchanged request, including reason and expected identity. An exact replay returns its original receipt even after another administrator restores the policy, without reapplying the old stop. Actor identity is part of idempotency; another actor cannot claim that receipt. A new mutation ID with the current expected identity records another revision even when the key is already stopped; idempotency is per mutation ID, not per key. The request deadline is ten seconds; a caller that loses the response uses the same retry procedure.

Emergency stops bypass the deployment-constraint and new-audit-index gates as specified in the plan, but never bypass binding, lineage, receipt integrity or authorization. The API has no content or notification dependency and does not wait for a fleet acknowledgement. Restore remains the shared writer's tested ReleaseStop operation pending the reviewed public preview/commit workflow; this slice does not expose a restore route or enroll an environment. Production remains refused.

Reader corrections accompanying the route count time spent reading toward freshness, and treat malformed BSON/argument failures as unavailable so they cannot escape an observation refresh. The catalog digest includes descriptions and definitions: changing either invalidates an old imported authority. Until the catalog-evolution workflow exists, do not enroll or mutate against a changed catalog; regenerate/review an import preview before first enrollment. Never rewrite a stored digest to bypass validation.

Bound preview, commit and restore

PR #4356 adds POST /api/runtime-feature-flags/{key}/preview and PUT /api/runtime-feature-flags/{key}/policy in explicit non-production Shared API mode. Both require ChangeRuntimeFeaturePolicy before policy access. The additive /policy route has a typed, lossless commit receipt; the legacy PUT keeps its existing response contract and remains refused in Shared mode.

Send mutationId, the complete decimal-revision expectedIdentity, nullable value, reason, optional confirmedDependencies, and action (Change or ReleaseStop). Preview is read-only, including under support impersonation: it writes no policy, receipt, index or lease. Dependency confirmation follows the existing typed 422 response; submit the exact confirmed key/value set before accepting the final preview. The response includes the proposed identity, saved override/baseline changes, before/proposed effective values, requested/dependency/induced-pin causes, and previewDigest. FleetReadiness is explicitly Unknown until expected hosts and acknowledgements are integrated. A preview is not a rollout approval.

Commit the same request plus previewDigest. The API requires that digest even locally. The store recomputes it inside the same transaction that writes policy and receipt; it includes the request and actor, exact authority identity, proposed policy, captured deployment flags and deployment-constraint guard. Record, extend and release each increment that guard and invalidate outstanding previews, even when the policy identity itself is unchanged. Changed requests or deployment inputs require a new preview; changed policy identity returns a revision conflict. Staging also requires a preview at the shared-store boundary. Emergency stops retain their independent capability/path and do not require preview, content, fleet or deployment-constraint approval. A Change-policy request cannot smuggle an EmergencyStop action.

To restore, preview and commit ReleaseStop with null value and no dependency changes. Saved values are preserved; effective values still obey deployed prerequisites. A no-op accepted under a new mutation ID intentionally advances revision and audit; exact replay uses the original receipt. If a commit returns 503 or its response is lost, retry the same ID and unchanged request, including digest and actor context. A missing receipt is not proof of abort while another host may still be committing. No environment is enrolled by this change. Public UI integration, expected-host inventory, coordinator, production admission, targeting and release/content integration remain required.

Resolving a lost mutation response

In explicit non-production Shared mode, an administrator with ChangeRuntimeFeaturePolicy can call GET /api/runtime-feature-flags/mutations/{mutationId}. Committed returns the original committedIdentity with its decimal-string revision, even after later changes or a stop restoration. It confirms the durable receipt, not current policy or fleet adoption. Unknown has no committed identity and does not prove an abort: retry the same mutation ID and unchanged original request. Do not mint a replacement ID merely because lookup is Unknown. Storage failures return 503 instead of an outcome. Outcome responses use Cache-Control: no-store so a previous Unknown result cannot be reused by a browser.

The lookup takes a read-only snapshot of the verified binding, current authority and receipts. It refuses foreign lineages, corrupt or future receipts, and missing current-authority evidence. It neither installs indexes nor runs migrations and remains available in the presence of a deployment constraint. Lineage-import IDs are not policy mutation IDs. Legacy and Shadow modes refuse this route. This adds no environment enrollment or production activation; the default mode remains Legacy.

Host acknowledgements and fleet inspection

In explicit non-production Shared mode, API and Project Management publish runtimePolicyHostAck after startup reconciliation and each jittered 8–12 second poll. Reads retain the five-second deadline; an acknowledgement write has a separate two-second deadline and cannot invalidate a valid local decision. Each acknowledgement includes pod UID, process start identity/time, image version, catalog and full policy identity, last successful authority check, state/error class, polling mode and last-seen time, and (since 6F PR A) policySchemaVersions, the policy document schemas the binary reads ([1, 2]; a row without it means [1]). Only checks younger than 30 seconds qualify as current. No user or project identifiers are included.

Publishing requires RuntimeFeatureFlags:InstanceUid (the pod UID, unique in the environment), ImageVersion, positive ExpectedReplicas for that service, and explicit Autoscaled=false for a fixed fleet. These are deployment metadata, not feature toggles. Missing UID/image/count produces no acknowledgement; missing or invalid autoscaling metadata is treated as autoscaled. Migration 0011 must already have installed the {service, lastSeenAtUtc} index. Publishing never creates indexes or runs migrations. Legacy and Shadow never publish. A late write from an older process cannot replace a restarted process's record for the same UID. The migration forward-rule guard explicitly enrolls this writer; its Mongo test pins the UID, service and BSON date fields, idempotent replacement and absence of TTL. Retained reports have corresponding shape coverage for their environment, collection timestamp and TTL expiry fields, and workload observations pin their service ID and BSON observation date with no TTL. Collections without an enrolled writer remain catalogue-only.

GET /api/runtime-feature-flags/{key}/fleet requires InspectRuntimeFeaturePolicy, Shared mode and a verified environment binding. Responses are not cached. The API needs explicit RuntimeFeatureFlags:FleetExpectedReplicas:api and RuntimeFeatureFlags:FleetExpectedReplicas:project-management counts matching each service's rendered ExpectedReplicas; no count is inferred from observed hosts. For a synthetic two-API/two-PM exercise, both expected counts are 2 and each process has its own UID. Missing or invalid expected inventory and any autoscaled host remain Unknown. A host's count disagreeing with inventory requires action. Exact count, compatible full identity and fresh evidence from every host are required for Active. Active means adoption of this policy, not that its value is On or that a user is authorized. With incomplete acknowledgements the state is Saved/Applying during the 60-second confirmation window, then Degraded. Per-host binding/catalog incompatibility is always reported; the aggregate prioritizes Unknown when inventory is missing or autoscaled, otherwise incompatibility requires action immediately.

This slice can confirm the audited immediate membership control. Statistics controls remain Unknown until separate domain-readiness evidence is integrated. Rows have no TTL; missing or silent hosts never disappear from the evidence. Extra old pod UIDs also prevent Active. Trusted workload collection and explicit retirement of old UIDs, chart rendering of metadata/counts, production admission, the admin UI, metrics and the multi-replica 200-commit freshness exercise remain required before real enrollment. These trial timing bounds are not a measured production guarantee. The timestamp comparisons assume synchronized UTC clocks across the database and hosts; clock skew can make evidence Unknown. The 200-commit trial must measure skew and the worst-case poll/read/write margin before live enrollment. No live configuration is changed by this PR.

Rendering host metadata

The API and PM charts can set runtimePolicyHostMetadata.enabled: true to derive each host's four acknowledgement metadata values from its own Deployment. The default is false and adds no variables. UID comes from metadata.uid (not pod name), image version is the exact container image including a debug suffix, expected replicas comes from replicaCount, and autoscaling comes from hpa.enabled. Raw env: entries cannot override these observations when enabled, including case/colon aliases. The chart does not select Shared mode or supply binding, migrations, or cross-service FleetExpectedReplicas values.

HPA remains explicitly autoscaled/Unknown. Scale-to-zero is allowed and renders zero, which does not publish a positive-count acknowledgement. Knative opt-in is refused pending trusted workload inventory. This is an opt-in deployment-metadata setting, not an additional runtime policy switch; the existing Legacy mode remains the default. No environment values or live deployments are changed. Trusted inventory must still reconcile cross-service desired counts and retire old pod UIDs before real enrollment.

Administrator deployment constraint recovery

PR #4375 exposes POST /api/environment/deployment-constraints/{rolloutId}/break-glass-release for a stranded coordinator. This is a separate BreakGlassRuntimeDeploymentConstraint application capability, assigned to Administrator; Change, Emergency stop and Manage releases are not fallback grants. Application authority off/shadow uses the existing administrator claim decision; enforced mode uses current PM authority and answers 503 when that authority is unavailable. A single authenticated human identity is required; automation scopes, multiple authenticated identities and support impersonation are refused.

The request contains mutationId, the full expectedIdentity (decimal-string revision), a nonempty reason (maximum 1,000 characters), and the reviewed expectedConstraintDigest. The bound store checks the exact policy identity, active rollout and constraint digest, releases the constraint and writes the before-state, real actor, reason and receipt in one majority-acknowledged transaction. Recovery does not assert rollout completion, change feature policy, or enable a feature. It deliberately bypasses the owning coordinator and completion-evidence requirements; it cannot cross an environment binding or authority lineage.

The flag decision is the existing deployment-only authority mode: this route is available only in explicit non-production Shared API mode. Legacy and Shadow refuse it and retain all constraint/audit data. It remains usable if the machine coordinator endpoint is disabled. No additional toggle or migration is needed for this recovery path. Production Shared mode is still refused.

Requests are limited to 16 KiB and ten seconds; responses are private/no-store. A 409 requires inspection of current policy/constraint state. A 503 may mean a committed operation: retry the same mutation ID and unchanged request, including actor context. Exact replay returns the original timestamp without releasing a subsequent constraint. Do not substitute a new mutation ID merely because the response was lost.

The audit record is durable. Surfacing the last break-glass event in the fleet view and the administrator recovery UI remain ordered follow-ups; this slice provides the authenticated recovery operation only. Tests use synthetic replica-set data; no live constraint or production environment was changed.

Lossless browser snapshot ordering

PR #4376 adds revisionText, the invariant decimal representation of the snapshot's Int64 revision. The numeric revision remains for existing clients. Bootstrap and the browser propagation service prefer revisionText, validate a canonical nonnegative Int64, and compare using exact integers. A present malformed string is refused even if the numeric field looks valid. Older servers remain readable only when their numeric revision is a nonnegative JavaScript safe integer. Unsafe legacy numbers are never rounded into an ordering decision.

An invalid bootstrap does not change deployed values. An invalid later response retains the last accepted snapshot, releases the initial-navigation wait and leaves future polls/reconnect fetches active. Adjacent revisions above 2^53 cannot overwrite each other in reverse order. SignalR remains an invalidation hint; its numeric payload is not used for ordering. No extra deployment flag is needed for this additive wire correction; runtime modes and feature decisions are unchanged, and no saved data is migrated. Full lineage/context identity, staleness UX and lossless administrator mutation requests remain follow-ups.

Production authority admission (default off)

PR #4377 introduces deployment-only RuntimeFeatureFlags:ProductionPolicyMode=Legacy|Runtime, defaulting to Legacy. In production, Legacy resolves the authority provider to Legacy even when the raw RuntimeFeatureFlags:Mode remains Shared. Returning to Runtime requires Shared mode, matching production environment identities and explicit verified binding. The mode changes no saved policy, baseline, receipt, audit or stop data. This supersedes the earlier blanket production Shared refusal; it does not enroll a deployment or authorize activation.

Only reviewed production-eligible, global, false-safe contracts may use runtime decisions or mutations. Currently this is disableMembership. Other flags retain normalized deployment values even if saved runtime overrides exist or authority refresh fails. Statistics serving and maintenance remain deployment controlled in production.

Ordinary enabling changes and stop releases that restore On require an exact preview digest and fresh acknowledgements of the current policy from the configured fixed API/PM fleet, rechecked inside the mutation transaction. Missing, stale, autoscaled or extra historical hosts refuse admission. Admission follows RuntimeFeatureFlags:FleetInventoryMode, the same operator-supplied setting as fleet inspection, which defaults to Static. Static admission counts every retained row, so it refuses after any Kubernetes rollout until the old rows are removed. With Collector, admission reads the workload observations and MongoDB's time in the same transaction and evaluates exactly as the inspect view does. Absent rows that MongoDB has not seen for 35 seconds (15 minutes for legacy rows) are retired, so an enable or restore is admitted about 35 seconds after a rollout's old pods stop. A live absent row, or missing or stale collector evidence, still refuses. An unsupported mode value refuses. A passing preview does not establish adoption of the candidate revision. Emergency stop remains independent of fleet readiness, in either mode, and preserves saved overrides.

The code slice is tested with production-like binding identities in an isolated replica-set fixture. Production activation remains blocked on the B/C prerequisites: complete compatibility evidence, promotion preflight parity checks for returning to Legacy, recovery exercise, freshness trial and explicit operator approval. Pods start in Legacy without consulting authority parity; the promotion preflight must prevent an unsafe mode rollback. No chart, live binding, credentials or production state is changed here.

Browser lineage guard

PR #4378 adds an optional authority envelope to global Shared snapshots: schema version 1 and the verified environment, binding/store epochs, exact decimal revision and catalog/baseline/policy digests. Legacy snapshots carry no authority. No targeting rules or user/project identity are included.

The browser pins the first accepted provider mode and lineage for the page lifetime. A changed environment or epoch, lost authority metadata, unsupported schema or conflicting equal revision stops further snapshot application and exposes snapshotFault on the service. Current values and unsaved work are retained; no forced reload occurs. Lower revisions are ignored; equal identities with identical values are idempotent. Equal policy identities with different effective values also fault, covering deployment fallback drift. A new page can bootstrap only from the server's verified reader; reconnect alone never acknowledges a new lineage. Legacy-only revision behavior is preserved. SHA-256 digests accept the server's uppercase hexadecimal representation and lowercase hex; case alone does not change a snapshot identity.

This is a global transport guard, not personalized authorization or a client security boundary. Server gates remain authoritative. Personalized request generations/context hashes and a user-facing recovery notice remain follow-ups; the service exposes the fault for that notice without destroying review state. No separate deployment flag is needed: authority metadata exists only in explicit Shared mode, and Legacy remains the default. No policy data, migrations or environment configuration change.

Feature registry (schema 2)

The generator also writes the feature inventory (docs/features/feature-inventory.md): counts by type and lifecycle, the complexity metrics (flags, evaluation seams, generators, commands and hand-edited files to add a flag, all computed from the sources), the features table, unclaimed flags and removed flags. It has no dates and no runtime state; "Ready to settle" is shown in each environment's admin page from its own history. Its diff in a PR is the reviewable complexity effect.

src/charts/syrf-common/feature-definitions.yaml is the one reviewed source for product features, the preset flag groups and the deployment-managed pins. PR #4379 introduced it (schema 1); R1b of wave 6R (#4432) made it schema 2. pnpm feature:new adds a feature (see Adding a New Feature Flag). After a hand edit, run pnpm generate (or only generate:feature-definitions) and then pnpm run validate:generated. The generator validates the registry against the runtime flag catalog and writes:

Output What reads it
SyRF.Mongo.Common/RuntimeFeatureReleases/FeatureDefinitions.generated.cs FeatureDefinitionCatalog.All (release preparation, 6D)
src/services/web/src/app/generated/feature-definitions.generated.ts Read-only metadata (no web consumer yet)
SyRF.Mongo.Common/RuntimeFeatureFlags/RuntimeFeatureFlagGroups.generated.cs RuntimeFeatureFlagGroups.All and Find
SyRF.Mongo.Common/RuntimeFeatureFlags/RuntimeFeatureFlagPins.generated.cs ProjectStatisticsRuntimeActivation (pinned keys)
src/services/web/src/assets/data/whats-new.json The home-page What's New feed (from docs/content/whats-new.yaml; see Update the What's New feed)

CI checks source and output checksums and rejects omitted generators.

Feature fields

Field Required Meaning
featureId Yes Stable, permanent ID (lower-case words joined by hyphens); never a flag key
type Yes Release, Operational, KillSwitch, Experiment or Permission. Sets the defaults below
owner Yes The team that owns the feature
settleDays No Overrides the type's settle period (whole days, 1 to 3650)
grain Yes Global, Project or Personal
lifecycle Yes Developing, Piloting, Released, Settled or Withdrawn (below)
flags Yes The flag keys this feature owns. A key belongs to at most one feature
permission Release: yes The one authorization policy using the feature requires: SignedIn, or an existing Project* or Application* policy name (below)
audience No Who the feature is for (6R.A): signedIn, projectMembers, projectAdministrators or applicationAdministrators. Defaults from permission (table below); an explicit value may only be narrower
consent No Whether people join or leave it (6F.7): none (default), offered or required. Always none for KillSwitch, Operational and Permission features and for a Global feature; a non-none feature needs a non-Draft guide page for one of its topics (its title names the beta). See Consent and environment settings
compatibleWorkflowVersions Yes Reviewed workflow versions (may be empty at any lifecycle; release admission refuses an empty list with WorkflowCompatibilityMissing)
topicIds Yes Permanent guidance topic IDs (guide pages declare them in their syrf.topics front matter)
capability Yes Typed Flag/AllOf/AnyOf expression over the feature's own flags
communication No Overrides of the type's communication defaults; every field is optional

permission says who may use the feature, so the availability endpoint (Feature availability, 6E.2) checks permission first and reports a feature only to callers allowed to use it. The value is SignedIn (any active, signed-in user) or the name of an existing ASP.NET policy that starts Project or Application, for example ProjectEditMembershipsPolicy (the policy on the route that disables a membership). A Stage* policy needs a stage, so it is refused. A Release must declare it; other types may omit it, and a feature without it is never reported as available (it fails closed). The registry validator checks the shape (pnpm run validate:registry); FeatureDefinitionPermissionTests in SyRF.API.Endpoint.Tests proves the name is a registered policy, so a misspelt name fails the build. pnpm feature:new --type Release requires --permission <value> and refuses, writing nothing, without it. permission is generated into FeatureDefinition.Permission and the TypeScript definitions; it does not enter the runtime catalog digest. A release Prepared before this field existed must be prepared again, because Apply refuses a changed definition.

Any registry change to a feature (including adding or changing audience or permission) changes its generated FeatureDefinition, so every release of that feature that was Prepared before the change must be prepared again, because Apply refuses a changed definition (#4504). No shared environment has a Prepared release today (RuntimeFeatureReleases:Enabled is off everywhere).

audience (6R.A). audience says who the feature is for, in words that What's New, walkthroughs, help topics and the release preview can show ("for project administrators"). It is presentation metadata: no authorization, route-gate or availability decision ever reads it (FeatureAudienceArchitectureTests fails if one does); permission and targeting stay the gates. Most features need no audience line: it defaults from permission by the table below, which lives in one place (PERMISSION_AUDIENCE in src/charts/syrf-common/scripts/generate-feature-definitions.ts). The default is generated into FeatureDefinition.Audience and the TypeScript registry, and shown in the feature inventory. The validator (pnpm run validate:registry) refuses, with an exact message, an audience that is not one of the four values, that is wider than the permission's default (signedIn ⊃ projectMembers ⊃ projectAdministrators; applicationAdministrators only with an application-administrator permission), or that has no permission to be checked against. A permission the table does not cover is refused too, so a new policy forces a decision (and FeatureAudienceArchitectureTests fails when the API registers a policy the table lacks). The default is classified from what each policy requires by default (ResourceSecurity.json grants and the application and machine-credential policies), not from its name.

Audience Default for these permission values
signedIn SignedIn; ProjectViewPolicy and ProjectRequestToJoinPolicy (granted to every signed-in user)
projectMembers ProjectViewStudiesPolicy, ProjectLeavePolicy, ProjectSubmitPdfCorrectionsPolicy, ProjectViewScreeningProgressGraphPolicy, ProjectViewScreeningProgressGraphDecisionsPolicy (granted to every project member)
projectAdministrators ProjectDesignPolicy, ProjectEditPolicy, ProjectBulkPdfUploadPolicy, ProjectApprovePdfCorrectionsPolicy, ProjectInvitePolicy, ProjectSetMembersStatusPolicy, ProjectBulkUpdateStudiesPolicy, ProjectImportSearchPolicy, ProjectViewSearchesPolicy, ProjectRemoveSearchPolicy, ProjectCalculateRobPolicy, ProjectEditMembershipsPolicy, ProjectViewMembershipsPolicy, ProjectDeleteMembershipsPolicy, ProjectExportDataPolicy (the project Administrators group), and the owner-only ProjectChangeOwnerPolicy, ProjectAssignPermissionsPolicy, ProjectDeletePolicy (the narrowest project audience there is)
applicationAdministrators ApplicationSendEmailPolicy, ApplicationListUsersPolicy, ApplicationListEmailTemplatesPolicy, ApplicationModifyEmailTemplatesPolicy, ApplicationBatchAdminProjectsPolicy, ProjectStatisticsParityReadPolicy, ProjectStatisticsOperatorPolicy

The labels on What's New, walkthroughs, help chips and the release preview, and the Audience column in the admin Features list, arrive with those screens.

communication fields: userGuide ({ kind: Required }, or { kind: Exempt, reason: "…" }), whatsNew (announce or quiet), notify (releaseAudience, administrators or none), alertingOwner (defaults to owner) and walkthroughId (a docs/content/walkthroughs.yaml ID; optional; checked both ways by the validator, see Walkthroughs). The schema 1 guidance field is now communication.userGuide and is refused at the top level.

Defaults by type (owner decision, 2026-10-23). The generator resolves them, so the generated C# and TypeScript carry final values and no reader applies a default:

Type User guide What's New Notify Settle period
Release Required announce releaseAudience 30 days
Operational, KillSwitch, Experiment, Permission Exempt ("<Type> control; no user-facing guidance") quiet none none

A once-only quiet release of a Release feature is a release-time choice (6D PR A adds it to the Prepare and Apply requests), not a registry edit.

Lifecycle. Developing, Piloting and Released describe a feature that still has a live flag. Settled means the clean-up PR has merged: no flag and no branch remain, and the feature is a permanent part of the product. Withdrawn means the feature was removed from the product. Both keep their former flags and capability keys as history, and the validator refuses either if one of those keys is still in the runtime catalog. They replace the schema 1 value Retired; in C#, FeatureLifecycle.Withdrawn keeps Retired's numeric value (3) and Settled is 4. Release preparation accepts only a Developing or Piloting feature: it refuses a Released one (already live for everyone) as well as Settled and Withdrawn. A Released or Settled feature is generally available and takes no consent: setting lifecycle: Released on a beta also means setting consent: none (registry check 10; see Moving a beta to general availability). (Release content's own publication class Retired is a different concept and is unchanged.)

Workflow compatibility is checked at release admission, not on the lifecycle label (Chris, 2026-10-23). lifecycle records the feature's real state, so a feature that shipped before the registry can be Released with an empty compatibleWorkflowVersions. A release that goes through the process must have reviewed workflow compatibility at 6D Apply admission. The content compiler still requires every content entry's workflowVersions to be a subset of its feature's list.

Retirement is fact-based, with no dates in the registry. The registry holds the intent: the type and its settle period. Each environment's own release history holds the fact: when the feature became available to everyone and stayed unstopped. A Release feature is ready to settle once that has lasted longer than its settle period; the admin page that lists it arrives later in 6R, and pnpm feature:settle does the clean-up. Nothing time-based ever fails a build. See §5A (6R.2, "Retirement is fact-based") of the remaining delivery plan.

Flag groups and deployment-managed pins

flagGroups lists the preset groups served in the runtime snapshot (RuntimeFeatureFlagSnapshot.Groups) and written by the legacy PUT api/runtime-feature-flags/groups/{key}. Each entry has key, name, description, keys and an optional advertised (default true). The advertised order is the file order. A group with advertised: false is a retained ID: an older browser snapshot may still submit it, so Find resolves it, but All never lists it. Group IDs are versioned contracts: never change the keys of an existing ID. Add a new ID and set the old one to advertised: false.

deploymentManaged lists the flags pinned to their deployed value because a service that never reads runtime overrides consumes them (today reviewEligibilityPolicy). Each entry has key, constraint (shown on the admin page) and refusal (the 400 message). The nine statistics maintenance gates stay in ProjectStatisticsRuntimeActivation, where a parity test checks them against the domain family map.

None of these fields enter RuntimeFeatureFlagDefinition. Its fields feed the catalog digest that every stored runtime policy identity carries, so RuntimeCatalogDigestPinTests pins the digest value; registry metadata goes on FeatureDefinition only.

Complexity effect (R1b). Flags 55, caller evaluation seams 5 and generators 5 are unchanged. The hand-written C# groups (RuntimeFeatureFlagGroups.cs data) and pins (PinnedKeys) are gone: a new group or pin is now one YAML entry plus generate:feature-definitions, instead of a hand edit in C#. That removes 2 of the up-to-6 situational hand edits for adding a flag. The cost is two more generated files, which validate:generated checks.

Feature registry validator

pnpm run validate:generated runs scripts/feature-registry.ts (alone: pnpm run validate:registry). It fails on any broken cross-link and names the file and entry:

  1. Flag keys: every capability, owned flags and deploymentManaged key is in the runtime catalog; no key is owned by two features or pinned twice; a capability uses only its feature's own flags. A Settled or Withdrawn feature is the reverse: none of its keys may still be in the catalog.
  2. Guidance: a feature whose user guide is Required (communication.userGuide, or the Release default) has a non-Draft guide page (syrf.topics front matter) for each topic once it leaves Developing (a Developing feature only warns). Each declaration's featureId/topicId resolves.
  3. Links: each What's New userGuidePath names a published user-guide/ page by path (.html or .md) or front-matter permalink: (a page with a permalink is reachable only there), and each optional featureId exists. Help topics resolve through help-topics.generated.ts, which must match the pages' front matter. communication.walkthroughId and the walkthrough's featureId point at each other; a walkthrough has 1-8 steps, bodies of at most 400 characters (no image or HTML), anchors that exist as data-tour-id="…" in a web template, and no walkthrough for a feature whose help topics are all restricted (6G.4 W1).
  4. Groups: members are catalogued and unique, and a group ID's keys never change compared with the base commit: HEAD locally (the pre-commit hook) and HEAD^1 in GitHub Actions, which is the base branch tip of a pull request's merge ref (the Validate Generated Code jobs check out with fetch-depth: 2). Without a readable base it prints "group history unchecked".
  5. Schemas: env-mapping.yaml against env-mapping.schema.json, and the generated whats-new.json against whats-new.schema.json.
  6. CI paths: every source it reads matches the PR Tests Validate Generated Code change regex and the pre-commit GENERATED_PATTERN, so a registry or guide edit cannot skip it.
  7. Runtime What's New (6G.1): each runtime: entry in docs/content/whats-new.yaml names a feature whose resolved communication.whatsNew is announce, one of its topicIds and valid stages, with at most one entry per feature and stage. Its body has no image, no raw HTML and only https:// or user-guide links that resolve. A feature that announces needs an entry once it leaves Developing (a Developing feature only warns).
  8. Consent (6F.7): consent is none, offered or required; KillSwitch, Operational and Permission features are always none; a non-none feature is Project or Personal grain and has a non-Draft release-content entry (a syrf.topics guide page for one of its topics); a Released or Settled feature (generally available) is always none (Moving a beta to general availability); and FeatureDefinitions.generated.cs carries each consent-eligible feature's title as the pages give it now (else "run pnpm generate").

Feature IDs are independent of flag keys and permanent topic IDs. The initial definition is project-membership-disable, owned by API maintainers, with the topic project-membership-status. It is a Release that owns disableMembership and composes the disableMembership=true requirement. Guidance is required (the Release default); the existing public membership guide supplies background, but a content-manifest entry and reviewed workflow compatibility are still needed before release. Its lifecycle is Developing (it is off everywhere), with no workflow versions; this catalog invents no milestone date.

Every existing flag is registered (registered in #4438 and reviewed by the owner before merge): 30 features claim the 54 catalogued flags, each key exactly once (FeatureDefinitionCatalogTests checks this, so a new flag needs its feature in the same change). By type: 20 Release, 9 Operational, 1 KillSwitch. Each lifecycle is the real state from the deployed values, all with empty workflow versions (see above): 9 Released, 9 Piloting, 12 Developing. The count of 12 includes the keyword-highlighting project-targeting pilot (6F.2), which keeps its own lifecycle and grain. Released and Piloting Release features are Exempt from guidance ("shipped before guidance was required" or "piloting before guidance was required") and quiet in What's New, because no guide page declares their topics and no runtime What's New entry exists (registry checks 2 and 8). Developing ones keep the Required and announce defaults with one topic named after the feature, so the validator warns until their content exists. Every Release declares the permission its page or action needs (registry check 6). # review: comments mark the type, owner, permission and grouping judgement calls. The file's header comment states the conventions.

Capability expressions are typed Flag/AllOf/AnyOf, bounded to eight nested levels and 64 nodes. Unknown flags, flags the feature does not own, scripts, empty expressions, duplicate IDs, unknown fields (including a reviewBy date) and malformed guidance or communication fail generation. Missing flag values never satisfy even a false requirement. Matching flags is only capability composition: it neither authorizes a user nor confirms availability, publication or a release. Permission-first availability, content publication classes and the release writer remain subsequent contracts. Do not place private article text, audience lists or secrets in these compiled metadata files.

Flag decision: not flagged. These are build-time sources, generation and validation; a flag cannot switch them. The groups, pins, REST golden and catalog digest are unchanged, which proves runtime behaviour is unchanged, and the new FeatureDefinition fields have no consumer until 6D PR A, which sits behind RuntimeFeatureReleases:Enabled. Existing feature decisions, policy, saved data and migrations are unchanged.

Release lifecycle (walkthrough)

A release moves one registry feature's flag requirement into effect in one environment and records when that happened. Everything below sits behind RuntimeFeatureReleases__Enabled=true (default false in every environment) and Shared authority mode, so production, which runs Legacy policy mode, cannot reach it. The steps, each detailed in the subsection it links to:

  1. Prepare (Prepared release intents). Preview, then prepare with that digest. The record is Prepared (version 1) with a Prepared event. No policy changes.
  2. Apply (Applying a release). Preview the Apply against current policy, then apply with that digest. One transaction commits the policy, moves the record to Applying (version 2), and writes PolicyCommitted, AnnouncementQueued and the outbox rows. Admission can refuse it (422). Apply is idempotent by release ID.
  3. Confirm (Confirming activation). The reconciler on each Shared API host confirms the stage milestone once the fleet is Active for the committed policy. The record becomes Active, gets confirmedAtUtc (MongoDB time) and one ActivationConfirmed. If the fleet is not Active within 60 s the record is Degraded, and it is still confirmed once later.
  4. Announce (Dispatching release announcements). Behind RuntimeFeatureReleases__OutboxDispatch__Enabled=true (default false). The in-app notice waits for confirmation, then the in-app notice handler completes it, and the What's New row is completed by its handler, the same way in production, staging, previews and local stacks (owner decision D60 A, 26 Oct 2026, which replaced OD-19's staging test sink and the preview and local refusal). Availability then discloses the entry and reports the notice delivered ("New to you") (6G.2). Neither handler sends email.
  5. Stop and restore (Stop and restore recording). An emergency stop or a pause uses the ordinary policy path, which never reads release data. The reconciler then records EmergencyStopped or Paused, cancels the unsent notice once, and moves the record to Paused. A restore records Restored, then the record returns to Active with the same milestone and no second confirmation.
  6. Import history (Importing past releases). A reviewed operator command, run outside startup, records a release that happened before release records existed: imported: true, no applied block and no outbox rows. The reconciler ignores these records, and availability counts them as released. Imported records are outside RT-22.

A beta's general availability adds a registry step: the PR that sets lifecycle: Released also sets consent: none and each environment's consent setting is cleared, so stored Left choices stop applying (registry check 10; see Moving a beta to general availability).

Crash recovery (RT-22). A crash at any step resumes without duplicates. A lost Prepare or Apply response is resolved by retrying the unchanged request with the same mutation ID, which returns the original receipt. A crash inside Apply writes nothing, and its retry applies once. A crash inside a reconcile writes nothing, and the next pass confirms once (ObservedAfterRecovery after a gap of more than three periods). A dead reconciler's lease expires after 30 s of MongoDB time, and a resumed holder whose lease has gone gets LeaseLost and writes nothing (the tested path; the version compare-and-set behind it is not separately driven). A dispatcher that dies holding a row lets the lease expire after 60 s, and the row is leased again. A re-delivery after the handler ran hits the handler's deduplication key, so the row is Done once. A stop that commits between confirmation and dispatch cancels the notice once and sends nothing. RuntimeReleaseCrashRecoveryTests (SyRF.Mongo.Common.Tests, replica set) proves each boundary. The release-lifecycle scenario of the runtime-fleet harness (bash e2e/run-local.sh --runtime-fleet --scenario release-lifecycle, local stack only, run on Bramble) runs the whole lifecycle through real API and Project Management processes, and SIGKILLs the API that holds the reconcile lease. It is never part of all. It needs a feature the registry can release: on main, project-membership-disable declares no compatibleWorkflowVersions, so its Apply preview refuses with WorkflowCompatibilityMissing and the scenario stops there. Its release profile mounts the server help bundle compiled from the checkout's docs/content (RuntimeFeatureReleases:GuidanceBundlePath), so an announced Apply finds the feature's runtime What's New entry (6G.1).

Before any production release (written prerequisites only; none is done by 6D): a releasable registry entry (reviewed compatibleWorkflowVersions, and guidance Required with a public article or a reviewed exemption); the compiled guidance bundle, which every API image built since 6G.7 carries (no value to set); ProductionPolicyMode and RuntimeFeatureFlags:Mode=Shared in production, with production fleet admission; migrations 0011–0022 applied and bound; and the release switches set per environment in cluster-gitops. The What's New and in-app notice handlers exist (6G.2); the screens that show their result are 6G PRs C and D.

Prepared release intents

PR #4380 adds a preparation-only slice behind RuntimeFeatureReleases__Enabled=true (default false) and Shared authority mode. It supports features with one exact flag requirement whose lifecycle is Developing or Piloting: a Global feature's single saved-value Set here, and a Project or Personal feature's targeting change as described in Targeted releases. A Released, Settled or Withdrawn feature is refused at preview and prepare with RuntimeReleaseAdmission.LifecycleNotReleasable ("Only a Developing or Piloting feature can be prepared or applied."). Preparation records the reviewed intent; it does not activate policy, verify guidance, schedule execution, publish content or create a confirmed milestone date. The initial Developing definition remains unreleased.

Preview and preparation require ManageFeatureReleases; retained reads require InspectRuntimeFeaturePolicy or ManageFeatureReleases. Each permission check precedes mode, request or data inspection:

  • POST /api/feature-releases/{featureId}/preview returns the exact release preview digest and expected and proposed policy identities. Required guidance is explicitly Unverified.
  • POST /api/feature-releases/{featureId}/prepare requires that preview digest and saves the intent plus its Prepared event atomically. The server binds the authenticated actor and environment.
  • GET /api/feature-releases/{releaseId} reads the retained preparation in the current environment binding. This read remains available in Shared mode when preparation is disabled.

ReleaseKey identifies a logical release within one environment and feature: a lowercase kebab-case identifier beginning with a letter, at most 100 characters. A new attempt for the same release retains its key; a genuinely separate release uses a new key and release ID. Migration 0021 adds the unique {environmentId, featureId, releaseKey} index alongside the unchanged 0011–0020 migrations. Preparation refuses missing or incompatible release indexes; flags never provision them.

The preview covers release metadata, actor, policy candidate and the definition snapshot. Preparation serializes against policy, constraint and lineage changes through the existing authority constraint guard, without advancing the policy revision or changing values. Every new prepare advances that shared constraint guard, which the preview digest includes, so any other outstanding preview becomes stale and must be previewed again (an exact retry of an already prepared request returns its receipt and does not advance the guard). The record and receipt share one transaction. An exact mutation retry returns its original receipt; changed requests cannot reuse a mutation ID. An unknown commit response requires retrying the unchanged request, not assigning another mutation ID. Foreign binding or store epochs are refused.

requestedAtUtc is server time. An optional scheduledForUtc is intent only; confirmedAtUtc remains null and milestones remain empty. Turning preparation off retains records and events; turning it on again can resolve the original receipt. Preparation and Apply accept only a Developing or Piloting feature; any other lifecycle is refused with 422 LifecycleNotReleasable (see below). No deployment or production enablement is included.

Applying a release

6D PR A adds Apply behind the same RuntimeFeatureReleases__Enabled=true (default false everywhere) and Shared authority mode. Production runs Legacy policy mode, so Apply is not reachable there. Both routes need ManageFeatureReleases, checked before mode, request or data:

  • POST /api/feature-releases/{releaseId}/apply/preview is side-effect free. Body: { mutationId, expectedIdentity, reason, announcement, quietReason }. It returns the Apply preview digest (domain syrf-release-apply-v1, over the stored preparation result, the definition snapshot, the policy preview of the fixed flag change against current policy, the request, the announcements, the guidance result and the matched What's New entry), the expected and proposed policy identities with the policy changes, the guidance result (guidanceState: Verified, Exempt or Unverified; guidanceBundleDigest, the verified bundle's digest; guidanceTopics, the feature's topics found in it), the admission result (admitted, admissionRefusal) and queuedAnnouncements. The digest also binds the guidance result and, for an Apply that queues MaterialiseWhatsNew, the matched runtime What's New entry (its entryId and revision) and the verified bundle's digest (#4473), so a changed bundle or entry needs a fresh preview. It is marked read-only safe for support impersonation.
  • POST /api/feature-releases/{releaseId}/apply takes the same body plus that previewDigest and returns the applied release: status: "Applying", version: "2", the committed policy identity, requestedAtUtc (from preparation), committedAtUtc, the announcement choice and confirmedAtUtc: null. Apply is a change for impersonation.

One MongoDB transaction re-reads the binding, the authority and its receipt chain; resolves a replay by mutation ID; requires a Prepared record whose definition snapshot equals the current definition; checks the expected identity and the preview digest; runs admission; commits the policy through the same path as a policy commit (policy preview digest, production fleet admission, deployment constraint, audit receipt with _id = the mutation ID); moves the record from version 1 to 2 (status: "Applying", an applied block and an unconfirmed milestones.{stage} entry with precision: "Instant"); inserts the "{mutationId}:PolicyCommitted" event; and inserts the announcement rows with an "{mutationId}:AnnouncementQueued" event. A failure anywhere before commit writes none of them. The mutation-outcome lookup resolves an Apply mutation ID like any policy mutation.

Idempotent by release ID. An exact retry returns the original receipt. The same mutation ID with a different body is refused ("Release mutation ID was reused with a different request."), and so is a second Apply of an applied release ("Only a Prepared release can be applied; this release is Applying."). After an unknown outcome, retry the unchanged request.

Policy drift. The flag change and the definition snapshot are fixed at preparation. If other policy changes have moved the revision since, preview the Apply again against current policy and apply with that digest. A stale identity returns 409; a stale or missing digest returns 400 ("The reviewed apply preview is missing or stale. Preview the apply against current policy and retry."). A changed definition is refused: "The feature definition changed after preparation; prepare a new release."

Announcements come from the feature's resolved registry communication settings: whatsNew: announce queues a MaterialiseWhatsNew row and notify other than none queues a NotifyInApp row, so a Release queues both and every other type queues none. announcement: "Quiet" (enum value 1) with a required quietReason queues none; the preview shows the choice, the digest binds it and the record and PolicyCommitted event keep it. A quiet reason without Quiet is refused. Rows are Pending, keyed {environmentId}:{bindingEpoch}:{releaseId}:{stage}:{releaseId}:{stage}:1:{kind} (unique by migration 0016), with requiresConfirmation: true and the committed identity as guardPolicy. The outbox dispatcher sends them (see Dispatching release announcements), and the release stays Applying until the reconciler confirms activation (below).

Admission refuses with 422 { code: "runtime_release_admission", reason, message }, checked in this order:

Reason Refused when
LifecycleNotReleasable the feature is not Developing or Piloting
WorkflowCompatibilityMissing the feature has no reviewed compatibleWorkflowVersions
GuidanceBundleMissing the user guide is Required and no compiled server bundle is readable: RuntimeFeatureReleases:GuidanceBundlePath names an absent or unreadable file, or it is unset (the default) and the API's release-guidance/ directory holds no single server-<sha256>.json (an image built without the 6G.7 step, or a local run)
GuidanceBundleDigestMismatch the bundle does not verify: its file name is not server-<sha256>.json, the digest recomputed from its content differs from that name or from its embedded digest, or its format is not a compiled server bundle. Also, deliberately, when the release was prepared with a contentBundleDigest and the mounted bundle is any other bundle: such a release applies only with that exact bundle, because Apply must use the content that was reviewed. The pin applies to the guidance check and, for an Apply that queues MaterialiseWhatsNew, to the What's New check too, so a guidance-exempt feature that announces is pinned as well (#4473). After recompiling the content, prepare the release again
GuidanceTopicMissing a topic in the feature's topicIds has no article for this feature in the bundle (or the feature names no topics)
GuidancePublicationNotReleased a topic has no PublicPreview or PublicReleased article (Draft, PrivatePilot and Retired do not count)
GuidanceWorkflowNotCovered a topic's public articles do not list every version in the feature's compatibleWorkflowVersions
WhatsNewEntryMissing the Apply would queue MaterialiseWhatsNew (the feature announces and the Apply is not quiet) but the verified server bundle has no runtime What's New entry covering this feature and stage, or no bundle verifies. A quiet Apply with a reason (D13) needs no entry (6G.1)
KeyStopped the release's flag is under an emergency stop at the current policy (the same stop test as confirmation and the dispatcher's guard; in production only an eligible key's stop counts). Restore the key with ReleaseStop first, then preview the Apply again: the restore moves the revision, so the earlier preview is stale anyway (#4456)
IncompatibleHost any acknowledged host is incompatible (foreign catalog digest, binding or lineage)
FleetActionRequired the fleet for the key reports Action required

So project-membership-disable (Developing, no workflow versions, guidance Required) is refused today. The Apply preview reports the same reason as admitted: false with admissionRefusal, and a refused Apply writes nothing. An emergency stop never runs admission and never reads the bundle, so it commits while a release of its key is Prepared or refused.

What's New admission (6G.1). An Apply that would queue MaterialiseWhatsNew reads the same verified server bundle, before the session opens, and needs a runtime What's New entry (whatsNew in a schema 2 bundle, compiled from docs/content/whats-new.yaml runtime:) whose featureId is the release's feature and whose stages include the release's stage. Without one it is refused WhatsNewEntryMissing, at the Apply preview and at Apply, and nothing is written. The same release applied with announcement: "Quiet" and a reason queues nothing and needs no entry, and a quiet Apply of a guidance-exempt feature never reads the bundle. A schema 1 bundle (compiled before 6G.1) still verifies but has no entries, so it admits only quiet or non-announcing releases. Flag decision: not separately flagged; this only tightens the existing default-off RuntimeFeatureReleases:Enabled Apply.

Which entry admitted the Apply (#4473). The PolicyCommitted event has a whatsNew block beside guidance: { entryId, revision, bundleDigest, refusal: null } for an Apply that queued MaterialiseWhatsNew, and null for one that queued none (quiet, or a feature that does not announce). It records the bundle digest even for a guidance-exempt feature, whose guidance.bundleDigest is null. The same block, with its refusal, is part of the Apply preview digest.

The prepared-bundle pin (#4473 decision). A release prepared with a contentBundleDigest is admitted only against that bundle for What's New as well as for guidance: one pin rule, so the event's whatsNew.bundleDigest is always the bundle the release was prepared against when one was named. A mismatch refuses GuidanceBundleDigestMismatch; a quiet Apply queues no What's New row, so it is not pinned by What's New. Leaving contentBundleDigest unset (the usual case) pins nothing, and Apply records whichever verified bundle admitted it. The pin is about audit and consistency only: availability always serves the current bundle's entry (see What's New and in-app notices), so readers see edited text as soon as a new bundle is deployed, with its new whatsNew.revision.

Guidance admission (6D PR E, RT-26, decision D-Q6). When the feature's resolved communication.userGuide is Required, Apply and its preview read the compiled server bundle that pnpm run compile:release-content wrote (server-<sha256>.json, see Audience-separated content artifacts) from the path in RuntimeFeatureReleases:GuidanceBundlePath (environment variable SYRF__RuntimeFeatureReleases__GuidanceBundlePath) or, when that is unset (the default), from the copy baked into the API image at /app/release-guidance/ (6G.7, below). There is no separate expected-digest setting: the digest is the one in the content-addressed file name, re-verified by recomputing the compiler's digest (SHA-256 of JSON.stringify of the payload without digest) from the parsed file, so reformatting cannot hide an edit. The bundle is read and verified before Apply's MongoDB session opens, so file IO, hashing and parsing never hold the transaction open; the transaction uses only that result. A release prepared with a contentBundleDigest verifies only against that bundle. Every topic must have an article for the feature, published for the release's audience, and the articles must cover the reviewed workflow versions. Admission checks topics and their publication class, not topic revisions: any article revision in the verified bundle satisfies a topic. Pinning the reviewed revision belongs to 6G's protected revisions (future work). A release reaches everyone (the Global audience is the only one until targeting, 6F), so only PublicPreview and PublicReleased articles count. The file is read only for guidance-required features and for Applies that would announce in What's New (6G.1). The last verified bundle is kept with the SHA-256 of its bytes (each Apply still reads and hashes the file, but an unchanged file is not parsed again). It is one entry replaced by a single reference write, so it is bounded without the racy clear it replaced (#4473); a second bundle loaded in the same process only costs a re-parse. No failure throws: every problem is one of the refusals above, so admission fails closed. Exempt features skip the check and record the exemption. The PolicyCommitted event's guidance block records state, exemptionReason, bundleDigest and topics. Flag decision: not separately flagged. It sits behind RuntimeFeatureReleases:Enabled and only makes admission stricter; a switch that bypassed it would defeat OD-5.

Where the bundle comes from (6G.7, owner decision D38 A). The API image carries it: CI compiles the bundle from the same commit and builds it into the image, so each API version carries the file made from its own code and content, and there is no extra step or value to keep in step. In the reusable image build (.github/workflows/_docker-build.yml, API job only), the step Compile release guidance bundle installs the root tooling from the lockfile and runs

pnpm run compile:release-content .release-guidance/server <job-private public directory>

A compiler error fails the job at that step with the compiler's own message. The API Dockerfile (generated from docs/architecture/dependency-map.yaml, image_files) then copies .release-guidance/server/server-*.json read-only (0444) to /app/release-guidance/; with no staged bundle the COPY fails, so an API image is never built without it. Before anything is pushed, Check release guidance bundles in the image exports the image filesystem and requires exactly one server bundle in the API image, at /app/release-guidance/<compiled name>, byte-identical to the compiled file and read-only, and none in the web or user-guide image (pinned by .github/scripts/test-api-release-guidance-image-contract.sh). With RuntimeFeatureReleases:GuidanceBundlePath unset, the API reads the single server-<sha256>.json in release-guidance/ beside its binaries (/app/release-guidance/ in the image); none, or more than one, is GuidanceBundleMissing. An explicitly set GuidanceBundlePath still wins (the runtime-fleet harness and tests use it). Because the bundle is built from the content, the API's change paths include docs/content/**, user-guide/**, feature-definitions.yaml and the compiler (detect-service-changes.sh, .github/services.json): a content-only change rebuilds the API image. To build the image locally, compile into an empty staging directory from the repository root first (a stale second bundle would make the API refuse both): rm -rf .release-guidance && pnpm run compile:release-content .release-guidance/server "$(mktemp -d)", then docker build -f src/services/api/SyRF.API.Endpoint/Dockerfile -t syrf-api .. The P12a Secret route is retired: the chart renders no bundle path, Secret volume or mount, and the former runtimeFeatureReleases.guidanceBundle.secretName and fileName values are ignored. The server bundle holds restricted Markdown, so the API image is as private as the bundle; images are pulled from private GHCR packages. Replace the bundle file, never edit it in place: the bundle cache stamps path, length and last-write time, so an in-place rewrite that keeps the length and time is not noticed (a new, content-addressed file name always is). Apply refuses missing or incompatible release and outbox indexes (0012–0017, 0021) before writing. Turning the switch off refuses new previews and Applies only: GET /api/feature-releases/{releaseId} still reads the record, now with its current status and version, and turning it on again resolves the original receipt. Flag decision: no new switch, because Prepare and Apply are one operator workflow behind RuntimeFeatureReleases:Enabled.

Confirming activation

6D PR B adds the confirmation reconciler (RT-21). RuntimeReleaseReconcilerService runs in each API process only where RuntimeFeatureFlags:Mode=Shared (the runtime-policy writer; decision D11). It is absent in Legacy mode, in production while ProductionPolicyMode is Legacy, and in Project Management. Every 10 s ± 2 s it lists this binding's applied releases (Applying, Degraded, ActionRequired, Active or Paused with an applied block, never an imported one, through the {environmentId, status} index) and reconciles each one under its own lease:

  • Lease. reconcileLease: { token, owner, expiresAtUtc } on the record, taken with findOneAndUpdate only when absent or expired by MongoDB's own clock ($$NOW), for 30 s. The commit releases it. Leases avoid duplicate work; the version compare-and-set (with the lease token in the same filter) and the unique event index make confirmation exactly-once, so any number of API replicas is safe.
  • One transaction. It reads MongoDB's time first, then the binding, the authority and its receipt chain, then the record. Every decision uses that MongoDB time and stored evidence only, never the API host's clock: the fleet's freshness checks, the 60 s window and the recovery gap. So a skewed API clock, or which replica holds the lease, changes neither a decision nor a confirmed instant (decision D12). Host acknowledgements still carry each host's own clock, as before; a host whose clock is far from MongoDB's reads Unknown and blocks confirmation.
  • Confirm when all of these hold: the status is Applying, Degraded or ActionRequired; the stage's milestone is unconfirmed; the feature's reviewed flag requirement (from the record's definition snapshot) holds for the current policy and its key is not stopped; the current store epoch equals the applied one with a revision at or above it; and the fleet for the key is Active at the current policy (every expected host Current; see Host acknowledgements and fleet inspection). It then sets milestones.{stage}.confirmedAtUtc and recordedAtUtc to the transaction's MongoDB time, confirmedPolicy to the current identity and semantics to ObservedActivation, or ObservedAfterRecovery when more than 30 s (three periods) passed since the last reconcile (or since the commit, if none ran): the instant is when MongoDB observed activation, never an earlier guess. The record becomes Active and the event "{applyMutationId}/{releaseId}:ActivationConfirmed" records confirmedBy (the pod's RuntimeFeatureFlags:InstanceUid), the confirmed policy and the Current hosts. A confirmed milestone is never rewritten.
  • Degrade. An Applying release whose fleet is not Active 60 s after its commit (or, after a restore, 60 s after the restore was recorded) becomes Degraded, or ActionRequired when the fleet reports Action required. No milestone or event is written, and reconciliation continues, so a later Active fleet still confirms once.
  • Observation. For an unconfirmed release, every pass records lastReconciledAtUtc (the heartbeat the recovery gap is measured from), lastFleetState and fleetReason. A confirmed Active or Paused release is evaluated without a lease and is written only when a new policy receipt exists or its fleet state or reason changes, so a stable release keeps its version and is not rewritten every pass. It needs no heartbeat: its milestone is never written again.

Only a key whose reviewed consumer contract decides at new-request-admission can reach a fleet Active: disableMembership and, since 6F PR I, the targeting pilot screeningKeywordHighlighting. Such a key is decided at each new request from the adopted policy, so identity adoption is activation. Any other key (the statistics keys, which decide per read operation and also need domain readiness, and keys without a contract) reads Unknown (DomainReadinessNotEstablished), so its release stays Applying and then Degraded, and never confirms. The fleet compares policy identities only, never a flag's value; for a targeted release, "the capability holds" is judged per subject by the reconciler (see Targeted releases). GET /api/feature-releases/{releaseId} returns milestones (keyed by stage) and confirmedAtUtc from them.

A pass that cannot list releases (release indexes 0012–0015 or 0021, the audit indexes 0009–0010 or the outbox index 0022 missing or incompatible, an unverified binding) logs one warning and writes nothing; a failed release is logged and retried on the next pass. Flag decision: not separately flagged. It only converges releases an operator already applied, which must continue when RuntimeFeatureReleases:Enabled goes off; it does nothing without applied records and is absent in Legacy mode.

Stop and restore recording

6D PR D records stops and restores in the release's history (RT-24). The stop path does not change: an emergency stop, a pause (Change to off) and a restore (ReleaseStop, or Change back on) still go through the ordinary policy commit, which reads no release record, event, outbox row or document. An emergency stop therefore commits with the release, event and outbox indexes dropped, the dispatcher failing and the reconciler stopped; the record catches up on the next pass after they return (RuntimeReleaseStopRestoreTests, replica set).

In the same reconcile transaction, before confirming anything, the reconciler reads the policy receipts after the record's applied.lastObservedRevision in {storeEpoch, revision} order (audit index 0010), at most 100 per pass, and advances lastObservedRevision past them. For each receipt that changes whether the release's capability holds (the same test as confirmation: the reviewed flag value and no stop on its key), it appends one event:

Receipt Event Record status
EmergencyStop that stops the capability EmergencyStopped Paused
Any other change that stops it (a Change to off) Paused Paused
ReleaseStop, or a Change, that makes it hold again Restored Applying, then Active once the fleet is Active

A receipt that changes neither (an unrelated key, a second stop while stopped) appends nothing. Each event's ID is "{receiptMutationId}/{releaseId}:{type}", so it is exactly-once by the 0013 index, and it records sourceMutationId, sourceRevision, action, key, stage, occurredAtUtc (the receipt's own recordedAtUtc), occurredPrecision: "Instant", provenance: "Observed", observedPolicy, observedBy and the status before and after. A stop and its restore inside one reconcile period are both recorded, in revision order.

Milestones are never rewritten. A restored release whose milestone was already confirmed goes back to Active once the fleet is Active for the restored policy, with no second ActivationConfirmed; its confirmedAtUtc, requestedAtUtc, release ID and announcement ID stay byte-identical. A release stopped before it was confirmed goes Paused rather than staying Applying, and is confirmed once, as usual, after its restore. Nothing is confirmed while the history scan is behind (more than 100 receipts pending). A restore sets lastRestoredAtUtc, from which the 60 s confirmation window then runs.

History assumes the capability held at the Apply revision. That holds because Apply refuses a key under an emergency stop (KeyStopped, #4456); before that refusal, a release applied while its key was stopped was recorded as Paused at the next, unrelated policy receipt. If the dispatcher leases an unsent notice between the reconciler's read and its cancellation, the reconciler's compare-and-set fails and the whole stop recording rolls back; the next pass records it, so Paused can appear one reconcile period late.

Stop-time cancellation. In the transaction that records a stop, the reconciler cancels the release's unsent in-app notices (Pending NotifyInApp rows, found through the {releaseId, state} index 0022) exactly as the dispatcher's guard does: Cancelled, cancelReason: "CapabilityStopped", cancelledByMutationId and one "{firstStopMutationId}/{releaseId}:AnnouncementCancelled" event, where the first stop is the first receipt since Apply that stopped the capability. Both writers derive that same ID and insert the event only if it is absent, so a notice is cancelled once with one event, whichever sees the stop first. A Leased row is left to the dispatcher holding it, whose guard cancels it under the same ID. MaterialiseWhatsNew rows are not cancelled: the dispatcher defers them while stopped. A restore never requeues a cancelled notice (D-Q5); a new notification needs a new announcement revision. Flag decision: not flagged, because it is part of the reconciler (absent in Legacy mode) and the stop path does not change.

Malformed notices (#4458). A Pending NotifyInApp row that does not parse (no guardPolicy, a guardPolicy without a GUID storeEpoch and an integer revision, or anything the dispatcher's row parse rejects) is poison. The stop-time cancellation skips it, so the stop event and Paused are still recorded and the well-formed notices still cancelled. After that transaction commits, the reconciler marks the row terminal in its own write: state: "Failed", lastErrorClass: "MalformedRow", failedAtUtc from $$NOW, with attempts untouched. It logs the row ID at Error. Later passes do not see the row again. Rows are only written by Apply, so this arises only from a bug or a manual edit. If the process dies between the commit and that write, or the dispatcher leased the row meanwhile, the dispatcher's guard marks it the same way.

Importing past releases

6D PR F adds a reviewed operator command for releases that happened before release records existed (RT-27, plan §8.5), RuntimeReleaseHistoryEntryPoint. Like the lineage command it runs the API image's normal entry point before startup, never as part of startup, migrations or a flag. It is a written operational prerequisite only: it has not been run in any shared environment, and the operator procedure is runbook 11. Flag decision: not flagged, because, like initialise, it does nothing unless an operator runs it.

The reviewed input is a JSON file with exactly these fields:

Field Value
featureId a feature in the generated feature definition catalog (any lifecycle)
releaseKey lowercase kebab-case, beginning with a letter, at most 100 characters
stage LimitedRelease or GeneralRelease
precision Date, Month or Unknown
occurredOn yyyy-MM-dd for Date, yyyy-MM for Month, absent or null for Unknown; never in the future
source where the date comes from, for example the whats-new.json entry (at most 1000 characters)
reviewer who reviewed it (at most 200 characters)
SYRF_RUNTIME_RELEASE_HISTORY_MODE=preview
SYRF_RUNTIME_RELEASE_HISTORY_INPUT=/reviewed/history-input.json
SYRF_RUNTIME_RELEASE_HISTORY_OUTPUT=/reviewed/history-preview.json

plus the Mongo configuration, RuntimeEnvironment and the SYRF_MIGRATION_BINDING_EXPECTED_* binding expectations, as for the lineage command. The preview is read-only and deterministic: the release ID and the mutation ID are derived from the binding, the feature and the release key, so the same input in the same binding always gives a byte-identical plan. It prints the exact record and event documents and the previewDigest; apply fills only the fields listed in its applyFills (recordedAtUtc, importedBy and previewDigest), which are null in the preview.

SYRF_RUNTIME_RELEASE_HISTORY_MODE=apply
SYRF_RUNTIME_RELEASE_HISTORY_INPUT=/reviewed/history-preview.json
SYRF_RUNTIME_RELEASE_HISTORY_OUTPUT=/reviewed/history-receipt.json
SYRF_RUNTIME_RELEASE_HISTORY_ACTOR=<operator identity>

Apply recomputes the plan from its input against the current binding and catalog and refuses unless the reviewed digest still matches. It needs the migration-owned release indexes (0012–0015, 0021) and never creates one. In one transaction it writes:

  • the record: status: "Active", version: 1, imported: true, no applied block, and milestones.{stage} with semantics: "Imported", the precision, the reviewed occurredOn literal and confirmedAtUtc set to the first instant (UTC) of that day or month, or null for Unknown;
  • one "{mutationId}:HistoricalImported" event: provenance: "Imported", provenanceDetail: { source, reviewer }, semantics: "Imported", occurredAtUtc, occurredPrecision, occurredOn, importedBy and the previewDigest.

It writes no policy, no audit row and no outbox row: nobody is notified about a past release. The confirmation reconciler lists only records with an applied block and without imported: true, and skips an imported record it is asked to reconcile by ID, so it never leases or writes one. Apply and its preview refuse an imported release ("An imported historical release is history only; it cannot be applied."), and GET /api/feature-releases/{releaseId}, which reads a release's preparation, returns 404 for one. An imported key also blocks a later preparation with the same key.

An exact replay of the same preview returns the original receipt and writes nothing, so after an unknown outcome retry the identical preview with a new output path. Refusals (exit 2, nothing written): an unknown feature ("Unknown feature: …"), a modified preview ("The reviewed import preview was modified …"), a stale one, for example after a rebind ("The reviewed import preview is stale: …"), a different import or a prepared release holding the same key ("A release with this feature and release key already exists …"), invalid input, a missing index or actor, an existing output path, and a combined migration or lineage mode.

Targeted releases

6F PR I (spec 6F.10) releases a Project or Personal feature to an audience instead of to everyone. It sits behind RuntimeFeatureReleases__Enabled=true and RuntimeFeatureTargeting__Enabled=true (both default false in every environment) and Shared authority mode. Production stays Legacy and refuses every targeting write, so it cannot be reached there. Global features keep the single Set path above, unchanged. No migration and no start-up work.

Prepare. The preparation body (POST /api/feature-releases/{featureId}/preview and .../prepare) sets policy.action to Targeting, leaves policy.value null and carries the key's complete replacement rules in targeting: { rules }, in the targeting rule shape. The rules' enabled allow subjects are the release's audience (a LimitedRelease to them). Every targeting save-time check applies, with its reason (runtime_targeting_refused: personalRuleOnProjectGrain, unknownSubject, parentOff, notYetSupported, limits), plus these release refusals (422 runtime_release_admission):

Reason When
TargetingRequired A Project or Personal feature is prepared with a saved-value Set, which would reach every subject
AudienceMissing The rules name no enabled allow subject (for example, exclusions only), or the feature's requirement is false (an allow can only turn a flag on)
RolloutAudienceNotSupported The rules include a rollout (any amount, 10,000 basis points included): a targeted release's audience is its named allow subjects. Refused before the targeting checks, so the answer does not depend on whether rollouts can be saved (6F.8)
RolloutPresent The key's current rules already hold a rollout, Active, Paused or Off (6F.8). A targeted release replaces the key's whole rule set, so it would drop the rollout; remove the rollout first. Refused at preview, prepare and Apply (a rollout added after preparation is refused at Apply with this reason, not TargetingChanged), before anything is written. Unlike the other Apply refusals, which come back as a 200 preview with Admitted: false and an AdmissionRefusal, Apply preview answers this one with a 422 (runtime_release_admission, the refusal's message), because the preview cannot be built with a rollout present; a release screen must show that message
PersonalAudienceNotSupported A Personal feature (6F PR E): its rules may name people, but confirmation decides the audience as projects without a person, so it could never confirm. Save its rules through the targeting routes instead

A Targeting action on a Global feature is refused (globalKey). The request stays within the 32 KiB release bound. The record also stores targetingBase, the key's rules at preparation (null when it had none).

Apply. Apply rebuilds the rules from the Prepared receipt and commits them through the ordinary targeting mutation path inside the Apply transaction: the globalPolicyMutation receipt carries request.action = "Targeting" and request.targeting = { key, rules }, with the targeting preview digest, the project existence check, the deployment constraint and schema-2 admission (schemaUnsupportedByFleet, a runtime_targeting_refused refusal from the commit; the Apply preview does not report it). Every existing admission check still runs: lifecycle, workflow compatibility, guidance, What's New, KeyStopped and the fleet (IncompatibleHost, FleetActionRequired). One more refusal: TargetingChanged when the key's rules differ from targetingBase, because Apply replaces the key's rules and would silently undo that change; prepare a new release against the current rules. An unrelated policy change since preparation still needs only a fresh Apply preview. The record's applied block gains targeting: { key, rules }, the rules Apply committed. Releasing an audience a tester already piloted through the targeting route works: the rules are committed again as a new revision.

Confirm. The reconciler confirms a targeted release when, besides the usual conditions, every project of its audience decides the flag on at the current policy (through the same deployment-aware evaluator requests use, as an operation context: stops, exclusions, required parents and pins apply) and RuntimeFeatureTargeting:Enabled is on in the reconciling API, so no milestone is confirmed while no host applies allows. The fleet must be Active (see Confirming activation). The same per-audience test drives the stop and restore history and the outbox guard: a later targeting write that keeps the audience on records nothing; one that excludes a project of the audience records Paused; allowing it again records Restored. Those tests read the policy as stored, so turning the switch off never writes release history.

Availability. available needs a confirmed release and enabled, so it holds exactly where the release's rules turn the feature on: in a project of the audience, not elsewhere, with no per-project release matching. The release block names the feature's release in every project.

GeneralRelease stays explicit (§8.1). No policy, however wide (a saved default On, or a rollout at 10,000 basis points once 6F.8 lands), adds a GeneralRelease milestone; it needs its own intent, a separate record with its own confirmation.

Personal audiences are refused (PersonalAudienceNotSupported) until releasing to people is built. RuntimeReleaseTargetedIntentTests (SyRF.Mongo.Common.Tests, replica set) and FeatureAvailabilityHttpTests.TargetedRelease.cs (Prepare, Apply and confirm an allow for P, then availability available in P and notReleased in Q) cover it.

Reading releases (admin Releases screen)

6G PR AD2 adds the read routes the admin Releases screen uses (the screen is AD2b). They are read-only, need InspectRuntimeFeaturePolicy or ManageFeatureReleases (either one is enough; a caller with neither gets 403, and the check precedes any read), and sit behind RuntimeFeatureReleases:Enabled: while it is false the list and events routes answer 404 feature_availability_disabled before authorization and read nothing. Every query is filtered to the current environment binding (environmentId plus bindingEpoch), so a release, event or announcement row written under another epoch or environment never appears. Nothing here writes, and there is no migration.

  • GET /api/feature-releases?featureId=&status=&before= lists this binding's releases, newest first (requestedAtUtc descending, an imported release last, then release ID), at most 50. status is a record status (Prepared, Applying, Active, Degraded, ActionRequired, Paused); an unknown one is 400. The response is { releases, nextBefore }: pass nextBefore as before for the next page (absent on the last page); a before that is not a release of this binding is 400. Each item carries the feature, release key, stage, status, version, imported, milestones, confirmedAtUtc and its announcements. An imported release lists here (with imported: true and no requestedAtUtc) although its detail GET is 404. The query uses migration 0012 (environmentId, status), or 0021 (environmentId, featureId, releaseKey) when featureId is given, by hint, and sorts the binding's releases in memory; an explain test pins both plans to an index scan.
  • GET /api/feature-releases/{releaseId}/events returns { releaseId, events, truncated }: the release's events in recorded order through 0014, at most 200 (truncated says more exist). An event shows its type, recorded and occurred times, stage, provenance and, for PolicyCommitted, the Apply reason; no actor, request body or digest. A release outside this binding, or unknown, is 404.
  • GET /api/feature-releases/{releaseId} gains announcements: [{ kind, state, attempts, lastErrorClass, cancelReason, completedAtUtc, testSink }], read from the release outbox through 0022. testSink is whether a finished delivery went to a test sink (null until something is delivered): false for every delivery since D60 A, true only on a staging row the retired test sink completed before then. While the switch is off the block is omitted and the outbox is not read; the rest of the detail is unchanged.

One malformed release row blocks the list. If a release row cannot be parsed (a FormatException from the prepared record, or from the imported-record check), the whole list answers 503 runtime_policy_unavailable, as the detail GET does for that release. Failing closed is intended: the list never silently skips a record. An operator who sees a blocked list should look for a bad row in the release collection (filter by the current environmentId and bindingEpoch), not for an outage.

Dispatching release announcements

6D PR C adds the release outbox dispatcher (RT-23; RT-25 as amended by D60 A) behind its own switch, RuntimeFeatureReleases__OutboxDispatch__Enabled=true (RuntimeFeatureReleases:OutboxDispatch:Enabled, default false everywhere). It is a plain configuration switch, like RuntimeFeatureReleases:Enabled, not a catalogued runtime flag. The hosted service RuntimeReleaseOutboxDispatcherService is registered only on an API host in Shared authority mode (the one writer host); Legacy (production today) and Shadow hosts never run it. Every 10 s ± 2 s it dispatches up to 20 due rows:

  1. Lease. One findOneAndUpdate against MongoDB's clock takes a Pending row, or a Leased row whose lease has expired, with nextAttemptAtUtc ≤ $$NOW, and sets state: "Leased", lease: { token, owner, leasedAtUtc } and nextAttemptAtUtc = $$NOW + 60 s, so a holder that dies releases the row through the same 0017 scan. Only rows this host can route are leased; a kind with no handler stays Pending and visible.
  2. Guard, in one snapshot transaction before any side effect, in this order:
Check Outcome
The row does not parse (wrong schemaVersion, a deduplication key that does not match, a missing field) terminal Failed, lastErrorClass: "MalformedRow", logged at Error
The row's environment or binding epoch differs from the current binding (a restored copy) Cancelled, cancelReason: "BindingChanged"
An outbound kind (NotifyInApp) under a binding whose environment ID is not valid (fail closed; no valid binding reaches this since D60 A) Cancelled, "DeliveryDisabled"
No guardPolicy, or one without a GUID storeEpoch and an integer revision terminal Failed, "MalformedRow", logged at Error
NotifyInApp, and some policy revision since Apply stopped the release's capability (emergency stop or a change that turns it off) Cancelled, "CapabilityStopped", plus one "{stopMutationId}/{releaseId}:AnnouncementCancelled" event; it stays cancelled after a restore (decision D14)
MaterialiseWhatsNew while the capability does not hold deferred ("CapabilityNotHeld"); What's New returns with its original date once restored
The release is not Active, or its milestone is unconfirmed deferred ("AwaitingConfirmation")

Deferral sets Pending, deferrals + 1 and nextAttemptAtUtc = $$NOW + backoff; it does not count as an attempt. A malformed row is poison (#4458): failedAtUtc from $$NOW, attempts untouched, the pass outcome Quarantined, and it is never leased again. The binding checks run between the two malformed-row checks, so a restored copy's row is still cancelled BindingChanged. 3. Handle. Each kind reaches the one handler registered for it, the same in every valid environment (D60 A, 26 Oct 2026). The binding's stored announcementDelivery is not read, so a binding written before D60 with testSink (staging) or disabled (previews, local stacks) delivers exactly as one written since, which stores production. No re-bind or data change is needed. Handlers are idempotent by the deduplication key. 4. Complete by compare-and-set on the lease token: Done, completedAtUtc, attempts + 1 and result ({ handler, testSink: false, announcementDelivery: "production" } in every environment, the shape production rows always had). A handler failure, including the handler's own timeout or cancellation, sets attempts + 1, lastErrorClass and Pending after the backoff; the 20th makes the row terminal Failed (failedAtUtc). A guard that cannot run (authority not initialised, binding changed mid-pass, the release record missing) sets guardErrors + 1 and lastGuardErrorClass and backs off without touching attempts, so no outage can make a row Failed. A malformed row is not a guard error: the guard marks it MalformedRow (step 2). Only a shutdown of the pass itself leaves the row leased until its lease expires. An operator requeue is a follow-up.

Readiness (the indexes and the binding) is checked once per pass, and each row's guard still re-reads the binding in its own transaction.

The backoff is 15 s, doubling per failure or deferral, capped at 1 hour. A stop is never delayed by any of this: the stop path does not read the outbox.

Handlers. The API registers one handler per kind (the dispatcher refuses two), the same in every environment:

Handler Kind Effect
RuntimeReleaseWhatsNewHandler (6G.2) MaterialiseWhatsNew none: completing the row is the record availability reads
RuntimeReleaseInAppNoticeHandler (6G.2) NotifyInApp none: completing the row (result.testSink: false) is what availability reports as delivered

Both are idempotent, so a repeated delivery changes nothing, and neither sends email or anything else outside the environment's own database. So in every environment, once the release is confirmed, the in-app notice and What's New end Done by these handlers (What's New is deferred while the capability does not hold; the notice is cancelled CapabilityStopped if a stop came first). There is no projection store: the outbox rows are the record of what was announced.

Before D60 A (26 Oct 2026; OD-19, superseded). Staging sent the in-app notice to RuntimeReleaseTestSinkHandler, which recorded it in the staging-only runtimeFeatureReleaseTestSink collection (result.testSink: true, never reported as delivered), and previews and local stacks cancelled it DeliveryDisabled. The handler and its routing are removed. Rows they finished are kept unchanged (rows are never rewritten): a staging notice completed by the sink stays undelivered and shows "Sent to test sink", and a cancelled one shows its reason. The runtimeFeatureReleaseTestSink collection is no longer written or read; any rows in it stay until a later clean-up.

Retention. Done, Cancelled and Failed rows are never deleted: there is no TTL index, no deleteAfterUtc and no clean-up job (ADR-023, forward only). Migration 0022 adds {releaseId, state} for the bounded per-release scan. Dispatch refuses to run (logs and writes nothing) until 0009, 0010, 0013, 0016, 0017 and 0022 exist, and refuses if the outbox has any TTL index.

Switching. Off, a pass reads and writes nothing, so rows stay Pending and kept. On, the rows are dispatched under the same guards, so a notice for a release that was stopped meanwhile is cancelled rather than sent. Off, on, off, on delivers nothing twice (RuntimeReleaseOutboxDispatchTests, replica set). Flag decision: flagged, because this is the only 6D component with outbound side effects and it needs its own kill switch. D60 A added no flag: it changes only non-production delivery (production's routing, results and responses are unchanged), it changes no stored data, and it is rolled back by reverting the change.

Feature availability

6E.2 answers "which released features can this person use here" for discovery surfaces (What's New and walkthroughs, 6G and 6G.4). It is a discovery fact, not a gate: UI and route gates keep reading the flag snapshot and IFeatureDecisions, and availability reads the same decisions, so there is no second evaluation path.

Route Admission
GET /api/features/availability Signed in. Features whose permission is a Project* policy are omitted without being evaluated (D25 option A)
GET /api/projects/{projectId}/features/availability ProjectViewPolicy, before the action runs (switch on), so an absent or deleted project answers exactly as every other ProjectView route (404, or 410 project-deleted for an active member)

Both answer Cache-Control: private, no-store. The server, per request:

  1. Takes the registry's features that are not Withdrawn and declare a permission.
  2. Permission first. Authorises each distinct policy once with IAuthorizationService (the existing handlers and gates; SignedIn passes for a signed-in caller). A feature the caller may not use is omitted, never reported with a reason, so the body is byte-identical to one from a registry without it. If a check finds the response already started (a deletion racing the request), the handler's answer stands.
  3. Makes one IFeatureDecisions.DecideAll over the union of the remaining features' flag keys.
  4. Reads the released records of exactly those features (RuntimeReleaseStateReader), only when a Global feature that is not Settled is among them (Developing/Piloting answers depend on the record, and a Released feature's release and announcement feed What's New and "New to you"). Legacy mode has no binding, so nothing counts as released there. Per feature the read takes the latest record in the captured authority binding (environment and binding epoch) with a confirmed stage milestone, or an imported Active record (Unknown precision included). GeneralRelease beats LimitedRelease, then the later confirmation. It is one bounded query (at most 1000 rows, projection only, majority read), cached per binding for 10 s, so a process issues at most one release query per 10 s. A record is announced only when it was applied with announcement mode Default and queued at least one kind (6G.2: a Default-mode release of a feature that is quiet with no notice announces nothing). In the same cached read, when an announced record is among them, one more query reads their outbox rows by _id $in (the deduplication keys, at most two per record; projection state and result.testSink), so a process issues at most one outbox query per 10 s too. A failed outbox query keeps the released records, reports nothing as done or delivered, logs a Warning and makes the response Degraded. Any failure of that query other than a cancelled caller (a Mongo error, a malformed record, an unexpected exception type) returns the reader's Failure, and the failure is cached for the same 10 s, so a bad read never becomes a 500. The failure is logged once per period at Warning with the exception and its type. A response whose permitted features are all Settled or not Global does not call the reader, so it is Current whether or not the read would have failed.
  5. Evaluates, first matching row wins:
Condition availability reason
Grain not Global (until 6F) unknown policyUnavailable
Settled available null
A decision is Stale unknown staleAuthority
Any other decision error unknown policyUnavailable
Capability false and a key Stopped unavailable stopped
Capability false unavailable notReleased
Capability true, Released available null
Capability true, Developing/Piloting, released record available null
Capability true, Developing/Piloting, no released record unavailable notReleased
Capability true, Developing/Piloting, release read failed unknown policyUnavailable

The response carries contextId (sha256: over the environment and binding, user, project and every field except checkedAtUtc, including rehearsal, delivered and each whatsNew entry; an equality token, never recomputed by the browser), policyVersion, rehearsal (retired, always false, below), freshness (checkedAtUtc, and state Current or Degraded when a reported decision, the release read, the outbox read, the discovery state read or a needed bundle read failed) and features ordered by featureId: featureId, availability, reason, release (stage, releasedAtUtc, precision; null without a released record), announcement ({releaseId}:{stage}, revision 1, delivered, and seen and dismissed from the caller's own discovery state below; null for a quiet, imported or unannounced release) and whatsNew (below). It never carries a flag key, store epoch, digest or targeting (RT-60), pinned by the FeatureAvailabilityResponse golden, FeatureAvailabilityContractTests and the endpoint catalogue's response-field pin.

Switching. Behind RuntimeFeatureReleases:Enabled (default false; false in every shared environment): off, both routes answer 404 { "code": "feature_availability_disabled" } and read nothing. The answer comes from RuntimeFeatureReleasesSwitchMiddleware, which Program.cs runs after authentication and before UseAuthorization (it reads the [RuntimeFeatureReleasesEnabled] endpoint metadata; ProgramMiddlewareOrderTests pins the order). So on the project route ProjectViewPolicy never runs and the project is never loaded: a non-member, a member, an administrator and an anonymous caller get the same disabled body. Two cases differ:

  • A subject that subject admission refuses (with applicationSuspension on) still gets that refusal, because admission runs first.
  • The 401 challenge comes from authorization, which has not run. With the switch on, anonymous gets 401.

The attribute's resource filter is the backstop where the middleware is absent. It is read-only and stateless, so on, off, on gives the same answers with any data (FeatureAvailabilityHttpTests). Flag decision: it shares the release switch rather than adding a flag, because availability exists only to serve release discovery. Enabling it in an environment also needs RuntimeFeatureFlags:Mode=Shared and a bound environment; production stays Legacy and off.

Seen and dismissed announcements

6E.3 remembers, per person, that they have seen or dismissed one announcement (OD-14), so 6G's "New to you" and 6G.4's walkthroughs do not show it again:

PUT /api/account/announcements/{announcementId}/state
{ "revision": 1, "seen": true, "dismissed": true }
  • Validation is by format only: {releaseId}:{LimitedRelease|GeneralRelease} (the announcement.id availability reports), revision at least 1, and at least one of seen and dismissed true. Anything else is 400, the same 400 whether or not the announcement exists.
  • It writes only when the announcement is one availability reports now: the latest released, non-quiet, non-imported record of a feature in this authority binding (read from the same cached RuntimeReleaseStateReader read, no second evaluation path). An unknown, quiet, imported, superseded or foreign-binding announcement, and Legacy mode, write nothing. Either way the answer is 204, so the route never says whether a release exists.
  • A revision ahead of the announcement's current one also gets 204 and writes nothing, so a client cannot record a future revision to hide its own later re-notification.
  • No permission or project check. The row is the caller's own preference, read back only through availability, which omits every feature the caller may not use before it joins the rows. A dismissal of a feature the caller cannot use is stored and never returned, and their availability body does not change.
  • Storage: runtimeFeatureDiscoveryState, one document per person and announcement, _id = "{userId:D}:{announcementId}" (MongoDB reference), written by one upsert by _id with MongoDB $$NOW timestamps. The same revision keeps the first seenAtUtc and dismissedAtUtc (retries and races leave one row); a higher revision replaces both (re-notification); a lower one is ignored. There is no un-dismiss. No secondary index, so no migration.
  • Join: availability reads the caller's rows by _id ($in over exactly the announcements in its response). seen and dismissed are true only for a row stored at a revision at least the announcement's. A failed read reports both false and freshness.state: Degraded.
  • Impersonation: the PUT is a change, so read-only support impersonation refuses it; under an edit context it writes the target's row, never the actor's (SupportImpersonationContractTests).
  • Switching: the same RuntimeFeatureReleases:Enabled switch. Off, the PUT answers 404 feature_availability_disabled before reading the body; rows are kept and ignored, and apply again when it is on.
  • Account deletion keeps these rows today, like every other per-user store (D27, Chris chose A on 2026-10-23); erasure belongs to the OD-6 subject-deletion work.

Tests: FeatureAvailabilityHttpTests (RT-41 API half, no leak and no junk with row counts, the switch round trip), RuntimeFeatureDiscoveryStoreTests (replica set: shape, races, revisions), FeatureAvailabilityEvaluatorTests (the join) and the impersonation suites.

What's New and in-app notices

6G.2 discloses runtime What's New text through availability, the one disclosure gate: there is no second personalised endpoint. Each feature entry may carry:

"announcement": { "id": "<releaseId>:LimitedRelease", "revision": 1, "delivered": true, "seen": false, "dismissed": false },
"whatsNew": { "id": "project-membership-disable", "revision": "sha256:…", "title": "…", "body": "…Markdown…",
  "topicId": "project-membership-status", "guidePath": "projects/membership/members-groups.html#disabling-and-re-enabling-a-membership" }
  • whatsNew is non-null only when all of these hold: the feature is available to the caller (so permission came first, and a stopped feature is unavailable); its release is announced with MaterialiseWhatsNew among its queued kinds; that outbox row is Done; and the current verified bundle has a runtime entry covering the release's stage. A confirmed release whose row is still Pending (or deferred) shows nothing. A stop hides the entry at once; a restore shows it again with the original release.releasedAtUtc. A missing or unverifiable bundle gives whatsNew: null and Degraded; a verified bundle without an entry for the stage gives null and stays Current. revision is sha256: and the entry's hash from the current bundle; no route takes a revision, and deploying a new bundle changes revision and contextId on the next read. The body is Markdown, which the browser renders only through MarkdownPipe in untrusted mode (6G PR D).
  • announcement.delivered ("New to you") is true only when the release's NotifyInApp row is Done with result.testSink: false and the feature's notify is releaseAudience (an administrators notice belongs to the admin Releases screen, 6G.11). Since D60 A (26 Oct 2026) the in-app notice handler completes the row the same way in production, staging, previews and local stacks, so "New to you" works in all of them. A cancelled notice is never Done, so after a stop and restore it stays false (D14). A staging row the retired test sink completed before D60 keeps result.testSink: true and stays undelivered.
  • rehearsal is retired and always false. Under OD-19 it was true outside production so the screens labelled runtime entries "Rehearsal"; D60 A (26 Oct 2026) made every environment deliver announcements as production does, so no environment is a rehearsal. The field stays so the response shape, production's bytes and production's contextId are unchanged (the hash input still carries "rehearsal":false); the browser no longer reads it, and removing it is a later clean-up. (Staging and preview context IDs change once, which only makes a browser re-read.)
  • Bundle cache. Availability reads the bundle through RuntimeReleaseGuidanceBundleCache, keyed by path, length and last-write time and re-checked at most every 10 s; the file is read and hashed again only when one of them changed, so no request hashes the file. It is read only when some entry passes the tests before the bundle (so a broken bundle never degrades a response that could not show What's New). Bundle files are content-addressed, so a new bundle normally has a new path; an in-place rewrite that keeps the length and last-write time is not noticed.
  • Cost. One release query, one outbox query and at most one bundle stat per 10 s per process, however many requests (RuntimeReleaseStateReaderTests and RuntimeReleaseGuidanceBundleTests, 50 concurrent callers).

Flag decision: not separately flagged. Reads sit behind RuntimeFeatureReleases:Enabled (404 and no reads when off) and the handlers behind RuntimeFeatureReleases:OutboxDispatch:Enabled; both are default false everywhere. No migration, no new collection (the reader only reads the outbox by _id), nothing at start-up. On, off, on gives the same bodies with any data. Tests: FeatureAvailabilityHttpTests (a reviewer, a non-member and the no-project route get bodies byte-identical to a registry without the feature, with the project administrator as the positive control; the outbox, stop and restore sequence; the switch round trip), FeatureAvailabilityEvaluatorTests, RuntimeReleaseStateReaderTests and RuntimeReleaseOutboxDispatchTests (production, staging, previews and local stacks with the registered handlers, including bindings that still store the legacy testSink or disabled value).

What's New in the app (6E.4 and 6G.3)

The browser reads availability through one service, FeatureAvailabilityService (src/services/web/src/app/core/services/feature-availability.service.ts, spec 6E §6E.4). Every discovery surface reads only this service; none fetches availability, flags or release records itself, and a review of a new discovery surface checks that.

  • One answer per context. It holds the answer for the current context: the signed-in (or impersonated) user, and the project in the URL (/projects/{projectId}/…), if any. Inside a project it calls the project route; elsewhere it calls the no-project route. Signed out, it fetches nothing.
  • When it fetches. On a context change (sign-in, sign-out, impersonation, entering or leaving a project) and after each runtime snapshot that RuntimeFeatureFlagsService accepts. That service already refetches on its slow fallback timer, a SignalR revision hint, reconnection and window focus or online, so availability adds no timer or listener of its own: one accepted revision hint gives one availability fetch.
  • Context safety. A context change clears the answer at once; a response for a context that is no longer current is dropped, so one project's or one person's entries never appear in another's.
  • Failures. Only the switch-off 404 is an empty answer: the body { code: "feature_availability_disabled" }, or a 404 with an empty body (an API build without the route). Any other error keeps the last answer for the same context. That includes another status, any other body and the project route's absent-project 404, whose body is not JSON. A new context has no answer yet, so surfaces render nothing.
  • Seen and dismissed. setAnnouncementState(id, revision, seen, dismissed) calls PUT api/account/announcements/{id}/state and updates the local answer at once; if the write fails, it fetches again so the screen shows the server's state.

Two surfaces show runtime What's New entries, with the same rules: the feature is available here, the response carries its whatsNew entry, and this person has not dismissed it.

  • Home page What's New (no-project context). Runtime entries merge with the static feed in date order (the static isoDate against the entry's Europe/London date) into one list of changes grouped by month: the two newest months open, older months collapsed. A runtime entry renders in the same format as a static change. Because of D25 A, features that need a project permission never appear here.
  • "New in this project" on the project overview (project context). Pending D36; built to recommendation A. A compact card that renders nothing without an entry. It is one self-contained component (project/project-overview/project-whats-new-card/), so it can move with the owner's answer.

Each entry (shared/whats-new/whats-new-entry.component.ts) shows:

  • The date. The confirmed milestone in Europe/London, in the static feed's style ("19th August 2026"). It is formatted with Intl.DateTimeFormat('en-GB', { timeZone: 'Europe/London' }) and never depends on the reader's machine zone (RT-29): 2026-08-18T23:30:00Z is "19th August 2026" (BST) and 2026-12-01T23:30:00Z is "1st December 2026" (GMT).
  • "New to you" when announcement.delivered is true and seen is false, in every environment (D60 A). There is no "Rehearsal" label any more (OD-19's label, superseded by D60 A on 26 Oct 2026).
  • The body. It is rendered only through MarkdownPipe in untrusted mode, so no image, raw HTML or remote resource reaches the page.
  • Actions. "Read the guide" is a link (guidePath through UserGuideUrlPipe). "Hide" is a button that records seen and dismissed for this person only. Opening a row, following the guide link or following a link in the body records seen.

With releases switched off, or no entry, both surfaces render exactly what they rendered before. A component spec pins the home page DOM against the DOM captured from main before this change.

Flag decision: not separately flagged. The server switch makes availability answer 404, so the browser shows only the static feed and no card. No migration, nothing at start-up. Tests: feature-availability.service.spec.ts (6E.4-1, 6E.4-2), shared/whats-new/**/*.spec.ts (RT-29 under three machine zones with a positive control, the entry rules, untrusted rendering, the actions), whats-new.component.spec.ts (the static-only DOM, merge order, no "Rehearsal", RT-41 rendering) and project-whats-new-card.component.spec.ts. On the runtime-fleet lane, e2e/tests/runtime-fleet-browser-whats-new.spec.ts (bash e2e/run-local.sh --runtime-fleet --spec runtime-fleet-browser-whats-new) runs the MVP boundary end to end: release switches on, the local binding left as stored (it delivers the notice as production does, D60 A), Apply and confirmation, then the card for a project administrator, nothing for a reviewer, a non-member or the home page, Hide for one person, a later joiner, and a stop and restore. It needs a releasable registry entry, so on main it fails at the Apply preview with WorkflowCompatibilityMissing, like the release-lifecycle scenario.

Public What's New for signed-out readers (6G.6)

GET /api/whats-new/public (anonymous, WhatsNewController, PublicWhatsNewProjection) answers the runtime What's New entries that general availability has made public, the same for every caller:

{ "rehearsal": false,
  "entries": [ { "id": "project-membership-disable", "revision": "sha256:…", "title": "…", "body": "…Markdown…",
    "topicId": "project-membership-status", "guidePath": "projects/membership/members-groups.html#…",
    "announcementId": "<releaseId>:GeneralRelease", "releasedAtUtc": "2026-11-12T23:30:00Z", "precision": "Instant",
    "audience": "projectAdministrators" } ] }

audience is the feature's registry audience in camelCase (signedIn, projectMembers, projectAdministrators, applicationAdministrators; null when the definition has none), and nothing else about the registry. It is presentation only (owner decision D57 = B, 26 Oct 2026): a feature that is fully released with a public guide page is still announced publicly, and an entry whose audience is projectAdministrators says so. The browser shows a "For project administrators" chip beside the date on the home page What's New, for both the public entry and the signed-in copy of the same announcement (the home page labels the signed-in copy from the public answer, because availability carries no audience). There is no label for the other audiences, and none on the project "New in this project" card, which only project administrators see. The field never decides access. It rides on the existing RuntimeFeatureReleases:Enabled switch, so there is no new flag.

An entry appears only for a Global feature (not Settled or Withdrawn) whose latest released record in the captured binding is a confirmed, announced GeneralRelease with its MaterialiseWhatsNew row Done, whose capability holds now (one DecideAll over exactly those features' keys, through the availability truth table, so a stop or a stale authority hides it and a restore brings it back with the same date), and whose current bundle entry covers GeneralRelease with a public guide page (guidePath not null). It is dated by the GeneralRelease milestone. A LimitedRelease-only, stopped or unreleased feature never appears, and the body has no clock, policy revision, flag key, environment or binding, so it is byte-identical to one from a registry without those features. Any failed read gives entries: [], never a guess: that includes a failed binding capture or a failed DecideAll, so the route never answers 500 (cancellation of the request itself still propagates). This is published content, not an anonymous availability view (D26 A).

rehearsal is retired and always false (D60 A, 26 Oct 2026): every environment delivers announcements as production does, so the answer no longer tells an anonymous caller anything about the environment. The field stays so the shape and production's bytes are unchanged.

With RuntimeFeatureReleases:Enabled off it answers 404 {"code":"feature_availability_disabled"} and reads nothing, as the other runtime release routes do. On, it answers Cache-Control: public, max-age=60, so after a stop a shared or browser cache may show the old entry for up to a minute (plus the 10 s release read).

In the browser, PublicWhatsNewService fetches it once and after each accepted runtime snapshot (a 404 is an empty answer; any other error keeps the last answer). The home page What's New merges, in date order:

  • signed out: the static feed and the public entries;
  • signed in: the static feed, the personalised entries and the public entries, without any public entry whose announcementId the no-project availability answer reports (whatever its availability or dismissal). So an entry the person can use shows once, with its own "Hide" and "New to you"; an entry they hid stays hidden; a feature the answer reports stopped is not shown from a cached public body. Public entries are held back for a signed-in reader until their availability answer has loaded, so a hidden entry never flashes; if availability fails, only the static feed shows (the conservative choice, since the dismissal cannot be checked).

Public entries carry no announcement state, so they have no "Hide" and no "New to you".

Flag decision: behind RuntimeFeatureReleases:Enabled (default false everywhere; the #4533 fixes use the same switch and add no flag); read-only, no migration, nothing at start-up. Tests: FeatureAvailabilityHttpTests.PublicWhatsNew.cs (RT-68 over HTTP: general release publishes dated that day, a limited release never does, stop and restore, the no-leak byte comparison with its positive control, one DecideAll, the switch), EndpointAuthorizationCatalogTests (anonymous classification, pinned response fields), public-whats-new.service.spec.ts, runtime-whats-new.spec.ts and whats-new.component.spec.ts (signed-out public only, deduplication, RT-68, the switched-off DOM equal to the pre-6G DOM).

Shared administrator preview and commit

Current page layout: the review panel, retained-request panel and saved receipt described here are now presented in the change dialog, the outcome banner and Details; see Using the Feature flags admin page. The preview, digest, commit and retry rules below are unchanged.

PR #4381 connects individual controls on the existing Feature Flags page to the Shared policy API when its verified snapshot carries authority metadata. Legacy snapshots retain the existing mutation flow. Shared group controls remain unavailable because there is no shared group transaction endpoint.

In Shared mode the snapshot's mutationAllowed is false: it describes only the legacy PUT, which the server refuses there. Each flag's runtimeManageable and managementReason describe the Shared preview and commit routes instead, using the same deployment constraints as the legacy store (for example the statistics maintenance gates). In production only eligible keys are manageable; the others read "This flag is deployment-controlled in production." The page offers On, Off and Default when the snapshot carries schema-1 authority and the flag is manageable, and shows the reason otherwise. The authority envelope also lists stoppedKeys: the ordinal-sorted keys whose emergency stop is in force at that policy (production lists only eligible keys). The field is additive within schema version 1; the browser's lineage guard accepts envelopes without it and refuses one whose stoppedKeys is not a list of key strings.

A change first requests a server preview. The page shows exact before/proposed revisions, effective values, the proposed override (including Deployed default), induced changes and fleet readiness. Confirm and save commits that reviewed request with its preview digest and one mutation ID. Cancel before commit writes nothing. A dependency refusal, at preview or at commit, returns to the existing explicit dependency-confirmation flow; it never retries or commits automatically, and a new preview and confirmation are required.

A successful receipt is shown as Policy saved, without claiming fleet adoption. After a lost response, keep the page open: controls remain locked to the original request, and Check outcome reads its receipt. An absent receipt means Unknown, not aborted. Retry unchanged request preserves every field and mutation ID. While the outcome is unknown, the page shows the retained request (flag, current and proposed values, override and revisions) so you can see what a retry would write; there is no Confirm and save button. A refusal of that retry does not prove that the earlier attempt failed. The mutation ID stays visible for diagnosis. Do not start a new request to resolve an unknown outcome. If the page is reloaded or closed and reopened in the same tab, Reload recovery (in Using the Feature flags admin page) picks the request up again by its mutation ID.

The server still owns permissions, environment binding, eligibility, dependencies, full-identity checks and fleet admission. A known initial refusal requires a fresh snapshot and another reviewed preview; changes never fall back to legacy PUT in Shared mode. The existing Shared mode is the opt-in boundary; this slice adds no environment configuration, migration, production activation or automatic page reload. Emergency-stop controls, full fleet-status UI and the hermetic multi-replica freshness exercise remain follow-ups before the complete B/C operator workflow is claimed.

Audience-separated content artifacts

PR #4382 adds an offline content compiler. Since 6G.5 E1 it has no manifest: it derives its entries from the syrf.topics front matter of user-guide/ and docs/content/protected/ pages (see Guide topics). It validates feature/topic references against the versioned feature definitions, explicit publication classes, workflow references and asset audiences. It produces separate server and public JSON artifacts with SHA-256 content identities. Source ordering does not affect those identities; article, asset or reader-class edits change the article revision. Private-only edits leave public bytes and the public digest unchanged.

pnpm run compile:release-content /tmp/syrf-content-private /tmp/syrf-content-public

Output directories must be separate trees outside the source root. Server artifacts use owner-only file permissions and are refused in the repository's public guide/web asset trees, including symlink aliases. Keep server bundles private: they contain full restricted Markdown, audience IDs and assets (the API image carries one, 6G.7, so treat the image as private too). Only copy public artifacts into a future public renderer. The compiler does not publish, render or serve either artifact; API reader authorization and old-revision access checks are still required before protected content is delivered. Markdown rendering/sanitization and URL resolution belong to that renderer.

Every declaration names featureId, stable topicId and workflowVersions, plus an optional section: a heading anchor on the page (as kramdown generates it), refused when the page has no such heading. publication defaults to PublicReleased in user-guide/ and is required in docs/content/protected/. The entry's variantId (the page path), title (the section heading or page title) and Markdown (the page without front matter) are derived. No page declares assets yet, so assets and assetIds are empty. Supported classes are Draft, PrivatePilot, PublicPreview, PublicReleased and Retired. Draft/private entries require an explicit documentation audienceId. Retired entries require a reviewed historicalPublication class and retain its audience; retirement never makes a private pilot public automatically.

Publication evidence (6G.1, #4382). A declaration alone is not proof, so the compiler checks the evidence it can see. A public class (PublicPreview, PublicReleased) is accepted only for a source inside user-guide/, which the public site publishes (Public content must live in the public user guide ...); a restricted class is accepted only outside it. A restricted entry's audienceId must equal its featureId, meaning "the people the feature is available to", until 6F defines audiences. A Retired entry or asset whose historicalPublication is public is refused, because nothing records which revisions were published (D40); Retired with PrivatePilot is still accepted. D40 was answered B on 2026-10-24: environments will record which public revisions their guide site served (6G.P, below), but the refusal stays until 6H consumes that record.

Server bundle schema 2 (6G.1). The server bundle is schemaVersion: 2: the schema 1 fields plus whatsNew, the runtime What's New entries from docs/content/whats-new.yaml (runtime:), each { id, featureId, stages, title, body, topicId, guidePath, revision }. guidePath is the topic's public article page and section (for example projects/membership/members-groups.html#disabling-and-re-enabling-a-membership), or null while the topic has no public article; revision is the SHA-256 of the entry. The public bundle stays schemaVersion: 1 and never carries whatsNew. The CLI reads docs/content/whats-new.yaml. The API's reader accepts schema 1 and schema 2, each with its exact field list.

Publication manifest (6G.P P1, D40 B). Beside the public bundle the compiler writes publication.json: { schemaVersion: 1, entries: [{ featureId, topicId, variantId, publication, revision, path, anchor }], digest }. It lists the public entries only (PublicPreview, PublicReleased), in bundle order, each with the same revision as its bundle entry, the guide page it is served at (path, the page with .md replaced by .html) and its section (anchor, or null). It carries no title or Markdown, and nothing new becomes public: the web bundle's help-topics.generated.ts already ships every public topic's feature, page and section. digest is the compiler's usual SHA-256 of the payload without digest, so the C# RuntimeReleaseGuidanceBundle.PayloadDigest verifies it (pinned by a golden digest in compile-release-content.test.ts and RuntimeHelpPublicationManifestTests). The user-guide image serves it at /syrf-publication.json: the reusable image build (_docker-build.yml, user-guide job only) runs Compile the publication manifest (both bundles to job-private temp; only publication.json is staged, read-only, as .help-publication/syrf-publication.json), the generated user-guide/Dockerfile copies it into the nginx stage outside Jekyll, and Check the publication manifest in the guide image requires exactly one /usr/share/nginx/html/syrf-publication.json, byte-identical to the compiled file and read-only, before anything is pushed (pinned by .github/scripts/test-user-guide-publication-image-contract.sh; the existing guidance check still refuses any server-*.json there). The compiler, guide-topics.ts, generate-feature-definitions.ts and feature-definitions.yaml are user-guide change paths, so a change to them rebuilds the guide image. To build the image locally, stage the manifest first (user-guide/README.md, "Building the production image locally"). "Served" will mean what the API's publication recorder (6G.P P2) observes: an anonymous GET of this manifest from the environment's own guide origin with a valid digest, then each listed page answering 200 without a redirect and containing its section anchor. Until P2 is merged and switched on, nothing reads the manifest. Flag decision: not flagged, because the file is static, public-only and read by nothing until P2, which sits behind RuntimeFeatureReleases:Enabled. Known gap: path ignores permalink: (eight guide pages use one, in whats-new/ and data-export/; none declares a topic today), so a page with a permalink would fail P2's page check and go unrecorded rather than be recorded wrongly.

Public output contains only explicitly public articles, their public search metadata and the public assets they reference. Restricted articles/assets, audience IDs and unreferenced assets cannot leak through this projection. Public articles cannot reference restricted assets; restricted assets must match the article's audience. Restricted sources themselves are refused under user-guide/ and web public assets: filtering one bundle cannot protect files already exposed by another public build. Keep future private sources in a separate non-public source tree. Relative paths, symlink containment, file/bundle sizes and UTF-8 are checked. The initial asset formats are PNG/JPEG/WebP; active-content formats require a later reviewed renderer.

The checked-in topic is declared in the front matter of the already-public membership guide (user-guide/projects/membership/members-groups.md, section: disabling-and-re-enabling-a-membership). PublicReleased describes that existing article, not the Developing feature's lifecycle. Its empty workflow list asserts no reviewed workflow compatibility. No release date, guidance readiness, activation, announcement or new publication is inferred from compilation. The compiler consumes no feature-flag values. Flag decision: no runtime switch is needed for this explicit offline tool, which is not registered in an application or publishing pipeline. Required guidance remains Unverified in release preparation; release Apply verifies the compiled server bundle (see Applying a release). Protected delivery to readers is still a later integration (6G).

Help publication record (6G.P)

The API records which public help topic revisions this environment's own public guide site has served, so that a later public archive of retired pages (6H) can rest on evidence rather than on a typed historicalPublication line (D40 B, 2026-10-24). Nothing reads the record yet except tests and evidence runs; there is no route and no UI.

What "served" means. The API's anonymous HTTPS GET of its guide origin returned the publication manifest that the guide image carries at /syrf-publication.json (built by the content compiler, 6G.P P1) with a valid digest and, for each manifest entry, the page itself: 200, no redirect, a text/html content type, a body containing <html (any case, so a soft 404 that answers 200 with plain text or a bare message is not a guide page) and id="{anchor}" when the entry has a section. Bodies are decoded as UTF-8 (the guide is Jekyll output). The guide site is the only thing that serves public help pages and is promoted separately from the API, so the API's own bundle is never used as the source. One row per environment and topic variant revision, in runtimeHelpPublication (see MongoDB reference): the verified binding's environmentId, bindingEpoch and clusterFingerprint, the entry, the origin, first and last observation times (MongoDB time), the SHA-256 of the HTML last served and lastGuideOrigin, the origin that served the latest observation (guideOrigin stays the first origin; rows written before lastGuideOrigin existed lack it and still read, with a null value).

Settings (API, read on every pass):

Setting Meaning
RuntimeFeatureReleases:Enabled The switch (default false everywhere). Off at start-up, the recorder is not registered at all; switched off later, a pass reads, fetches and writes nothing. Rows stay
RuntimeFeatureReleases:PublicationRecord:GuideOrigin The environment's public guide origin, scheme and host only, for example https://help.staging.syrf.org.uk. Unset or anything but an origin: the recorder idles (NotConfigured)
RuntimeFeatureReleases:PublicationRecord:IntervalMinutes Minutes between passes, default 60, minimum 1, maximum 1440, each ± 10 %. The first pass waits a jittered 1-5 minutes (never during start-up)

Environment variables use the usual SYRF__ prefix and double underscores, for example SYRF__RuntimeFeatureReleases__PublicationRecord__GuideOrigin. Setting the origin per environment in cluster-gitops is an operational step that has not been taken: production https://help.syrf.org.uk, staging https://help.staging.syrf.org.uk, a preview or local stack its own guide service.

Guards.

  • Registered only on the Shared-mode API writer (the reconciler's guard) with the switch on. Legacy mode, production in Legacy policy mode and every other host register nothing.
  • Each pass verifies the environment binding (one read plus hello) before any HTTP request; an unverified binding writes nothing (BindingUnverified).
  • Origin guard. A production binding records only https://help.syrf.org.uk (the code constant RuntimeHelpPublicationEvidence.ProductionGuideOrigin); every other binding refuses that origin; http is allowed only under a local: or preview: binding. So a staging API pointed at the production site, or production pointed elsewhere, records nothing (OriginRefused).
  • Whole-manifest refusal. Exact field lists, schemaVersion: 1, a recomputed digest, at most 200 entries, every entry PublicPreview or PublicReleased, every path a relative .html path without .., scheme or query, and every anchor null or a kramdown ID. Any failure refuses the whole manifest (ManifestRefused), so nothing from it is recorded.
  • Bounded. Per replica and pass: one binding check, one manifest GET, at most 200 page GETs (each distinct page once), one unordered bulk upsert. The named client sends no cookies or credentials and never follows a redirect; each request has 10 s, the manifest 1 MiB and a page 4 MiB, and the pass 60 s (entries left when it ends are not recorded, BudgetExceeded). Replicas upsert the same rows idempotently.
  • Fail-soft. Every failure writes nothing for that pass or entry. Each reason is logged with its own detail when it first appears, again at most once every 24 hours while it persists (so a recorder that fails every pass is not silent), and when it clears: Disabled, NotConfigured, OriginRefused, BindingUnverified, ManifestMissing (an older guide image has no manifest), ManifestRefused, PageNotServed, BudgetExceeded, StoreUnavailable. The recorder never stops the host and adds nothing to readiness or health.

Production evidence. RuntimeHelpPublicationEvidence.IsProductionEvidence(row, binding) is true only for a production row read under the reading host's verified production binding with the same epoch and cluster, the production origin and a public class. Staging, preview and local rows never satisfy it, nor a local stack bound as production against a local origin, and production rows copied into another database by a reseed fail there. RuntimeHelpPublicationStore.FindAsync(environmentId, topicId, variantId) reads one environment's rows for a topic variant by an anchored _id range; together they are the contract 6H will consume. One read returns at most 1000 rows and the result carries a Truncated flag that is true when the cap was hit. The cap takes rows by _id (which ends in a revision hash, not an age), so beyond 1000 rows both the subset and its order by first observation are approximate: treat a truncated result as incomplete.

Reading the rows. On a local or Bramble stack (read-only):

db.runtimeHelpPublication.find({ _id: { $gte: "local:<stack>:project-membership-status:", $lt: "local:<stack>:project-membership-status;" } })

Flag decision. Behind the existing RuntimeFeatureReleases:Enabled (spec 6G.P, Q-P1 recommendation A, pending Chris): the recorded topics exist for runtime releases and the record needs the same verified binding, so no new switch. The switch is read through RuntimeHelpPublicationRecorder.SwitchedOn alone, so a dedicated switch (Q-P1 option B) would change one method and the registration. GuideOrigin and IntervalMinutes are configuration, not switches. Off means no reads, no HTTP and no writes; rows stay, and on again resumes the upserts, so round trips are safe with data from either state. No migration. Production records nothing until it is bound, runs Shared mode and has the switch on.

Inspecting fleet adoption

Current page layout: there is no per-flag Inspect fleet button any more. The summary line reads the membership control's fleet (the only key with tracked adoption, which reports the whole policy), and the host table is in Details; see Using the Feature flags admin page.

PR #4383 adds Inspect fleet beside each flag when the administrator catalog carries Shared authority. It calls the existing inspect-capability endpoint and shows its exact policy revision, adoption state, expected replica counts, host states, reasons and last-seen times. A saved mutation is not evidence that hosts adopted it. Domain readiness remains unknown for controls the server cannot yet assess.

The response must match the selected flag and every field of the catalog policy identity. Refresh the catalog if a newer policy was committed. Selecting another flag or receiving a new catalog identity cancels the previous lookup. Checking again clears the previous observation immediately; failed or forbidden reads show Unknown. Observations expire after 15 seconds and require Check fleet to obtain fresh evidence. This is an on-demand inspection view, not a continuous convergence monitor. It changes no policy, enrollment, configuration or data. The existing Shared-mode opt-in is the flag boundary; Legacy has no fleet action. Emergency-stop controls, automatic convergence monitoring and the multi-replica exercise remain follow-ups.

Administrator emergency stop

Current page layout: Emergency stop… in the summary card opens a dialog that needs a reason; Stop prepares and sends the stop in one confirmed step. Check and unchanged retry are unchanged; see Using the Feature flags admin page.

PR #4384 exposes a separate Prepare emergency stop action for the audited global disableMembership control in Shared mode. It fetches the current catalog identity within the same environment/binding/store lineage, shows its exact revision and audit reason, and requires Confirm emergency stop before invoking the dedicated stop endpoint. The audit reason uses the membership control's existing comment, or an explicit administrator-page default. The server independently checks the emergency-stop capability and safe-state contract. Stop forces the control's audited safe value (Off) without discarding saved overrides.

The stop flow is independent of ordinary preview/save state: an ordinary mutation with an unknown outcome does not disable it. An ambiguous stop keeps one immutable request and mutation ID. Check stop outcome reads that receipt; Retry unchanged stop sends the same request. Missing receipts and later refusals do not prove an earlier attempt aborted. After a reload or navigation in the same tab, Reload recovery (in Using the Feature flags admin page) looks the stop up again by its mutation ID and never sends it again. An initial known conflict/refusal can be prepared afresh.

Emergency stop saved confirms only the durable receipt and names the committed revision. Confirming reloads the catalog even if an ordinary mutation is unresolved, without replacing that original mutation. The API's policy reader can lag a commit by a few seconds; until the page shows the saved revision the receipt says so and offers Refresh. The receipt goes once dismissed or once a refresh reaches its revision. Inspect fleet adoption separately.

The section shows the current stop state (Stopped or Not stopped) after every catalog load, including a page reload. It reads authority.stoppedKeys from the snapshot (added in #4381), so no extra request or capability is needed. While a saved receipt is newer than the snapshot the section shows Stop state updating instead of a stale answer. Prepare emergency stop is offered only when no stop is known to be active. A server that omits stoppedKeys leaves the state unknown; the section says so and keeps the stop available, and the server still refuses a stop that does not apply. The stop does not call release preparation, guidance or announcement services. Releasing a stop is a separate action: see Restoring a stopped membership control. Hermetic multi-replica timing evidence remains open. Flag decision: the existing Shared authority opt-in and server capability are the gates. This adds no production activation, deployment configuration, migration or audience enrollment.

Restoring a stopped membership control

Current page layout: Restore… in the summary card opens a dialog that shows the server preview's restored value and needs a reason; see Using the Feature flags admin page.

PR #4385 adds Preview restore (accessible name "Preview restore of membership control") to the emergency-stop section of the Shared administrator page. It is offered while the section shows Stopped, or when the server does not report the stop state; Prepare emergency stop is offered only when the control is not known to be stopped. It uses the existing ReleaseStop policy action with no override or dependency edits. The server preview must remove the selected stop while preserving its saved override. Review the before/after effective values, saved values and Emergency stop column (for example Stop released), then confirm. A control with no matching stop cannot be confirmed through this restore flow.

Restoration uses the ordinary policy-change permission, preview digest, exact identity, deployment constraint and production-readiness checks. It does not use the emergency-stop bypass. Preparation and confirmation are blocked while the local emergency-stop request is preparing, confirming, committing, resolving or unknown. The original restore request is retained after ambiguous responses, with the same receipt lookup and unchanged retry as ordinary policy edits. A confirmed restore dismisses an older known stop receipt, refreshes the catalog and shows a Policy saved receipt with the committed revision rather than claiming adoption; inspect fleet adoption separately. The server preserves saved overrides and stop/restore data; the browser changes no migrations or release milestone dates. Existing Shared-mode opt-in remains the flag boundary. Production configuration remains unchanged. Hermetic convergence/stop/restore timing remains a follow-up; browser recovery after a reload is covered by Reload recovery (in Using the Feature flags admin page).