Runtime feature releases: current-code matrix (Increment A)¶
This is the evidence half of Increment A in the runtime feature releases plan.
It records, for each requirement of the flag architecture roadmap
(P1–P10) and each element of Chris's 15 October 2026 functional proposal, what main actually does today.
Baseline. syrf origin/main at e5a3155c4 (2026-10-15). cluster-gitops main at 5bdaa00d6, read
through the GitHub API only. Every row was checked against the code, not against older milestones or PR
descriptions. Line numbers refer to that baseline. No test was run for this matrix; "Done" means the code
and its tests exist on main, not that the behaviour has been proven in a deployed fleet.
Status key. Done: the requirement is met as written. Partial: some of it exists, the gap is
named. Missing: nothing on main meets it.
Implementation update: 20 October 2026¶
The original 56-row matrix below is historical evidence at e5a3155c4; its counts are not current completion
counts. Source revalidation at 4e24039ca4 confirms the binding and audit/release index foundations from
4258, #4272 and #4281 are merged. PmMigrations owns those migrations; their presence in source is not¶
proof they ran in an environment. No production database or live configuration was queried for this update.
| Area | Current source evidence / #4344 change | Remaining promotion or targeting prerequisite |
|---|---|---|
| Binding and indexes (E.1–E.2) | Explicit bind/pre-write verification and migrations 0009–0021 exist | Operator-reviewed environment enablement and independently verified applied state |
| Shared writer/evaluator (P2.3, OD-24) | #4344 extracts the existing catalog, dependencies, compatibility rules and writer into SyRF.Mongo.Common; API contracts remain the same |
Complete immutable shared evaluation; audited consumer/safe-state metadata |
| Catalog source (P2.3) | #4345 generates all 54 existing shared definitions from canonical YAML, with compiled-contract parity and generated-file guards; deployment-only constraints are preserved | Consumer enrollment, safe-state/grain metadata and authority digests still require audited evidence |
| Explicit lineage transition (P3.2, P4) | #4347 adds an operator-only preview/apply command with reviewed baseline input, binding verification, stopped-writer evidence, source checks and an atomic idempotent receipt; #4346 prevents legacy replacement of epoch-bearing state | Authority-capable consumers, rebind/restore procedure, runtime mutation receipts, compatibility collector and staging rehearsal remain required; no live import has run |
| Bound authority observation (P3.5–P3.6) | #4348 validates binding, envelope and import receipt in a coherent read; immutable capture, first-failure refusal, monotonic expiry and sticky quarantine; optional API/PM Shadow observation and authenticated API local status | Ordinary mutation receipts, actual consumer enrollment, expected-host inventory/acks and compatibility reports still precede shared admission |
| Ordinary authority mutations (P3.3) | #4349 adds exact-identity transactions, deterministic request receipts and replay resolution; independent readers validate their results | API Shared-mode wiring, coordinator constraint interpretation and fleet admission remain required; no public write path is enrolled |
| Durability (P3.1) | #4344 sets snapshot read concern, primary read preference and majority commit; snapshot reads are majority | Mutation identity/receipt and authority envelope, including reviewed lineage transition |
| Runtime propagation (P3.5–P3.11) | API legacy 30-second refresh; #4352 applies a 30-second freshness limit to shared consumer decisions; the observer diagnostic ceiling remains 60 seconds | #4353 supplies bootstrap/reconnect and #4358 acknowledgements; trusted inventory, multi-replica timing proof and integrated lifecycle status remain outstanding |
| Cross-service gates | Statistics maintenance and review eligibility remain deployment-managed; shared reader remains explicit and non-production only | Per-consumer operation/lifecycle proof before runtime enrollment; no implicit initial statistics build |
| Release targeting (E–F) | No general project/user audience writer or evaluator on this baseline | Binding/identity/compatibility safety foundations and accepted grain/precedence contracts; permissions must constrain every allow |
| Browser recovery (B, P3.8–P3.9) | #4353 seeds the service from the exact bootstrap snapshot and refreshes on initial connection, reconnect, online and focus; older responses cannot overwrite bootstrap | Lossless revision transport, context/digest validation and lifecycle active-state reporting remain required |
| Operation capabilities (OD-2) | PR #4354 routes existing inspection through InspectRuntimeFeaturePolicy and mutations through ChangeRuntimeFeaturePolicy; registers separate stop and release capabilities, all initially Administrator | Bound preview/commit and stop/restore routes follow in #4355–#4356; separate browser presentation remains required |
| Emergency-stop transport | #4355 provides a bound, idempotent, EmergencyStop-authorized POST in explicit non-production Shared mode with a decimal-string committed identity | Public restore/preview/commit follows in #4356; production admission and fleet Active acknowledgement remain required |
| Bound preview/commit | #4356 provides read-only previews and transactional digest checks, typed decimal-string receipts and saved-policy restore through a separate bound PUT route | Mutation-status GET follows in #4357; UI, expected inventory, coordinator and production admission remain required; fleet readiness is Unknown |
| Mutation outcome recovery | PR #4357 adds bound, read-only GET mutations/{mutationId} with original committed identity or Unknown (never inferred abort), including later-edit/stop-restore and lineage-corruption coverage |
Non-production Shared only; UI recovery flow and fleet/coordinator readiness remain required |
| Fleet acknowledgement | #4358 adds API/PM acknowledgements, explicit static expected counts and Inspect-authorized fleet status; membership adoption can be Active only with complete fresh evidence | Trusted workload retirement, statistics readiness, admin UI, metrics and multi-replica timing proof remain required; production stays refused |
| Host deployment metadata | #4359 adds default-off API/PM chart metadata derived from pod UID, image, replicaCount and HPA state; raw environment aliases cannot replace it | No mode selection or cross-service expected-count inference; trusted workload collection/retirement remains required |
| Bound consumer admission (B, P3/P5) | #4352 connects explicit non-production Shared mode to API membership and API/PM statistics serving/selectors; every audited decision uses the current 30-second capture | Public mutations follow in #4355–#4356 and reconnect in #4353; lossless browser identity, fleet acknowledgements and coordinator admission remain required |
| Emergency stop primitive (C, P6.6) | #4351 adds distinct audited stop/release actions, digest-bound stop state and safe-state evaluation; stop does not need the new audit indexes or coordinator constraint clearance | Public capability/preview routes, consumer enrollment, fleet confirmation and production mode remain required |
| Deployment safety (B/C, S.1–S.4) | #4360 adds bound, audited constraint record/extend/release with a shared transaction guard and compatible policy edits; no complete authenticated report or coordinator endpoint | Scoped machine transport, trusted report/intersection generation, collector-backed release, administrator-only break-glass routing and production gate remain required |
| Scoped environment reports | #4361 adds a default-off report-only machine route, OpenIddict client definition, Auth0 machine admission and retained sanitized snapshot evidence with 15-minute expiry / 30-day retention | Reports are Incomplete unless a purpose is given and its required sources are Current in Collector mode (#4405); trusted live collectors, remaining domain evidence and coordinator transport remain required; migration verification and bounded revalidation are in #4365–#4366; no credentials or environments configured |
| Deployment constraint report source | #4362 verifies the constraint receipt and lineage in the policy snapshot, projects safe operational fields and reports expiry as ActionRequired; shared-guard source revisions remain lossless | No automatic release or deployment authority; remaining trusted live collectors and domain evidence still block Complete |
| Workload collector boundary | #4363 adds a separate default-off machine publication scope, bound snapshot CAS/replay, pinned pod-image evidence and per-service report freshness | Synthetic runtime/API/Identity tests pass; CI/review pending; no live collector, automatic host retirement or complete checker; optional desired evidence follows in #4364 |
| Retained report retrieval | #4368 adds default-off machine GET by report ID, with exact binding, immutable historical timestamps and retention checks | Human Inspect access/UI and live readiness remain outstanding; retrieval never refreshes evidence |
| Workload acquisition command | #4370 captures two matching explicit-context Kubernetes ownership projections into a bounded sanitized publication payload | Live scheduling/credentials/publication, desired Argo acquisition and rollout/retirement checks remain outstanding; no cluster was queried |
| Lossless browser propagation | #4376 adds canonical decimal-string revisions to API snapshots and exact bootstrap/poll ordering; unsafe legacy numeric revisions are refused | Full lineage/context identity, staleness UX and admin mutation revision transport remain follow-ups |
| Administrator constraint recovery | #4375 adds a separate administrator capability and human-only break-glass HTTP route, exact identity/digest checks and atomic audit/replay | Last-event fleet display and recovery UI remain follow-ups; non-production Shared only |
| Authenticated coordinator API | #4374 adds default-off machine PUT/DELETE, a dedicated single-purpose scope and verified normal release; actor identity is server-derived | Live credential/workflow enrollment remains outstanding; separate administrator recovery follows in #4375; production mode is not enabled |
| Verified coordinator completion | #4373 records immutable target/rollback image expectations and checks fresh collector/host evidence transactionally before normal release | HTTP coordinator follows in #4374 and administrator break-glass in #4375; old internal constraints cannot use verified release |
| Collector-backed fleet inventory | #4372 adds opt-in fresh bound pod-UID reconciliation, preserving static counts and historical acknowledgements; absent older hosts are marked Retired only in the view | Static remains default; autoscaling/statistics remain Unknown; target rollout proof and coordinator release are not implemented |
| Workload publication command | #4371 sends one saved capture with an explicit API origin, binding and machine token; validates the receipt and preserves exact retry bytes | Token provisioning and scheduling remain outstanding; no live publication was performed; receipt success does not establish readiness |
| Explicit workload evidence | #4369 requires ready/terminating presence before publication and when parsing stored payloads; valid false values and exact replay remain supported | Live acquisition, trusted retirement and rollout completion checks remain required |
| Statistics control evidence | #4367 projects the durable singleton mode, epochs and versions in the report snapshot without actor/project content | Fleet maintenance/configuration/protocol evidence and per-project readiness remain unproven |
| Report source consistency | #4366 compares independent source snapshots up to three times, checks end-of-collection freshness and retains one final result | Persistent changes are Unstable; host heartbeats no longer count (#4405); statistics-touching policy still prevents Complete |
| Migration report evidence | #4365 adds fresh read-only catalogue checksum/index verification, sanitized digests and Ready/NotReady/Unknown prerequisites | Current-binary evidence only; #4366 revalidates sources, while remaining domain prerequisites are outstanding |
| Desired deployment provenance | #4364 accepts optional application/resource-version, GitOps revision, chart and rendered-settings digests through the existing collector; matching fresh API/PM evidence projects desired state | Missing, stale or conflicting evidence remains unknown; live collection and remaining prerequisites still block Complete; #4366 adds cross-source validation |
| Browser global lineage (P3.8–P3.9) | #4378 adds Shared schema/identity metadata and sticky page-lifetime rejection of changed lineage or conflicting equal revisions | Personalized context generation/hash, recovery notice and staleness UX remain outstanding; server gates remain authoritative |
| Production admission (P10, OD-8) | #4377 adds default-Legacy production mode, eligible-only runtime decisions, deployment fallback for other keys and transaction-time fixed-fleet checks for enable/restore; #4407 adds server-time collector retirement, used by admission when FleetInventoryMode=Collector |
No activation; rollback preflight, complete compatibility evidence, recovery and freshness trials still required |
| Report completeness by purpose (6A.0) | #4405 adds purpose (policy-change, promotion, legacy-return), per-purpose required sources, Collector-only Complete, heartbeat-insensitive stability, RT-15 legacyParity and appliedMigrations |
Offline checker (6A.1), CI lane (6A.2), live collectors and credentials remain; statistics-touching policy cannot be Complete |
| Release definitions (D) | #4379 adds validated versioned feature definitions and generated C#/TS capability metadata; membership is Developing without release dates; #4450 adds the per-feature permission |
Content manifest remains outstanding; permission-first availability is the 6E.2 row |
| Release Apply (D, RT-20) | #4443 (6D PR A) adds default-off Apply preview and Apply: one transaction commits the fixed flag change through the shared policy path, moves the record to Applying (version CAS), and inserts the PolicyCommitted event and Pending announcement rows from the registry communication defaults (or none, with a quiet reason). Idempotent by release ID; drift needs a fresh preview; admission refuses lifecycle, missing workflow compatibility, unverified required guidance (next row) and incompatible or action-required fleets |
A releasable registry entry is a production prerequisite (project-membership-disable has no compatibleWorkflowVersions) |
| Release confirmation (D, RT-21) | #4444 (6D PR B) adds RuntimeReleaseReconcilerService on the Shared-mode API only: a leased per-release transaction that confirms the stage milestone (confirmedAtUtc from MongoDB $$NOW, ObservedActivation or ObservedAfterRecovery, ActivationConfirmed event with confirmedBy) only when the reviewed capability holds unstopped on the applied lineage and the fleet is Active, and marks an unconfirmed release Degraded or ActionRequired after 60 s. Every decision uses MongoDB time; GET returns the milestones |
Only disableMembership can confirm (other keys read Unknown); no shared environment runs the reconciler |
| Release outbox dispatch (D, RT-23, RT-25) | #4445 (6D PR C) adds a default-off dispatcher (RuntimeFeatureReleases:OutboxDispatch:Enabled) on the Shared API writer: a $$NOW lease, a transactional guard (binding or epoch changed, delivery disabled, capability stopped since Apply, unconfirmed release), the test-sink handler as the only handler, CAS completion, backoff to a kept Failed after 20 attempts, migration 0022 and no TTL |
Real What's New, in-app and public handlers (6G); an operator requeue for Failed rows |
| Release guidance admission (D, RT-26) | #4446 (6D PR E) verifies required guidance at Apply against the compiled server bundle at RuntimeFeatureReleases:GuidanceBundlePath (file-name digest re-verified from content; D-Q6): every topic present, published PublicPreview/PublicReleased, covering the reviewed workflow versions. Fails closed with GuidanceBundleMissing, GuidanceBundleDigestMismatch, GuidanceTopicMissing, GuidancePublicationNotReleased or GuidanceWorkflowNotCovered; the preview and PolicyCommitted event show the state, digest and topics; emergency stops never read it; chart mount off by default |
No bundle is mounted in any environment (production prerequisite P12a). Admission checks topics and publication class, not topic revisions: revision pinning (6G protected revisions), protected delivery to readers (6G) and per-audience guidance for targeted releases (6F) remain |
| Release stop and restore recording (D, RT-24) | #4453 (6D PR D) makes the reconciler read the policy receipts after each release's last observed revision (audit index 0010, 100 per pass) and append one EmergencyStopped, Paused or Restored event per receipt that changes the release's capability (occurredAtUtc from the receipt, Instant, Observed). The record goes Paused, then Applying and Active again with no second ActivationConfirmed; milestones, release ID and announcement ID never change. The stop's transaction cancels unsent in-app notices (0022) under the dispatcher's own AnnouncementCancelled ID, so a notice is cancelled once; a restore never requeues it. The stop path is unchanged |
No shared environment runs the reconciler |
| Release history import (D, RT-27) | #4447 (6D PR F) adds RuntimeReleaseHistoryEntryPoint, a reviewed operator command (SYRF_RUNTIME_RELEASE_HISTORY_MODE preview or apply): a deterministic preview of the exact documents and digest, then one transaction writing an Active record with imported: true and no applied block, and one HistoricalImported event (provenance: "Imported", provenanceDetail source and reviewer, semantics: "Imported"). Date and Month precision are kept, Unknown stays unknown; no policy, audit or outbox write. Exact replay is idempotent; a modified or stale preview, an unknown feature and a taken key are refused. The reconciler never lists, leases or writes an imported record; Apply refuses one |
Written only: not run in any shared environment, and no chart template runs it; features without a definition (for example bulk PDF) need one before their import |
| Release crash recovery (D, RT-22) | #4461 (6D PR G) adds RuntimeReleaseCrashRecoveryTests (replica set, 11 cases over the eight RT-22 boundaries, each with a positive control that the fault fired) using the existing Apply, reconcile and dispatch fault points: each case resumes to one PolicyCommitted, one ActivationConfirmed, one outbox row per dedup key, one test-sink row (or none and one AnnouncementCancelled when a stop lands between confirmation and dispatch), Active, and policy revisions of start plus one per commit. Also the named-only release-lifecycle runtime-fleet scenario: dark deploy, Prepare, Apply, SIGKILL of the API holding the reconcile lease, confirmation, stop and restore through real processes. Test and harness only; no product change |
On main the scenario stops at the Apply preview (WorkflowCompatibilityMissing); the full Bramble run used a rehearsal-only registry entry. Imported records (#4447) are outside RT-22 |
| Release hardening (D, RT-24, RT-23) | #4464 refuses release Apply and its preview with KeyStopped (422 runtime_release_admission, nothing written) while the release's key is under an emergency stop, using the stop test confirmation and the dispatcher share; the stop itself never runs admission (#4456). A malformed outbox row (no or invalid guardPolicy, or a row the item parse rejects) is poison: the reconciler records the stop and Paused anyway and marks the row Failed with lastErrorClass: "MalformedRow" in its own write after the stop commits, and the dispatcher's guard marks it the same way instead of backing off forever; both log it at Error (#4458) |
No shared environment runs Apply, the reconciler or the dispatcher; no operator requeue for a Failed row |
| Feature availability (E, 6E.2) | #4457 (6E PR B) adds GET /api/features/availability and GET /api/projects/{projectId}/features/availability (ProjectViewPolicy), default-off behind RuntimeFeatureReleases:Enabled (404 feature_availability_disabled, nothing read). Permission first through IAuthorizationService (each distinct policy once; unpermitted features omitted, body byte-identical to a registry without them; Project* features omitted without a project), one DecideAll, then RuntimeReleaseStateReader: the latest confirmed or imported-Active record of the captured binding, one bounded projection-only query cached 10 s per binding. Truth table, contextId, RT-60 golden |
seen/dismissed and the PUT (6E.3), the browser client (6E.4), project-aware and personal grains (6F, unknown until then), any discovery surface (6G, 6G.4) |
| Discovery state (E, 6E.3, RT-41 API half) | #4460 (6E PR C) adds PUT /api/account/announcements/{announcementId}/state (own account state, always 204, format-only 400, a change for read-only impersonation) and runtimeFeatureDiscoveryState (RuntimeFeatureDiscoveryStore: one upsert by _id = "{userId:D}:{announcementId}", $$NOW timestamps, first timestamps kept at the same revision, a higher revision replaces, a lower one is ignored, no index or migration). It writes only for an announcement availability reports now (the same cached release read); availability joins the caller's rows by _id $in, so unpermitted features' rows never change the body. Same RuntimeFeatureReleases:Enabled switch (404) |
The browser client and its setAnnouncementState (6E.4); "New to you" rendering (6G, RT-41 rendering half); erasure on account deletion (D27: Chris chose A on 2026-10-23) |
| Release What's New content (G, 6G.1) | 6G PR A (#4466) adds runtime: entries to docs/content/whats-new.yaml (id, featureId, stages?, title, body, topicId; no date), compiled only into server bundle schema 2 (whatsNew with guidePath and a SHA-256 revision); whats-new.json and the public bundle never carry them, and static updates gain a strictly parsed isoDate. The compiler enforces #4382's publication evidence (public class only under user-guide/, restricted audienceId == featureId, no Retired historical-public claim, D40); RuntimeReleaseGuidanceBundle verifies schema 1 or 2; Apply and its preview refuse WhatsNewEntryMissing when an announced release has no entry for its feature and stage (a quiet Apply is admitted, a stop never reads the bundle); the validator checks entries and coverage. project-membership-disable has its entry and its content points at the members-groups section |
Disclosure through availability and the real handlers (6G PR B); rendering (PR D); a bundle in the API image (PR G, D38) |
| Release read routes (G, 6G.11 API half) | 6G PR AD2 adds GET /api/feature-releases (newest first, 50 a page, before cursor, featureId/status filters) and GET /api/feature-releases/{releaseId}/events (recorded order, 200 at most), and an additive announcements block on GET /api/feature-releases/{releaseId} from the outbox. Read-only; InspectRuntimeFeaturePolicy or ManageFeatureReleases; list and events 404 behind RuntimeFeatureReleases:Enabled (nothing read); every query bound to the verified environment and epoch and hinted to 0012/0021/0014/0022; no migration; goldens and catalogue entries |
The Releases screen itself (6G.11 web half, AD2 web), Ready to settle (6G.12) |
| Browser safe fallback (6B.4 G3, RT-14) | 6B.4 PR 4 generates RUNTIME_SAFE_VALUES (production-eligible contracted keys only) and adds the deployment-static web setting browserRuntimeSafeFallback (Helm runtimeFeatureFlags.browserSafeFallback, default false): when on, a failed bootstrap shows those safe values until the first accepted snapshot; a later failure keeps the last snapshot. Nothing is persisted; on → off → on is unit-tested, and the E2E case is runtime-fleet-browser-recovery.spec.ts |
Enabling it in any environment is a separate reviewed GitOps change; no environment sets it |
| Release content (D/H) | Static feed and public guide remain | Confirmed milestone projection and permission-first protected content, after current safety priority |
Promotion assessment: not established by these changes. Legacy production mode must preserve its existing behavior; accepting runtime production policy or targeted releases remains blocked on the named B/C prerequisites and separate operational approval. Existing flags that are deployment-managed must not be represented as working live controls. The consumer manifest inventories code seams, not complete per-flag lifecycle certification; the per-flag/service audit remains required before enrollment.
Targeting policy is already decided: plan §11 / OD-13 uses constraints → emergency safe-state stops → exclusions/consent → combined, user, investigator and project allows → cohort → environment override → accepted baseline, with dependencies evaluated in the same trusted context. Project-grain behavior cannot vary by member; personal targeting is restricted to private UI experiences. Baseline Off is not an emergency stop. No new semantics are introduced by #4344.
Audited runtime consumer contracts¶
PR #4350 records the initial audited subset in YAML runtimePolicy.consumerContract. The generated
catalog includes owner, grain, safe value, host list, decision boundary, maximum age, failure
tolerance, production eligibility and this evidence link in its digest. #4352 adds the three existing PM
consumer selectors to preserve their behavior during authority cutover. 6F PR B (#4472) adds the first
grain: Project contract, the targeting pilot. The other 48 catalog entries have no reviewed consumer contract:
they remain unreviewed and cannot receive an authoritative evaluation. Catalog eligibility is not deployment
readiness or authorization to activate a production control.
| Key | Hosts / boundary | Safe state and retained data | Evidence / eligibility |
|---|---|---|---|
disableMembership |
API only; new disable-request admission | False hides the disable route. Existing membership status/history is retained; re-enabling remains available and permission-checked while the flag is off | Route and permission boundary, routing/on-off-on tests. Production candidate accepted by OD-1; no PM consumer is claimed |
materializedProjectStatisticsServing |
API and PM; new materialized-read admission | False stops materialized serving. Maintenance, family gates, persisted projections and durable lifecycle controls stay separate; enabling still requires deployed serving and writes | PM adapter, prerequisite/failure tests, API propagation tests. Non-production only in this contract |
| materializedProjectStatisticsPages | API and PM; new page-read admission | False removes materialized page selection; global serving, family and durable readiness remain prerequisites; projections and maintenance are retained | Authority PM adapter, stop/failure/selector tests. Non-production only |
| materializedProjectStatisticsStageOverview | API and PM; new Stage Overview read admission | False removes the Stage Overview selector; pages, annotation family and durable readiness still constrain it | Same adapter/tests; existing legacy selector round trip. Non-production only |
| materializedProjectStatisticsSignalR | API and PM; new statistics invalidation consumption | False removes materialized invalidation selection; existing authorization and durable eligibility remain; no retained state is erased | Same adapter/tests and legacy selector round trip. Non-production only |
| screeningKeywordHighlighting (Project grain; pilot pending D33, built to recommendation A) | API only; new request admission (the keyword-list route gate and, from 6F PR C, the project-context gate) | False hides the keyword-list route and highlighting. Stored keyword lists are retained; a project rule only selects which projects get the existing behaviour. Project rules are stored in the policy document (schema 2); with none, the document stays schema 1 | Route gate, write path and refusals, HTTP and switch tests, pilot golden and Shared-mode value parity. productionEligible: false: production stays deployment-controlled. No PM consumer is claimed |
All six contracts permit decisions only from observations younger than 30 seconds (monotonic time), with zero tolerated read failures.
Pilot compatibility event (6F PR B). Adding the pilot's contract changes the catalog digest (every stored identity
names the new digest after deployment, as for any catalog change) and moves the key's decisions in Mode=Shared only:
from the API's compatibility values to the bound authority. With no rule saved, Legacy and Shadow mode decide
byte-identically to main on both hosts (all 48 non-Shared worlds of the parity table), and in Shared mode the value
equals the API's compatibility projection whenever the policy is fresh, outside and in production Runtime (where it is
deployment-controlled); a stale or unavailable policy now gives the contract's safe value, off. Project Management in
Shared mode reports the key UnsupportedHost (safe value off); PM has no consumer of it. No deployed environment runs
Mode=Shared. Evidence: ThePilotDecidesAsOnMainOutsideSharedModeAndAsItsReviewedGoldenInside and
ThePilotsSharedModeValueEqualsTheApiCompatibilityProjectionWithNoRules. The existing diagnostic observation may remain Ready for up to 60 seconds;
that does not authorize a decision older than its catalog limit. The authenticated local evaluation
endpoint exposes the distinction. Explicit non-production Shared mode enrolls the audited consumers in
4352; the default remains Legacy and no actual environment has been enrolled. The accepted baseline supplies values;¶
current deployment constraints remain a separate input and cannot be overridden by imported policy.
Summary¶
| Status | Count |
|---|---|
| Done | 5 |
| Partial | 19 |
| Missing | 32 |
| Total rows | 56 |
The gaps that matter most for the plan:
- Production does not run any of this code. Production's API and PM run
104eca9, which has no runtime-flag store, controller, shared reader or migration entry point (row X.1). Everything below describesmain, which production reaches only through the CD plan's catch-up promotion. - On
main, production ignores stored policy on both the read and the write side, and the shared reader refuses to start in production. Enabling production policy is three code changes plus a deployment mode, not one controller check (rows P10.1–P10.3). - No authority envelope. The policy document has
_id,revision,overridesandupdatedAtUtconly: no environment identity, lineage (storeEpoch), catalog or baseline digest, or mutation ID (P3.2, P3.3). - No fleet acknowledgement. Nothing records which host applied which revision, and nothing knows how many hosts there should be, so "Active" cannot be computed (P3.10).
- Propagation is polling only (API 30 s, shared reader 15 s). The legacy API provider keeps its last values forever on failure. The shared reader fails closed on the first failed, missing or inconsistent read; its 60 s age limit applies only if refreshes stop altogether (P3.4, P3.5).
- The migration entry point never checks which environment it is writing to. It takes the database name from configuration and writes. The pre-deploy Job is enabled in no environment, and previews cannot enable it in its current form (E.1, E.2).
- Nothing exists yet for release records, outbox, availability, discovery or protected content (rows R.1–R.10).
- No compatibility report or production machine identity. Staging has OpenIddict and a single-scope machine-client precedent; production is still on Auth0 with no machine client accepted by the API (S.1–S.4).
P1: generated contract and write safety¶
| # | Requirement | Status | Evidence on main |
Gap |
|---|---|---|---|---|
| P1.1 | .NET catalog, defaults and metadata generated from the YAML schema | Partial | RuntimeFeatureFlagCatalog.cs is a handwritten allow-list (Flag(...)/ReloadRequired(...), lines 102–139, 273–282); a catalog test compares it with env-mapping.yaml (operating guide, "Step 2") |
Not generated. Owner, scopes, stale behaviour and grain are not declared anywhere |
| P1.2 | Machine-checked consumer manifest; CI fails on a new unclassified read | Done | consumer-manifest.json (141 consumers), checked by pnpm run validate:generated (guide) |
Static evidence only, as the guide says |
| P1.3 | Expected revision mandatory on every mutation route | Done | Controller RuntimeFeatureFlagsController.cs:115-116; store EnsureExpectedRevision RuntimeFeatureFlagStore.cs:451-457; group path :99-100 |
None |
| P1.4 | Activation class and lifecycle exceptions declared per flag | Partial | RuntimeFeatureFlagActivation has immediate and pageReload only (RuntimeFeatureFlagCatalog.cs:28-32) |
No stale-state behaviour, allowed scopes, safe state or production eligibility |
P2: shared snapshot seam¶
| # | Requirement | Status | Evidence on main |
Gap |
|---|---|---|---|---|
| P2.1 | One immutable snapshot per process, captured once per operation, in API and PM | Partial | SharedRuntimeFeatureFlagReader publishes a frozen SharedRuntimeFeatureFlagSnapshot and Capture() reads it once (SharedRuntimeFeatureFlagReader.cs:16-20, 56-63); used by API FeatureGatedAttribute.cs:52-53 and PM SharedProjectStatisticsFlagSource.cs:34-46 |
It reads the legacy override document only. The legacy API provider still writes values into the mutable FeatureFlags singleton (RuntimeFeatureFlagProvider.cs:83-146) and is what most API consumers use |
| P2.2 | Minimum diagnostic endpoints | Done | API GET /api/runtime-feature-flags/provider-status (RuntimeFeatureFlagsController.cs:65-88); PM GET /api/runtime-feature-flags/reader-status (RuntimeFeatureFlagReaderController.cs:10-16); diagnostic health check (SharedRuntimeFeatureFlagRegistration.cs:74-104) |
Local process only, labelled as such |
| P2.3 | Shared evaluator (dependencies, compatibility, statistics constraints) usable by every host | Missing | Normalisation lives in the API store (RuntimeFeatureFlagStore.cs:337-391); the shared snapshot's Evaluate is "override or deployed default" (SharedRuntimeFeatureFlagReader.cs:19) |
PM cannot evaluate dependencies; a dependency-normalised value differs between API and PM |
P3: usable propagation slice¶
| # | Requirement | Status | Evidence on main |
Gap |
|---|---|---|---|---|
| P3.1 | State and audit committed atomically | Done | WithTransactionAsync around ReplaceOneAsync + InsertOneAsync (RuntimeFeatureFlagStore.cs:88-92, 112-115, 182-186) |
Write and read concern are not explicit at the call site; staging's cluster address carries no w=majority option (cluster-gitops staging.values.yaml:17) |
| P3.2 | Authority envelope: environment ID, storeEpoch, catalog, baseline and policy digests |
Missing | BuildState writes _id, revision, overrides, updatedAtUtc (RuntimeFeatureFlagStore.cs:489-496). The shared reader rejects any document with storeEpoch (SharedRuntimeFeatureFlagReader.cs:112) |
Whole envelope, plus the reviewed legacy-to-lineage migration (README §3) |
| P3.3 | Mutation ID resolves a lost response without a second mutation | Missing | RuntimeFeatureFlagMutation has value, revision, reason and dependency fields only (RuntimeFeatureFlagModels.cs:57-62). The admin UI refreshes instead of replaying (guide) |
Mutation ID, unique index, receipt lookup |
| P3.4 | Change-stream watch with race-free bootstrap | Missing | Flags poll only: API PeriodicTimer(30 s) (RuntimeFeatureFlagProvider.cs:172), shared reader 15 s (SharedRuntimeFeatureFlagRegistration.cs:69). Change-stream code exists for other collections: MongoContext.GetCachedCollectionChangeStream (MongoContext.cs:180-228) and the realtime ChangeFeedSource.cs:69-77 |
Not wired to policy. See the freshness assessment for whether it is needed |
| P3.5 | Periodic reconciliation; a stale observation fails closed for risky admission | Partial | Shared reader: 15 s poll, 5 s read deadline; a single failed, missing or equal-revision/different-digest read sets Unavailable or Inconsistent at once, and Evaluate returns no decision for any state but Current (SharedRuntimeFeatureFlagReader.cs:65-69, 84-104); MaximumAge 60 s marks an observation Stale only when refreshes stop (:45, 56-63) |
Legacy API provider logs and keeps the last values indefinitely (RuntimeFeatureFlagProvider.cs:177-199). No per-flag stale behaviour |
| P3.6 | Revision regression and equal-revision/different-digest detected | Partial | Shared reader marks Inconsistent (SharedRuntimeFeatureFlagReader.cs:90-95) |
Legacy provider accepts an equal revision with any content (RuntimeFeatureFlagProvider.cs:56); so does the browser (runtime-feature-flags.service.ts:111) |
| P3.7 | Deployment-controlled provider mode with parity-checked rollback | Partial | RuntimeFeatureFlags:SharedReaderMode = Disabled/Shadow/Membership/StatisticsServing (SharedRuntimeFeatureFlagRegistration.cs:24-40) |
README proposed ProviderMode=Legacy|Shadow|Shared; the name differs. Rollback parity is a manual runbook step (guide), not a check |
| P3.8 | Browser bootstrap seeds the service with the same envelope | Missing | main.ts:284-291 applies the snapshot; the service's counter starts at -1 (runtime-feature-flags.service.ts:74) |
A slower first poll can apply an older revision than bootstrap did |
| P3.9 | Browser recovery and boundaries: reconnect fetch, revision as a decimal string, context generation, safe fallback | Partial (safe fallback: Done behind a default-off setting; 6B.4 G3) | Five-minute poll plus SignalR hint, 3 s timeout, two retries (runtime-feature-flags.service.ts:32, 86-99). 6B.4 G3: with the deployment-static browserRuntimeSafeFallback on, a failed bootstrap applies the generated RUNTIME_SAFE_VALUES (safe value of each production-eligible contracted key, today disableMembership: false) until the first accepted snapshot (applyRuntimeFeatureFlagBootstrapFailure, called from main.ts) |
Reconnect and decimal revisions came later (#4353, #4376); no context generation. The setting is false everywhere, so until a reviewed rollout a failed bootstrap still keeps deployed defaults, which is permissive once production can stop a default-On flag |
| P3.10 | Fleet acknowledgement against an expected-host inventory | Missing | The guide says there is no fleet registry (manage-feature-flags.md, provider-status paragraph). syrf_metadata records instances and versions (DatabaseMetadataWriter.cs:160-168), not applied policy |
Host acknowledgement store, expected inventory, Active/Degraded computation |
| P3.11 | Each API pod notifies its own SignalR clients after it applies a revision | Done (Shared mode, per pod; 6B.4 G1) | In Mode=Shared each API pod's RuntimePolicyAuthorityRefreshService notifies IRuntimePolicyRevisionListener once per newly observed identity (never for a successful start-up read; after a failed one, the first identity read later is hinted), and RuntimePolicyBrowserHint sends RuntimeFeatureFlagsRevision to that pod's own connections. The writing pod is not hinted at commit time: a Shared commit, stop or restore wakes its own refresh (IRuntimePolicyRefreshNudge, #4425), so the hint follows once its reader can serve the new revision. Still no backplane |
Legacy mode keeps the writing-pod-only hint. Browser timing evidence on two pods is 6B.4 PR 3 |
P4: authoritative baseline import¶
| # | Requirement | Status | Evidence on main |
Gap |
|---|---|---|---|---|
| P4.1 | Reviewed import of an accepted baseline; pods never reseed | Missing | Deployed values come from configuration at startup (RuntimeFeatureFlagCatalog.cs:262-271, store constructor :67) |
Baseline bundle, digest, import command |
| P4.2 | Clearing an override previews the baseline it reveals | Partial | The admin page shows a Deployed column and a Default action (guide) | No side-effect-free preview; the revealed baseline is per-pod configuration |
P5 and P5a: consumers¶
| # | Requirement | Status | Evidence on main |
Gap |
|---|---|---|---|---|
| P5.1 | Migrate one host and flag family at a time with captured reads | Partial | API disableMembership route gate (FeatureGatedAttribute.cs:52-53); PM statistics serving and four consumer selectors (SharedProjectStatisticsFlagSource.cs:34-46) |
Every other consumer is on the legacy provider or startup FeatureFlags. README §13 lists only the membership slice; the PM StatisticsServing mode is newer |
| P5.2 | signalRActive backend/browser lifecycle audit |
Missing | Still special-cased: pinned to deployment while reviewer tracking is on (RuntimeFeatureFlagModels.cs:151, 163-168) |
Audit and classification |
| P5.3 | Grouped controls (not in the roadmap) | Done | RuntimeFeatureFlagGroups.cs; atomic group mutation RuntimeFeatureFlagStore.cs:95-118 |
Ahead of the roadmap; keep |
| P5a.1 | "Use precomputed statistics" operable at runtime in API and PM | Partial | StatisticsServing mode in PM; group statistics-consumers-v2 (guide) |
Non-production only; stops reads but cannot start deployment-disabled serving |
| P5a.2 | "Calculate and maintain statistics" as a coordinated runtime transition | Missing | Maintenance gates pinned to deployment (RuntimeFeatureFlagProvider.cs:71-74, ProjectStatisticsRuntimeActivation.cs) |
Whole requirement; stays its own track |
P6: audit and administration depth¶
| # | Requirement | Status | Evidence on main |
Gap |
|---|---|---|---|---|
| P6.1 | Audit history read API and UI | Missing | Audit is written (RuntimeFeatureFlagStore.cs:72, 184-186) and never read by any endpoint |
Paginated, authorised history |
| P6.2 | Side-effect-free preview with a plan digest | Partial | The 422 dependency plan and exact confirmed set (RuntimeFeatureFlagStore.cs:238-253) |
No preview endpoint, no digest binding revision, catalog and actor |
| P6.3 | Restore as a forward revision from history | Partial | Default/clear exists | No restore-from-history |
| P6.4 | Real-actor authorisation under impersonation; both identities audited | Partial | Audit records real and impersonated identities (RuntimeFeatureFlagsController.cs:162-174, RuntimeFeatureFlagStore.cs:516-521). But the authorisation decision uses currentUser.GetUserId(), which returns the impersonated user (CurrentUserService.cs:26-31), and under support impersonation the claim groups are empty (:48-50). Authorisation is the target's authority, safe today only because impersonation cannot escalate |
Decide which identity is authorised for production policy; plan RT-13 asserts it per impersonation path |
| P6.5 | Separate capabilities: change policy, emergency stop, inspect targeting | Partial | One capability, ApplicationCapability.ManageRuntimeFeatureFlags, mapped to Administrator (ApplicationAuthorityPolicy.cs:42, 85) |
Split capabilities |
| P6.6 | Emergency stop distinct from Off | Missing | The UI offers On, Off and Default only (guide) | Stop semantics, safe state, separate permission |
P7–P10¶
| # | Requirement | Status | Evidence on main |
Gap |
|---|---|---|---|---|
| P7.1 | Project-scoped allow/deny | Missing | The guide: "This MVP deliberately does not include percentage or audience targeting" | Whole phase |
| P8.1 | User, investigator and combined targeting; beta consent | Missing | Same | Whole phase |
| P9.1 | Stable basis-point cohorts | Missing | Same | Whole phase |
| P10.1 | Production mutation permitted under an explicit policy | Missing | IsMutationAllowed allow-lists non-production environment names (RuntimeFeatureFlagStore.cs:467-487); the controller returns 403 (RuntimeFeatureFlagsController.cs:109-110) |
Deployment-controlled production mode (see ADR-025) |
| P10.2 | Production reads honour stored policy | Missing | BuildEffectiveValues ignores overrides when mutation is not allowed (RuntimeFeatureFlagStore.cs:352); descriptors hide them (:199-202) |
The proposal mentions only the mutation refusal; the read side must change too |
| P10.3 | Shared reader runs in production | Missing | Enabled modes throw unless the environment is in a non-production list (SharedRuntimeFeatureFlagRegistration.cs:31-36) |
Same production mode switch |
Administrator restore (PR #4385)¶
The Shared page now exposes the existing previewed ReleaseStop action for disableMembership and renders before/after stop state. It preserves saved overrides, requires confirmation and retains the original request through ambiguous outcomes. An unresolved local emergency stop blocks both restore preparation and confirmation. This completes the basic operator controls, not B/C's hermetic convergence, recovery or promotion-readiness evidence. No environment is activated and no release milestone is inferred.
Administrator emergency stop (PR #4384)¶
The Shared administrator page now prepares and confirms disableMembership stops using the dedicated
endpoint, independently of unresolved ordinary changes. It retains the original request through unknown
outcomes and resolves/retries that receipt. The stopped catalog can refresh without replacing an unresolved
ordinary mutation. This advances the B/C operator flow; restore controls, durable browser recovery and
RT-12/RT-40's multi-replica timing evidence remain unimplemented. No environment is activated.
Administrator fleet inspection (PR #4383)¶
The Shared administrator page now exposes on-demand fleet inspection against the full displayed policy identity. It renders expected counts, host observations and the server adoption state, clears failed or superseded observations, and expires observations after 15 seconds. This advances the B/C operator view; it does not prove RT-11's multi-replica convergence or provide emergency-stop controls. No actual fleet, environment binding or production configuration is changed.
Content artifact update (PR #4382)¶
R.7/R.8 now have an offline versioned-manifest compiler on the stacked branch. Public artifacts omit private/draft articles, search entries and assets; retired content retains an explicit historical reader class. Content-addressed revisions and separate output trees make artifact inspection reproducible. The checked-in public membership overview is background guidance, not an activation or compatibility claim. RT-28's build-separation portion has synthetic tests; protected API requests, historical revisions and renderer integration are still missing. Preparation guidance remains Unverified until delivery and workflow compatibility are proven. No public site, private pilot or environment has been published or enrolled.
Shared administrator flow update (PR #4381)¶
The stacked implementation connects individual admin controls to a server-generated Shared preview, explicit confirmation, exact commit and mutation-outcome lookup. Unknown outcomes retain the original request on the open page; no new mutation can replace it. Legacy behavior remains unchanged and unsupported Shared group writes are refused in the UI. Saved is distinguished from fleet adoption. This advances P6's operator surface but does not complete B/C: emergency-stop UI, fleet observation, durable browser recovery and the specified multi-replica exercises remain outstanding. No production enrollment is included.
Prepared release implementation update (PR #4380)¶
The historical main evidence below remains the original audit. On the stacked implementation branch,
R.2 now has default-off global preview/prepare/read routes, atomic record and Prepared event persistence,
exact mutation replay, environment/store lineage checks and an additive 0021 logical-release-key index.
Permission checks precede mode/body/data access. Preparation preserves policy identity and values;
turning it off retains readable bound history. Scheduled time is intent and no confirmed milestone is set.
R.2 remains partial: Apply/reconciliation and subsequent lifecycle events are missing. R.3–R.8 remain
open; preparation grants no availability or content access. See the
preparation contract.
Release Apply update (PR #4443, 6D PR A)¶
R.2 gains Apply (RT-20): the policy commit, the record's move to Applying, the PolicyCommitted event and the
announcement rows share one transaction, and a fault at each of the three Apply points writes none of them
(RuntimeReleaseApplyTests, replica set). R.3 is partial: Apply writes Pending outbox rows with the plan's
deduplication key, but nothing dispatches them yet. Confirmation (ConfirmedAtUtc), stop and restore events and
historical import remain open. Default-off behind RuntimeFeatureReleases:Enabled; see
Applying a release.
Release confirmation update (PR #4444, 6D PR B)¶
R.2 gains confirmation (RT-21): the reconciler sets milestones.{stage}.confirmedAtUtc from the transaction's MongoDB
time, once, only when every expected host is Current for the current policy and the committed capability still
holds on the same lineage (RuntimeReleaseReconciliationTests, replica set: Behind, Unknown, Incompatible, missing
host, missing inventory, reversed capability, stop and foreign epoch never confirm; a ±5 min API clock skew changes
nothing; two reconcilers confirm once; a crash before commit writes nothing). Not separately flagged: it converges
applied work and is absent in Legacy mode. Stop and restore events and historical import remain open; see
Confirming activation.
Release guidance admission update (PR #4446, 6D PR E)¶
R.2's admission now verifies required guidance instead of refusing it outright (RT-26): Apply reads the compiled server
bundle from a configured read-only path, re-verifies its content-addressed digest, and refuses with a stable reason
and no write when the bundle is absent, does not verify, lacks a topic, has no article published for everyone, or does
not cover the reviewed workflow versions. The emergency stop of the same flag still commits with no bundle
(RuntimeReleaseGuidanceAdmissionTests, replica set). The C# digest agrees with the compiler's on a fixture the real
compiler generates (RuntimeReleaseGuidanceBundleTests, compile-release-content.test.ts). Not separately flagged:
it sits behind RuntimeFeatureReleases:Enabled and only tightens Apply. See
Applying a release.
Release outbox dispatch update (PR #4445, 6D PR C)¶
R.3 gains a consumer: the outbox dispatcher leases a due row against MongoDB time, guards it in a transaction and
completes it by compare-and-set on the lease token. RT-23: a stop committed between lease and guard cancels the in-app
notice with one AnnouncementCancelled event and nothing delivered, and the notice stays cancelled after a restore.
RT-25: staging reaches only the test sink, previews and local stacks end DeliveryDisabled, and a restored copy ends
BindingChanged. Rows are kept permanently (no TTL; migration 0022 indexes {releaseId, state}). Evidence:
RuntimeReleaseOutboxDispatchTests (replica set), Migration22_IndexesTheOutboxByReleaseAndStateWithoutExpiry and the
0011–0022 Up/Down/Up and adoption tests. R.3 remains partial: real handlers arrive in 6G. Default-off behind
RuntimeFeatureReleases:OutboxDispatch:Enabled; see
Dispatching release announcements.
Release stop and restore recording update (PR #4453, 6D PR D)¶
R.2 gains stop and restore history (RT-24): confirm, stop, restore and confirm again keeps the release ID, the stage
milestone's confirmedAtUtc and requestedAtUtc and the announcement ID byte-identical, with exactly one
ActivationConfirmed, one EmergencyStopped (or Paused, for a change to off) and one Restored. A stop and a restore
inside one reconcile period are both recorded in revision order; an unconfirmed release that is stopped goes Paused
instead of staying Applying, and is confirmed after the restore. The emergency stop commits with the release, event
and outbox indexes dropped, the dispatcher throwing and the reconciler stopped, and the record catches up afterwards.
R.3: a stop cancels the unsent in-app notice once, with one AnnouncementCancelled, whether the reconciler or the
dispatcher sees it first, and a restore does not requeue it (D-Q5). Evidence: RuntimeReleaseStopRestoreTests and the
updated RuntimeReleaseReconciliationTests (replica set). Not flagged: part of the reconciler, and the stop path does
not change. See Stop and restore recording.
Feature availability update (PR #4457, 6E PR B)¶
6E.2 adds the first reader of release records outside the release writers: RuntimeReleaseStateReader (read-only, in
SyRF.API.RuntimeAdministration, enrolled in RuntimeReleaseCollectionsForwardRuleTests) and the availability routes
in SyRF.API.Endpoint/FeatureAvailability/. A dropped feature is never decided or looked up
(FeatureAvailabilityEvaluatorTests, recording fakes); a confirmed record of a foreign binding epoch or environment,
and Legacy mode, leave a Developing feature notReleased (RuntimeReleaseStateReaderTests, replica set, and
FeatureAvailabilityHttpTests); 50 concurrent callers share one release query per 10 s. The HTTP suite drives the real
authorization handler with the e2e users (administrator as the positive control) and compares an absent and a deleted
project with another ProjectView route. Release records are read in the shapes Apply, the reconciler, the historical
import (#4447, imported: true, Unknown precision) and stop recording (#4453, Paused after confirmation) write; the
last two are document fixtures until those PRs merge. Default-off behind RuntimeFeatureReleases:Enabled; see
Feature availability.
Release history import update (PR #4447, 6D PR F)¶
R.2 gains historical import (RT-27): a reviewed command records a past release as an Active, imported record with
no applied block and one HistoricalImported event, keeping Date or Month precision and leaving Unknown
unknown, with the source and reviewer (RuntimeReleaseHistoryImportTests, replica set: precision, provenance, outbox and
policy unchanged, deterministic preview, modified, stale, unknown-feature and duplicate-key refusals, idempotent replay;
RuntimeReleaseHistoryEntryPointTests through the API entry point after every migration). A reconcile pass leaves an
imported record untouched while it confirms an applied one in the same pass. Not flagged: like initialise, it does
nothing unless an operator runs it, and it is written only (runbook 11). See
Importing past releases.
Discovery state update (PR #4460, 6E PR C)¶
6E.3 adds the first per-person runtime release store: RuntimeFeatureDiscoveryStore (in SyRF.API.RuntimeAdministration,
the only source naming runtimeFeatureDiscoveryState, guarded by RuntimeReleaseCollectionsForwardRuleTests) and
AccountAnnouncementsController in SyRF.API.Endpoint/FeatureAvailability/. The PUT finds the announcement through
IRuntimeReleaseStateReader.FindAnnouncementAsync, which answers from the same cached per-binding read as availability,
so there is no second evaluation path. Evidence: FeatureAvailabilityHttpTests (RT-41 API half with the e2e users and a
later joiner; an unknown, quiet, imported, other-stage and foreign-binding announcement and Legacy mode each 204 with no
row, a real one writes one; a format-only 400; a reviewer's stored dismissal of an unpermitted feature leaves their body
byte-identical; on, off, on with rows written while on), RuntimeFeatureDiscoveryStoreTests (replica set: shape,
$$NOW, 24 concurrent identical writes, revisions, the $in read), FeatureAvailabilityEvaluatorTests (join and
failure), SupportImpersonationContractTests (read-only refuses, an edit context writes the target's row) and the
endpoint catalogue. Default-off behind RuntimeFeatureReleases:Enabled; see
Seen and dismissed announcements.
Release crash recovery update (PR #4461, 6D PR G)¶
With historical import (#4447) merged, R.2 is Done for isolated environments: intent, record and append-only events
cover Prepare, Apply, confirmation, stop, restore and import, and RT-22 shows a crash at each lifecycle
boundary resumes with one event per milestone (RuntimeReleaseCrashRecoveryTests, replica set). The cases are: a
process lost after Prepare; each of the three Apply fault points; a lost Apply response; a fault inside confirmation,
with and without a gap of more than three periods (ObservedAfterRecovery); a reconciler killed after its commit; a
dispatcher dead after its lease or after its handler; and a stop between confirmation and dispatch. R.3 is Done for
the test sink and Partial overall: one row per dedup key, delivered once or cancelled once whatever the crash, kept
permanently. Real What's New, in-app and public handlers (6G) remain. The release-lifecycle runtime-fleet scenario
(e2e/harness/runtime-fleet/release-lifecycle.mjs, named-only) runs the same lifecycle through two API and two PM
processes on a local binding, and SIGKILLs the API that holds the reconcile lease. On main it stops at the Apply
preview, because the registry's only feature is not releasable (WorkflowCompatibilityMissing). Its Bramble evidence
used a rehearsal-only registry entry in a scratch clone. That run passed 28 of 28 assertions: the API holding the lease was
SIGKILLed with the lease still on the record, and the survivor confirmed once after the lease expired by MongoDB's
clock. Test and harness only: no product code, flag or migration changes. See Release lifecycle (walkthrough).
Release hardening update (PR #4464)¶
Two fixes from the 6D reviews. R.2: Apply can no longer commit a policy change the kill switch keeps off, so stop and
restore history keeps its assumption that the capability held at the Apply revision (#4456). Evidence:
ApplyUnderAStopOfItsKeyIsRefusedKeyStoppedWritesNothingAndAppliesOnceRestored, replica set: the preview reports
KeyStopped, Apply is refused and writes nothing, the stop commits, and after a ReleaseStop restore a fresh preview
applies once. Also AdmissionRefusalIsUnprocessableWithItsStableReason (422 body). R.3: a malformed outbox row no longer
blocks stop recording or retries forever (#4458). Evidence: AMalformedNoticeIsMarkedFailedOnceAndTheStopIsStillRecorded
and TheGuardMarksAMalformedRowFailedInsteadOfBackingOffForever, replica set, each over three malformations next to a
well-formed notice. Not separately flagged: both only tighten paths already behind RuntimeFeatureReleases:Enabled and
RuntimeFeatureReleases:OutboxDispatch:Enabled (default off), and the emergency stop path does not change. See
Applying a release and
Dispatching release announcements.
What's New disclosure update (PR #4474, 6G PR B)¶
R.3 gains its production handlers: RuntimeReleaseWhatsNewHandler (MaterialiseWhatsNew) and
RuntimeReleaseInAppNoticeHandler (NotifyInApp), registered beside the test sink, one per kind and audience. Both are
idempotent no-ops: the dispatcher's completion of the row is the record. Under a production-delivery binding fixture both
rows complete once with result.testSink: false; under testSink the notice reaches only the sink; under disabled it is
cancelled DeliveryDisabled; the What's New row completes under all three
(ProductionDeliveryCompletesBothRowsOnceThroughTheProductionHandlers,
OutsideProductionTheNoticeIsNeverDeliveredButWhatsNewStillCompletes, replica set). R.3 is Done for in-app delivery in
isolated environments; the public projection (6G PR F) remains.
R.6: availability now discloses runtime What's New text, only to people the feature is available to, only after the
release's MaterialiseWhatsNew row is Done, and only when the current verified bundle has an entry for the stage. The
release-state reader joins the announced records' outbox rows by _id in the same cached read (one release query and one
outbox query per 10 s per process), and the bundle is read through a cache keyed by path, length and last-write time
(re-checked at most every 10 s, so no request hashes the file). The response gains rehearsal,
announcement.delivered and whatsNew. Evidence: FeatureAvailabilityHttpTests (a reviewer, a non-member and the
no-project route get bodies byte-identical to a registry without the feature; the project administrator gets the entry),
WhatsNewWaitsForTheOutboxHidesOnAStopAndReturnsOnRestoreWithoutANotice, FeatureAvailabilityEvaluatorTests and
RuntimeReleaseStateReaderTests (50 concurrent callers, one query of each kind per period). The announced-record defect
is fixed: a record is announced only with mode Default and at least one queued kind. #4473: the matched entry's
revision and the bundle digest are bound into the Apply preview digest and recorded on PolicyCommitted (whatsNew
block), and a release prepared with a bundle digest is pinned to it for What's New as for guidance. Rendering remains
(6G PRs C and D). Not separately flagged: behind RuntimeFeatureReleases:Enabled (reads) and
RuntimeFeatureReleases:OutboxDispatch:Enabled (handlers), both default off; no migration, no new collection. See
What's New and in-app notices.
Proposal elements¶
| # | Element | Status | Evidence on main |
Gap |
|---|---|---|---|---|
| E.1 | Environment binding verified before any migration write | Missing | MigrationEntryPoint.RunAsync reads ConnectionStrings:MongoConnection:DatabaseName and opens it (MigrationEntryPoint.cs:124-131); RuntimeEnvironment is used only for metadata (:138-141). syrf_metadata.meta.environment is a best-effort SetOnInsert from any pod start (DatabaseMetadataWriter.cs:239-246) |
Binding record and pre-write check (plan §6.3) |
| E.2 | ADR-023 migrations can provision the release stores | Partial | Runner, lock file and PreSync Job template exist; latest PM migration is 0008 (PmMigrations.cs:115-119, migrations.lock.json). Only CreateIndexMigration and UnsetFieldsMigration kinds exist |
The Job is enabled in no environment (no migrations.enabled in cluster-gitops; pre-deploy plan M22): staging needs pre-deploy R1, production its catch-up enablement, previews the DatabaseLifecycle switch to up (the template refuses migrations.enabled with preview.waitForDatabase, _migrations-job.tpl:39). A second catalogue in the same database would see the first one's syrf_migrations records as unknown (MigrationRunner.cs:83, 151). Expected binding values cannot come through env:, which the Job strips of SYRF_MIGRATION_* (_migrations-job.tpl:51-55) |
| E.3 | Global policy identity separate from release ID | Missing | Only a global revision exists |
Release ID |
| R.1 | Feature definition with stable identity, separate from flag keys | Missing | Closest: RuntimeFeatureFlagGroup(Key, Name, Description, Keys) (RuntimeFeatureFlagGroups.cs:6) |
Feature catalog |
| R.2 | Release intent, record and append-only events | Missing | None | Whole store |
| R.3 | Durable outbox for announcements | Missing | Pattern exists: pmActivityClaimRevocationOutbox (MongoDB reference) |
Release outbox |
| R.4 | Availability response per trusted context | Missing | Anonymous GET /api/runtime-feature-flags returns global values only (RuntimeFeatureFlagsController.cs:59-63) |
Context-aware endpoint |
| R.5 | Discovery preference (seen/dismissed) | Missing | Closest: ReviewerWorkspaceSettings.FirstRunHintSeen (ReviewerWorkspaceSettings.cs:54, 67), scoped to the review workspace |
Store and endpoint |
| R.6 | What's New driven by confirmed milestones | Partial | Static whats-new.json, shipped with the web image; date is a hand-typed string such as "18th August 2026" (whats-new.json:6) |
Dates follow web deployment, not activation. Runtime entries and isoDate exist (6G PR A); signed-in disclosure through availability, dated by the confirmed milestone, exists (6G PR B, #4474); rendering remains (6G PRs C and D). The feed how-to describes the updates array and the server-only runtime list |
| R.7 | Guide shows availability and milestone dates | Partial | Public Jekyll site; dates appear in page titles (user-guide/whats-new/whats-new-bulk-pdf-upload.md, front matter) |
No authenticated or availability-aware view |
| R.8 | Protected (private pilot) content, assets and search | Missing | user-guide/ is a public static site |
Protected surface |
| R.9 | Declared editorial timezone for date-only headings | Missing | Storage and scheduling convention is UTC (Quartz schedules set TimeZoneId = "UTC", e.g. ProjectStatisticsMaintenanceSchedule.cs:21; statistics charts render UTC, observed-history.ts:689). No declared display zone for date-only headings; feed dates carry no zone |
Decision OD-9 (UTC or Europe/London for display) |
| R.10 | Staging rehearsals cannot create production records | Missing | No release records exist; staging and production databases are separate (cluster-gitops staging.values.yaml:13-17, production.values.yaml:19-20) |
Binding and environment checks on every write |
| C.1 | CD plan G2 production render comparison | Missing | Planned only; .github/promotion/ does not exist on main |
Built by the CD plan; made conditional by this plan |
| X.1 | Production runs the runtime-flag mechanism described above | Missing | cluster-gitops production/api/config.yaml pins 9.44.1-restore.104eca9 (chart api-v9.44.0) and PM 11.43.1-restore.104eca9; git grep -il 'RuntimeFeatureFlag\|MigrationEntryPoint' 104eca9 -- src returns nothing |
The CD plan's catch-up promotion; every production row above describes main |
| S.1 | Environment compatibility report through a scoped automation view | Missing | Only the anonymous snapshot exists (RuntimeFeatureFlagsController.cs:59-63); the statistics staging preflight script combines that snapshot, /health/live versions and cluster-gitops files itself (statistics STATUS) |
Report contract, revision vector, endpoint (plan §14) |
| S.2 | Environment-bound machine identity with a read-only scope | Partial | Staging: OpenIddict client-credentials (Identity.Endpoint/Program.cs:142); the single-scope syrf-statistics-evidence client (StatisticsParityEvidenceClient.cs:4-23) and the statistics:operate credential, accepted on specific routes (API Program.cs:481-485). Production: Identity disabled (cluster-gitops production/identity/config.yaml) and no OpenIddict validation configured, so only the Auth0 JWT scheme authenticates there; it would authenticate an Auth0 machine token for the API audience, but no authorisation path for a machine principal exists in production, and the tenant's grants are not visible from the repository |
Production automation identity and its authorisation path (plan OD-21) |
| S.3 | Workload observations from a scoped collector | Partial | syrf_metadata instance records (DatabaseMetadataWriter.cs:160-168); staging-only read-only Kubernetes identity syrf-stats-preflight (#3806) |
Collector publication route and production identity |
| S.4 | Deployment compatibility constraint enforced on policy writes | Missing | None | Constraint record and write-path check (plan §14.5) |
Superseded assumptions¶
The full list, with what replaces each, is in plan §3.