Skip to content

Runtime feature flags isolated rehearsal checklist

This checklist is for slice 6C.2 of the remaining delivery plan: an isolated rehearsal in the local E2E stack that follows the operator runbooks end to end, with the transcript recorded.

Outcome of the 6C.2 rehearsal (2026-10-22)

EXECUTED in the local stack only. Run on 2026-10-22 against main at 45091ad72fe7eba816e1e7da271b61824daa63a6, on Bramble, in one isolated stack (syrf-e2e-137) with synthetic data, then torn down with teardown.sh. No shared environment or production credential was used. Staging and production remain unexecuted.

This checklist: 60 rows, 47 PASS, 0 FAIL, 13 NOT EXERCISED. The evidence README splits some rows further (65 rows: 52 PASS, 0 FAIL, the same 13 NOT EXERCISED). The 13 NOT EXERCISED rows, for the owner to accept as staging entry gaps: - R1.3 render review - start-up configuration refusal - R5.1–R5.4 - zero replicas or autoscaled - R6.4 counts together - the R7 dependency refusal (422) - the R7 production refusals - the R8 production restore refusal - R9.5 quarantine - R9.6 break-glass - R9.7 rebind - R10.4 mode switch - R10.5 confirm

R10.4 and R10.5 are covered by #4414 (6C.5). Evidence: handover/runtime-foundations-20261020-01a1519c-evidence/6C2-rehearsal/ (outside the repo, with SHA256SUMS for 157 files). One PASS (R4.3) ran the real capture script against a kubectl stand-in, because the stack has no Kubernetes. Both driver runs wrote evidence.json with pass: true, every scenario pass: true.

  • Purpose reports: with both APIs in FleetInventoryMode=Collector and the R3.4 loop publishing, policy-change and legacy-return reports were Complete (every required source Current, migrations prerequisite Ready). In Static mode both were Incomplete with fleetInventoryMode:Static. promotion stays Incomplete on gitops (known gap 8).
  • Checker: Collector policy-change and legacy-return reports: pass (exit 0). Static reports: insufficient-evidence (exit 2). Legacy return with a stop active: refuse (exit 1, legacy-parity-stop-active).
  • Runbook 10: the STOP box holds. One API host started in Legacy mode against the imported authority refused: Legacy runtime policy provider cannot serve initialized authority. Deploy the authority-capable adapter before restarting this host. R10.4 and R10.5 were not run.

Evidence (outside the repository): handover/runtime-foundations-20261020-01a1519c-evidence/6C2-rehearsal/ in the SyRF workspace, with README.md (the full step table: expected and actual lines), logs, the scripts used and SHA256SUMS. Below, <evidence>/ stands for that directory.

Deviations from the plain harness, all recorded in <evidence>/README.md:

  1. One mock OIDC mapping for the report scope (environment:report:read) was added in the rehearsal's own checkout, not committed. Main's mock maps only the collector scope, so no report can be collected without it.
  2. The APIs were started (through runtime-fleet-harness.sh --host-control) with the compatibility-report and workload-collector client settings exported.
  3. R4.3 used a read-only kubectl stand-in that describes the four harness hosts.
  4. Runbooks 1 and 2 were also run step by step on a second database (syrf_rehearsal_6c2) in the stack's MongoDB, with main's own functions, so that each preflight and refusal could be checked. The fleet database was never edited by hand.

Outcome of the 6B.4 browser rows (2026-10-23)

The 6B.4 browser rows (added 2026-10-23, beside R7.6, R8.3, R8.5 and R9.1) are not part of the 6C.2 count. Their outcomes come from their own runs on Bramble, in the local stack only (2026-10-23):

  • Before #4425 (main at f92b496fe), browser convergence FAILED: a tab on the API that took the change missed it (the writer hinted before its own reader had the commit, and the later refresh was not hinted).
  • With #4425 (merged in main at 412289008), the four browser rows run so far pass: R7.6 adoption and recovery, R8.3 and R8.5. The confirmation is at this branch's head 981ffaad7 (run 5). Tabs on the writing API catch up in 77–266 ms and tabs on the other API in 1.8–9.7 s. The R9.1 browser row waits for the recovery spec.

Evidence: handover/runtime-foundations-20261020-01a1519c-evidence/6B4-browser-convergence/ (outside the repo, with README.md and SHA256SUMS).

How to run it

The runtime-fleet lane is local only (Bramble, per delivery plan 6B.1). It starts two API and two PM processes in Shared mode against the stack's own database, bound to local:<stack>, with synthetic data. It never reaches a shared environment.

# The plan only: nothing is started, written or contacted
bash e2e/scripts/runtime-fleet-harness.sh --dry-run

# The whole lane: stack, bind, up, lineage import, four hosts, then every scenario
bash e2e/run-local.sh --runtime-fleet --iterative

# Re-run chosen scenarios against the kept stack (see the note below: the hosts must be running)
E2E_PORT_OFFSET=<n> bash e2e/scripts/runtime-fleet-harness.sh \
  --scenario convergence,mongo-outage,restart,scale-up-unexpected,stop-under-outage

# Offline contract checks that need no stack
bash e2e/scripts/test-runtime-fleet-harness.sh
python3 scripts/test-runtime-compatibility-check.py
python3 scripts/test-capture-runtime-workload.py
python3 scripts/test-publish-runtime-workload.py

--iterative keeps the containers and the database, but run-local.sh stops the four .NET hosts when it exits. Before re-running scenarios, start them again with bash e2e/scripts/runtime-fleet-harness.sh --host-control start <host> static for each of api-1, api-2, pm-1, pm-2, from a shell that has sourced e2e/scripts/ports.sh (with E2E_PORT_OFFSET=<n>, E2E_PORTS_SKIP_AUTO_RESOLVE=true and e2e_configure_ports <n>) and e2e/scripts/service-env.sh. Without service-env.sh the hosts start with the offset-0 container ports.

The browser rows run the lane's Playwright spec instead of the harness driver:

bash e2e/run-local.sh --runtime-fleet --spec runtime-fleet-browser-convergence

It writes e2e/test-results/runtime-fleet-browser/<UTC timestamp>/ (browser-timings.csv, evidence.json) and prints each verdict as [RTF_BROWSER] PASS <name> -- <detail> or [RTF_BROWSER] FAIL <name> -- <detail>.

Evidence lands in e2e/test-results/runtime-fleet/<UTC timestamp>/ (timings.csv, evidence.json). The lineage artifacts are in /tmp/<stack>-runtime-fleet/. Store the console transcript with them.

Driver assertions print as PASS <name> -- <detail> or FAIL <name> -- <detail>. The run ends with evidence written to <dir> (pass=true).

What the local stack differs on

  • RuntimeEnvironment=local and binding local:<stack>: production admission (ProductionPolicyMode=Runtime) and the production fleet refusal are not exercised.
  • Fleet inventory is Static (no FleetInventoryMode) by default, with fixed instance UIDs api-1, api-2, pm-1, pm-2. The collector-scale-up scenario (#4409) switches both APIs to Collector mode and exercises collector retirement; the 6C.2 rehearsal also ran both APIs in Collector mode with the R3.4 loop.
  • No Kubernetes, so no real capture, no chart rendering and no ArgoCD Jobs.
  • The mock OIDC issues the collector machine token (#4409). It has no report or coordinator mapping, and the harness configures no report client, so reports need the two rehearsal additions above. No coordinator client exists.
  • E2E deploys DisableMembership=true (e2e/run-local.sh, appsettings.e2etest.json), while the harness baseline sets every key, disableMembership included, to false.
  • The admin is the E2E admin user, logged in through /api/e2e-auth/login.

Checklist

"Gap" means the harness does not exercise the step. The last column says what would. "Outcome" is the 6C.2 result and its evidence file under <evidence>/.

1. Environment binding

Step How it is exercised Expected transcript line Outcome (6C.2) Gap and what would exercise it
R1.1 status e2e/scripts/pm-migrations.sh, run by run-local.sh; rehearsal on a second database, with and without expected values === Reading the live cluster fingerprint (SYRF_MIGRATION_MODE=status) ===, then Environment binding: none in database <database>; live cluster fingerprint <fingerprint> on a fresh stack PASS: Environment binding: none in database syrf_e2e; live cluster fingerprint rs0@localhost; with expected values, the mismatch warning before bind and Environment binding matches … after (runA/pm-migrations.log, s2-db2-r1-r2.log) None for the binary. The cluster status delivery is a known gap
R1.2–R1.4 bind, up pm-migrations.sh; rehearsal repeat bind === Binding the stack's database (SYRF_MIGRATION_MODE=bind): environment local:<stack>, database <database>, cluster <fingerprint> ===, Environment bound: local:<stack>, … announcement delivery disabled, then === Applying project-management migrations (ADR-023, SYRF_MIGRATION_MODE=up) === and === Project-management migrations applied === PASS: all four lines; 0001–0021 applied; a repeated bind printed Environment binding already in place: …; nothing changed (runA/pm-migrations.log, s2-db2-r1-r2.log) The chart Jobs: src/services/api/.chart/tests/migrations_binding_test.yaml (helm unittest)
R1.3 render review Not exercised n/a NOT EXERCISED: no chart rendering in the local stack Chart unit tests above
R1.5 verify Rehearsal: pm_migrations_run verify on the fleet database and the second database Verify: every migration this binary knows is applied and every declared index exists PASS: that line, exit 0 (s3-fleet-verify.log, db2/logs/14-mig-verify.log) Closed by the rehearsal step; the harness could add the same call
Refusal: wrong cluster Rehearsal: bind with a wrong SYRF_MIGRATION_BINDING_EXPECTED_CLUSTER, plus a missing value, an invalid environment ID, a wrong database, a different binding, up before bind and a bad mode Exit 2, messages as the runbook's refusal table PASS: exit 2 each, e.g. Bind refused: the live cluster fingerprint 'rs0@localhost' does not equal the expected cluster 'wrongset@rehearsal.invalid'. (s2-db2-r1-r2.log) Also EnvironmentBindingTests and EnvironmentBindingIntegrationTests

2. Lineage preview and apply

Step How it is exercised Expected transcript line Outcome (6C.2) Gap and what would exercise it
R2.1 baseline rtf_baseline_json: every generated catalog key, all false 54 keys on stderr PASS: 54 keys (runA-console.log.gz) None
R2.2 preview rtf_lineage_run preview [runtime-fleet] lineage preview; lineage-preview.log holds Runtime policy lineage preview completed; artifact: …/lineage-preview.json. No consumer was enrolled. PASS: both lines; the legacy-document preflight showed no storeEpoch (runA/lineage/, s2-db2-r1-r2.log) None
R2.3 review Rehearsal: the preview's fields read without reformatting previewDigest, sourceRevision, target.storeEpoch recorded PASS: kind legacyLineageImport, planVersion {"$numberInt": "1"}, sourceRevision "0", the bound environment and epoch, a new store epoch, previewDigest (s2-db2-r1-r2.log) Closed by the rehearsal step
R2.4 writer freeze rtf_seed_legacy seeds an empty syrf-api registry; no API process is running yet; rehearsal negative control Nothing printed (the seed state is captured). On a stack that is already imported: [runtime-fleet] authority already imported on this database; skipping the lineage import PASS: registry instances: {}; with one writer entry, apply refused API writers remain registered. Stop/freeze all legacy writers before lineage import; there is no bypass switch. exit 2 (s2-db2-r1-r2.log) Closed by the rehearsal's negative control; also RuntimePolicyLineageImportTests
R2.5 apply rtf_lineage_run apply with an actor and a freeze acknowledgement [runtime-fleet] lineage apply, [runtime-fleet] lineage imported (receipt: /tmp/<stack>-runtime-fleet/lineage-receipt.json) PASS: both lines; every receipt field present; receipt storeEpoch equals authority-status (runA/lineage/lineage-receipt.json) None
R2.6 retry Rehearsal: apply again with the same preview and a new output path The same receipt PASS: exit 0, retry receipt byte-identical (db2/) Closed by the rehearsal step
R2.7 no Legacy restart Rehearsal negative control: one API with RuntimeFeatureFlags__Mode=Legacy after the import Legacy runtime policy provider cannot serve initialized authority… and no listener PASS: that refusal, never listened. The refused host still wrote a syrf-api writer-registry entry (s2-r2-7-legacy-api.txt) Superseded by #4414: a Legacy API now starts against the imported authority, serves deployed values and marks its registry entry read-only; the runtime-fleet legacy-return scenario restarts all four hosts with only Mode=Legacy (legacy-return R10.5: api-1 serves the deployed values with the preserved policy state …)
Refusals Rehearsal: no legacy document, combined modes, same input and output, existing output, no actor, an imported source Runtime policy lineage refused: …, exit 2 PASS: each message as in the runbook table, exit 2 (s2-db2-r1-r2.log) Also RuntimePolicyLineageImportTests

2a. Initialise (no legacy document, 6C.6)

Run with bash e2e/run-local.sh --runtime-fleet-initialise --iterative --scenario initialise-policy-change (add convergence to flip on the initialised authority too). The stack's database has no global flag document: the bring-up initialises the authority from the deployed values before any host starts, and its artifacts and logs stay in /tmp/<stack>-runtime-fleet/initialise-*. The named-only scenario initialise-policy-change (never part of all) judges them and copies them to <out>/initialise/. Outcomes are from the 6C.6 rehearsal on Bramble at 29e1ee568 (evidence 6C6-initialise/ in the programme handover directory; 40 assertions, 0 failed).

Step How it is exercised Expected transcript line Outcome (6C.6) Gap and what would exercise it
Before R2.I2: no global document rtf_seed_registry (seeds only the writer registry) [runtime-fleet] preflight: no global flag document (initialise may run); PASS initialise preflight: the database held no global flag document before initialise -- RTF_GLOBAL_DOCUMENTS=0; RTF_STATE=absent PASS: RTF_GLOBAL_DOCUMENTS=0; RTF_STATE=absent (initialise-preflight.txt) None
R2.I1 input rtf_initialise_input_json (approval and provenance only) Nothing printed; initialise-input.json kept PASS: approval and provenance only (initialise-input.json) None
R2.I2 preview, twice rtf_lineage_run preview … SYRF_RUNTIME_POLICY_SOURCE=initialize to two paths, then cmp [runtime-fleet] initialise preview (initialise-preview), … (initialise-preview-again); PASS initialise preview: two previews of the same input are byte-identical -- previewDigest <digest> PASS: both previews byte-identical, digest 0B350ECD…8A05 None
R2.I3 review Driver reads the preview PASS initialise preview: kind deployedValueInitialization, revision 0, no overrides, no stops, a Boolean for every key -- <n> baseline keys; on: <keys> PASS: 54 baseline keys; on: disableMembership, maxInProgressSessions (the deployed values) No human review; the evidence directory holds the file
R2.I5 apply rtf_lineage_run apply with an actor and a freeze acknowledgement [runtime-fleet] initialise apply; PASS initialise apply: the receipt is kind deployedValueInitialization for the reviewed preview digest at revision 0 -- receipt <id>, store epoch <epoch> PASS: receipt 70ee3bbc-…, revision 0, store epoch fa4fbee1-… None
R2.I6 replay Apply again with the same preview to a new path [runtime-fleet] initialise replay (…); PASS initialise replay: the identical reviewed preview returned the original receipt PASS: initialise-receipt-replay.json identical to the receipt None
Refusal after apply Preview again after apply PASS initialise refusal: a preview after apply exits 2 because an authority exists -- refused: An initialized runtime policy authority already exists; initialise never replaces it. PASS: exit 2 with that message Other refusals: RuntimePolicyLineageInitializeTests
After runbook 3 Four Shared hosts start on the initialised authority PASS initialised authority: stored at revision 0 …, PASS initialised authority: api-1 reports Shared/Ready at the receipt's store epoch, revision 0 (and api-2), PASS initialised authority: api-1 evaluates disableMembership to the deployed value (true) at revision 0 (and api-2), PASS initialise-policy-change (Static inventory): fleet is Active before the disruption PASS: stored at revision 0; both APIs Shared/Ready at the receipt's store epoch, disableMembership true; the Static fleet Active None
R6.3, R3.4 Collector mode Both APIs restarted with the report profile (collector inventory plus the report reader); the driver publishes the inventory PASS initialise-policy-change: in Collector mode, before any inventory is published, the fleet is Unknown with a Collector reason, PASS initialise-policy-change: with the inventory published, the fleet reaches Active in Collector mode PASS: Unknown (CollectorInventoryMissingOrInvalid) before publication, then Active A real collector and Kubernetes rollout: staging
R7.3 report and checker GET …/compatibility-reports/current?purpose=policy-change with the mock OIDC report token; scripts/runtime-compatibility-check.py PASS initialise-policy-change: the policy-change report is Complete, every required source Current, migrations Ready, PASS initialise-policy-change: the checker passes the policy-change report (exit 0, verdict pass, no reasons), PASS initialise-policy-change: control, the same report under a foreign binding epoch is refused (exit 1, binding-mismatch) PASS: report 50a6155c-… Complete on the first attempt; checker {"exitCode": 0, "reasons": [], "verdict": "pass"}; the foreign-epoch control exit 1 (binding-mismatch) None
Production admission of the initialised authority Not exercised (RuntimeEnvironment=local) n/a n/a RuntimeProductionInitializeIntegrationTests (production-like configuration: readers, evaluator, Static and Collector fleet admission)

3. Consumer enrolment

Step How it is exercised Expected transcript line Outcome (6C.2) Gap and what would exercise it
R3.1–R3.3 configuration rtf_instance_env sets Mode=Shared, SharedReaderMode=Disabled, the binding, InstanceUid, ImageVersion, ExpectedReplicas=2, Autoscaled=false and, on APIs, FleetExpectedReplicas 2/2 [runtime-fleet] started api-1 (api, static inventory) on port <port>, pid <pid>, log <file> for each of the four hosts PASS: four started lines (runA-console.log.gz, s3-start-static.txt) Chart rendering: runtime_policy_host_metadata_test.yaml (API and PM charts)
R3.4 checks Driver; rehearsal requests PASS admin logged in on both API processes -- api-1, api-2, then PASS fleet is Active for disableMembership before any flip -- state Active and PASS fleet lists four hosts -- 4 hosts PASS: the three lines; authority-status Shared/Ready on both APIs, evaluation Evaluated, snapshot with authority, Legacy PUT 403 Runtime changes are not available in this mode. (s3-r3-4-preflight.txt) Closed: the rehearsal recorded authority-status
R3.4 publish loop Rehearsal: the runbook's loop with the real capture (stand-in kubectl) and publish scripts Fleet Unknown with a Collector… reason, then Active within 60 s PASS: Unknown CollectorInventoryStaleOrInvalidClock, Active 2.2 s after the loop started; cycles 1.3–1.5 s plus a 2 s pause (s7-collector-mode.txt) A real cluster for the capture
Refusal: start-up configuration Not exercised n/a NOT EXERCISED: not attempted (the R2.7 one-off launch would serve) RuntimeProductionPolicyTests and RuntimePolicyAuthorityRegistrationTests; a negative control could start one host with SharedReaderMode=Membership and expect Shared authority mode cannot also enroll the legacy-format shared reader.

4. Collector identity and scope

Step How it is exercised Expected transcript line Outcome (6C.2) Gap and what would exercise it
R4.1 API settings Rehearsal: both APIs started with the WorkloadCollector settings (Auth0, syrf-e2e-runtime-collector) No start-up refusal; publications accepted PASS (s3-start-static.txt, publish-receipts.log) Also EnvironmentCollectorAuthorizationTests, EnvironmentCollectorHttpTests
R4.2 token Mock OIDC client credentials (#4409 mapping) A token with only the collector scope PASS: claims sub syrf-e2e-runtime-collector@clients, azp, gty client-credentials, scope environment:observations:write, the API audience (s4-collector-static-reports.txt) None
R4.3 capture Rehearsal: scripts/capture-runtime-workload.py against a kubectl stand-in Exit 0, a new file; never overwritten PASS (stand-in): exit 0; to an existing path, Capture refused: Output already exists; preserve it for exact replay exit 2 (s4/) A real cluster: the D1 PR preview
R4.4 publish Rehearsal: scripts/publish-runtime-workload.py to POST /api/environment/observations on loopback Exit 0; stdout only {"observationId": …, "digest": …} PASS; refusals: HTTP 409 exit 1, local refusal exit 2, report-scope token HTTP 401 exit 1 (s4-collector-static-reports.txt) None
R4.5 project-management Rehearsal As R4.4 PASS None
R4.6 continuous publication Rehearsal: the R3.4 loop Both workloads:* sources Current while the loop runs PASS: 22 cycles, 44 accepted, 0 refused (publish-receipts.log) The OD-23 scheduler

5. Coordinator identity

Step How it is exercised Expected transcript line Outcome (6C.2) Gap and what would exercise it
R5 preflight Rehearsal reports deploymentConstraint source Current, projection null PASS (s7/) None
R5.1–R5.4 Not exercised n/a NOT EXERCISED: no coordinator client in the mock OIDC; nothing builds the capability intersection or completion targets (gap 10) RuntimeDeploymentCoordinatorTests, EnvironmentCoordinatorHttpTests, RuntimeDeploymentConstraintTests, RuntimeDeploymentCompletionTests

6. Expected inventory and Collector mode

Step How it is exercised Expected transcript line Outcome (6C.2) Gap and what would exercise it
R6.1–R6.2 static inventory Harness configuration (2 API, 2 PM) PASS fleet lists four hosts -- 4 hosts PASS None
Unexpected extra host scale-up-unexpected scenario PASS scale-up-unexpected: the unexpected api-3 acknowledged and appears in the fleet, PASS scale-up-unexpected: while api-3 is listed the fleet is never Active and is Applying, Degraded or Action required, PASS scale-up-unexpected: the fleet returned to Active within 60s of the cleanup PASS: all three (runB-console.log) The cleanup deletes api-3's row by hand (test only). Collector retirement replaces this in Kubernetes
Silent host convergence stop/resume part PASS stop: fleet never Active while one API process is stopped, and showed Applying or Degraded -- states seen …, PASS resume: fleet reached Active PASS: states Saved, Applying, Degraded; Active 512 ms after resume (runA-console.log.gz) None
R6.3 Collector mode collector-scale-up scenario (#4409); rehearsal: both APIs restarted with --host-control start <api> collector and the R3.4 loop PASS collector-scale-up: in Collector mode, before any inventory is published, the fleet is Unknown with a Collector reason, the a, b and c lines, … every inventory publication during the scenario was accepted and the two cleanup lines PASS: all 13 lines; a stopped api-3 listed Retired (AbsentFromFreshCollectorInventory), Active 35.1 s after its last serverSeenAtUtc (runB-console.log, s7-collector-mode.txt) A real Kubernetes rollout still needs the D1 preview or staging
Non-production key Rehearsal: the fleet for newStageOverview Unknown / DomainReadinessNotEstablished PASS (s3-r3-4-preflight.txt) None
Zero replicas, autoscaled Not exercised n/a NOT EXERCISED: the harness fixes ExpectedReplicas=2 and Autoscaled=false RuntimePolicyFleetTests; the runtime_policy_host_metadata_test.yaml chart tests
R6.4 counts together Not exercised n/a NOT EXERCISED: a cluster-gitops review rule Staging

7. Change one flag

Step How it is exercised Expected transcript line Outcome (6C.2) Gap and what would exercise it
R7.1–R7.2 Driver reads authority-status and the fleet before each flip PASS fleet is Active for disableMembership before any flip -- state Active PASS None
R7.3 report and checker Rehearsal: reports from the stack, synthetic candidate manifests, the checker Static: Incomplete with fleetInventoryMode:Static. Collector: Complete; checker {"exitCode": 0, "reasons": [], "verdict": "pass"} PASS: Static Incomplete, checker insufficient-evidence exit 2; Collector Complete, checker pass exit 0 (s4/, s7/) Main's harness needs a report mapping and client (deviations 1 and 2)
R7.4–R7.5 preview and commit convergence: preview, then commit with the digest, for each flip; rehearsal with "action": "Change" flip 1: disableMembership=false committed at revision <n> PASS: flips 1–4 at revisions 1–4; rehearsal preview 200 then commit at revision 6 (s5-r7-change.txt) None
R7.6 adoption convergence PASS flip 1: every API decision adopted within 60s -- {…}, PASS flip 1: every process acknowledged within 60s -- {…}, PASS flip 1: fleet reached Active -- <ms> ms after commit, repeated per flip PASS: 12 lines, Active 8.0–12.0 s after each commit None
R7.6 browser adoption (6B.4) runtime-fleet-browser-convergence (run-local.sh --runtime-fleet --spec runtime-fleet-browser-convergence): tab A on api-1, tab B on api-2 (membership page), off, on, off, on committed alternately on each API [RTF_BROWSER] PASS every change is visible in both tabs (api-1 and api-2) within 60 s of the commit response -- …, [RTF_BROWSER] PASS no focus, online or offline event reached tab A or B while it converged, and neither tab reloaded -- …, [RTF_BROWSER] PASS every convergence in tab A or B followed a revision hint frame (not a window event or the 5-minute fallback poll) -- …, [RTF_BROWSER] PASS a tab opened after each change renders the committed revision and value from its bootstrap read (api-1 and api-2) -- … PASS (6B4-browser-convergence/ runs 4 and 5 with #4425, 2026-10-23; run 4 figures): every flip in both tabs, all after the hint. Tab on the writing API: 84–267 ms (median 245 ms). Tab on the other API: 0.5–11.1 s (median 6.8 s). No event and no reload: PASS. Fresh tabs render the committed revision: PASS, 12 of 12. Before #4425 (run 3): FAIL, because the writing API's tab caught none of its changes within 60 s None (#4425 merged)
R7.6 browser recovery (6B.4) runtime-fleet-browser-convergence (run-local.sh --runtime-fleet --spec runtime-fleet-browser-convergence): tab C on api-2 with every RuntimeFeatureFlagsRevision hub message dropped [RTF_BROWSER] PASS tab C's hub connection carried the revision hint and the dropper removed it (step 7), [RTF_BROWSER] PASS with the hint dropped and no event, tab C stays stale for 75 s after the commit, [RTF_BROWSER] PASS with the hint dropped, tab C shows the change within 15 s of a focus event, [RTF_BROWSER] PASS with the hint dropped, tab C misses a change while its requests fail and shows it within 15 s of an online event PASS (run 3 and 6B4-browser-convergence/ runs 4 and 5 with #4425, 2026-10-23; run 4 figures): stale for 75 s with the hint dropped and no event; then 252 ms after focus and 240 ms after online (requests aborted, then the event dispatched; context.setOffline resets every socket in the browser) Recovery after a hub reconnect: the runtime-fleet-browser-recovery spec (next 6B.4 PR)
R7.7 lookup Rehearsal: GET /api/runtime-feature-flags/mutations/<mutation-id> outcome: Committed PASS: Committed with committedIdentity; an unknown ID gives Unknown with no identity Closed by the rehearsal step
Refusals (409, stale digest, reuse) Rehearsal 409 runtime_policy_revision_conflict; 400 The reviewed preview is missing or stale…; 400 Mutation ID was already used with a different request. PASS: all three, and an unchanged retry returned the original receipt Also RuntimePolicyMutationStoreTests
Refusal 422 (dependencies) Not exercised n/a NOT EXERCISED: disableMembership has no dependencies API administration tests
Production refusals Not exercised n/a NOT EXERCISED: RuntimeEnvironment=local RuntimeProductionPolicyIntegrationTests

8. Emergency stop and restore

Step How it is exercised Expected transcript line Outcome (6C.2) Gap and what would exercise it
Prior override stop-under-outage sets the override On first PASS stop-under-outage: the prior override (On) is established and the fleet is Active -- revision <n> (only if it was not already On) PASS: not printed, the override was already On None
R8.2 stop with a PM host paused stop-under-outage; rehearsal stop PASS stop-under-outage: the emergency stop commit succeeds while pm-2 is paused -- HTTP 200 revision <n> PASS: revision 9; rehearsal stop revision 7 (runB-console.log, s5-r8-stop-restore.txt) None
R8.3 stop adoption stop-under-outage PASS stop-under-outage: every running API evaluates the stopped value (false) at revision <n> within 60s, PASS stop-under-outage: the fleet is not Active while pm-2 is paused (and a non-Active state was observed), PASS stop-under-outage: after SIGCONT every process acknowledges the stop and the fleet is Active PASS: all three; a retried stop with the same mutation ID returned the same revision None
R8.3 browser stop visible (6B.4) runtime-fleet-browser-convergence (run-local.sh --runtime-fleet --spec runtime-fleet-browser-convergence), step 5: emergency stop committed on api-1 [RTF_BROWSER] PASS every stop is visible in both tabs (api-1 and api-2) within 60 s of the commit response -- …, [RTF_BROWSER] PASS the admin feature-flags page shows the membership stop after the stop and not after the restore -- … PASS (6B4-browser-convergence/ runs 4 and 5 with #4425, 2026-10-23; run 4 figures): the control was hidden 264 ms after the stop on the writing API's tab and 1.9 s after on the other's. The admin page showed the stop: PASS. Before #4425 (run 3): FAIL, the writing API's tab not within 90 s None (#4425 merged)
R8.4 restore stop-under-outage (ordinal action); rehearsal ("action": "ReleaseStop") PASS stop-under-outage: the ReleaseStop preview succeeds, PASS stop-under-outage: the preview restores the prior override, PASS stop-under-outage: the ReleaseStop commit succeeds -- HTTP 200 revision <n> PASS: all three; the name form was accepted too None
R8.5 restore adoption stop-under-outage PASS stop-under-outage: every API evaluates the restored prior override (true) at revision <n> within 60s, PASS stop-under-outage: the fleet is Active after the release PASS None
R8.5 browser restore visible (6B.4) runtime-fleet-browser-convergence (run-local.sh --runtime-fleet --spec runtime-fleet-browser-convergence), step 6: ReleaseStop committed on api-2 [RTF_BROWSER] PASS every restore is visible in both tabs (api-1 and api-2) within 60 s of the commit response -- … PASS (6B4-browser-convergence/ runs 4 and 5 with #4425, 2026-10-23; run 4 figures): the restored override showed 251 ms after the commit on the writing API's tab and 10.9 s after on the other's. The admin page showed not stopped: PASS. Before #4425 (run 3): FAIL, 85.3 s by the 5-minute poll None (#4425 merged)
Production restore refusal Not exercised n/a NOT EXERCISED: local environment RuntimeProductionPolicyIntegrationTests

9. Recovery and rebind

Step How it is exercised Expected transcript line Outcome (6C.2) Gap and what would exercise it
R9.1 lost outcome Rehearsal, as R7.7 Committed or Unknown PASS None
R9.1 browser reload recovery (6B.4) runtime-fleet-browser-recovery (next 6B.4 PR; not written yet): a reload during an unresolved commit resolves by the original mutation ID Saved by GET mutations/<original id>; no preview, policy or emergency-stop request after the reload; exactly one new audit receipt Pending: spec not written yet The recovery spec, then a Bramble run
R9.2 lineage retry Rehearsal, as R2.6 The original receipt PASS None
R9.3 publication replay Rehearsal: the exact accepted capture published again The same receipt PASS: same observationId and digest, exit 0 The exit 3 path itself: test-publish-runtime-workload.py
R9.4 Mongo outage mongo-outage PASS mongo-outage: api-1 left Evaluated within 35s with value null and safe value false -- … (and api-2), PASS mongo-outage: the fleet was never Active during the outage -- …, PASS mongo-outage: every API returned to Evaluated at revision <n> within 60s of the unpause, PASS mongo-outage: fleet Active within 60s of the unpause PASS: fail-closed after 12.0 s and 18.0 s; Active 12.1 s after the unpause None
Restarted host restart PASS restart: the fleet is not Active from ~30s after api-2 stopped (checked from 35s), PASS restart: the restarted process's first evaluation is Evaluated at the committed revision <n> with the prior value, PASS restart: the acknowledgement row carries a new processStartId, PASS restart: fleet Active within 60s of starting api-2 PASS: all four None
R9.5 quarantine Not exercised n/a NOT EXERCISED: causing one needs a hand edit of the binding or authority RuntimePolicyAuthorityReaderTests; the recovery procedure itself is not written
R9.6 break-glass Not exercised n/a NOT EXERCISED: needs an active deployment constraint (runbook 5) RuntimeDeploymentBreakGlassIntegrationTests, RuntimeDeploymentBreakGlassHttpTests
R9.7 rebind Not possible n/a NOT EXERCISED: rebind is not built None until it is built

10. Legacy return

Step How it is exercised Expected transcript line Outcome (6C.2) Gap and what would exercise it
R10.1 values agree Rehearsal: a forward revision to the deployed value (true in the E2E defaults) legacyValue equal to runtimeValue PASS: the Static report showed legacyValue true, runtimeValue false; after the R7 revision both were true (s4/, s7/) None
R10.2 report Rehearsal, Collector mode Complete, legacyParity.agrees: true, stopActive: false PASS (s7/report-collector-legacy-return.json) Closed by #4414: the harness's report mapping and the legacy-return scenario (PASS legacy-return R10.2: the legacy-return report is Complete, its parity agrees and no stop is active)
R10.3 checker Rehearsal, plus the offline fixtures (test_legacy_return, test_legacy_stop_active_names_key, test_legacy_differs_names_key, test_legacy_parity_missing) {"exitCode": 0, "reasons": [], "verdict": "pass"} PASS: pass exit 0; with a stop active, refuse exit 1 with legacy-parity-stop-active (s8/) None
R10 STOP-box proof Rehearsal: one API with Mode=Legacy against the fleet's imported authority Legacy runtime policy provider cannot serve initialized authority. PASS: the refusal, no listener; the Shared fleet stayed Active (s8-legacy-return.txt) Superseded by #4414: the STOP box is replaced by R10.4 to R10.6
R10.4 mode switch legacy-return scenario (#4414): api-2, api-1, pm-2, pm-1 restarted with only RuntimeFeatureFlags__Mode=Legacy PASS legacy-return: after 35s in Legacy mode the authority document digest and the audit row count are unchanged NOT EXERCISED: blocked (STOP box, owner decision D7) Exercised by the legacy-return scenario
R10.5 confirm legacy-return scenario (#4414): snapshot, provider status, authority-status, a refused legacy change, the registry marker PASS legacy-return R10.5: api-1 authority-status is Legacy, Disabled, LegacyModeAuthorityPreserved, with no identity NOT EXERCISED: follows R10.4 Exercised by the legacy-return scenario
R10.6 return to Shared legacy-return scenario (#4414): every host restarted in Shared static mode PASS legacy-return: Shared resumes at the same revision and identity (all seven fields) n/a (added by #4414) None

Acceptance for 6C.2

  1. Every row whose "How it is exercised" is not "Not exercised" shows its expected transcript line. Met: every exercised row is PASS.
  2. evidence.json has pass: true, and every scenario has pass: true. Met for both driver runs.
  3. Each gap row is either closed by an added harness step (with its transcript line recorded here) or carried into the staging entry criteria as an accepted gap. Partly met: the rehearsal closed R1.5, R2.3, R2.4, R2.6, R2.7, R3.4, R4.1–R4.6, R7.3, R7.7, the R7 refusals and R10.1–R10.3 with rehearsal steps, not harness steps. The 13 NOT EXERCISED rows are for Chris to accept as staging entry gaps.
  4. No step used a shared environment or a production credential. Met.