Runtime feature flags isolated rehearsal checklist¶
This checklist is for slice 6C.2 of the remaining delivery plan: an isolated rehearsal in the local E2E stack that follows the operator runbooks end to end, with the transcript recorded.
Outcome of the 6C.2 rehearsal (2026-10-22)¶
EXECUTED in the local stack only. Run on 2026-10-22 against
mainat45091ad72fe7eba816e1e7da271b61824daa63a6, on Bramble, in one isolated stack (syrf-e2e-137) with synthetic data, then torn down withteardown.sh. No shared environment or production credential was used. Staging and production remain unexecuted.
This checklist: 60 rows, 47 PASS, 0 FAIL, 13 NOT EXERCISED. The evidence README splits some rows further (65 rows: 52 PASS, 0 FAIL, the same 13 NOT EXERCISED). The 13 NOT EXERCISED rows, for the owner to accept as staging entry gaps: - R1.3 render review - start-up configuration refusal - R5.1–R5.4 - zero replicas or autoscaled - R6.4 counts together - the R7 dependency refusal (422) - the R7 production refusals - the R8 production restore refusal - R9.5 quarantine - R9.6 break-glass - R9.7 rebind - R10.4 mode switch - R10.5 confirm
R10.4 and R10.5 are covered by #4414 (6C.5). Evidence: handover/runtime-foundations-20261020-01a1519c-evidence/6C2-rehearsal/ (outside the repo, with SHA256SUMS for 157 files). One PASS (R4.3) ran the real capture script against a kubectl
stand-in, because the stack has no Kubernetes. Both driver runs wrote evidence.json with pass: true, every
scenario pass: true.
- Purpose reports: with both APIs in
FleetInventoryMode=Collectorand the R3.4 loop publishing,policy-changeandlegacy-returnreports wereComplete(every required sourceCurrent,migrationsprerequisiteReady). In Static mode both wereIncompletewithfleetInventoryMode:Static.promotionstaysIncompleteongitops(known gap 8). - Checker: Collector
policy-changeandlegacy-returnreports:pass(exit 0). Static reports:insufficient-evidence(exit 2). Legacy return with a stop active:refuse(exit 1,legacy-parity-stop-active). - Runbook 10: the STOP box holds. One API host started in Legacy mode against the imported authority refused:
Legacy runtime policy provider cannot serve initialized authority. Deploy the authority-capable adapter before restarting this host.R10.4 and R10.5 were not run.
Evidence (outside the repository): handover/runtime-foundations-20261020-01a1519c-evidence/6C2-rehearsal/ in the
SyRF workspace, with README.md (the full step table: expected and actual lines), logs, the scripts used and
SHA256SUMS. Below, <evidence>/ stands for that directory.
Deviations from the plain harness, all recorded in <evidence>/README.md:
- One mock OIDC mapping for the report scope (
environment:report:read) was added in the rehearsal's own checkout, not committed. Main's mock maps only the collector scope, so no report can be collected without it. - The APIs were started (through
runtime-fleet-harness.sh --host-control) with the compatibility-report and workload-collector client settings exported. - R4.3 used a read-only
kubectlstand-in that describes the four harness hosts. - Runbooks 1 and 2 were also run step by step on a second database (
syrf_rehearsal_6c2) in the stack's MongoDB, with main's own functions, so that each preflight and refusal could be checked. The fleet database was never edited by hand.
Outcome of the 6B.4 browser rows (2026-10-23)¶
The 6B.4 browser rows (added 2026-10-23, beside R7.6, R8.3, R8.5 and R9.1) are not part of the 6C.2 count. Their outcomes come from their own runs on Bramble, in the local stack only (2026-10-23):
- Before #4425 (
mainatf92b496fe), browser convergence FAILED: a tab on the API that took the change missed it (the writer hinted before its own reader had the commit, and the later refresh was not hinted). - With #4425 (merged in
mainat412289008), the four browser rows run so far pass: R7.6 adoption and recovery, R8.3 and R8.5. The confirmation is at this branch's head981ffaad7(run 5). Tabs on the writing API catch up in 77–266 ms and tabs on the other API in 1.8–9.7 s. The R9.1 browser row waits for the recovery spec.
Evidence: handover/runtime-foundations-20261020-01a1519c-evidence/6B4-browser-convergence/ (outside the repo, with
README.md and SHA256SUMS).
How to run it¶
The runtime-fleet lane is local only (Bramble, per delivery plan 6B.1). It starts two API and two PM processes
in Shared mode against the stack's own database, bound to local:<stack>, with synthetic data. It never
reaches a shared environment.
# The plan only: nothing is started, written or contacted
bash e2e/scripts/runtime-fleet-harness.sh --dry-run
# The whole lane: stack, bind, up, lineage import, four hosts, then every scenario
bash e2e/run-local.sh --runtime-fleet --iterative
# Re-run chosen scenarios against the kept stack (see the note below: the hosts must be running)
E2E_PORT_OFFSET=<n> bash e2e/scripts/runtime-fleet-harness.sh \
--scenario convergence,mongo-outage,restart,scale-up-unexpected,stop-under-outage
# Offline contract checks that need no stack
bash e2e/scripts/test-runtime-fleet-harness.sh
python3 scripts/test-runtime-compatibility-check.py
python3 scripts/test-capture-runtime-workload.py
python3 scripts/test-publish-runtime-workload.py
--iterative keeps the containers and the database, but run-local.sh stops the four .NET hosts when it exits. Before
re-running scenarios, start them again with bash e2e/scripts/runtime-fleet-harness.sh --host-control start <host> static
for each of api-1, api-2, pm-1, pm-2, from a shell that has sourced e2e/scripts/ports.sh (with
E2E_PORT_OFFSET=<n>, E2E_PORTS_SKIP_AUTO_RESOLVE=true and e2e_configure_ports <n>) and
e2e/scripts/service-env.sh. Without service-env.sh the hosts start with the offset-0 container ports.
The browser rows run the lane's Playwright spec instead of the harness driver:
It writes e2e/test-results/runtime-fleet-browser/<UTC timestamp>/ (browser-timings.csv, evidence.json) and prints each
verdict as [RTF_BROWSER] PASS <name> -- <detail> or [RTF_BROWSER] FAIL <name> -- <detail>.
Evidence lands in e2e/test-results/runtime-fleet/<UTC timestamp>/ (timings.csv, evidence.json). The lineage
artifacts are in /tmp/<stack>-runtime-fleet/. Store the console transcript with them.
Driver assertions print as PASS <name> -- <detail> or FAIL <name> -- <detail>. The run ends with
evidence written to <dir> (pass=true).
What the local stack differs on¶
RuntimeEnvironment=localand bindinglocal:<stack>: production admission (ProductionPolicyMode=Runtime) and the production fleet refusal are not exercised.- Fleet inventory is Static (no
FleetInventoryMode) by default, with fixed instance UIDsapi-1,api-2,pm-1,pm-2. Thecollector-scale-upscenario (#4409) switches both APIs to Collector mode and exercises collector retirement; the 6C.2 rehearsal also ran both APIs in Collector mode with the R3.4 loop. - No Kubernetes, so no real capture, no chart rendering and no ArgoCD Jobs.
- The mock OIDC issues the collector machine token (#4409). It has no report or coordinator mapping, and the harness configures no report client, so reports need the two rehearsal additions above. No coordinator client exists.
- E2E deploys
DisableMembership=true(e2e/run-local.sh,appsettings.e2etest.json), while the harness baseline sets every key,disableMembershipincluded, tofalse. - The admin is the E2E
adminuser, logged in through/api/e2e-auth/login.
Checklist¶
"Gap" means the harness does not exercise the step. The last column says what would. "Outcome" is the 6C.2 result and
its evidence file under <evidence>/.
1. Environment binding¶
| Step | How it is exercised | Expected transcript line | Outcome (6C.2) | Gap and what would exercise it |
|---|---|---|---|---|
R1.1 status |
e2e/scripts/pm-migrations.sh, run by run-local.sh; rehearsal on a second database, with and without expected values |
=== Reading the live cluster fingerprint (SYRF_MIGRATION_MODE=status) ===, then Environment binding: none in database <database>; live cluster fingerprint <fingerprint> on a fresh stack |
PASS: Environment binding: none in database syrf_e2e; live cluster fingerprint rs0@localhost; with expected values, the mismatch warning before bind and Environment binding matches … after (runA/pm-migrations.log, s2-db2-r1-r2.log) |
None for the binary. The cluster status delivery is a known gap |
R1.2–R1.4 bind, up |
pm-migrations.sh; rehearsal repeat bind |
=== Binding the stack's database (SYRF_MIGRATION_MODE=bind): environment local:<stack>, database <database>, cluster <fingerprint> ===, Environment bound: local:<stack>, … announcement delivery disabled, then === Applying project-management migrations (ADR-023, SYRF_MIGRATION_MODE=up) === and === Project-management migrations applied === |
PASS: all four lines; 0001–0021 applied; a repeated bind printed Environment binding already in place: …; nothing changed (runA/pm-migrations.log, s2-db2-r1-r2.log) |
The chart Jobs: src/services/api/.chart/tests/migrations_binding_test.yaml (helm unittest) |
| R1.3 render review | Not exercised | n/a | NOT EXERCISED: no chart rendering in the local stack | Chart unit tests above |
R1.5 verify |
Rehearsal: pm_migrations_run verify on the fleet database and the second database |
Verify: every migration this binary knows is applied and every declared index exists |
PASS: that line, exit 0 (s3-fleet-verify.log, db2/logs/14-mig-verify.log) |
Closed by the rehearsal step; the harness could add the same call |
| Refusal: wrong cluster | Rehearsal: bind with a wrong SYRF_MIGRATION_BINDING_EXPECTED_CLUSTER, plus a missing value, an invalid environment ID, a wrong database, a different binding, up before bind and a bad mode |
Exit 2, messages as the runbook's refusal table | PASS: exit 2 each, e.g. Bind refused: the live cluster fingerprint 'rs0@localhost' does not equal the expected cluster 'wrongset@rehearsal.invalid'. (s2-db2-r1-r2.log) |
Also EnvironmentBindingTests and EnvironmentBindingIntegrationTests |
2. Lineage preview and apply¶
| Step | How it is exercised | Expected transcript line | Outcome (6C.2) | Gap and what would exercise it |
|---|---|---|---|---|
| R2.1 baseline | rtf_baseline_json: every generated catalog key, all false |
54 keys on stderr |
PASS: 54 keys (runA-console.log.gz) |
None |
| R2.2 preview | rtf_lineage_run preview |
[runtime-fleet] lineage preview; lineage-preview.log holds Runtime policy lineage preview completed; artifact: …/lineage-preview.json. No consumer was enrolled. |
PASS: both lines; the legacy-document preflight showed no storeEpoch (runA/lineage/, s2-db2-r1-r2.log) |
None |
| R2.3 review | Rehearsal: the preview's fields read without reformatting | previewDigest, sourceRevision, target.storeEpoch recorded |
PASS: kind legacyLineageImport, planVersion {"$numberInt": "1"}, sourceRevision "0", the bound environment and epoch, a new store epoch, previewDigest (s2-db2-r1-r2.log) |
Closed by the rehearsal step |
| R2.4 writer freeze | rtf_seed_legacy seeds an empty syrf-api registry; no API process is running yet; rehearsal negative control |
Nothing printed (the seed state is captured). On a stack that is already imported: [runtime-fleet] authority already imported on this database; skipping the lineage import |
PASS: registry instances: {}; with one writer entry, apply refused API writers remain registered. Stop/freeze all legacy writers before lineage import; there is no bypass switch. exit 2 (s2-db2-r1-r2.log) |
Closed by the rehearsal's negative control; also RuntimePolicyLineageImportTests |
| R2.5 apply | rtf_lineage_run apply with an actor and a freeze acknowledgement |
[runtime-fleet] lineage apply, [runtime-fleet] lineage imported (receipt: /tmp/<stack>-runtime-fleet/lineage-receipt.json) |
PASS: both lines; every receipt field present; receipt storeEpoch equals authority-status (runA/lineage/lineage-receipt.json) |
None |
| R2.6 retry | Rehearsal: apply again with the same preview and a new output path | The same receipt | PASS: exit 0, retry receipt byte-identical (db2/) |
Closed by the rehearsal step |
| R2.7 no Legacy restart | Rehearsal negative control: one API with RuntimeFeatureFlags__Mode=Legacy after the import |
Legacy runtime policy provider cannot serve initialized authority… and no listener |
PASS: that refusal, never listened. The refused host still wrote a syrf-api writer-registry entry (s2-r2-7-legacy-api.txt) |
Superseded by #4414: a Legacy API now starts against the imported authority, serves deployed values and marks its registry entry read-only; the runtime-fleet legacy-return scenario restarts all four hosts with only Mode=Legacy (legacy-return R10.5: api-1 serves the deployed values with the preserved policy state …) |
| Refusals | Rehearsal: no legacy document, combined modes, same input and output, existing output, no actor, an imported source | Runtime policy lineage refused: …, exit 2 |
PASS: each message as in the runbook table, exit 2 (s2-db2-r1-r2.log) |
Also RuntimePolicyLineageImportTests |
2a. Initialise (no legacy document, 6C.6)¶
Run with bash e2e/run-local.sh --runtime-fleet-initialise --iterative --scenario initialise-policy-change
(add convergence to flip on the initialised authority too). The stack's database has no global flag document:
the bring-up initialises the authority from the deployed values before any host starts, and its artifacts and logs
stay in /tmp/<stack>-runtime-fleet/initialise-*. The named-only scenario initialise-policy-change (never part
of all) judges them and copies them to <out>/initialise/. Outcomes are from the 6C.6 rehearsal on Bramble at
29e1ee568 (evidence 6C6-initialise/ in the programme handover directory; 40 assertions, 0 failed).
| Step | How it is exercised | Expected transcript line | Outcome (6C.6) | Gap and what would exercise it |
|---|---|---|---|---|
| Before R2.I2: no global document | rtf_seed_registry (seeds only the writer registry) |
[runtime-fleet] preflight: no global flag document (initialise may run); PASS initialise preflight: the database held no global flag document before initialise -- RTF_GLOBAL_DOCUMENTS=0; RTF_STATE=absent |
PASS: RTF_GLOBAL_DOCUMENTS=0; RTF_STATE=absent (initialise-preflight.txt) |
None |
| R2.I1 input | rtf_initialise_input_json (approval and provenance only) |
Nothing printed; initialise-input.json kept |
PASS: approval and provenance only (initialise-input.json) |
None |
| R2.I2 preview, twice | rtf_lineage_run preview … SYRF_RUNTIME_POLICY_SOURCE=initialize to two paths, then cmp |
[runtime-fleet] initialise preview (initialise-preview), … (initialise-preview-again); PASS initialise preview: two previews of the same input are byte-identical -- previewDigest <digest> |
PASS: both previews byte-identical, digest 0B350ECD…8A05 |
None |
| R2.I3 review | Driver reads the preview | PASS initialise preview: kind deployedValueInitialization, revision 0, no overrides, no stops, a Boolean for every key -- <n> baseline keys; on: <keys> |
PASS: 54 baseline keys; on: disableMembership, maxInProgressSessions (the deployed values) |
No human review; the evidence directory holds the file |
| R2.I5 apply | rtf_lineage_run apply with an actor and a freeze acknowledgement |
[runtime-fleet] initialise apply; PASS initialise apply: the receipt is kind deployedValueInitialization for the reviewed preview digest at revision 0 -- receipt <id>, store epoch <epoch> |
PASS: receipt 70ee3bbc-…, revision 0, store epoch fa4fbee1-… |
None |
| R2.I6 replay | Apply again with the same preview to a new path | [runtime-fleet] initialise replay (…); PASS initialise replay: the identical reviewed preview returned the original receipt |
PASS: initialise-receipt-replay.json identical to the receipt |
None |
| Refusal after apply | Preview again after apply | PASS initialise refusal: a preview after apply exits 2 because an authority exists -- refused: An initialized runtime policy authority already exists; initialise never replaces it. |
PASS: exit 2 with that message | Other refusals: RuntimePolicyLineageInitializeTests |
| After runbook 3 | Four Shared hosts start on the initialised authority | PASS initialised authority: stored at revision 0 …, PASS initialised authority: api-1 reports Shared/Ready at the receipt's store epoch, revision 0 (and api-2), PASS initialised authority: api-1 evaluates disableMembership to the deployed value (true) at revision 0 (and api-2), PASS initialise-policy-change (Static inventory): fleet is Active before the disruption |
PASS: stored at revision 0; both APIs Shared/Ready at the receipt's store epoch, disableMembership true; the Static fleet Active |
None |
| R6.3, R3.4 Collector mode | Both APIs restarted with the report profile (collector inventory plus the report reader); the driver publishes the inventory |
PASS initialise-policy-change: in Collector mode, before any inventory is published, the fleet is Unknown with a Collector reason, PASS initialise-policy-change: with the inventory published, the fleet reaches Active in Collector mode |
PASS: Unknown (CollectorInventoryMissingOrInvalid) before publication, then Active |
A real collector and Kubernetes rollout: staging |
| R7.3 report and checker | GET …/compatibility-reports/current?purpose=policy-change with the mock OIDC report token; scripts/runtime-compatibility-check.py |
PASS initialise-policy-change: the policy-change report is Complete, every required source Current, migrations Ready, PASS initialise-policy-change: the checker passes the policy-change report (exit 0, verdict pass, no reasons), PASS initialise-policy-change: control, the same report under a foreign binding epoch is refused (exit 1, binding-mismatch) |
PASS: report 50a6155c-… Complete on the first attempt; checker {"exitCode": 0, "reasons": [], "verdict": "pass"}; the foreign-epoch control exit 1 (binding-mismatch) |
None |
| Production admission of the initialised authority | Not exercised (RuntimeEnvironment=local) |
n/a | n/a | RuntimeProductionInitializeIntegrationTests (production-like configuration: readers, evaluator, Static and Collector fleet admission) |
3. Consumer enrolment¶
| Step | How it is exercised | Expected transcript line | Outcome (6C.2) | Gap and what would exercise it |
|---|---|---|---|---|
| R3.1–R3.3 configuration | rtf_instance_env sets Mode=Shared, SharedReaderMode=Disabled, the binding, InstanceUid, ImageVersion, ExpectedReplicas=2, Autoscaled=false and, on APIs, FleetExpectedReplicas 2/2 |
[runtime-fleet] started api-1 (api, static inventory) on port <port>, pid <pid>, log <file> for each of the four hosts |
PASS: four started lines (runA-console.log.gz, s3-start-static.txt) |
Chart rendering: runtime_policy_host_metadata_test.yaml (API and PM charts) |
| R3.4 checks | Driver; rehearsal requests | PASS admin logged in on both API processes -- api-1, api-2, then PASS fleet is Active for disableMembership before any flip -- state Active and PASS fleet lists four hosts -- 4 hosts |
PASS: the three lines; authority-status Shared/Ready on both APIs, evaluation Evaluated, snapshot with authority, Legacy PUT 403 Runtime changes are not available in this mode. (s3-r3-4-preflight.txt) |
Closed: the rehearsal recorded authority-status |
| R3.4 publish loop | Rehearsal: the runbook's loop with the real capture (stand-in kubectl) and publish scripts |
Fleet Unknown with a Collector… reason, then Active within 60 s |
PASS: Unknown CollectorInventoryStaleOrInvalidClock, Active 2.2 s after the loop started; cycles 1.3–1.5 s plus a 2 s pause (s7-collector-mode.txt) |
A real cluster for the capture |
| Refusal: start-up configuration | Not exercised | n/a | NOT EXERCISED: not attempted (the R2.7 one-off launch would serve) | RuntimeProductionPolicyTests and RuntimePolicyAuthorityRegistrationTests; a negative control could start one host with SharedReaderMode=Membership and expect Shared authority mode cannot also enroll the legacy-format shared reader. |
4. Collector identity and scope¶
| Step | How it is exercised | Expected transcript line | Outcome (6C.2) | Gap and what would exercise it |
|---|---|---|---|---|
| R4.1 API settings | Rehearsal: both APIs started with the WorkloadCollector settings (Auth0, syrf-e2e-runtime-collector) |
No start-up refusal; publications accepted | PASS (s3-start-static.txt, publish-receipts.log) |
Also EnvironmentCollectorAuthorizationTests, EnvironmentCollectorHttpTests |
| R4.2 token | Mock OIDC client credentials (#4409 mapping) | A token with only the collector scope | PASS: claims sub syrf-e2e-runtime-collector@clients, azp, gty client-credentials, scope environment:observations:write, the API audience (s4-collector-static-reports.txt) |
None |
| R4.3 capture | Rehearsal: scripts/capture-runtime-workload.py against a kubectl stand-in |
Exit 0, a new file; never overwritten | PASS (stand-in): exit 0; to an existing path, Capture refused: Output already exists; preserve it for exact replay exit 2 (s4/) |
A real cluster: the D1 PR preview |
| R4.4 publish | Rehearsal: scripts/publish-runtime-workload.py to POST /api/environment/observations on loopback |
Exit 0; stdout only {"observationId": …, "digest": …} |
PASS; refusals: HTTP 409 exit 1, local refusal exit 2, report-scope token HTTP 401 exit 1 (s4-collector-static-reports.txt) |
None |
| R4.5 project-management | Rehearsal | As R4.4 | PASS | None |
| R4.6 continuous publication | Rehearsal: the R3.4 loop | Both workloads:* sources Current while the loop runs |
PASS: 22 cycles, 44 accepted, 0 refused (publish-receipts.log) |
The OD-23 scheduler |
5. Coordinator identity¶
| Step | How it is exercised | Expected transcript line | Outcome (6C.2) | Gap and what would exercise it |
|---|---|---|---|---|
| R5 preflight | Rehearsal reports | deploymentConstraint source Current, projection null |
PASS (s7/) |
None |
| R5.1–R5.4 | Not exercised | n/a | NOT EXERCISED: no coordinator client in the mock OIDC; nothing builds the capability intersection or completion targets (gap 10) | RuntimeDeploymentCoordinatorTests, EnvironmentCoordinatorHttpTests, RuntimeDeploymentConstraintTests, RuntimeDeploymentCompletionTests |
6. Expected inventory and Collector mode¶
| Step | How it is exercised | Expected transcript line | Outcome (6C.2) | Gap and what would exercise it |
|---|---|---|---|---|
| R6.1–R6.2 static inventory | Harness configuration (2 API, 2 PM) | PASS fleet lists four hosts -- 4 hosts |
PASS | None |
| Unexpected extra host | scale-up-unexpected scenario |
PASS scale-up-unexpected: the unexpected api-3 acknowledged and appears in the fleet, PASS scale-up-unexpected: while api-3 is listed the fleet is never Active and is Applying, Degraded or Action required, PASS scale-up-unexpected: the fleet returned to Active within 60s of the cleanup |
PASS: all three (runB-console.log) |
The cleanup deletes api-3's row by hand (test only). Collector retirement replaces this in Kubernetes |
| Silent host | convergence stop/resume part |
PASS stop: fleet never Active while one API process is stopped, and showed Applying or Degraded -- states seen …, PASS resume: fleet reached Active |
PASS: states Saved, Applying, Degraded; Active 512 ms after resume (runA-console.log.gz) |
None |
| R6.3 Collector mode | collector-scale-up scenario (#4409); rehearsal: both APIs restarted with --host-control start <api> collector and the R3.4 loop |
PASS collector-scale-up: in Collector mode, before any inventory is published, the fleet is Unknown with a Collector reason, the a, b and c lines, … every inventory publication during the scenario was accepted and the two cleanup lines |
PASS: all 13 lines; a stopped api-3 listed Retired (AbsentFromFreshCollectorInventory), Active 35.1 s after its last serverSeenAtUtc (runB-console.log, s7-collector-mode.txt) |
A real Kubernetes rollout still needs the D1 preview or staging |
| Non-production key | Rehearsal: the fleet for newStageOverview |
Unknown / DomainReadinessNotEstablished |
PASS (s3-r3-4-preflight.txt) |
None |
| Zero replicas, autoscaled | Not exercised | n/a | NOT EXERCISED: the harness fixes ExpectedReplicas=2 and Autoscaled=false |
RuntimePolicyFleetTests; the runtime_policy_host_metadata_test.yaml chart tests |
| R6.4 counts together | Not exercised | n/a | NOT EXERCISED: a cluster-gitops review rule | Staging |
7. Change one flag¶
| Step | How it is exercised | Expected transcript line | Outcome (6C.2) | Gap and what would exercise it |
|---|---|---|---|---|
| R7.1–R7.2 | Driver reads authority-status and the fleet before each flip |
PASS fleet is Active for disableMembership before any flip -- state Active |
PASS | None |
| R7.3 report and checker | Rehearsal: reports from the stack, synthetic candidate manifests, the checker | Static: Incomplete with fleetInventoryMode:Static. Collector: Complete; checker {"exitCode": 0, "reasons": [], "verdict": "pass"} |
PASS: Static Incomplete, checker insufficient-evidence exit 2; Collector Complete, checker pass exit 0 (s4/, s7/) |
Main's harness needs a report mapping and client (deviations 1 and 2) |
| R7.4–R7.5 preview and commit | convergence: preview, then commit with the digest, for each flip; rehearsal with "action": "Change" |
flip 1: disableMembership=false committed at revision <n> |
PASS: flips 1–4 at revisions 1–4; rehearsal preview 200 then commit at revision 6 (s5-r7-change.txt) |
None |
| R7.6 adoption | convergence |
PASS flip 1: every API decision adopted within 60s -- {…}, PASS flip 1: every process acknowledged within 60s -- {…}, PASS flip 1: fleet reached Active -- <ms> ms after commit, repeated per flip |
PASS: 12 lines, Active 8.0–12.0 s after each commit | None |
| R7.6 browser adoption (6B.4) | runtime-fleet-browser-convergence (run-local.sh --runtime-fleet --spec runtime-fleet-browser-convergence): tab A on api-1, tab B on api-2 (membership page), off, on, off, on committed alternately on each API |
[RTF_BROWSER] PASS every change is visible in both tabs (api-1 and api-2) within 60 s of the commit response -- …, [RTF_BROWSER] PASS no focus, online or offline event reached tab A or B while it converged, and neither tab reloaded -- …, [RTF_BROWSER] PASS every convergence in tab A or B followed a revision hint frame (not a window event or the 5-minute fallback poll) -- …, [RTF_BROWSER] PASS a tab opened after each change renders the committed revision and value from its bootstrap read (api-1 and api-2) -- … |
PASS (6B4-browser-convergence/ runs 4 and 5 with #4425, 2026-10-23; run 4 figures): every flip in both tabs, all after the hint. Tab on the writing API: 84–267 ms (median 245 ms). Tab on the other API: 0.5–11.1 s (median 6.8 s). No event and no reload: PASS. Fresh tabs render the committed revision: PASS, 12 of 12. Before #4425 (run 3): FAIL, because the writing API's tab caught none of its changes within 60 s |
None (#4425 merged) |
| R7.6 browser recovery (6B.4) | runtime-fleet-browser-convergence (run-local.sh --runtime-fleet --spec runtime-fleet-browser-convergence): tab C on api-2 with every RuntimeFeatureFlagsRevision hub message dropped |
[RTF_BROWSER] PASS tab C's hub connection carried the revision hint and the dropper removed it (step 7), [RTF_BROWSER] PASS with the hint dropped and no event, tab C stays stale for 75 s after the commit, [RTF_BROWSER] PASS with the hint dropped, tab C shows the change within 15 s of a focus event, [RTF_BROWSER] PASS with the hint dropped, tab C misses a change while its requests fail and shows it within 15 s of an online event |
PASS (run 3 and 6B4-browser-convergence/ runs 4 and 5 with #4425, 2026-10-23; run 4 figures): stale for 75 s with the hint dropped and no event; then 252 ms after focus and 240 ms after online (requests aborted, then the event dispatched; context.setOffline resets every socket in the browser) |
Recovery after a hub reconnect: the runtime-fleet-browser-recovery spec (next 6B.4 PR) |
| R7.7 lookup | Rehearsal: GET /api/runtime-feature-flags/mutations/<mutation-id> |
outcome: Committed |
PASS: Committed with committedIdentity; an unknown ID gives Unknown with no identity |
Closed by the rehearsal step |
| Refusals (409, stale digest, reuse) | Rehearsal | 409 runtime_policy_revision_conflict; 400 The reviewed preview is missing or stale…; 400 Mutation ID was already used with a different request. |
PASS: all three, and an unchanged retry returned the original receipt | Also RuntimePolicyMutationStoreTests |
| Refusal 422 (dependencies) | Not exercised | n/a | NOT EXERCISED: disableMembership has no dependencies |
API administration tests |
| Production refusals | Not exercised | n/a | NOT EXERCISED: RuntimeEnvironment=local |
RuntimeProductionPolicyIntegrationTests |
8. Emergency stop and restore¶
| Step | How it is exercised | Expected transcript line | Outcome (6C.2) | Gap and what would exercise it |
|---|---|---|---|---|
| Prior override | stop-under-outage sets the override On first |
PASS stop-under-outage: the prior override (On) is established and the fleet is Active -- revision <n> (only if it was not already On) |
PASS: not printed, the override was already On | None |
| R8.2 stop with a PM host paused | stop-under-outage; rehearsal stop |
PASS stop-under-outage: the emergency stop commit succeeds while pm-2 is paused -- HTTP 200 revision <n> |
PASS: revision 9; rehearsal stop revision 7 (runB-console.log, s5-r8-stop-restore.txt) |
None |
| R8.3 stop adoption | stop-under-outage |
PASS stop-under-outage: every running API evaluates the stopped value (false) at revision <n> within 60s, PASS stop-under-outage: the fleet is not Active while pm-2 is paused (and a non-Active state was observed), PASS stop-under-outage: after SIGCONT every process acknowledges the stop and the fleet is Active |
PASS: all three; a retried stop with the same mutation ID returned the same revision | None |
| R8.3 browser stop visible (6B.4) | runtime-fleet-browser-convergence (run-local.sh --runtime-fleet --spec runtime-fleet-browser-convergence), step 5: emergency stop committed on api-1 |
[RTF_BROWSER] PASS every stop is visible in both tabs (api-1 and api-2) within 60 s of the commit response -- …, [RTF_BROWSER] PASS the admin feature-flags page shows the membership stop after the stop and not after the restore -- … |
PASS (6B4-browser-convergence/ runs 4 and 5 with #4425, 2026-10-23; run 4 figures): the control was hidden 264 ms after the stop on the writing API's tab and 1.9 s after on the other's. The admin page showed the stop: PASS. Before #4425 (run 3): FAIL, the writing API's tab not within 90 s |
None (#4425 merged) |
| R8.4 restore | stop-under-outage (ordinal action); rehearsal ("action": "ReleaseStop") |
PASS stop-under-outage: the ReleaseStop preview succeeds, PASS stop-under-outage: the preview restores the prior override, PASS stop-under-outage: the ReleaseStop commit succeeds -- HTTP 200 revision <n> |
PASS: all three; the name form was accepted too | None |
| R8.5 restore adoption | stop-under-outage |
PASS stop-under-outage: every API evaluates the restored prior override (true) at revision <n> within 60s, PASS stop-under-outage: the fleet is Active after the release |
PASS | None |
| R8.5 browser restore visible (6B.4) | runtime-fleet-browser-convergence (run-local.sh --runtime-fleet --spec runtime-fleet-browser-convergence), step 6: ReleaseStop committed on api-2 |
[RTF_BROWSER] PASS every restore is visible in both tabs (api-1 and api-2) within 60 s of the commit response -- … |
PASS (6B4-browser-convergence/ runs 4 and 5 with #4425, 2026-10-23; run 4 figures): the restored override showed 251 ms after the commit on the writing API's tab and 10.9 s after on the other's. The admin page showed not stopped: PASS. Before #4425 (run 3): FAIL, 85.3 s by the 5-minute poll |
None (#4425 merged) |
| Production restore refusal | Not exercised | n/a | NOT EXERCISED: local environment | RuntimeProductionPolicyIntegrationTests |
9. Recovery and rebind¶
| Step | How it is exercised | Expected transcript line | Outcome (6C.2) | Gap and what would exercise it |
|---|---|---|---|---|
| R9.1 lost outcome | Rehearsal, as R7.7 | Committed or Unknown |
PASS | None |
| R9.1 browser reload recovery (6B.4) | runtime-fleet-browser-recovery (next 6B.4 PR; not written yet): a reload during an unresolved commit resolves by the original mutation ID |
Saved by GET mutations/<original id>; no preview, policy or emergency-stop request after the reload; exactly one new audit receipt |
Pending: spec not written yet | The recovery spec, then a Bramble run |
| R9.2 lineage retry | Rehearsal, as R2.6 | The original receipt | PASS | None |
| R9.3 publication replay | Rehearsal: the exact accepted capture published again | The same receipt | PASS: same observationId and digest, exit 0 |
The exit 3 path itself: test-publish-runtime-workload.py |
| R9.4 Mongo outage | mongo-outage |
PASS mongo-outage: api-1 left Evaluated within 35s with value null and safe value false -- … (and api-2), PASS mongo-outage: the fleet was never Active during the outage -- …, PASS mongo-outage: every API returned to Evaluated at revision <n> within 60s of the unpause, PASS mongo-outage: fleet Active within 60s of the unpause |
PASS: fail-closed after 12.0 s and 18.0 s; Active 12.1 s after the unpause | None |
| Restarted host | restart |
PASS restart: the fleet is not Active from ~30s after api-2 stopped (checked from 35s), PASS restart: the restarted process's first evaluation is Evaluated at the committed revision <n> with the prior value, PASS restart: the acknowledgement row carries a new processStartId, PASS restart: fleet Active within 60s of starting api-2 |
PASS: all four | None |
| R9.5 quarantine | Not exercised | n/a | NOT EXERCISED: causing one needs a hand edit of the binding or authority | RuntimePolicyAuthorityReaderTests; the recovery procedure itself is not written |
| R9.6 break-glass | Not exercised | n/a | NOT EXERCISED: needs an active deployment constraint (runbook 5) | RuntimeDeploymentBreakGlassIntegrationTests, RuntimeDeploymentBreakGlassHttpTests |
| R9.7 rebind | Not possible | n/a | NOT EXERCISED: rebind is not built |
None until it is built |
10. Legacy return¶
| Step | How it is exercised | Expected transcript line | Outcome (6C.2) | Gap and what would exercise it |
|---|---|---|---|---|
| R10.1 values agree | Rehearsal: a forward revision to the deployed value (true in the E2E defaults) |
legacyValue equal to runtimeValue |
PASS: the Static report showed legacyValue true, runtimeValue false; after the R7 revision both were true (s4/, s7/) |
None |
| R10.2 report | Rehearsal, Collector mode | Complete, legacyParity.agrees: true, stopActive: false |
PASS (s7/report-collector-legacy-return.json) |
Closed by #4414: the harness's report mapping and the legacy-return scenario (PASS legacy-return R10.2: the legacy-return report is Complete, its parity agrees and no stop is active) |
| R10.3 checker | Rehearsal, plus the offline fixtures (test_legacy_return, test_legacy_stop_active_names_key, test_legacy_differs_names_key, test_legacy_parity_missing) |
{"exitCode": 0, "reasons": [], "verdict": "pass"} |
PASS: pass exit 0; with a stop active, refuse exit 1 with legacy-parity-stop-active (s8/) |
None |
| R10 STOP-box proof | Rehearsal: one API with Mode=Legacy against the fleet's imported authority |
Legacy runtime policy provider cannot serve initialized authority. |
PASS: the refusal, no listener; the Shared fleet stayed Active (s8-legacy-return.txt) |
Superseded by #4414: the STOP box is replaced by R10.4 to R10.6 |
| R10.4 mode switch | legacy-return scenario (#4414): api-2, api-1, pm-2, pm-1 restarted with only RuntimeFeatureFlags__Mode=Legacy |
PASS legacy-return: after 35s in Legacy mode the authority document digest and the audit row count are unchanged |
NOT EXERCISED: blocked (STOP box, owner decision D7) | Exercised by the legacy-return scenario |
| R10.5 confirm | legacy-return scenario (#4414): snapshot, provider status, authority-status, a refused legacy change, the registry marker |
PASS legacy-return R10.5: api-1 authority-status is Legacy, Disabled, LegacyModeAuthorityPreserved, with no identity |
NOT EXERCISED: follows R10.4 | Exercised by the legacy-return scenario |
| R10.6 return to Shared | legacy-return scenario (#4414): every host restarted in Shared static mode |
PASS legacy-return: Shared resumes at the same revision and identity (all seven fields) |
n/a (added by #4414) | None |
Acceptance for 6C.2¶
- Every row whose "How it is exercised" is not "Not exercised" shows its expected transcript line. Met: every exercised row is PASS.
evidence.jsonhaspass: true, and every scenario haspass: true. Met for both driver runs.- Each gap row is either closed by an added harness step (with its transcript line recorded here) or carried into the staging entry criteria as an accepted gap. Partly met: the rehearsal closed R1.5, R2.3, R2.4, R2.6, R2.7, R3.4, R4.1–R4.6, R7.3, R7.7, the R7 refusals and R10.1–R10.3 with rehearsal steps, not harness steps. The 13 NOT EXERCISED rows are for Chris to accept as staging entry gaps.
- No step used a shared environment or a production credential. Met.