Runtime feature flags staging rehearsal plan and production prerequisites

This page holds two written plans from the remaining delivery plan:

• the staging rehearsal plan (6C.3), with entry and exit criteria for Chris's later gate;
• the production prerequisites list (6C.4), with an approval record template.

Every step points to a runbook step in Operate runtime feature flags (R<runbook>.<step>). The local rehearsal that comes first is in the isolated rehearsal checklist.

Roll-forward means the next reviewed forward change if a step fails or must be reversed. Deployments and data only move forward (ADR-023); a rollback is a planned replacement release.

Staging rehearsal plan (6C.3)

NOT EXECUTED. This is a plan. No step below has been run in staging, in a PR preview or in production. Staging execution needs Chris's explicit decision (owner gate, delivery plan §8).

Scope

• One control: disableMembership (D2). Statistics keys are not enrolled as runtime decisions.
• API and PM are promoted together (D4).
• The rehearsal proves the mechanism: bind, import, enrol, collect, change, stop, restore and the Legacy-return checks. It is evidence about the mechanism, never authorisation for production.
• The RT-69 part of OD-26 (guide page and What's New entry appearing at the moment of the switch) needs the release and content slices (6D, 6G). It is listed in the exit criteria as not yet possible.

Stage 0: throwaway PR preview (D1)

Before staging, the same sequence may run in a throwaway PR preview with its own database and no real users, then be torn down. Gaps that block it today:

• previews do not run bind or up, so migrations 0009–0021 are never applied there (runbook 1 preconditions);
• the lineage command has no in-cluster delivery mechanism (runbook 2);
• the machine-client secrets have no chart wiring on Identity (runbook 4).

Until those are resolved, Stage 0 is limited to what the isolated rehearsal covers.

Entry criteria for Chris's gate

Each criterion names how it is verified.

1. 6A and 6B are merged (#4407, server-time retirement, merged as 9174f9e9). Verify: PR states.
2. The isolated rehearsal (6C.2) transcript shows every row of the checklist as expected, or as an accepted gap. Verify: the stored evidence directory.
3. Staging's API and PM run the same main build, which contains migrations 0009–0021. Verify: /health/live versions and migrations.lock.json at that commit.
4. The pre-deploy migration Job can be enabled in staging (pre-deploy plan R1). Verify: the reviewed cluster-gitops change.
5. The known gaps that block a step are resolved or explicitly accepted by Chris: status/verify delivery, lineage delivery, Identity secret wiring, collector scheduler, writer freeze mechanism. All 16 items are tracked in #4403; items 14 and 15 wait on owner decisions D7 and D8 on the programme progress page. Verify: each item has a linked PR or a recorded decision.
6. A staging read-only Kubernetes identity for the collector exists (precedent syrf-stats-preflight). Verify: its role definition reference.
7. No active deployment constraint and no other runtime-policy change is planned during the rehearsal window. Verify: report deploymentConstraint projection is null.
8. A rehearsal window and a named operator are recorded in the approval record. Verify: the record.

Steps

Step	Runbook ref	Owner	Approver	Evidence	Roll-forward
S1 Read the live fingerprint	R1.1	Operator	Chris	status output	Read-only; none needed
S2 Enable the migration Job and bind	R1.2–R1.4	Operator	Chris (cluster-gitops PR)	PR URL; bind and up logs with epoch	A failed PreSync keeps the previous release serving; fix forward with a new commit. A wrong binding cannot be replaced (rebind not built): stop and escalate
S3 Verify migrations	R1.5	Operator	n/a	verify output or report migrations: Ready	Forward migration fix in a new release
S4 Provision the report and collector clients	R4.1–R4.2, R7.3	Operator	Chris (secrets owner)	Client registration change (no secrets)	Revoke by removing the secret (Identity revokes the client)
S5 Run the collector for API and PM	R4.3–R4.6	Operator	n/a	Capture files and receipts	Replay the exact capture on exit 3; new capture otherwise
S6 Review the baseline JSON	R2.1	Operator	Chris	Baseline file, approval, provenance	Edit and re-review before S7
S7 Lineage preview	R2.2–R2.3	Operator	Chris (reviews the file)	Preview file and digest	Discard; make a new preview
S8 Freeze and scale the API to zero	R2.4	Operator	Chris	Freeze reference, zero-pod proof	Restore replicas with a forward change if S9 is abandoned before apply
S9 Lineage apply	R2.5–R2.6	Operator	Chris	Receipt file, logs	No undo. Correct values with a reviewed policy revision (S13). A wrong lineage needs rebind (not built)
S10 Enrol API and PM with inventory and Collector mode	R3.1–R3.3, R6.1–R6.3	Operator	Chris (cluster-gitops PR)	PR URL, rendered env diff	Forward values fix. Never return the API to Legacy (it refuses the imported authority)
S11 Resume the manual capture-and-publish loop (cycle under 30 s), post-enrolment checks; end the freeze	R3.4	Operator	n/a	Loop receipts; authority-status, authority-evaluation, fleet responses (Unknown with a Collector… reason is expected until fresh observations land)	Runbook 9 cases
S12 Policy-change report and checker	R7.3	Operator	n/a	Report file, checker output	Fresh report
S13 Flip disableMembership on	R7.1–R7.6	Operator	Chris	Preview, receipt, fleet Active timing	Reviewed forward revision (off) or stop (S15)
S14 Flip it off	R7.1–R7.6	Operator	Chris	As S13	Reviewed forward revision
S15 Emergency stop	R8.1–R8.3	Operator	Chris	Stop receipt, per-host adoption	Restore (S16)
S16 Restore	R8.4–R8.5	Operator	Chris	Preview, receipt, fleet Active	Another stop
S17 Lost-response drill	R7.7, R9.1	Operator	n/a	Lookup result; exact replay returns the original receipt	None needed
S18 Rollout drill: one ordinary API and PM rollout	R6 notes	Operator	Chris	Fleet states during rollout; old rows Retired; Active again	Forward change
S19 Legacy-return checks only (no mode switch)	R10.1–R10.3	Operator	n/a	Report legacyParity, checker output	Read-only. R10.4 is executable since #4414 but is not part of this rehearsal
S20 Optional: coordinator record, extend, release	R5.1–R5.4	Operator	Chris	Receipts, report before and after	Release, or break-glass (S21)
S21 Optional: break-glass drill	R9.6	Operator (human administrator)	Chris	Reason, receipt, report	None needed
S22 Store the evidence and approval record	All	Operator	Chris	Run record	n/a

Exit criteria

1. Every executed step has its evidence stored with the run record.
2. Each flip (S13, S14) reaches fleet Active on its committed identity within the 60-second confirmation window, with per-host adoption times recorded.
3. The stop (S15) reaches every API decision and every host acknowledgement; restore (S16) brings back the saved override.
4. A policy-change report is Complete in Collector mode, and the checker passes it (S12).
5. The rollout drill (S18) returns to Active without operator action.
6. The Legacy-return checks (S19) pass.
7. No step needed a production credential.
8. RT-69 (OD-26): not possible until 6D and 6G exist; recorded as outstanding.
9. Chris reviews the record and decides on production separately.

Production prerequisites (6C.4)

NOT EXECUTED. This is a list of prerequisites, not a record of anything done. Every production step needs Chris's approval at the time; merging a production cluster-gitops PR deploys it.

Step	Runbook ref	Owner	Approver	Evidence	Roll-forward
P1 Catch-up promotion: production on main's runtime-flag code in Legacy mode, API and PM at the same build (D4)	CD plan, D4	Release operator	Chris (production PR)	Promotion PR, ArgoCD sync, /health/live versions for both services	A planned replacement release. The planned rollback to 104eca9 with migration 4's Down is the one built exception (ADR-023)
P2 ADR-023 Job enabled in production, with the binding in the same change	R1.1–R1.5	Release operator	Chris (production PR)	status output, bind and up logs, verify output	Failed PreSync keeps the previous release; fix forward. A wrong binding needs rebind (not built)
P3 OD-21 automation identity: Auth0 machine application, scope environment:report:read, credential in a main-only GitHub environment	R7.3	Release operator	Chris (production-shared Auth0 tenant change)	Tenant change record (no secrets), API CompatibilityReport settings PR	Remove the grant or the application
P4 OD-23 collector: read-only Kubernetes identity and scheduler in camarades-infrastructure, plus an Auth0 machine application for environment:observations:write	R4.1–R4.6	Infrastructure operator	Chris (camarades-infrastructure PR, Auth0 change)	Infrastructure PR, first capture and receipt per service	Remove the identity or disable WorkloadCollector
P5 Coordinator identity and workflow, scope environment:deployments:coordinate	R5.1–R5.4	Release operator	Chris	Workflow PR, Auth0 change, first receipts	Disable DeploymentCoordinator; break-glass for a stranded constraint
P6 Chart and configuration values for API and PM: Mode=Shared, ProductionPolicyMode=Runtime, SharedReaderMode=Disabled, binding, runtimePolicyHostMetadata.enabled, hpa.enabled: false, fixed replicaCount; API also FleetInventoryMode=Collector and both FleetExpectedReplicas	R3.1, R6.1–R6.4	Release operator	Chris (production PR)	Rendered env diff, post-rollout checks	Forward values fix, or ProductionPolicyMode=Legacy (runbook 10, once P11 is in the production build)
P7 D4 together-deploy and the #4354 mixed-version window	D4	Release operator	Chris	Both services on one build before any admin step	Finish the rollout; retry after Retry-After
P8 Collector-mode requirement: production admission follows FleetInventoryMode (since #4407), so set Collector and keep the collector publishing (P4)	R6.3, R3.4	Release operator	Chris	A production build containing #4407; a rollout drill showing old rows Retired after the 35-second quiet period (up to 15 minutes for rows from pre-#4407 images) and the fleet Active again	In Static mode (the default) enables and restores are refused after any rollout; stops still work
P9 Initialise the production authority from the deployed values, with the writer freeze (API outage). Production has no legacy policy document (legacy mutation is disabled there), so it uses initialise, not the lineage import (D8=A)	R2.I1–R2.I7	Release operator	Chris (window, reconciled rendered values, preview)	Input, both previews and the cmp result, reconciled API and PM rendered values, freeze reference, receipt, verify output	No undo. Correct values with a reviewed forward policy revision; a wrong lineage needs rebind (not built). Needs an in-cluster delivery mechanism first (gap 3). Rehearsed only on an isolated stack (run-local.sh --runtime-fleet-initialise, checklist)
P10 Recovery exercise and freshness trial (increment C)	Runbook 9, 6B evidence	Release operator	Chris	Exercise record; measured adoption times	n/a
P11 Legacy-return path in the production build: #4414 (D7=A) promoted to production before any import there	Runbook 10	Developer	Chris	Merged PR, the runtime-fleet legacy-return scenario evidence, and the production build containing it	n/a
P12 First enable as a release (OD-26): guidance under OD-5 and the RT-69 pre-enable check	R7, RT-69	Product owner	Chris	Guide page and What's New check before and after	Emergency stop (runbook 8)
P12a Release guidance bundle mounted read-only for the API (6D.5, D-Q6): compile the reviewed manifest, store the server-<sha256>.json bundle in a Secret keyed by that file name, and set runtimeFeatureReleases.guidanceBundle.secretName and fileName in the production API values. Without it every guidance-required Apply is refused with GuidanceBundleMissing (emergency stops are unaffected)	Applying a release	Release operator	Chris (production PR, Secret source)	Compiler output and digest, the Secret's provenance (no content in the record), rendered API env and mount diff, an Apply preview showing guidanceState: Verified with the expected guidanceBundleDigest	Unset the values (admission refuses again, fail closed), or mount a newly compiled bundle under its new name
P13 Production enable of disableMembership	R7.1–R7.6	Release operator	Chris	Report, checker, preview, receipt, fleet Active	Emergency stop, then a reviewed forward revision
P14 Approval record completed for each of P1–P13 (including P12a)	Template below	Release operator	Chris	Stored record	n/a

Approval record template

Copy this block once per step or per rehearsal run. Use placeholders for anything secret.

Approval record
---------------
Step ID(s):                <S… or P…>
Environment:               <env> (binding epoch <binding-epoch>)
Runbook steps:             <R…>
Operator:                  <name>
Approver:                  <name>
Approval reference:        <chat message, PR approval or issue comment URL>
Window (UTC):              <start> to <end>
Freeze reference:          <reference, if runbook 2>
Builds:                    api <image@sha256>, project-management <image@sha256>
Preconditions checked:     <list, with the evidence for each>
Commands or requests run:  <list; no tokens, no secrets>
Expected output seen:      <yes/no per preflight check, with excerpts>
Refusals and handling:     <code, action taken, retry mutation IDs>
Evidence stored at:        <path or link>
Roll-forward needed:       <none / description and PR>
Outcome:                   <completed / stopped at step …>
Follow-ups:                <issue links>