Operate runtime feature flags (operator runbooks)

These runbooks cover the operator side of runtime feature flags: binding an environment, importing the legacy policy lineage, enrolling API and Project Management (PM), running the workload collector and the deployment coordinator, setting the expected inventory, changing, stopping and restoring a flag, recovery, and the return to Legacy. They are slice 6C.1 of the remaining delivery plan.

The feature behaviour, endpoint contracts and design history are in Managing Feature Flags. This guide does not repeat them. Where the two seem to differ, the code on main decides, and the difference is listed in Known gaps.

Related plans:

• Staging rehearsal plan and production prerequisites (6C.3, 6C.4);
• Isolated rehearsal checklist (6C.2, run in the local stack on 2026-10-22).

Status and boundary

NOT EXECUTED in staging or production. None of these runbooks has been run against staging or production. Nothing in this repository binds, imports, enrols or activates an environment. A merged PR is not evidence that an environment is ready. The isolated rehearsal (6C.2) ran them in a local E2E stack on 2026-10-22; its outcomes are in the isolated rehearsal checklist.

• Production candidate (decision D2). disableMembership is the first and only production candidate. It is the only key whose consumer contract is productionEligible: true, grain: Global, safeValue: false (src/charts/syrf-common/env-mapping.yaml; RuntimeProductionPolicy.Eligible). Statistics keys stay non-production.
• Deploy API and PM together (decision D4). See Mixed-version window.
• Rehearsals (decision D1). A rehearsal beyond the local stack may use a throwaway PR preview environment. Staging and production remain written plans until Chris approves each step.
• Production prerequisite. Production enables and restores need RuntimeFeatureFlags:FleetInventoryMode=Collector and a running collector (runbook 4). Since #4407 (server-time retirement, merged as 9174f9e9), production admission, the fleet view and deployment completion follow the configured inventory mode. The default is still Static, which keeps every old pod's acknowledgement row: after any Kubernetes rollout those rows stop the fleet reaching Active, so production enables and restores are refused in Static mode. The remaining prerequisite is operational: set Collector mode and keep the collector publishing.
• Forward only. Deployments and data only move forward (ADR-023). A rollback is a planned replacement release. Every "roll-forward" column in the plans means a new reviewed change, never an undo.
• GitOps only. Configuration changes are cluster-gitops PRs. Never use kubectl apply, helm install or direct database edits.

Before you start

Placeholders

Placeholder	Meaning
<env>	Binding environment ID: production, staging, preview:pr-<n> or local:<stack>
<runtime-env>	The RuntimeEnvironment the binding runs as: the part of <env> before the colon
<database>	The configured MongoDB database name
<cluster-fingerprint>	The live cluster fingerprint that status prints, for example <setName>@<hostSuffix>
<binding-epoch>	The binding's epoch GUID, from the bind log line or the authority identity
<api-origin>	The API's HTTPS origin, with no path
<identity-origin>	The Identity service origin (OpenIddict environments only)
<admin-session>	An authenticated real administrator: browser session or user bearer token. Never a machine token
<report-token>, <collector-token>, <coordinator-token>	Machine access tokens, one scope each (runbooks 4 and 5)
<key>	The flag key. For production this is always disableMembership
<identity>	The complete policy identity object, copied unchanged from authority-status (below)
<mutation-id>	A new GUID per intended change. Reuse it only to retry the same request

Never put real values, tokens or secrets in this document, a PR description or a log excerpt.

How settings reach a host

The .NET hosts read SYRF__-prefixed environment variables, with __ meaning :. So the setting RuntimeFeatureFlags:Mode is the variable SYRF__RuntimeFeatureFlags__Mode.

Only two groups of these settings are chart values:

• migrations.enabled and migrations.binding.* (API and PM charts), rendered into the migration Jobs as SYRF_MIGRATION_BINDING_EXPECTED_*;
• runtimePolicyHostMetadata.enabled (API and PM charts), which renders InstanceUid, ImageVersion, ExpectedReplicas and Autoscaled from the Deployment.

Every other RuntimeFeatureFlags:* setting in these runbooks has no env-mapping.yaml entry. It reaches the host only as a raw env: entry in the service's cluster-gitops values. Raw env: entries are literal strings (_deployment-dotnet.tpl), so they must never carry a secret.

TO CONFIRM: whether reviewed chart values should replace these raw env: entries before production.

The policy identity

Every policy request carries the complete current identity. Read it from the API, then copy it unchanged:

GET /api/runtime-feature-flags/authority-status
<admin-session>

Expected (Shared mode, healthy):

{
  "mode": "Shared",
  "host": "api",
  "fleetScope": "local-process",
  "state": "Ready",
  "identity": {
    "environmentId": "<env>",
    "bindingEpoch": "<binding-epoch>",
    "storeEpoch": "<store-epoch>",
    "revision": "<revision>",
    "catalogDigest": "<catalog-digest>",
    "baselineDigest": "<baseline-digest>",
    "policyDigest": "<policy-digest>"
  },
  "observedAtUtc": "<time>",
  "lastAttemptAtUtc": "<time>",
  "reason": null
}

revision is a decimal string. state is one of Ready, Uninitialized, Stale (MaximumAgeExceeded), Unavailable (AuthorityNotInitialized, AuthorityReadFailed), Legacy, Quarantined (InvalidBinding, AuthorityNotInitialized, IdentityOrRevisionRegression) or, in Legacy mode, Disabled with reason LegacyMode, or LegacyModeAuthorityPreserved once that Legacy process has found an initialized authority (runbook 10). This is one API process's view, not the fleet's.

Retry rules

• A 503 with runtime_policy_commit_unknown, a lost response or a timeout does not prove failure. Retry the same mutation ID and the unchanged request, or look the outcome up (R7.7).
• Never mint a new mutation ID because an outcome is unknown.
• A 409 runtime_policy_revision_conflict means the identity moved. Read it again and decide again.

Exit codes of the operator commands

Command	0	1	2	3
Migration entry point (SYRF_MIGRATION_MODE)	Success	Failure (verify problems)	Usage or binding refusal; nothing written	Lease or writer wait timed out
Lineage command (SYRF_RUNTIME_POLICY_MODE)	Artifact written	Unexpected failure; retry the identical reviewed plan	Refused (Runtime policy lineage refused: …)	n/a
Historical import (SYRF_RUNTIME_RELEASE_HISTORY_MODE)	Artifact written	Unexpected failure; retry the identical reviewed preview	Refused (Historical release import refused: …)	n/a
scripts/capture-runtime-workload.py	Capture written	n/a	Capture refused: …	n/a
scripts/publish-runtime-workload.py	Receipt matched	HTTP 3xx/4xx refusal (not 408/425/429)	Local refusal	Outcome unknown; replay the exact capture
scripts/runtime-compatibility-check.py	pass	refuse	insufficient-evidence	unusable-input

Mixed-version window (D4)

When application authority is enforced, an API that asks an older PM about the operation-specific runtime capabilities gets a retryable 503 on the flag-admin endpoints until both run the same version (#4403, from #4354). The 503 is ProblemDetails with code: "authority-unavailable" and a Retry-After header. So:

• promote and sync API and PM at the same build, in one change;
• do not start any runbook step that calls an admin route until both services report the same image;
• treat a 503 authority-unavailable during a rollout as the expected window, not a failed step.

ADR-023 also expects API and PM to run different builds briefly during a rollout. Migrations stay additive for that window.

Evidence store

Keep every artifact named in "Evidence to capture" with the run record (see the approval record template). Preview and receipt files are review evidence: never edit or reformat them.

Runbook order

#	Runbook	Writes to the environment
1	Environment binding	Binding record (once)
2	Lineage preview and apply with writer freeze	Authority document and receipt
3	Consumer enrolment	Deployment configuration
4	Collector identity and scope	Workload observations
5	Coordinator identity	Deployment constraints
6	Expected inventory and Collector mode	Deployment configuration
7	Change one flag	Policy revision and receipt
8	Emergency stop and restore	Policy revision and receipt
9	Recovery and rebind	Depends on the case
10	Legacy return	Deployment configuration
11	Historical release import (written only)	One release record and one event

Runbooks 1 to 3 run once per environment, in that order. Runbook 2 needs a full API outage (no API process may be registered), and runbook 3 must follow it before the outage ends: an API in Legacy mode starts against an imported authority but serves only the deployed values (runbook 10), so the saved runtime policy applies only once runbook 3 has enrolled the hosts.

Runbook 4 is numbered after runbook 3 but must be working before runbook 2 starts. Runbook 3 sets Collector mode, and a Collector-mode fleet is Active only while both services have observations younger than 30 seconds. So configure the collector (R4.1–R4.2) and prove one capture and publication per service (R4.3–R4.5) first, then keep publishing from the moment the API is back (R3.4).

1. Environment binding

Purpose

Bind the database once to one environment ID, database name and cluster fingerprint. Migrations 0009 to 0021 declare RequiresBinding, and every runtime authority read and write checks the binding.

Preconditions

• The service images contain migrations 0009–0021 (check migrations.lock.json at the image's commit).
• The ADR-023 pre-deploy migration Job is available in the API and PM charts (migrations.enabled).
• For staging: Chris has approved the step (owner gate, plan §8). For production: Chris approves the specific cluster-gitops PR.
• No binding exists yet, or the existing one is identical. bind never overwrites a different binding.
• PR previews: previews do not run up or bind today. Their database lifecycle Job runs index-init only, and the migration Job refuses preview.waitForDatabase. TO CONFIRM: how a D1 preview rehearsal gets bind and up.

Steps

1. R1.1 Read the live fingerprint. Run the API or PM image once with:

SYRF_MIGRATION_MODE=status
SYRF_MIGRATION_BINDING_EXPECTED_ENVIRONMENT=<env>        # optional on this first run
SYRF_MIGRATION_BINDING_EXPECTED_DATABASE=<database>      # optional
SYRF_MIGRATION_BINDING_EXPECTED_CLUSTER=<cluster-fingerprint>  # optional

TO CONFIRM: no chart template renders a status Job. In the local stack e2e/scripts/pm-migrations.sh runs it. For staging and production, the reviewed way to run a one-off status (and verify) is not defined. 2. R1.2 Prepare the cluster-gitops change for both the API and the PM service values:

migrations:
  enabled: true
  binding:
    bind: true
    expectedEnvironment: "<env>"
    expectedDatabase: "<database>"
    expectedCluster: "<cluster-fingerprint>"

Copy expectedCluster from R1.1. Both charts need the three expected values, because each chart's up Job reads them when a RequiresBinding migration is pending. bind: true adds a PreSync bind Job in sync wave −1, ahead of up. A second identical bind is a no-op. Whether bind stays set after the first bind is open (#4263). 3. R1.3 Review the rendered Jobs in the cluster-gitops PR diff: one bind Job and one up Job per chart, each with the three SYRF_MIGRATION_BINDING_EXPECTED_* variables. 4. R1.4 Merge with the required approval and let ArgoCD sync. Do not sync API and PM at different builds (D4). 5. R1.5 Verify with SYRF_MIGRATION_MODE=verify (same TO CONFIRM as R1.1), or, once reports are enabled (runbook 4), with the report's migrations prerequisite.

Preflight checks and expected output

Check	Expected output
R1.1 status, no binding yet	Environment binding: none in database <database>; live cluster fingerprint <cluster-fingerprint> and, without expected values, Environment binding: no SYRF_MIGRATION_BINDING_EXPECTED_* value is set; nothing compared. Exit 0
R1.1 status with expected values set	Environment binding matches the expected environment '<env>', database '<database>', cluster '<cluster-fingerprint>', or one Environment binding mismatch: … warning per problem. Exit 0 either way
R1.4 bind Job log	Environment bound: <env>, database <database>, cluster <cluster-fingerprint>, epoch <binding-epoch>, announcement delivery <mode>, where <mode> is production, testSink (staging) or disabled (previews, local). On a repeat: Environment binding already in place: …; nothing changed. Exit 0
R1.4 up Job log	Environment binding verified: <env>, database <database>, cluster <cluster-fingerprint>, epoch <binding-epoch>, then each migration applied. Exit 0
R1.5 verify	Verify: every migration this binary knows is applied and every declared index exists. Exit 0

Evidence to capture

• The status output from R1.1 (fingerprint).
• The cluster-gitops PR URL and its approval.
• The bind and up Job logs, including the binding epoch.
• The verify output, or a report whose migrations prerequisite is Ready.

If a step is refused

Every binding refusal exits 2 and writes nothing. For up there is no partial apply; the previous release keeps serving because the PreSync phase fails.

Message (prefix Bind refused: or Up refused before applying anything (…):)	Action
SYRF_MIGRATION_BINDING_EXPECTED_… is not set; … are all required.	Set all three values
SYRF_MIGRATION_BINDING_EXPECTED_ENVIRONMENT must be production, staging, preview:pr-<number> or local:<stack> (got '…')	Fix the environment ID
the configured RuntimeEnvironment '…' does not match the expected environment '…'	The values file renders the wrong environment. Stop and investigate; never change the expected value to fit
the configured database '…' does not equal the expected database '…'	Same as above
the live cluster fingerprint '…' does not equal the expected cluster '…'	Wrong cluster, or the cluster changed. See runbook 9
database '…' is already bound to …; a different binding is never overwritten.	Runbook 9. Do not edit the record
database '…' has no environment binding; run SYRF_MIGRATION_MODE=bind first.	Add the bind Job (R1.2)
SYRF_MIGRATION_MODE must be one of status, up, down, verify, bind (got '…')	Fix the mode
Exit 3	The migration lease or writer wait timed out. Find the other runner before retrying

2. Lineage preview and apply with writer freeze

Purpose

Turn the legacy runtimeFeatureFlagOverrides document into a bound runtime authority with a reviewed baseline, keeping the legacy revision and overrides exactly. This is an operator command in the API image, separate from migrations and start-up. It does not enrol any consumer.

Two branches. Where the legacy document exists, import it (R2.1–R2.7 below). Where no global flag document exists at all, as in production, use Initialise (no legacy document) instead: it creates the authority from the deployed values (owner decision D8=A). Check which applies with the first preflight of each branch. The two never mix: import refuses a missing document and initialise refuses any document.

Preconditions

• Runbook 1 is complete and the binding matches.
• Migrations 0009 and 0010 are applied with their expected definitions. The command never creates indexes.
• The legacy policy document exists (_id: "global"). Where there is none (production: legacy mutation is disabled there), do not create one: use Initialise (no legacy document).
• syrf_metadata has a well-formed syrf-api instance registry.
• The image used for the command is the API build that will serve the imported authority. The catalog digest is part of the identity; a preview made with another catalog is rejected later.
• The consumer enrolment change (runbook 3) is reviewed and ready to merge straight after apply.
• Approval: Chris for staging and production. D1 preview rehearsals follow the staging plan.
• TO CONFIRM: no chart template runs the lineage command in a cluster, and none mounts the reviewed input or keeps the output artifact. The delivery mechanism for staging and production is not defined.

Steps

1. R2.1 Write the reviewed baseline JSON. It needs a Boolean for every key of the generated catalog (54 keys on main), plus approval and provenance text:

{
  "values": { "<key-1>": false, "<key-2>": true },
  "approval": "<reviewed baseline decision reference>",
  "provenance": "<source configuration and report identities used to reconcile it>"
}

List the keys with:

grep -oE '^        new\("[A-Za-z0-9]+", "' \
  src/libs/mongo/SyRF.Mongo.Common/RuntimeFeatureFlags/RuntimeFeatureFlagCatalog.generated.cs \
  | sed -E 's/^ +new\("([A-Za-z0-9]+)".*/\1/'

The keys are the first argument of each new("<key>", …) entry in that generated file, which is the catalog's key list; read it there if grep is unavailable.

Reconcile each value against the rendered values of every consuming service. The command never takes the API's current settings as the baseline. 2. R2.2 Preview (read-only). Run the API image's normal entry point once with the usual Mongo configuration and RuntimeEnvironment=<runtime-env>, plus:

SYRF_RUNTIME_POLICY_MODE=preview
SYRF_RUNTIME_POLICY_INPUT=/reviewed/baseline.json
SYRF_RUNTIME_POLICY_OUTPUT=/reviewed/lineage-preview.json
SYRF_MIGRATION_BINDING_EXPECTED_ENVIRONMENT=<env>
SYRF_MIGRATION_BINDING_EXPECTED_DATABASE=<database>
SYRF_MIGRATION_BINDING_EXPECTED_CLUSTER=<cluster-fingerprint>

SYRF_MIGRATION_MODE must not be set at the same time. 3. R2.3 Review the preview file. Check kind: legacyLineageImport, planVersion: 1, sourceRevision (equal to the legacy revision), target.environmentId = <env>, target.bindingEpoch = <binding-epoch>, a new target.storeEpoch, the baseline values and previewDigest. It is Canonical Extended JSON: do not reformat it. Numbers carry their type wrapper, so planVersion reads {"$numberInt": "1"} and target.revision {"$numberLong": "<revision>"}; sourceRevision is a decimal string; the baseline values are in target.baseline. 4. R2.4 Freeze writers. Start the freeze recorded in the approval: no cluster-gitops merges for this environment except this procedure, no auto-promotion, no autoscaling. Scale the API to zero replicas through cluster-gitops and wait until every API pod has terminated. TO CONFIRM: the freeze mechanism for staging's auto-merged promotion PRs. 5. R2.5 Apply the exact reviewed preview with the same image and binding values:

SYRF_RUNTIME_POLICY_MODE=apply
SYRF_RUNTIME_POLICY_INPUT=/reviewed/lineage-preview.json
SYRF_RUNTIME_POLICY_OUTPUT=/reviewed/lineage-receipt.json
SYRF_RUNTIME_POLICY_ACTOR=<operator identity>
SYRF_RUNTIME_POLICY_WRITER_FREEZE_ACK=<freeze reference from the approval record>
SYRF_MIGRATION_BINDING_EXPECTED_ENVIRONMENT=<env>
SYRF_MIGRATION_BINDING_EXPECTED_DATABASE=<database>
SYRF_MIGRATION_BINDING_EXPECTED_CLUSTER=<cluster-fingerprint>

1. R2.6 If the outcome is unknown (exit 1, lost output), rerun R2.5 with the identical preview and a new output path. A committed import returns its original receipt without writing again.
2. R2.7 Go straight to runbook 3, then end the freeze. An API restarted in Legacy mode would start, but it serves only the deployed values and refuses legacy changes (runbook 10); the imported policy applies only in Shared mode. An older image (before #4414) instead refuses to start against the imported authority, and it writes its syrf-api writer-registry entry (DatabaseMetadataWriter) before it refuses, so a refused start still leaves an entry behind (seen in the 6C.2 rehearsal).

Preflight checks and expected output

Check	Expected output
Before R2.2: legacy document	db.runtimeFeatureFlagOverrides.findOne({_id: "global"}, {_id: 1, revision: 1}) returns a document with no storeEpoch field (read-only query)
Before R2.5: writer registry	db.syrf_metadata.findOne({_id: "meta"}, {"services.syrf-api.instances": 1}) shows instances: {}. Apply checks this itself, twice
Before R2.5: no API pods	The API Deployment reports 0 ready and 0 total pods
R2.2 log	Runtime policy lineage preview completed; artifact: /reviewed/lineage-preview.json. No consumer was enrolled. Exit 0
R2.5 log	Runtime policy lineage apply completed; artifact: /reviewed/lineage-receipt.json. No consumer was enrolled. Exit 0
R2.5 receipt	kind: legacyLineageImport, the preview's mutationId and previewDigest, environmentId, bindingEpoch, storeEpoch, revision equal to the source revision, actor, writerFreezeAcknowledgement, recordedAtUtc

A registry entry from a pod that did not shut down cleanly stays until a later API start prunes entries older than 24 hours (DatabaseMetadataWriter). There is no bypass. The one entry the freeze does not count is a Legacy API that has already found an initialized authority: it marks its own registration runtimePolicyReadOnly (bound to that registration's connectedAt) because it can never write runtime policy again (runbook 10). Every other entry, including one from an older image, is counted. TO CONFIRM: the approved way to clear a stale entry younger than 24 hours, other than waiting.

Evidence to capture

• The baseline JSON, its approval and provenance references.
• The preview file and its previewDigest.
• The freeze reference, start and end times, and proof of zero API pods.
• The receipt file and the command logs (no secrets).

If a step is refused

Refusals exit 2 with Runtime policy lineage refused: <message>. Nothing is written.

Message	Action
SYRF_RUNTIME_POLICY_MODE must be preview or apply. / … is required.	Fix the invocation
Lineage import and schema migration modes cannot be combined.	Unset SYRF_MIGRATION_MODE
Input and output paths must differ; preserve the reviewed input. / Output already exists; choose a new receipt/preview path.	Use a new output path
Input must be an existing JSON file of at most 1 MiB.	Fix the input path or file
Lineage import requires an exact verified environment/database/cluster binding: …	Runbook 1, or runbook 9
Missing legacy policy; lineage import cannot invent an initial state.	Initialise the legacy state with a reviewed edit first
Legacy source has unknown fields; review its schema before importing.	Stop; an unknown legacy shape needs review (it may already be imported)
API writer registry is missing or malformed; absence of evidence is not a writer freeze.	Investigate the registry; do not create it by hand without approval
API writers remain registered. Stop/freeze all legacy writers before lineage import; there is no bypass switch.	Finish R2.4
Required migration-owned audit index is missing or incompatible: …	Apply 0009/0010 (runbook 1)
The reviewed lineage preview was modified or has an unsupported version.	Make and review a new preview
The reviewed lineage preview belongs to a different environment binding. / Environment binding changed after preview.	Make a new preview against the current binding
Legacy source changed after preview; obtain and review a new preview.	A writer changed the policy: the freeze failed. New preview
Lineage import must preserve the exact legacy revision and overrides. / Legacy source audit references changed after preview.	New preview
Mutation ID was already used with a different lineage plan.	Never reuse a preview's mutation ID for a different plan
Too many ambiguous source audit references; review the legacy history before importing.	Stop and review the audit history

Initialise (no legacy document)

Purpose. Create the first bound runtime authority where no global flag document exists. The baseline is the deployed values: the FeatureFlags configuration the API image renders for this command, for every catalog key. The authority starts at revision 0 with no overrides and no stops. Production uses this branch instead of the import (D8=A): legacy mutation is disabled there, so it has nothing to import. The command, its digest discipline, writer freeze, receipts and output path rules are the import's; only the source differs.

Preconditions. As for the import, except:

• No global document exists, legacy or initialized. The command refuses either.
• Every migration the image knows is applied and every declared index exists (verify is clean; runbook 1). The command runs the same read-only check and refuses otherwise. It never creates indexes.
• The command runs with the rendered values the API will serve: the same image and the same FeatureFlags settings as the enrolled API (runbook 3). Reconcile them with Project Management's rendered values before review. The command reads only the API's.
• The same image and settings run preview and apply. Apply reads the deployed values again and refuses if they changed.

Steps.

1. R2.I1 Write the reviewed input. Only approval and provenance; the baseline is never supplied by hand:

{
  "approval": "<owner approval reference for the initial baseline>",
  "provenance": "<rendered values (API and PM) reconciled for this baseline>"
}

1. R2.I2 Preview (read-only). As R2.2, plus SYRF_RUNTIME_POLICY_SOURCE=initialize:

SYRF_RUNTIME_POLICY_MODE=preview
SYRF_RUNTIME_POLICY_SOURCE=initialize
SYRF_RUNTIME_POLICY_INPUT=/reviewed/initialise-input.json
SYRF_RUNTIME_POLICY_OUTPUT=/reviewed/initialise-preview.json
SYRF_MIGRATION_BINDING_EXPECTED_ENVIRONMENT=<env>
SYRF_MIGRATION_BINDING_EXPECTED_DATABASE=<database>
SYRF_MIGRATION_BINDING_EXPECTED_CLUSTER=<cluster-fingerprint>

The preview is deterministic: the same input, image, values, binding and migrations always give a byte-identical file (the store epoch and mutation ID are derived from them). Run it twice to two paths and compare (cmp) to prove nothing varies. 3. R2.I3 Review the preview file. Check kind: deployedValueInitialization, planVersion: 1, source: deployedValues, migrationCatalogueDigest, target.revision: 0, target.overrides: {}, target.killSwitches: {}, target.environmentId = <env>, target.bindingEpoch = <binding-epoch>, target.storeEpoch, previewDigest, and every value in target.baseline against the reconciled rendered values. Do not reformat it. 4. R2.I4 Freeze writers. As R2.4. 5. R2.I5 Apply the exact reviewed preview with the same image, values and binding:

SYRF_RUNTIME_POLICY_MODE=apply
SYRF_RUNTIME_POLICY_SOURCE=initialize
SYRF_RUNTIME_POLICY_INPUT=/reviewed/initialise-preview.json
SYRF_RUNTIME_POLICY_OUTPUT=/reviewed/initialise-receipt.json
SYRF_RUNTIME_POLICY_ACTOR=<operator identity>
SYRF_RUNTIME_POLICY_WRITER_FREEZE_ACK=<freeze reference from the approval record>
SYRF_MIGRATION_BINDING_EXPECTED_ENVIRONMENT=<env>
SYRF_MIGRATION_BINDING_EXPECTED_DATABASE=<database>
SYRF_MIGRATION_BINDING_EXPECTED_CLUSTER=<cluster-fingerprint>

1. R2.I6 If the outcome is unknown, rerun R2.I5 with the identical preview and a new output path. A committed initialise returns its original receipt without writing, even after the API is back. Any other preview is refused once the authority exists.
2. R2.I7 As R2.7: go straight to runbook 3, then end the freeze.

Preflight checks and expected output.

Check	Expected output
Before R2.I2: no global document	db.runtimeFeatureFlagOverrides.countDocuments({_id: "global"}) returns 0 (read-only query)
Before R2.I2: migrations	SYRF_MIGRATION_MODE=verify prints Verify: every migration this binary knows is applied and every declared index exists, exit 0
Before R2.I5: writer registry, no API pods	As for the import (R2.5 rows above)
R2.I2 log	Runtime policy initialise preview completed; artifact: /reviewed/initialise-preview.json. No consumer was enrolled. Exit 0
R2.I2 repeated	A second preview to another path is byte-identical (cmp prints nothing, exit 0)
R2.I5 log	Runtime policy initialise apply completed; artifact: /reviewed/initialise-receipt.json. No consumer was enrolled. Exit 0
R2.I5 receipt	kind: deployedValueInitialization, _id and mutationId equal to the preview's mutationId, the preview's previewDigest, source: deployedValues, migrationCatalogueDigest, sourceOverrides: {}, baseline, environmentId, bindingEpoch, storeEpoch, revision: 0, actor, writerFreezeAcknowledgement, recordedAtUtc
After R2.I5	A new preview is refused with An initialized runtime policy authority already exists; initialise never replaces it. (exit 2)
After runbook 3	authority-status is Shared/Ready at the receipt's store epoch and revision 0

Evidence to capture. The input file, both preview files and the cmp result, the reconciled rendered values, the freeze reference and zero-pod proof, the receipt, and the command logs (no secrets).

If a step is refused. Refusals exit 2 with Runtime policy lineage refused: <message>. Nothing is written. The import's invocation, path, binding and writer messages (table above) apply unchanged. Initialise adds:

Message	Action
SYRF_RUNTIME_POLICY_SOURCE must be import or initialize when set.	Fix the invocation
A legacy runtime policy document already exists; initialise never replaces it. Import it with the lineage import (SYRF_RUNTIME_POLICY_SOURCE unset) instead.	Use the import branch
An initialized runtime policy authority already exists; initialise never replaces it.	Stop: the environment is already initialised. If this was a retry, use the identical preview (R2.I6)
Migrations are not Ready; apply every migration this binary knows before initialising: …	Runbook 1 (up, then verify)
Initialise needs this binary's migration catalogue to prove that migrations are Ready.	Use the API image's entry point; the catalogue is built in
Initialise input takes only approval and provenance; the baseline is always the deployed values.	Remove values (and anything else) from the input
Deployed values changed after preview; obtain and review a new preview.	The apply ran with other FeatureFlags values than the preview: use the reviewed image and values, or make a new preview
The migration catalogue changed after preview; obtain and review a new preview.	The image changed between preview and apply: new preview
The reviewed plan was made for another lineage source; use the SYRF_RUNTIME_POLICY_SOURCE it was previewed with.	Set SYRF_RUNTIME_POLICY_SOURCE to match the preview
An initialise plan must create revision 0 from the deployed values, with no overrides and no stops.	The preview was altered: discard it, make a new one

3. Consumer enrolment

Purpose

Switch API and PM to the bound runtime authority (RuntimeFeatureFlags:Mode=Shared) with deployment-owned host metadata, so that their decisions and acknowledgements follow the imported policy.

Preconditions

• Runbook 2 is complete, and the freeze from R2.4 is still held with the API at zero.
• API and PM images are the same build (D4), and that build matches the catalog in the receipt.
• Migration 0011 is applied (acknowledgement index).
• Runbook 4 is ready: the API's collector settings (R4.1) are deployed, a token can be obtained (R4.2), and one capture and publication per service (R4.3–R4.5) has succeeded. R3.1 sets Collector mode, so the fleet can only be Active while both services have fresh observations (younger than 30 seconds).
• An operator is ready to run the manual capture-and-publish loop (R3.4) as soon as the API is back. While the API is at zero nothing can be published, because the observations endpoint is on the API.
• Approval for the cluster-gitops change.
• Staging only: staging values may set SYRF__RuntimeFeatureFlags__SharedReaderMode on API and PM (they did in cluster-gitops on 2026-10-17). Shared mode requires Disabled, so the same change must set it.

Steps

1. R3.1 Prepare one cluster-gitops change for API and PM. Shape for the API service values (PM is the same without the API-only lines):

replicaCount: <api-replicas>        # fixed, positive
hpa:
  enabled: false                    # an autoscaled fleet is always Unknown
runtimePolicyHostMetadata:
  enabled: true
env:
  SYRF__RuntimeFeatureFlags__Mode: "Shared"
  SYRF__RuntimeFeatureFlags__SharedReaderMode: "Disabled"
  SYRF__RuntimeFeatureFlags__Binding__Environment: "<env>"
  SYRF__RuntimeFeatureFlags__Binding__Database: "<database>"
  SYRF__RuntimeFeatureFlags__Binding__Cluster: "<cluster-fingerprint>"
  # Production only, API and PM:
  SYRF__RuntimeFeatureFlags__ProductionPolicyMode: "Runtime"
  # API only (runbook 6):
  SYRF__RuntimeFeatureFlags__FleetExpectedReplicas__api: "<api-replicas>"
  SYRF__RuntimeFeatureFlags__FleetExpectedReplicas__project-management: "<pm-replicas>"
  SYRF__RuntimeFeatureFlags__FleetInventoryMode: "Collector"

In production, Mode: Shared with ProductionPolicyMode: Legacy (the default) resolves to Legacy, so both are needed. Production also requires RuntimeEnvironment and ActiveEnvironment to be production. 2. R3.2 Review the rendered Deployments in the PR diff: four SYRF__RuntimeFeatureFlags__ host metadata entries from the chart, and the raw env: entries above. Raw env: cannot override the four host metadata entries. 3. R3.3 Merge and sync API and PM together, restoring the API replica count. This ends the API outage. 4. R3.4 Resume publishing, then check every host (preflight below), then end the freeze. No OD-23 scheduler exists yet, so run the documented capture and publication (R4.3–R4.4) for both services by hand, in a loop whose full cycle, including the pause, stays under 30 seconds: each observation must be under 30 s old when the fleet is evaluated. Time one capture and publication per service before relying on the loop. If both together take more than about 20 s, drop the pause or run one loop per service. The previous IDs are the last accepted observation IDs from R4.5. --expected-observation-id is required here; only a service's first-ever publication omits it (R4.3). Shape (bash, one terminal per environment):

#!/usr/bin/env bash
set -euo pipefail
declare -A previous=([api]="<last-api-id>" [project-management]="<last-pm-id>")
declare -A deployment=([api]="<api-deployment>" [project-management]="<pm-deployment>")
declare -A container=([api]="<api-container>" [project-management]="<pm-container>")
while true; do
  for service in api project-management; do
    out="observations/${service}-$(date -u +%Y%m%dT%H%M%SZ).json"
    python3 scripts/capture-runtime-workload.py --context "<reviewed-context>" --namespace "<namespace>" \
      --service "$service" --deployment "${deployment[$service]}" --container "${container[$service]}" \
      --environment "<env>" --binding-epoch "<binding-epoch>" \
      --expected-observation-id "${previous[$service]}" --output "$out"
    receipt="$(SYRF_RUNTIME_COLLECTOR_TOKEN="<collector-token>" python3 scripts/publish-runtime-workload.py \
      --api-origin "<api-origin>" --environment "<env>" --binding-epoch "<binding-epoch>" \
      --service "$service" --input "$out")"
    previous[$service]="$(python3 -c 'import json,sys; print(json.loads(sys.argv[1])["observationId"])' "$receipt")"
  done
  sleep 2  # counts towards the 30 s budget
done

The loop stops on the first non-zero exit. On publish exit 3, replay the same $out file before capturing again (R9.3); on exit 1 or 2, fix the cause (runbook 4 refusals) and restart with the last accepted IDs. Tokens expire: obtain a new one (R4.2) before it does. Never put the token in a file or in shell history.

Preflight checks and expected output

Check	Expected output
API start-up	No InvalidOperationException from authority options (see refusals). /health/live healthy on every pod
GET /api/runtime-feature-flags/authority-status	mode: Shared, state: Ready, identity.environmentId: <env>, identity.bindingEpoch: <binding-epoch>, identity.storeEpoch equal to the receipt's
GET /api/runtime-feature-flags/authority-evaluation/disableMembership	state: Evaluated, value equal to the baseline or saved override, safeValue: false, productionEligible: true, maximumAgeSeconds: 30
GET /api/runtime-feature-flags/disableMembership/fleet	Until the loop has published fresh observations of both services: state: Unknown with a Collector… reason (for example CollectorInventoryMissingOrInvalid or CollectorInventoryStaleOrInvalidClock). This is expected. Then state: Active within 60 s, one host row per expected pod of each service, each state: Current; rows of pods that no longer exist show Retired
GET /api/runtime-feature-flags (anonymous snapshot)	200, with an authority envelope. A 503 with runtime_policy_unavailable means no current authority
Legacy PUT PUT /api/runtime-feature-flags/<key>	403 Runtime changes are not available in this mode. This is expected in Shared mode

PM has no admin authority routes. Its adoption shows only in the fleet host list.

Evidence to capture

• The cluster-gitops PR URL, approval and the rendered env diff.
• The authority-status, authority-evaluation and fleet responses after rollout.
• The end time of the freeze.

If a step is refused

Start-up refusals stop the host before it serves:

Start-up message	Cause
RuntimeFeatureFlags:Mode must be Legacy, Shadow or Shared.	Bad mode value
Shared authority mode cannot also enroll the legacy-format shared reader.	SharedReaderMode is not Disabled
Authority observation requires matching environment identities and explicit production Runtime admission.	RuntimeEnvironment/ActiveEnvironment differ, or production without Runtime admission
Authority observation requires explicit binding expectations: …	A binding value is missing or invalid
Production Runtime requires Shared mode and both production environment identities.	Production ProductionPolicyMode=Runtime without Mode=Shared
ProductionPolicyMode must be Legacy or Runtime.	Bad value
runtimePolicyHostMetadata requires boolean hpa.enabled (render) and similar	Fix the chart values
Legacy runtime policy provider cannot read the runtime policy document. Inspect the document before restarting this host.	The legacy store could not read the global document at all. Inspect it (read-only) and stop. An imported authority is not this case: a Legacy API starts against it and serves the deployed values (runbook 10)
Legacy runtime policy provider cannot serve initialized authority. Deploy the authority-capable adapter before restarting this host.	Only an image older than #4414, started in Legacy mode against the imported authority. Finish R3.1, or deploy a current image. The refused host has already written a syrf-api writer-registry entry (R2.7)

After start-up, authority-status Quarantined or Unavailable and fleet states other than Active are handled in runbooks 6 and 9.

4. Collector identity and scope

Purpose

Give the environment one workload collector that publishes API and PM Deployment and pod facts to POST /api/environment/observations, using a machine identity whose only scope is environment:observations:write. Collector evidence is what lets reports become Complete and what Collector inventory mode reads.

Preconditions

• Migration 0018 is applied.
• A read-only Kubernetes identity with get on the named Deployment and list on ReplicaSets and Pods in the namespace. Staging has the precedent syrf-stats-preflight. Production needs a camarades-infrastructure change (OD-23).
• A machine client for the scope:
• OpenIddict environments: Identity seeds syrf-environment-collector only from an explicit strong EnvironmentCollector:ClientSecret (at least 32 characters);
• production: an Auth0 machine application with the same scope name, a production-shared tenant change that needs Chris's approval.
• TO CONFIRM: no chart value renders EnvironmentCollector:ClientSecret into Identity, and raw env: must not carry secrets. A chart change following identityStatisticsParityEvidence is needed.
• TO CONFIRM: the scheduler that runs capture and publication repeatedly does not exist (OD-23).

Steps

1. R4.1 Configure the API (raw env:, API values):

env:
  SYRF__RuntimeFeatureFlags__WorkloadCollector__Enabled: "true"
  SYRF__RuntimeFeatureFlags__WorkloadCollector__Provider: "<OpenIddict|Auth0>"
  SYRF__RuntimeFeatureFlags__WorkloadCollector__ClientId: "<collector-client-id>"
  SYRF__RuntimeFeatureFlags__WorkloadCollector__Audience: "<api-audience>"

Audience must equal IdentityService:Audience (OpenIddict) or Auth0:ApiAudience (Auth0). The SYRF__RuntimeFeatureFlags__Binding__* values (as in R3.1) must also be set, and <runtime-env> must match them. The collector does not need Shared mode, so this can run before enrolment; Legacy mode ignores the binding values. 2. R4.2 Obtain a token (OpenIddict), the same form as scripts/fetch-statistics-parity.py:

POST <identity-origin>/connect/token
Content-Type: application/x-www-form-urlencoded

grant_type=client_credentials&client_id=<collector-client-id>&client_secret=<secret>&scope=environment:observations:write

TO CONFIRM: the token request for the production Auth0 machine application (audience parameter). 3. R4.3 Capture one service, read-only, with an explicit reviewed context:

python3 scripts/capture-runtime-workload.py \
  --context "<reviewed-context>" --namespace "<namespace>" \
  --service api --deployment "<api-deployment>" --container "<api-container>" \
  --environment "<env>" --binding-epoch "<binding-epoch>" \
  --expected-observation-id "<last-accepted-observation-id>" \
  --output api-observation.json

The chart default Deployment and container name is the chart name (syrf-api, syrf-projectmanagement); check the rendered manifest. Omit --expected-observation-id only for a known first publication. 4. R4.4 Publish it at once (publication refuses snapshots older than 30 seconds):

SYRF_RUNTIME_COLLECTOR_TOKEN="<collector-token>" python3 scripts/publish-runtime-workload.py \
  --api-origin "<api-origin>" --environment "<env>" \
  --binding-epoch "<binding-epoch>" --service api --input api-observation.json

1. R4.5 Repeat R4.3–R4.4 for project-management. Keep each service's last accepted observation ID.
2. R4.6 Keep publishing. Collector inventory needs observations strictly younger than 30 seconds for both services, so both must be republished well inside 30 seconds for as long as Collector-mode Active is needed.

Preflight checks and expected output

Check	Expected output
R4.3 capture	Exit 0, api-observation.json created; never overwritten
R4.4 publish	Exit 0; stdout only {"observationId": "<observation-id>", "digest": "<64 hex>"}
After R4.5: report (runbook 7, R7.3)	Sources workloads:api and workloads:project-management are Current
After R4.5: Collector-mode fleet (runbook 6)	state: Active; replaced pods' rows show Retired

Evidence to capture

• The client registration change and its approval (no secret values).
• Each capture file and its publish receipt (observation ID and digest).
• The Kubernetes identity's role definition reference.

If a step is refused

Result	Meaning and action
Capture exit 2 Capture refused: Output already exists; preserve it for exact replay	Use a new output path; keep the old file
Capture exit 2 Capture refused: Workload changed during capture; acquire a new observation	A rollout is in progress; capture again
Capture exit 2 Capture refused: Capture clock or freshness budget invalid	Reads took over 20 seconds or the clock moved back
Capture exit 2 Capture refused: Observation exceeds publication body limit	Over 512 KiB; investigate the pod count
Publish exit 2 Publication refused locally; check origin, binding, capture and credential inputs	Fix the inputs. HTTPS only, no path; capture must match --environment, --binding-epoch, --service
Publish exit 1 Publication refused: HTTP 400	Invalid workload observation: contract failure, for example a pod without explicit ready/terminating
Publish exit 1 Publication refused: HTTP 409	Workload observation refused. Causes in the store: expected observation missing or not advanced, an observation ID reused with different evidence, a stale or future snapshot, a changed binding, a missing 0018 index, or publication disabled or bound elsewhere. Capture again with the right previous ID
Publish exit 1 HTTP 401 or HTTP 403	Token scope, client, audience or binding mismatch, or the collector is disabled
Publish exit 3 Publication outcome unknown; preserve and replay the exact capture	408/425/429, 5xx (503 Workload publication outcome unknown…) or transport loss. Replay the same file
API start-up Enabled workload collection requires explicit matching provider, client, audience and environment binding.	Fix R4.1

5. Coordinator identity

Purpose

Give the promotion workflow a machine identity, scope environment:deployments:coordinate, that records, extends and releases one deployment compatibility constraint per rollout through PUT and DELETE /api/environment/deployment-constraints/<rollout-id>.

Preconditions

• The API is in Shared mode with the binding (runbook 3).
• A machine client:
• OpenIddict: Identity seeds syrf-environment-coordinator only when EnvironmentCoordinator:ClientSecret has at least 32 characters;
• production: an Auth0 machine application (tenant change, Chris's approval).
• TO CONFIRM: no chart value renders EnvironmentCoordinator:ClientSecret into Identity.
• TO CONFIRM: no workflow calls the coordinator routes yet, and how the coordinator builds the capability intersection and completion targets from a trusted report is not implemented.

Steps

1. R5.1 Configure the API (raw env:):

env:
  SYRF__RuntimeFeatureFlags__DeploymentCoordinator__Enabled: "true"
  SYRF__RuntimeFeatureFlags__DeploymentCoordinator__Provider: "<OpenIddict|Auth0>"
  SYRF__RuntimeFeatureFlags__DeploymentCoordinator__ClientId: "<coordinator-client-id>"
  SYRF__RuntimeFeatureFlags__DeploymentCoordinator__Audience: "<api-audience>"

1. R5.2 Record a constraint before a rollout. Always send action; an omitted action means Record:

PUT /api/environment/deployment-constraints/<rollout-id>
Authorization: Bearer <coordinator-token>
Content-Type: application/json

{
  "mutationId": "<mutation-id>",
  "expectedIdentity": <identity>,
  "action": "Record",
  "reason": "<rollout reason>",
  "expiresAtUtc": "<future time>",
  "servingCatalogDigests": ["<serving-catalog-digest>"],
  "targetCatalogDigest": "<target-catalog-digest>",
  "allowedCapabilities": { "policySchemaVersion": 1, "enabledKeys": ["<key>"] },
  "completionTargets": {
    "target":   { "catalogDigest": "<target-catalog-digest>",  "serviceImages": { "api": "<repo>@sha256:<digest>", "project-management": "<repo>@sha256:<digest>" } },
    "rollback": { "catalogDigest": "<serving-catalog-digest>", "serviceImages": { "api": "<repo>@sha256:<digest>", "project-management": "<repo>@sha256:<digest>" } }
  }
}

1. R5.3 Extend with action: "Extend", the reviewed expectedConstraintDigest and a later expiresAtUtc.
2. R5.4 Release after the collector shows every replica on the target (or a completed rollback):

DELETE /api/environment/deployment-constraints/<rollout-id>
Authorization: Bearer <coordinator-token>
Content-Type: application/json

{ "mutationId": "<mutation-id>", "expectedIdentity": <identity>, "action": "Release",
  "reason": "<reason>", "expectedConstraintDigest": "<constraint-digest>", "completionKind": "Target" }

Clients and proxies must keep the JSON body on DELETE.

Preflight checks and expected output

Check	Expected output
Before R5.2: report (purpose=promotion)	deploymentConstraint source Current; projection null (no active constraint)
R5.2–R5.4 response	200 {"mutationId": "<mutation-id>", "rolloutId": "<rollout-id>", "action": "<action>", "constraintDigest": "<digest or null after release>", "recordedAtUtc": "<time>"}
After R5.2: report	deploymentConstraint.state: Active, rolloutId, constraintDigest

Evidence to capture

• Each request and receipt (mutation ID, rollout ID, constraint digest).
• The report before record and after release.

If a step is refused

Response	Meaning and action
400 Invalid coordinator operation	Wrong verb for the action (Record/Extend need PUT, Release needs DELETE) or a rollout ID that is not one ASCII segment of up to 128 [A-Za-z0-9-_.:]
400 Invalid coordinator request	Malformed identity or fields
409 Policy identity changed; read a fresh report	Revalidate against the new policy, then start again
409 Deployment constraint operation refused	The store refused. Possible causes: another active constraint, policy outside the capability intersection, missing completion expectations, unproven completion, not the owning coordinator, expiry not moved forward. The response does not name the cause
503 Coordinator outcome unknown; retry the exact mutation ID and request	Retry unchanged
401/403	Wrong scope, client, audience or binding; coordinator disabled
API start-up Enabled deployment coordination requires explicit matching provider, client, audience and environment binding.	Fix R5.1

A stranded constraint is released by an administrator through break-glass (runbook 9). An expired constraint keeps blocking incompatible changes; expiry is never a release.

6. Expected inventory and Collector mode

Purpose

Tell the API how many API and PM hosts must acknowledge a policy before the fleet is Active, and let it read that membership from fresh collector observations instead of every stored acknowledgement row.

Preconditions

• Runbook 3 (hosts publish acknowledgements) and runbook 4 (fresh observations for both services).
• Fixed replica counts and hpa.enabled: false for API and PM.
• For production enables and restores: Collector mode with the collector publishing (see Status and boundary). Static mode refuses them after any rollout.

Steps

1. R6.1 Set the per-host metadata through the chart (runtimePolicyHostMetadata.enabled: true). Each pod renders InstanceUid from metadata.uid, ImageVersion from the exact image, ExpectedReplicas from replicaCount and Autoscaled from hpa.enabled.
2. R6.2 Set the fleet inventory on the API: SYRF__RuntimeFeatureFlags__FleetExpectedReplicas__api and SYRF__RuntimeFeatureFlags__FleetExpectedReplicas__project-management, each equal to that service's replicaCount. No count is inferred from observed hosts.
3. R6.3 Set SYRF__RuntimeFeatureFlags__FleetInventoryMode: "Collector" on the API. The default is Static. Any other value refuses inspection.
4. R6.4 Change counts only together. A replica change edits replicaCount and the matching FleetExpectedReplicas value in the same cluster-gitops change.

Notes:

• replicaCount: 0 renders ExpectedReplicas: "0" as metadata only; a zero-count host publishes no acknowledgement. FleetExpectedReplicas must be positive, so a zero-count service leaves the fleet Unknown (ExpectedInventoryMissingOrInvalid).
• Acknowledgements count as current only if younger than 30 seconds. Hosts poll every 8–12 seconds, and 6B measured adoption up to 11.5 seconds. That margin is close to the limit; treat brief Applying as normal.
• Static mode suits only hosts whose instance UIDs stay the same (the local harness). In Kubernetes every replaced pod has a new UID, and old rows have no TTL, so a Static fleet reads Applying, then Degraded, after every rollout.
• In Collector mode a row absent from fresh collector inventory is retired only once MongoDB has not seen it rewritten for the 35-second quiet period (30-second freshness plus 5 seconds, server time; #4407). So a rollout reaches Active about 35 seconds after the old pods stop. Rows with no server stamp, written by images older than #4407, need 15 minutes, the assumed bound on host clock skew, so the first rollout away from those images can take up to 15 minutes.

Preflight checks and expected output

GET /api/runtime-feature-flags/disableMembership/fleet (Cache-Control: no-store):

Fleet state / reason	Meaning
Active / null	Every expected host of both services is Current on this identity
Saved or Applying / HostsUnconfirmed	Within the 60-second confirmation window
Degraded / HostsUnconfirmed	Window passed with hosts missing or behind
Unknown / ExpectedInventoryMissingOrInvalid	A FleetExpectedReplicas value is missing, non-positive or names another service
Unknown / AutoscaledInventory	A host reports Autoscaled
Unknown / DomainReadinessNotEstablished	Any key other than disableMembership
Unknown / Collector… reasons	CollectorInventoryMissingOrInvalid, CollectorBindingMismatch, CollectorInventoryStaleOrInvalidClock, CollectorExpectedCountDisagrees, CollectorRolloutIncomplete, CollectorHostServiceDisagrees, CollectorHostObservationDisagrees
Action required / ExpectedInventoryDisagrees, DuplicateInstanceIdentity, IncompatibleHost, InvalidCommitClock	Fix the configuration or the incompatible host

Host rows carry state Current, Behind (PolicyNotApplied), Unknown (HostMetadataMissing, StaleOrInvalidObservationClock, or the host's error class), Incompatible (IdentityOrCatalogMismatch) or, in Collector mode, Retired (AbsentFromFreshCollectorInventory). Retired rows stay stored and visible.

Evidence to capture

• The values change and the fleet response showing Active with the expected host count.
• In Collector mode, a fleet response after a rollout showing old rows Retired.

If a step is refused

• 403 Fleet inspection requires Shared API authority mode.: runbook 3 is not complete on this API.
• 503 runtime_policy_unavailable: the snapshot could not be read; retry.
• An unsupported FleetInventoryMode value refuses inspection (Unsupported fleet inventory mode.).

7. Change one flag

Purpose

Make one reviewed policy change to <key> (in production, disableMembership only): preview it, commit exactly the previewed change, then confirm that every expected host has adopted it.

Preconditions

• Runbooks 3 and 6 complete; fleet Active for the current identity.
• The operator holds ChangeRuntimeFeaturePolicy (Administrator, OD-2).
• For production enables: Collector mode with the collector publishing, the release guidance (OD-26) and Chris's approval.
• The admin page's preview/commit UI (#4381) is not on main; use the API routes.

Steps

1. R7.1 Read the identity (authority-status, above). Expect state: Ready.
2. R7.2 Check the fleet (GET /api/runtime-feature-flags/<key>/fleet). Expect state: Active.
3. R7.3 Check compatibility. Collect a report with the report token, save it, then run the checker:

GET /api/environment/compatibility-reports/current?purpose=policy-change
Authorization: Bearer <report-token>

python3 scripts/runtime-compatibility-check.py --report report.json --candidate candidate.json \
  --environment "<env>" --binding-epoch "<binding-epoch>" --purpose policy-change

The candidate is the serving build (syrf.runtime-candidate/v1). Never pass --now for a real decision. Reports need RuntimeFeatureFlags:CompatibilityReport settings (Enabled, Provider, ClientId, Audience) and the client syrf-environment-report with EnvironmentReport:ClientSecret (same chart gap as runbook 4). 4. R7.4 Preview (read-only, allowed under support impersonation):

POST /api/runtime-feature-flags/<key>/preview
<admin-session>
Content-Type: application/json

{
  "mutationId": "<mutation-id>",
  "expectedIdentity": <identity>,
  "value": true,
  "reason": "<reason>",
  "action": "Change"
}

value is true, false or null (clear the override). If the response is 422 featureFlagDependenciesRequired, review the set and repeat with "confirmedDependencies": {…}, the exact key/value set. Keep includeDependencies false. 5. R7.5 Commit the same body plus the preview digest:

PUT /api/runtime-feature-flags/<key>/policy
<admin-session>
Content-Type: application/json

{ …the R7.4 body unchanged…, "previewDigest": "<preview-digest>" }

1. R7.6 Confirm adoption. Poll the fleet until state: Active and every host row shows the committed revision with state: Current. Check authority-evaluation/<key> on the API for the new revision and value. Browsers are not part of adoption. The commit wakes the writing pod's own refresh, so its tabs get a RuntimeFeatureFlagsRevision hint as soon as that pod's reader has the new identity (one authority read, bounded by its 5 s deadline); tabs on other API pods get one when that pod's next refresh (8–12 s) observes it. Either way the tab refetches the snapshot. A failed hint is only a log line (Runtime policy revision … hint failed or … hint unavailable); it never changes the receipt.
2. R7.7 If the response was lost, look up the receipt, then retry the unchanged request if needed:

GET /api/runtime-feature-flags/mutations/<mutation-id>
<admin-session>

Preflight checks and expected output

Check	Expected output
R7.3 report	completeness: Complete, purpose: policy-change, every entry of requiredSources Current. In Static mode reports stay Incomplete (fleetInventoryMode:Static in unknownOrUnsupported)
R7.3 checker	{"exitCode": 0, "reasons": [], "verdict": "pass"}, exit 0
R7.4 preview	200 with previewDigest, before (the current identity), proposed (revision + 1), changes containing {"key": "<key>", "proposedOverride": true, "beforeEffective": false, "proposedEffective": true, "cause": "Requested", …} and fleetReadiness: the current fleet state in production (Active needed to enable), Unknown elsewhere
R7.5 commit	200 {"mutationId": "<mutation-id>", "identity": {…, "revision": "<revision + 1>"}}. Committed, not yet adopted
R7.6 fleet	Active for the new identity within the 60-second window; 6B measured adoption up to 11.5 seconds
R7.7 lookup	{"mutationId": "<mutation-id>", "outcome": "Committed", "committedIdentity": {…}}, or outcome: Unknown with no identity, which does not prove an abort

Evidence to capture

• The report ID, the report file and the checker output.
• The preview response, the commit receipt and the fleet responses until Active, with timestamps.

If a step is refused

Response	Meaning and action
403 (no body)	The caller lacks ChangeRuntimeFeaturePolicy
503 authority-unavailable	Application authority could not decide; retry after Retry-After. During a rollout, see D4
403 Policy administration requires Shared API authority mode.	This API is not enrolled
409 runtime_policy_revision_conflict with actualRevision	Someone changed the policy. Re-read and decide again
422 featureFlagDependenciesRequired	Confirm dependencies (R7.4). canApplyAutomatically: false means a deployment change is needed first
400 runtime_policy_constraint, message: The reviewed preview is missing or stale. Preview the current policy and retry.	Preview again; the digest no longer matches
… Production enable or restore requires fresh fixed-fleet acknowledgement of the current policy.	Fleet not Active. In Static mode this follows every rollout; set Collector mode and keep the collector publishing (runbook 6)
… This key is not eligible for production runtime policy.	Only disableMembership
… A production plan cannot change ineligible dependencies or safety pins.	The change would touch other keys in production
… Policy requires a capability outside the deployment intersection.	An active deployment constraint forbids it; wait for release
… Emergency stops use their independent safe-state path.	Use runbook 8
… Authority mutation requires an exact verified environment binding., Environment binding changed before the authority transaction., Authority identity does not match the verified binding., The request belongs to a different authority lineage.	Runbook 9
… Mutation ID was already used with a different request.	Use a new mutation ID for a new request
503 runtime_policy_commit_unknown	Retry the same mutation ID and unchanged request (R7.7)
503 runtime_policy_unavailable	Preview or read failed; nothing was written
Checker exit 1 or 2	See the reason codes in Checking a candidate offline

8. Emergency stop and restore

Purpose

Force an audited global safe-false control to its safe value at once, keeping the saved baseline and overrides, and later restore the saved policy through a reviewed preview and commit.

Preconditions

• The API is in Shared mode. The operator holds EmergencyStopRuntimeFeatures for a stop and ChangeRuntimeFeaturePolicy for the restore.
• A stop needs no fleet readiness, deployment-constraint clearance, content or new audit indexes. It still needs the binding, lineage and receipt integrity.
• A production restore that makes the control effective On needs the fleet Active on the current (stopped) policy, so the same Collector-mode and running-collector prerequisite applies.

Steps

1. R8.1 Read the identity (authority-status).
2. R8.2 Stop:

POST /api/runtime-feature-flags/<key>/emergency-stop
<admin-session>
Content-Type: application/json

{ "mutationId": "<mutation-id>", "expectedIdentity": <identity>, "reason": "<reason>" }

1. R8.3 Confirm the stop reached every host: authority-evaluation/<key> shows value: false at the stop revision on the API, and the fleet host rows show the stop revision Current. A 503 or lost response is retried with the same mutation ID and body.
2. R8.4 Restore when the incident is resolved. Read the identity again, then preview and commit ReleaseStop with a null value and no dependency changes:

POST /api/runtime-feature-flags/<key>/preview
<admin-session>
Content-Type: application/json

{ "mutationId": "<new-mutation-id>", "expectedIdentity": <identity>, "value": null,
  "reason": "<reason>", "action": "ReleaseStop" }

Then PUT /api/runtime-feature-flags/<key>/policy with the same body and previewDigest. The local harness sends action as its ordinal (2). The API reads request bodies with Newtonsoft.Json, which also accepts the name: the 6C.2 rehearsal sent "ReleaseStop" (and "Change" in R7.4) as names and both were accepted. 5. R8.5 Confirm adoption of the restore revision as in R7.6.

Preflight checks and expected output

Check	Expected output
R8.2 response	200 {"mutationId": "<mutation-id>", "identity": {…}}: durably committed, not yet adopted
R8.3	Every API evaluates false at the stop revision; the 6B harness checks this within 60 seconds with a PM host paused
R8.4 preview	The changes entry shows wasStopped: true, willBeStopped: false and proposedOverride equal to the saved override
R8.5	Every API evaluates the restored saved value; fleet Active

A new mutation ID always records a new revision, even if the key is already stopped. Idempotency is per mutation ID.

Evidence to capture

• The reason, the stop receipt and the per-host adoption times.
• The restore preview, receipt and fleet Active response.

If a step is refused

Response	Meaning and action
403 (no body)	The caller lacks EmergencyStopRuntimeFeatures
403 Emergency stops require Shared API authority mode.	Not enrolled
409 runtime_policy_revision_conflict	Re-read the identity and decide; a stop is often still wanted
400 runtime_policy_constraint: Stop actions require an audited global safe-state contract and cannot change saved values or dependencies.	The key has no audited safe-false contract
503 runtime_policy_commit_unknown	Retry the same mutation ID and request
Restore refusals	As runbook 7, including the production fleet refusal

9. Recovery and rebind

Purpose

Recover from the failure cases the code reports, without bypassing a binding, a lineage or an audit record. Every recovery is a forward action.

Preconditions

• The failure is identified from a response, log line or state below.
• Approval as for the original step. Anything that would edit the binding, the authority document or the audit by hand needs Chris and is not covered here.

Steps

1. R9.1 Lost or unknown policy outcome. GET /api/runtime-feature-flags/mutations/<mutation-id>. On Committed, record it and stop. On Unknown, retry the unchanged original request.
2. R9.2 Lost lineage apply outcome. Rerun R2.5 with the identical preview and a new output path.
3. R9.3 Unknown workload publication (exit 3). Replay the exact capture file. Do not change evidence under an existing observation ID.
4. R9.4 Mongo outage. No operator action. Decisions fail closed to the safe value on the first failed read and recover by themselves after the outage (6B mongo-outage scenario). Confirm with R7.6 afterwards.
5. R9.5 Reader quarantine. authority-status shows Quarantined (InvalidBinding, IdentityOrRevisionRegression, or AuthorityNotInitialized after a snapshot). There is no automatic reset. Find out why the identity, binding or revision changed, then restart the affected pods with a reviewed cluster-gitops change. TO CONFIRM: the reviewed recovery procedure that must precede the restart is not written.
6. R9.6 Stranded deployment constraint. A real administrator (no impersonation, no machine scope) calls:

POST /api/environment/deployment-constraints/<rollout-id>/break-glass-release
<admin-session>
Content-Type: application/json

{ "mutationId": "<mutation-id>", "expectedIdentity": <identity>,
  "reason": "<reason, max 1000 characters>", "expectedConstraintDigest": "<64-character digest>" }

It needs BreakGlassRuntimeDeploymentConstraint and Shared mode. Expected: 200 with the coordinator receipt shape. 7. R9.7 Restored database or changed cluster (rebind). A restored copy carries the source binding, so up refuses with exit 2 and the reader quarantines. rebind is not built: no supported command replaces a binding. Stop and raise it with Chris. Do not edit runtimeEnvironmentBinding by hand.

Preflight checks and expected output

Check	Expected output
R9.1	outcome: Committed with committedIdentity, or Unknown
R9.4 after recovery	authority-evaluation state: Evaluated at the committed revision; fleet Active
R9.5 after restart	authority-status state: Ready with the expected identity
R9.6	200 {"mutationId", "rolloutId", "action", "constraintDigest": null, "recordedAtUtc"}; the report's constraint projection is null
R9.7	status shows the binding record and the live fingerprint, and the mismatch

Evidence to capture

• The failure response or log line, the lookup result and the retry receipt.
• For break-glass: the reason, the before-state report and the receipt.
• For R9.7: the status output and the escalation record.

If a step is refused

Response	Action
Mutation lookup 403 Mutation outcomes require Shared API authority mode.	Use a Shared-mode API
Mutation lookup 503	Storage failure: no outcome. Retry the lookup
Break-glass 403	Impersonation, a machine scope, more than one identity, or a missing capability
Break-glass 403 Break-glass requires Shared API authority mode.	Not enrolled
Break-glass 400 Invalid rollout identity	Fix the rollout ID
Break-glass 409 Policy identity changed; inspect current state / Deployment constraint recovery refused	Read the current policy and constraint, then decide again
Break-glass 503 Break-glass outcome unknown; retry the same mutation ID and unchanged request	Retry unchanged

10. Legacy return

A host in Legacy mode against an imported authority (owner decision D7=A, spec 6C.5) starts and serves the deployment-controlled values for every key. It never applies the authority's saved overrides or stops, never writes the authority document, refuses legacy changes, and registers as a read-only reader so it cannot block a later lineage freeze. The saved policy, baseline, receipts, audit and stops stay exactly as they were, so returning to Shared (or production Runtime) resumes the same identity. The legacy-return scenario of the runtime-fleet harness rehearses this runbook end to end on an isolated stack.

Purpose

Return API and PM from runtime policy (Shared, or Runtime in production) to deployment-controlled values without any change in behaviour (RT-15), and come back without losing the saved policy.

Preconditions

• No active stop on any production-eligible key.
• For every production-eligible key (today disableMembership), the runtime decision equals the deployed FeatureFlags value.
• Collector mode, so the report can be Complete.
• Chris's approval of the mode-switch change.

Steps

1. R10.1 Make the values agree. Either commit a reviewed forward revision so the runtime value equals the deployed value (runbook 7), or change the deployed value in GitOps (featureFlags.disableMembership) to the runtime value. Release any stop first (runbook 8).
2. R10.2 Collect a legacy-return report:

GET /api/environment/compatibility-reports/current?purpose=legacy-return
Authorization: Bearer <report-token>

1. R10.3 Run the checker with a candidate whose targetMode is Legacy (the build that will serve):

python3 scripts/runtime-compatibility-check.py --report report.json --candidate candidate.json \
  --environment "<env>" --binding-epoch "<binding-epoch>" --purpose legacy-return

Record the authority identity from authority-status (all seven fields) before the switch. 4. R10.4 Switch the mode for API and PM in one cluster-gitops change, changing nothing else: production SYRF__RuntimeFeatureFlags__ProductionPolicyMode: "Legacy"; elsewhere SYRF__RuntimeFeatureFlags__Mode: "Legacy". Saved policy, baseline, receipts, audit and stops are not changed by the switch. 5. R10.5 Confirm after the rollout, on each API: - authority-status returns mode: Legacy, state: Disabled, reason: LegacyModeAuthorityPreserved and no identity (a Legacy host never reads or verifies the authority); - the anonymous snapshot GET /api/runtime-feature-flags has policyState: DeploymentControlledAuthorityPreserved, mutationAllowed: false, no overrideValue on any flag, and disableMembership at the report's legacyValue; - provider-status has policyState: DeploymentControlledAuthorityPreserved and policyStateDetail saying the runtime policy is preserved but not applied; - the admin page shows "Deployment-controlled; runtime policy preserved" and offers no changes. 6. R10.6 Return to runtime policy when wanted: revert only the mode (production ProductionPolicyMode: "Runtime"; elsewhere Mode: "Shared") for API and PM together. authority-status returns Ready with the identity recorded in R10.3, unchanged in all seven fields, and the fleet reaches Active.

Preflight checks and expected output

Check	Expected output
R10.2 report	completeness: Complete, purpose: legacy-return, legacyParity.agrees: true, each entry stopActive: false, legacyValue equal to runtimeValue
R10.3 checker	{"exitCode": 0, "reasons": [], "verdict": "pass"}, exit 0
R10.5 legacy change	PUT /api/runtime-feature-flags/<key> (or /groups/<key>) answers 409 with code: runtime_policy_authority_preserved; nothing is written
R10.5 registry	db.syrf_metadata.findOne({_id: "meta"}, {"services.syrf-api.instances": 1}) (read-only) shows each Legacy API with runtimePolicyReadOnly: {reason: "LegacyReaderAuthorityPreserved", connectedAt: <its own connectedAt>}
R10.5 authority unchanged	The global document and the audit row count read before R10.4 are unchanged (read-only queries)
R10.6 return	authority-status Ready with the R10.3 identity; fleet Active

Evidence to capture

• The report ID and file, the candidate manifest and the checker output.
• The forward revision or GitOps change from R10.1.
• The identity before R10.4 and after R10.6, and the R10.5 outputs.
• The mode-switch PRs and their approvals.

If a step is refused

Checker reason	Exit	Action
legacy-parity-stop-active	1	Release the named key's stop (runbook 8)
legacy-parity-differs	1	R10.1 for the named key
legacy-parity-missing	2	The report has no bound policy; collect it while the hosts are still in Shared/Runtime
binding-mismatch	1	Wrong --environment/--binding-epoch, or a report from another environment
report-incomplete, required-source-unknown, report-expired, report-unstable, purpose-mismatch	2	Collect a fresh report with purpose=legacy-return in Collector mode
migration-not-applied, catalog-key-missing, catalog-digest-incompatible	1	The target build cannot honour the current state; plan a replacement release
Report 400 environment_report_purpose_invalid	n/a	purpose must be exactly policy-change, promotion or legacy-return, given once
Report 503 environment_report_unavailable	n/a	Missing binding, indexes or evidence; retry, then investigate

The checker exits 2 whenever any insufficient-evidence reason is present, even if refusal reasons are listed too. For example, a Static-mode report whose values differ gives exit 2 with both report-incomplete and legacy-parity-differs.

A Legacy host never refuses to start over the authority. If R10.5 shows a runtime value instead of the deployed one, the host is not in Legacy mode: check the rendered Mode/ProductionPolicyMode.

11. Historical release import (written only)

Written only. This runbook has not been run in any environment beyond the isolated tests. It is an operational prerequisite for showing past releases (for example the August 2026 What's New entries) with their real dates, and needs Chris's approval before any shared environment.

Purpose

Record a release that happened before release records existed (6D PR F, RT-27), with its date precision, source and reviewer, and without notifying anyone. Contract, fields and refusals: Importing past releases.

Preconditions

• Runbook 1 done (verified binding) and migrations 0012–0015 and 0021 applied (SYRF_MIGRATION_MODE=verify).
• The feature exists in the deployed image's feature definition catalog. Historical features without a definition (for example bulk PDF) need a feature-definitions.yaml change first.
• A reviewed input file per release: featureId, releaseKey, stage, precision, occurredOn, source, reviewer. Use Unknown when no date is evidenced; never guess one.
• No API outage or writer freeze is needed: the command writes no policy.

Steps

1. R11.1 Preview with SYRF_RUNTIME_RELEASE_HISTORY_MODE=preview, ..._INPUT (the reviewed input) and ..._OUTPUT (a new path), plus the Mongo settings, RuntimeEnvironment and the three SYRF_MIGRATION_BINDING_EXPECTED_* values.
2. R11.2 Review the preview: record (status Active, imported: true, no applied), event (HistoricalImported, provenanceDetail), the precision and occurredOn, and previewDigest.
3. R11.3 Apply the unmodified preview with SYRF_RUNTIME_RELEASE_HISTORY_MODE=apply, ..._INPUT (the preview), a new ..._OUTPUT and SYRF_RUNTIME_RELEASE_HISTORY_ACTOR=<operator identity>.

Preflight checks and expected output

Check	Expected output
R11.1	Exit 0; Historical release import preview completed; preview file written; nothing in the database changed
R11.3	Exit 0; receipt with the stored record and event; runtimeFeatureReleaseOutbox and runtimeFeatureFlagAudit counts unchanged
Replay	Applying the same preview to a new output path exits 0 with the original receipt and writes nothing

Evidence to capture

• The reviewed input, preview and receipt files, unedited, and the reviewer's approval.

If a step is refused

Refusals exit 2 with Historical release import refused: <message>. Nothing is written.

Message starts with	Action
Unknown feature	Add the feature definition first, in its own PR
The reviewed import preview was modified	Preview again; never edit a preview
The reviewed import preview is stale	The binding changed (runbook 9) or the input no longer gives the plan; preview again
A release with this feature and release key already exists	The key is taken by another import or a prepared release; check the existing record, or use a new key for a genuinely separate release
Required release index is missing or incompatible	Apply migrations (runbook 1 preconditions); the command never creates indexes
Output already exists	Choose a new output path

An exit 1 is an unexpected failure: retry the identical preview with a new output path. This includes a failure to write the receipt file: the receipt is written after the database commit, so the import may already be committed. The retry is an exact replay, which writes nothing and returns the original receipt.

Known gaps and TO CONFIRM items

These were found while checking the runbooks against main. None is fixed by this docs-only change. They are tracked, with the same numbers, in #4403. Each needs a tooling PR or an owner decision before the staging rehearsal (entry criterion 5).

1. No chart-rendered way to run SYRF_MIGRATION_MODE=status or verify in a cluster (R1.1, R1.5).
2. PR previews do not run bind or up (only index-init), so a D1 preview rehearsal cannot apply 0009–0021 today.
3. No chart template runs the lineage command, mounts its reviewed input or keeps its output (runbook 2).
4. The freeze mechanism for staging's auto-merged promotions is not defined (R2.4).
5. No approved way to clear a stale writer-registry entry younger than 24 hours (runbook 2).
6. No chart values for EnvironmentReport:ClientSecret, EnvironmentCollector:ClientSecret or EnvironmentCoordinator:ClientSecret on Identity, and raw env: must not carry secrets (runbooks 4, 5, 7).
7. Whether reviewed chart values should replace the raw env: runtime settings (Mode, Binding, ProductionPolicyMode, FleetInventoryMode, FleetExpectedReplicas, report, collector and coordinator sections) before production.
8. No scheduler runs capture and publication continuously (OD-23), and the capture script supplies no desired GitOps provenance, so promotion reports cannot be Complete yet. Until a scheduler exists, the manual loop in R3.4 is the only way to keep observations fresh.
9. The token request for production Auth0 machine applications (its audience parameter) is not documented.
10. No workflow calls the coordinator routes, and nothing builds its capability intersection or completion targets from a report.
11. No tool emits a candidate manifest; the catalog digest is computed at run time only.
12. The reviewed recovery procedure before restarting a quarantined reader is not written (R9.5).
13. rebind is not built (R9.7).
14. Resolved by #4414 (owner decision D7=A, spec 6C.5): a Legacy host against an imported authority serves the deployed values, writes nothing and registers read-only, so runbook 10 is executable; the runtime-fleet legacy-return scenario rehearses it.
15. Resolved by owner decision D8=A (slice 6C.6): production uses Initialise (no legacy document) instead of the lineage import, so no legacy policy document has to be created. Its in-cluster delivery is still gap 3.

16. The E2E mock OIDC provider issues the collector machine token since #4409 and the report-scope token (environment:report:read) since #4414, which the runtime-fleet legacy-return and initialise-policy-change scenarios use to collect a compatibility report. It still has no mapping for the coordinator scope.

17. No chart template runs the historical release import command or keeps its artifacts (runbook 11), as for the lineage command (gap 3).

Not a gap since #4407: production enables and restores need Collector mode and a running collector. Static, the default, refuses them after any rollout. That is a configuration and operations prerequisite (runbook 6).