E2E Testing Infrastructure¶

Problem Statement¶

SyRF has no end-to-end tests. Unit and integration tests cover individual components (saga state transitions, service methods, concurrency handling), but nothing validates the full user journey: clicking a button in the Angular frontend through to data persisted in MongoDB.

Core system flows — search upload, data export, bulk study update, screening, annotation — are only verified manually. Regressions in cross-service wiring, auth integration, or message routing go undetected until someone manually tests or a user reports a bug.

Goals¶

Validate core user journeys end-to-end: browser → Angular → .NET API → MassTransit → MongoDB
Test the real authentication flow (OIDC) without depending on Auth0
Test the real S3 upload → Lambda → RabbitMQ → saga flow without depending on AWS
Run in CI with no external service dependencies — fully self-contained
Two tiers: fast smoke suite for on-demand PR validation, comprehensive suite on merge to main
Replace the incomplete MockAuthProvider with a proper mock OIDC server
Allow concurrent local worktrees to run isolated E2E stacks without port, container, process, log, or build-artifact collisions

Non-Goals¶

Testing Auth0-specific features (social account linking, Management API)
Testing Elasticsearch integration (search indexing)
Testing SES email delivery
Testing Quartz scheduled jobs (deferred to future iteration)
Performance or load testing
Visual regression testing

Architecture¶

┌──────────────────────────────────────────────────────────────────────┐
│                     Playwright Test Runner                           │
│                                                                      │
│  ┌────────────┐  ┌──────────────────┐  ┌──────────────────────────┐ │
│  │ Tier 1     │  │ Tier 2           │  │ Fixtures                 │ │
│  │ @smoke     │  │ @full            │  │ - auth (storageState)    │ │
│  │ ~3 min     │  │ ~15 min          │  │ - db (MongoDB client)    │ │
│  │ label-gated│  │ merge to main    │  │ - api (HTTP client)      │ │
│  └────────────┘  └──────────────────┘  └──────────────────────────┘ │
└───────────────────────────┬──────────────────────────────────────────┘
                            │ HTTP (Chromium)
                            ▼
                 ┌─────────────────────┐
                 │  Angular Dev Server  │  (ng serve, port 4200)
                 │  OidcAuthProvider (generic OIDC)
                 │  issuer → mock OIDC server
                 └──────────┬──────────┘
                            │
          ┌─────────────────┼─────────────────┐
          ▼                 ▼                  ▼
  ┌──────────────┐  ┌──────────────┐  ┌──────────────────┐
  │ .NET API     │  │ .NET PM      │  │ Mock OIDC Server  │
  │ (dotnet run) │  │ (dotnet run) │  │ (Docker)          │
  │ port 5080    │  │ port 5081    │  │ port 9000         │
  │              │  │              │  │ navikt/mock-       │
  │ JWT Authority│  │ JWT Authority│  │ oauth2-server      │
  │ → :9000      │  │ → :9000      │  └──────────────────┘
  └──────┬───────┘  └──────┬───────┘
         │                 │
         ▼                 ▼
  ┌──────────────────────────────┐   ┌────────────────────────────┐
  │  MongoDB (Docker)             │   │  RabbitMQ (Docker)          │
  │  port 27017                   │   │  port 5672                  │
  │  DB: syrf_e2e                 │   │  MassTransit transport      │
  └──────────────────────────────┘   └─────────────┬──────────────┘
                                                    │
                                    ┌───────────────┴───────────────┐
                                    │  LocalStack (Docker)           │
                                    │  S3 (port 4566)                │
                                    │  Lambda: s3-notifier           │
                                    │  S3 event → Lambda → RabbitMQ  │
                                    └────────────────────────────────┘

Key Principle¶

The Angular app and .NET services run production code. The only configuration differences from a real deployment are:

OIDC authority URL → mock OIDC server instead of Auth0
MongoDB connection → local Docker instead of Atlas
S3/Lambda → LocalStack instead of AWS
RabbitMQ → local Docker instead of in-cluster broker
External services (SES, Elasticsearch, Elastic APM) → disabled

Local multi-stack execution¶

Local runs resolve a stable preferred offset with (PR number mod 41) * 137 and apply it to every host-facing service port. Compose containers carry a worktree ownership label, and pid files, logs, Lambda artifacts, Playwright configuration, Angular runtime configuration, and the interactive OIDC page all use the same resolved stack. CI retains offset 0 and its established ports.

See the local multi-stack runbook for commands, the collision invariant, ownership behaviour, and safe cleanup guidance.

Infrastructure Components¶

1. Mock OIDC Server — `navikt/mock-oauth2-server`¶

Replaces Auth0 for E2E tests. Acts as a real OIDC provider:

Serves /.well-known/openid-configuration and JWKS endpoints
Built-in login page where Playwright selects test users and configures claims
Issues real signed JWTs that .NET services validate via standard JWT Bearer middleware
Pre-configured test users carry SyRF-specific claims (https://claims.syrf.org.uk/user_id, syrf_groups)

How it integrates:

Angular: The @auth0/auth0-angular SDK is not a generic OIDC client — it hardcodes Auth0-specific URL paths (/oauth/token, /authorize) rather than using OIDC discovery. Pointing its domain at the mock server will not work. Instead, a new OidcAuthProvider implementation of the existing IAuthProvider interface will use a standard OIDC client library (e.g., angular-auth-oidc-client) and be registered via the AUTH_PROVIDER injection token when running in E2E mode. The @auth0/auth0-angular package remains for production. This is exactly what the IAuthProvider abstraction was designed for.
.NET: Standard .AddJwtBearer() with Authority pointed at the mock server. Same JWT validation pipeline, same claims extraction, same [Authorize] policies. No backend code changes required.
Auth0-specific features (social account linking, Management API, silent logout) are guarded by feature-availability checks and will be no-ops when those features are unavailable from the mock OIDC server.

Angular environment injection: The app loads config from appConfig.default.json at runtime (not compile-time environments). For E2E, the config is overridden via environment variable substitution in env.template.generated.json, setting auth0CustomDomain and auth0ClientId to the mock OIDC server coordinates. An authProvider config key selects between Auth0AuthProvider (production) and OidcAuthProvider (E2E). The ng serve proxy config routes API calls to the local .NET services.

Test users (seeded in both mock OIDC config and MongoDB pmInvestigator):

User	Role	Purpose
`e2e-admin@test.com`	Administrator, SyrfAdmin	Full access for setup/admin tests
`e2e-reviewer@test.com`	Member	Screening and annotation tests
`e2e-user@test.com`	Member	Standard user journey tests

2. MockAuthProvider Removal¶

The existing MockAuthProvider (frontend-only, hardcoded user, X-Dev-Api-Key header) is incomplete — the backend has no handler for X-Dev-Api-Key (noted as a TODO in the interceptor code). It will be removed and replaced by the mock OIDC server approach.

Files to remove: - src/services/web/src/app/core/auth/mock-auth.provider.ts - src/services/web/src/app/core/auth/mock-auth.provider.spec.ts

Configuration to remove (complete scope): - mockAuth flag from appConfig.default.json, appConfig.default.generated.json, env.template.generated.json - mockAuth branching in auth-interceptor.service.ts — remove _handleMockAuth method, X-Dev-Api-Key logic, DEV_API_KEY import - mockAuth references in app-settings.interface.ts, mock-config.ts - mockAuthMode property in nav.component.ts and template usage - mockAuth guard in admin.effects.ts - Auto-generated files referencing mockAuth: - generated/feature-flags.constants.ts (FEATURE_FLAG_DEFAULTS, ENV_VAR_NAMES, JSON_PATHS) - generated/feature-flags.types.ts (AppConfig, ExternalConfig interfaces) - generated/feature-flags.parser.ts (mockAuth parsing) - generated/feature-flags.selectors.ts (selectMockAuth selector) - app-config.generated.ts, external-config.generated.ts, load-config.generated.ts - Note: generated files are produced by pnpm run generate:flags — the generator config must be updated, then files regenerated - app-config.service.ts — remove parseMockAuth special-casing - .chart/values.yaml for web — remove mockAuth feature flag

mockAuth guard replacement strategy for auth.effects.ts:

Effect	Current guard	Replacement
`connectAccount$`	`!mockAuth`	`!!config.auth0ManagementApiAudience`
`disconnectAccount$`	`!mockAuth`	`!!config.auth0ManagementApiAudience`
`managementAccessToken$`	`!mockAuth`	`!!config.auth0ManagementApiAudience`
`signInActions$`	`mockAuth → EMPTY`	Remove guard — `OidcAuthProvider.loginWithRedirect()` handles this
`refreshTokenSilentlyRefreshed$`	`!mockAuth`	Remove guard — `OidcAuthProvider.getAccessTokenSilently()` handles this
`userRegistered$`	`!mockAuth`	Remove guard — `OidcAuthProvider.loginWithRedirect()` handles this
`silentSignOut$`	`!mockAuth`	`!!config.auth0CustomDomain` (hardcoded Auth0 logout URL)
`signInRecorded$`	`!mockAuth`	Remove guard — both providers emit `user$`
`mockUserSignInRecorded$`	`mockAuth`	Remove entirely — `signInRecorded$` covers both

Hardcoded Auth0 URL in silentSignOut$: The effect contains a hardcoded https://signin.syrf.org.uk/v2/logout?client_id=... URL. This should be made configurable (read from auth0CustomDomain config) and guarded by !!config.auth0CustomDomain.

What replaces MockAuthProvider: - For E2E tests: mock OIDC server (full OIDC flow, real JWT validation) - For local dev without Auth0: developers run the mock OIDC server locally via docker compose -f e2e/docker-compose.e2e.yml up mock-oidc

3. MongoDB — `mongo:7`¶

Database: syrf_e2e (isolated from dev/staging)
Seeded per test run with baseline data (test users in pmInvestigator)
Test data uses per-test unique identifiers to avoid collisions during parallel execution
Read-only reference data (user accounts) shared across tests
Mutable data (projects, studies, annotations) created per test

4. RabbitMQ — `rabbitmq:3-management`¶

Standard MassTransit transport configuration
Management UI available on port 15672 for debugging
Used by both .NET services and the Lambda (via LocalStack)
No service uses the administrator (broker principals plan, Phase 0.3). e2e/scripts/setup.sh creates one restricted user per principal, before the Lambda is verified and before any service starts: syrf-e2e-api, syrf-e2e-pm, syrf-e2e-quartz, syrf-e2e-s3notifier (the notifier Lambda and the bulk-PDF reconciler) and syrf-e2e-pdfagent. Each holds exactly the regexes committed in its service's broker-permissions.json, the same contract the staging and production principals use. The principals, their test-only passwords and their placeholder values are listed in e2e/broker-principals.json; scripts/broker-principals.py renders the files and creates the users (rabbitmqctl, as guest). A bus-endpoint placeholder is rendered for this host: MassTransit names a bus endpoint <host>_<process>_bus_<id>, and E2E runs each service's apphost through dotnet run, so the API's prefix is <host>_SyRFAPIEndpoint_bus. The API, project management and Quartz read their user from appsettings.e2etest.json; the Lambda, the reconciler and the agent read theirs from the manifest.
guest remains the administrator for setup only: user creation, deploy-lambda.sh's and CI's management-API checks, and debugging through the management UI.
A refused operation is visible in two places. The broker logs, for example, operation queue.declare caused a channel exception access_refused: configure access to queue 'quartz' in vhost '/' refused for user 'syrf-e2e-quartz' (docker logs <stack>-rabbitmq). The service logs Declare queue faulted: name: quartz with ACCESS_REFUSED - configure access to queue 'quartz' in vhost '/' refused for user 'syrf-e2e-quartz', then retries ReceiveTransport faulted and never starts serving HTTP, so run-local.sh fails its health wait. A refused publish or send outside a consumer fails that request instead. A regex that is too narrow therefore fails E2E.
Render or inspect a principal's grant by hand: python3 scripts/broker-principals.py render src/services/api/broker-permissions.json --set "{busEndpointPrefix}=$(python3 scripts/broker-principals.py bus-endpoint-prefix SyRF.API.Endpoint)". The renderer fails on an unknown or missing placeholder (python3 scripts/test-broker-principals.py).
The application-authority lane (e2e/authority/) runs its own broker without this setup and still connects as guest.

5. LocalStack — S3 + Lambda¶

Emulates the AWS S3 → Lambda flow with the real Lambda code.

Why LocalStack instead of MinIO + substitute:

The S3 notifier Lambda contains significant business logic that must be tested: - Two distinct workflows based on uploadkind metadata (search upload vs bulk study update) - Complex JSON deserialization of ScreeningImportSettings (includes Dictionary fields) and BulkStudyUpdateJobInfo (nested record type) - LibraryFileType enum parsing with validation - 12 required metadata fields across both workflows with specific error messages - Different MassTransit patterns: Publish<T>() for events vs SubmitJob<T>() for commands

A substitute that bypasses this logic would leave real bugs untested.

Setup flow: 1. E2E setup script builds Lambda package: dotnet lambda package 2. Deploys to LocalStack via AWS CLI: aws --endpoint-url=http://localhost:4566 lambda create-function ... 3. Creates S3 bucket (syrfappuploads) with event notification → Lambda trigger 4. Lambda environment variables configured with RabbitMQ connection pointing to local Docker, as syrf-e2e-s3notifier

Trade-off: LocalStack image is ~1.9 GB and takes 30-60s to start. This is acceptable because: - One-time cost per CI run, not per test - The Lambda's metadata extraction, routing, and deserialization are critical business logic - Maintaining a substitute creates divergence risk

6. .NET Services — Native `dotnet run`¶

API on port 5080, PM on port 5081 (offset from default dev ports to avoid conflicts)
Environment: E2ETest with appsettings.e2etest.json:
MongoDB → mongodb://localhost:27017/syrf_e2e
JWT Authority → http://localhost:9000
RabbitMQ → amqp://localhost:5672 as the service's own restricted user (syrf-e2e-api, syrf-e2e-pm, syrf-e2e-quartz)
S3 endpoint → http://localhost:4566 (LocalStack)
Elasticsearch, SES, Elastic APM, Sentry → disabled
Pre-deploy migrations first. Before the services start, e2e/scripts/pm-migrations.sh applies the project-management migrations to the stack's database (ADR-023). It uses the project-management build's entry point, as the deployed PreSync Job does, bound first.
When it runs. run-local.sh runs it after the .NET build (log: /tmp/<stack>-pm-migrations.log) for every lane except the FEAT-024 production-copy rehearsal. The CI workflow has the same step ("Apply project-management migrations").
Why. Without it the deletion indexes (migrations 0005 to 0008) are missing, and the search DELETE routes answer the readiness 503 deletion_lifecycle_unavailable.
The binding. Migrations 0009 onwards require the environment binding (ADR-023 §8), so up alone refuses an unbound database with exit 2. The script therefore makes three entry-point runs, all with SYRF__RuntimeEnvironment=local:
1. status reads the live cluster fingerprint (rs0@localhost for the stack's single-member replica set);
2. bind binds the database to local:<E2E_STACK>, with the database from appsettings.e2etest.json (syrf_e2e, or SYRF__ConnectionStrings__MongoConnection__DatabaseName) and that fingerprint (a no-op when the identical binding exists);
3. up applies the pending migrations.
The web hosts keep e2etest. A failing step's exit code is the script's: 2 for a binding refusal, an invalid stack name or an unreadable fingerprint. e2e/scripts/pm-migrations.test.sh is the hermetic contract, run by .github/scripts/test-e2e-concurrency.sh. - Reversible deletion on. e2e/scripts/service-env.sh sets FeatureFlags__DeletionLifecycle=true and a one-minute deletion runner schedule (DeletionLifecycle__CronExpression; five minutes in deployments), so the search DELETE routes record a reversible deletion (202) that the runner completes within about two minutes. Specs: search-deletion-lifecycle.spec.ts, search-deletion-in-flight.spec.ts (deletion lifecycle testing §4). Production keeps the flag off, but the flag-off (legacy immediate) delete path is covered by the API and project-management tests, not by E2E.

7. Angular Dev Server — `ng serve`¶

Port 4200 with proxy config routing /api/* to .NET services
E2E runtime config selects OidcAuthProvider (generic OIDC) instead of Auth0AuthProvider
OidcAuthProvider implements the existing IAuthProvider interface using angular-auth-oidc-client, configured with the mock OIDC server's issuer URL
All downstream code (interceptors, effects, guards, ngrx state) runs identically — they depend on IAuthProvider, not on Auth0 directly

Test Tiers and CI Integration¶

Tier 1: Smoke Suite (@smoke)¶

Trigger: PR label run:e2e-smoke, or auto-trigger on paths touching saga/auth/export code

Duration target: ~3 minutes

Tests: - Login → land on dashboard - Create a project - Upload a systematic search → verify studies appear in project - Bulk study update (PDF path, Custom ID) → verify changes reflected - Export data → verify CSV downloads with correct content

Tier 2: Comprehensive Suite (@full)¶

Trigger: PR label run:e2e-full, merge to main, or workflow_dispatch

Duration target: ~15 minutes

Tests: - Everything in Tier 1 - Screening workflow (include/exclude studies across reviewers) - Annotation workflow (add annotations, verify persistence) - Multi-user scenarios (two users screening the same project) - Data integrity checks (export matches what was entered via screening/annotation/bulk update) - Error paths (invalid reference files, upload timeouts) - Bulk study update with all supported field types

Separate opt-in lane: application authority¶

e2e/authority/ is not a tier of this suite: it has its own Compose stack (two API replicas, native Identity, Valkey, Mongo replica set, PM, RabbitMQ), Playwright config and teardown, uses real native sign-in instead of /api/e2e-auth/login, and runs only through bash e2e/authority/run.sh. See Run the application-authority fixture.

Separate local lane: realtime pipeline parity¶

bash e2e/run-local.sh --realtime --realtime-pipeline-v2 off|on runs the realtime specs against one deployed value of the server-only realtimePipelineV2 flag (realtime server plan PR-R2.1). Run it once per value. Before any spec runs, the lane checks the API log and fails closed. It runs the two-context project, listing, export and presence specs serially, plus an offline/online convergence test, then restarts the API mid-session in a second invocation. The restart is local only: it needs a restartable API, and the restart drops the in-memory BFF sessions. @full in CI runs the parallel-safe specs with the flag off. --realtime rejects --smoke, --spec (including the Bulk PDF specs), E2E_BULK_PDF=1 and the FEAT-024 modes. After the restart, the spec asserts the API log gained exactly one Realtime dispatcher started line with the flag on and none with it off. Commands, the spec list and how the restart works: e2e/README.md.

Separate local lane: runtime fleet (Shared authority, two API + two PM)¶

bash e2e/run-local.sh --runtime-fleet is a local lane of its own, built for the runtime feature releases programme (remaining delivery plan, 6B). It starts the ordinary stack, applies the project-management migrations (bind to local:<stack>, then up), then:

seeds the legacy runtimeFeatureFlagOverrides policy and an empty syrf-api writer registry if absent;
generates a baseline holding a Boolean (all false, the deployed e2e defaults) for every key of the generated runtime catalog, then runs the lineage import preview and apply through the API image's operator entry point (SYRF_RUNTIME_POLICY_MODE), with a synthetic actor and freeze reference, before any API process starts;
starts two API and two PM processes, each in its own process group, with RuntimeFeatureFlags__Mode=Shared, the three pinned binding values read from the stack's own database, matching non-production RuntimeEnvironment/ActiveEnvironment (local), a unique instance UID (api-1, api-2, pm-1, pm-2), ExpectedReplicas=2, Autoscaled=false and, on the APIs, FleetExpectedReplicas of 2 for both services (Quartz is unchanged);
runs e2e/scripts/runtime-fleet-harness.sh, whose driver (e2e/harness/runtime-fleet/run.mjs) logs in as the e2e admin, waits for the disableMembership fleet to be Active, then flips the control off, on, off, on (preview, then commit with the digest). After each commit it polls each API's authority-evaluation and the fleet host list every 500 ms until every process reports the committed revision or 90 s pass, and asserts every adoption delta is 60 s or less and the fleet reaches Active. PM has no admin routes, so its adoption is read from its acknowledgement row in the fleet view. Last, it sends SIGSTOP to the second API's process group, flips once, asserts the fleet is never Active and shows Applying or Degraded, sends SIGCONT and asserts convergence.

It writes e2e/test-results/runtime-fleet/<UTC timestamp>/{timings.csv,evidence.json}. --runtime-fleet rejects --smoke, --realtime, the Bulk PDF specs and the FEAT-024 modes, and every --spec except its own browser specs (below): the legacy runtime-flag routes the other specs use are refused in Shared mode. Re-run the driver against a kept stack (--iterative) with E2E_PORT_OFFSET=<n> bash e2e/scripts/runtime-fleet-harness.sh; --dry-run prints the plan and each process's configuration. Hermetic contract: bash e2e/scripts/test-runtime-fleet-harness.sh.

Browser specs (6B.4). bash e2e/run-local.sh --runtime-fleet --spec runtime-fleet-browser-<name> brings up the same four Shared-mode processes, applies the config overlay and, instead of the harness driver, runs only that spec in the Playwright project @runtime-fleet, serially, with E2E_RUNTIME_FLEET=1 (each spec skips without it). --runtime-fleet-initialise runs the harness only and refuses --spec. The project matches only tests/runtime-fleet-browser-*.spec.ts, and every other project in every config ignores those files (scripts/test-smoke-selection.sh and the harness contract check both). There is no round-robin proxy and BFF sessions are per process, so a spec pins each browser context to one API: the app's own origin (the dev-server proxy) reaches api-1, and a context with its own api-2 login and an apiOrigin override of appConfig.env.json reaches api-2. Changes go through e2e/helpers/runtime-policy.ts (Shared preview then commit, emergency stop, restore, outcome and fleet reads as the e2e admin). runtime-fleet-browser-convergence times each change from the commit response to the visible "Disable membership" control in a tab on each API (bound: the 60 s confirmation window), and checks that a tab whose RuntimeFeatureFlagsRevision hub messages are dropped recovers within 15 s of a focus event and of an online event (its API requests are aborted until then; Playwright's context.setOffline is avoided because its offline-to-online transition resets every socket in the browser, in every context). It writes e2e/test-results/runtime-fleet-browser/<UTC timestamp>/{browser-timings.csv,evidence.json}.

Disruption scenarios. --scenario <name>[,<name>] (default all; a dry run lists them) selects what the driver runs, each with its own assertions in the same evidence.json (scenarios.<name>, and a scenario field on every assertion) and rows in timings.csv. Each disruption starts by proving the fleet Active.

Scenario	Disruption	Asserted
`convergence`	the flips above, then `SIGSTOP`/`SIGCONT` of api-2	as above
`mongo-outage`	`docker pause` of this stack's own `<stack>-mongodb` for 75 s	both APIs leave `Evaluated` within 35 s with `value` null and `safeValue` false; fleet never `Active`; after the unpause both APIs are `Evaluated` at the committed revision and the fleet is `Active` within 60 s, with no operator action
`restart`	`SIGTERM` api-2, then start it again (same `InstanceUid`)	fleet not `Active` from about 30 s; the new process's first evaluation is at the committed revision; the acknowledgement's `processStartId` changes; fleet `Active` within 60 s
`scale-up-unexpected`	start a third API (`api-3`, `ExpectedReplicas=2`)	while api-3 is listed the fleet is never `Active` and reads `Applying`, `Degraded` or `Action required`; after stopping api-3 and deleting its acknowledgement row (explicit test cleanup) the fleet is `Active`
`stop-under-outage`	`SIGSTOP` pm-2, then `POST {key}/emergency-stop`	the commit succeeds; every API evaluates the stopped value within the bound; fleet not `Active` until `SIGCONT`; then a `ReleaseStop` preview and commit restores the prior override. Clean-up runs in a `finally`: even after a failed assertion, an error or a lost response, pm-2 is resumed and, once the run has sent an emergency stop it did not release, a `ReleaseStop` preview decides whether the key is stopped and, if so, it is released (or a loud warning says the stack is still stopped)
`collector-scale-up`	restart both APIs with `RuntimeFeatureFlags:FleetInventoryMode=Collector`; publish a synthetic inventory; start an unlisted `api-3`, then stop it	before any publication the fleet is `Unknown` with a `Collector…` reason (the mode is in effect); a. with the inventory listing exactly the four harness UIDs, the fleet is `Active` (positive control); b. while api-3 is live the fleet is never `Active` and is `Unknown` with `CollectorHostObservationDisagrees`; c. after api-3 stops, with the inventory still republished, the fleet is `Active` again within 65 s (the 35 s quiet period plus 30 s), no sooner than 35 s after api-3's last `serverSeenAtUtc`, and lists api-3 as `Retired` (`AbsentFromFreshCollectorInventory`); d. the stop-to-`Active` time is recorded. In a `finally`, so on failing paths too, Static mode is restored and the fleet must be `Active`

First-time initialisation (slice 6C.6). bash e2e/run-local.sh --runtime-fleet-initialise is the same lane on a database with no global flag document (production's situation). Instead of seeding a legacy document and importing it, the bring-up seeds only the empty syrf-api writer registry and runs the API image's initialize lineage source (SYRF_RUNTIME_POLICY_SOURCE=initialize): the preview twice (the two files must be byte-identical), the apply, an exact replay to a new receipt path, and a preview after apply that must be refused. The authority's baseline is the stack's deployed FeatureFlags values, at revision 0, with no overrides and no stops. Then the four Shared hosts start as usual. The scenario initialise-policy-change is run only when named (never by all), and only on such a stack: it judges the bring-up artifacts (copied to <out>/initialise/), checks the stored authority and what both APIs serve, restarts both APIs in Collector mode with the compatibility-report client (syrf-e2e-runtime-report; the mock OIDC maps environment:report:read like the collector scope), publishes the inventory, takes a policy-change report, and runs scripts/runtime-compatibility-check.py on it with a synthetic Runtime candidate (pass, plus a foreign-binding-epoch control that must refuse). Its finally restores Static mode. Runbook: Initialise (no legacy document).

scale-up-unexpected runs in Static mode, where a surplus host reads Applying then Degraded, not Action required: Static mode counts every retained acknowledgement, so it cannot tell a surplus host from a retired one. collector-scale-up is the collector-mode counterpart.

The collector scenario's inventory. Every 5 s the driver publishes one observation per service through the API's real endpoint, POST /api/environment/observations. Each lists exactly api-1, api-2 (api) and pm-1, pm-2 (project management) as ready, non-terminating pods, with an observed generation, a desired count of 2 and a synthetic pinned image digest, chained on the last accepted observation ID. The bearer token comes from this stack's own mock OIDC: a client-credentials request for exactly environment:observations:write. e2e/docker-compose.yml maps only that scope, ahead of the catch-all mapping, to the Auth0 machine-token shape the collector authorization requires (sub <client>@clients, azp, gty client-credentials), so no other token request changes. The restarted APIs also set RuntimeFeatureFlags:WorkloadCollector (enabled, provider Auth0, client syrf-e2e-runtime-collector, audience the e2e Auth0:ApiAudience). The stack itself always starts in Static mode. Unlike scale-up-unexpected, retirement needs no row deletion: the api-3 row stays stored and is shown as Retired. The clean-up deletes it afterwards only because the restored Static mode would count it.

Hermetic checks: bash e2e/scripts/test-runtime-fleet-harness.sh also runs e2e/harness/runtime-fleet/stop-under-outage-cleanup.test.mjs. That self-test runs the real driver's stop-under-outage against a fake API on loopback, with a real stand-in pm-2 process group. One case forces a failed assertion after the emergency stop is committed; another commits the stop but drops its response, with lsof and ps missing from PATH. Both must end with pm-2 resumed and exactly one release. A refused stop must leave nothing to release, and a positive control asserts that a passing run releases exactly once. The contract runs in CI (Validate Workflows, through .github/scripts/test-e2e-concurrency.sh) on a hosted image that does not declare lsof. So the driver reads listeners and process groups from /proc when lsof or ps is not installed, and a skip on Linux fails the contract. The contract also pins the product constants the collector scenario copies (scope, the 30 s freshness and 35 s quiet period, reason names) against the C# sources.

Safety: signals go only to a process group named by this stack's own pid file, after a pure check (verifySignalTarget, unit-tested) proves the pid leads its own group and every listener on that process's port is in it. Docker is touched only for the exact container <stack>-mongodb (assertStackContainer; "contains" is not enough, syrf-e2e-1370-mongodb must not match stack syrf-e2e-137). api-3 uses the stack's spare listener slot (E2E_NOTIFIER_AUTH_PORT, used only by the Bulk PDF lane, which this lane refuses) and the pid file ports.sh reserves for it, which teardown also stops. A mongo pause or a stopped group is also undone on exit and on SIGINT. Tokens come only from this stack's own mock OIDC, over loopback.

What it proves: real, separate API and PM processes (not in-process containers) in Shared mode adopt a committed membership policy, publish acknowledgements, and show the fleet inspection's Applying/Degraded states when one replica is silent, converging when it returns, against synthetic data on one local stack. The in-process counterpart is RuntimePolicyPropagationHarnessTests.

collector-scale-up additionally shows that real API processes in Collector mode refuse Active while a live host is missing from the published inventory, and retire a stopped host's row by MongoDB's clock after the quiet period.

What it does not prove: operational readiness. It is not a deployed environment (no Kubernetes, ingress, Helm-rendered metadata, rolling deploys, autoscaling or real network partitions), uses one host clock and one Mongo, covers only the audited disableMembership control, and its timings are trial measurements, not a production guarantee. Production Shared mode remains refused.

Tier 3: Bulk PDF lane (@bulk-pdf)¶

Trigger: PR label run:e2e-bulk-pdf; automatically on a PR that changes the Bulk PDF pipeline (the PDF agent, the S3 notifier, the upload dialog, the BulkPdf* API/PM/core code or the lane's own harness) when no other rule selects a suite; workflow_dispatch with suite: bulk-pdf; and every full run, as its fourth entry.

Spec: e2e/tests/bulk-pdf-upload.spec.ts, Playwright project @bulk-pdf (the @full project ignores it). Locally: bash e2e/run-local.sh --spec bulk-pdf-upload.

What it proves, with durable capture on:

The browser refuses a folder holding a non-PDF and a renamed non-PDF before anything uploads. The client checks the extension and the %PDF- signature, as the agent does, so a corrupt file cannot reach the agent through the UI.
The dialog's preview matches three studies, reports one unclaimed file and one study without a PDF, and the browser uploads the client-side ZIP through the server-owned multipart session.
The notifier Lambda records one Pending hold row for the exact S3 version and publishes nothing. With publication paused, three reconciler passes leave the job Uploaded.
Lifting the pause publishes the hold. Project Management claims it, the PDF agent scans it with a real clamd and delivers the files, and the job reaches Complete. The report lists copied/unmatched/missing, and each matched study serves a PDF carrying this run's id.
Cleanup, asserted directly because retention is PT5S in this lane: cleanup is registered, the exact staging version is deleted, the release record reaches CleanupReceipt, the hold row is retired and its tombstone purged, and nothing of the project is left under the reserved prefix.

Stack differences (E2E_BULK_PDF=1, e2e/scripts/bulk-pdf-env.sh):

The e2e/docker-compose.bulk-pdf.yml overlay adds clamd (clamav/clamav:1.4; the image ships the signature database and freshclam fetches only updates) and LocalStack DynamoDB and SQS. SQL Server and Quartz are not started: nothing in the flow needs them, and dropping them keeps the entry inside the listener's four-core Docker allowance (clamd has 1.5 CPUs and 3 GiB).
The notifier Lambda gets the chart's durable-capture variables. The hold table (e2e/infra/bulk-pdf-hold-table.json) and the variable names are pinned to the s3-notifier chart by e2e/scripts/test-bulk-pdf-infra-contract.sh.
The reconciler is the deployed handler, run in-process by e2e/tools/bulk-pdf-reconciler-runner once per spec call instead of on the EventBridge schedule. It runs on the host because every host .NET listener binds to loopback.
API and Project Management impersonate the staging root, as the local process-compose stack does, because the Bulk PDF root is fail-closed to production, staging and previews/pr-N. The E2E auth and setup gates read the hosting environment (e2etest), not ActiveEnvironment.
Notifier authority: the real Identity service (OpenIddict), started by e2e/scripts/bulk-pdf-lane.sh (#3622). It seeds the notifier client from BulkPdfNotifierAuthority:Clients, as staging does, plus the API's syrf-api introspection client, with fixed E2E-only secrets. It issues the reconciler's client-credentials token at /connect/token and answers the API's introspection (IdentityService:BaseUrl). Identity runs outside Development, so its full production configuration contract (IdentityHostOptions.BindAndValidate) applies, with disposable values for the clients the lane never reaches (Google, the Project Management provisioner, SMTP). OpenIddict requires HTTPS. Kestrel serves it with a per-run throwaway CA and certificate (e2e/scripts/bulk-pdf-identity-material.sh, which also makes the signing, encryption and data-protection keys). Only the API and the reconciler trust that CA, through SSL_CERT_FILE, which .NET on Linux reads through OpenSSL alongside the system store. No product code relaxes transport security for E2E. A fresh all-zero activation proof is generated at start (the database is dropped first).
The PDF agent does not use the broker administrator. syrf-e2e-pdfagent (created by setup.sh, and again by e2e/scripts/bulk-pdf-lane.sh, through scripts/broker-principals.py) holds exactly the Enabled regexes committed in src/services/pdf-agent/broker-permissions.json (the syrf-<env>-pdfagent contract), with the bus-endpoint prefix derived from this host's name. The reconciler connects as syrf-e2e-s3notifier, like the notifier Lambda. An operation those regexes do not allow is refused with ACCESS_REFUSED and the job never completes. The scheduler and fault paths, which this lane does not reach, are proved by PdfAgentBrokerPermissionTests.
The reconciler's retention (PT5S), failure-queue quiet period (PT2S), retry delay (PT2S) and publication replay interval (PT5S) are seconds, so one test follows a hold from publication to reclamation. The last two are optional variables that no deployed environment sets (defaults 1 and 15 minutes). Plain-HTTP reconciler URLs are accepted on loopback hosts only.

CI Workflow¶

Label-triggered, same pattern as preview environments:

on:
  pull_request:
    types: [labeled, synchronize]
  push:
    branches: [main]
  workflow_dispatch:
    inputs:
      suite:
        type: choice
        options: [smoke, full]

jobs:
  e2e-smoke:
    if: >
      contains(github.event.pull_request.labels.*.name, 'run:e2e-smoke')
      || (github.event_name == 'workflow_dispatch' && inputs.suite == 'smoke')

  e2e-full:
    if: >
      github.event_name == 'push'
      || contains(github.event.pull_request.labels.*.name, 'run:e2e-full')
      || (github.event_name == 'workflow_dispatch' && inputs.suite == 'full')

Infrastructure: Docker containers declared as GitHub Actions services: (health-checked automatically). .NET services started via dotnet run in background with health check polling. Angular started via Playwright's webServer config.

Label lifecycle: Label automatically removed after run completes. Re-add to re-trigger.

Artifacts on failure: - Playwright HTML report (14-day retention) - Traces on first retry - Screenshots on failure - Video retained on failure

Browser network namespace (pomegranate-01)¶

Chromium's AddressTrackerLinux watches NETLINK_ROUTE in its own network namespace. On pomegranate-01 the sibling runners' Docker stacks add and remove veth interfaces constantly, and each event makes Chromium flush its sockets and fail in-flight requests with net::ERR_NETWORK_CHANGED (#3423). Matrix entries on pomegranate-01 set browser_netns, exported as E2E_BROWSER_NETNS=1, and playwright.config.ts then launches Chromium through e2e/scripts/chromium-netns.sh:

unshare --user --map-root-user --net gives the browser a private namespace (no packages; Playwright already passes --no-sandbox). Inside it lo is up, plus a dummy0 interface with an address (without one Chromium reports navigator.onLine === false) and an unreachable default route.
e2e/scripts/netns-bridge.cjs carries the browser-facing loopback ports across through unix sockets in a private 0700 directory under $E2E_JOB_ROOT: web, API, PM, Quartz, mock OIDC, LocalStack and the Bulk PDF static server (E2E_{WEB,API,PM,QUARTZ,OIDC,LOCALSTACK,PDF_STATIC}_PORT). Plain TCP piping, so websockets (SignalR, ng serve) work unchanged.
The same bridge carries an HTTP CONNECT proxy, passed to Chromium as --proxy-server, for the external assets the app loads (Google Fonts and Material Icons from fonts.googleapis.com / fonts.gstatic.com); loopback bypasses the proxy. It tunnels only to those two hosts on port 443 and answers anything else with 403 Forbidden.
The real binary is the headless shell Playwright would pick itself (E2E_CHROMIUM_REAL).

"Prove Chromium without job sudo" runs e2e/scripts/test-chromium-netns.cjs on those entries: the Chromium browser process must be in a different network namespace from the runner shell and must load a page and round-trip a websocket from a throwaway loopback server. Run it locally with node e2e/scripts/test-chromium-netns.cjs [--external <https-url>]. Local runs and the Bramble guest (Ubuntu 24.04, unprivileged user namespaces restricted) keep the default launch.

Project Structure¶

e2e/                                    # Top-level, spans all services
├── package.json                        # Playwright, MongoDB driver, helpers
├── tsconfig.json
├── playwright.config.ts                # Projects, webServer, tiers
├── docker-compose.e2e.yml              # MongoDB, RabbitMQ, mock OIDC, LocalStack
├── scripts/
│   ├── setup-localstack.sh             # Build + deploy Lambda to LocalStack
│   ├── seed-db.ts                      # Baseline MongoDB seeding
│   └── wait-for-services.ts            # Health check polling
├── config/
│   ├── mock-oidc/                      # navikt test users + client config
│   ├── angular-e2e-env.json            # Angular environment overrides
│   └── appsettings.e2etest.json        # .NET service config
├── global-setup.ts                     # Start infra, seed DB, start services
├── global-teardown.ts                  # Cleanup
├── fixtures/
│   ├── auth.fixture.ts                 # Authenticated page contexts
│   ├── auth.setup.ts                   # Login via mock OIDC, save storageState
│   ├── db.fixture.ts                   # MongoDB client for seeding/assertions
│   └── api.fixture.ts                  # Direct API client for test setup
├── pages/                              # Page Object Models
│   ├── dashboard.page.ts
│   ├── project.page.ts
│   ├── screening.page.ts
│   ├── annotation.page.ts
│   ├── data-export.page.ts
│   ├── search-upload.page.ts
│   └── bulk-study-update.page.ts
├── tests/
│   ├── smoke/                          # @smoke tagged — Tier 1
│   │   ├── login.spec.ts
│   │   ├── create-project.spec.ts
│   │   ├── search-upload.spec.ts
│   │   ├── bulk-study-update.spec.ts
│   │   └── data-export.spec.ts
│   └── full/                           # @full tagged — Tier 2
│       ├── screening-workflow.spec.ts
│       ├── annotation-workflow.spec.ts
│       ├── multi-user-screening.spec.ts
│       ├── data-integrity.spec.ts
│       └── error-paths.spec.ts
├── helpers/
│   ├── mongo-client.ts                 # CSUUID-aware MongoDB queries
│   ├── test-data-factory.ts            # Generate unique test data per test
│   └── wait-for-saga.ts               # Poll API for saga/job completion
└── playwright/
    └── .auth/                          # gitignored — storageState files

Test Patterns¶

Per-Test Data Isolation¶

Every test creates its own project/data with unique identifiers to avoid collisions during parallel execution:

test('can upload a systematic search @smoke', async ({ authenticatedPage, db }) => {
  const projectName = `e2e-search-upload-${Date.now()}`;
  // Create project via UI or API fixture
  // Upload search file
  // Assert studies appear
  // Assert MongoDB state
});

Waiting for Async Operations¶

MassTransit saga processing is asynchronous. Tests poll for expected outcomes:

// Preferred: wait for UI to reflect the change
await expect(page.getByText('5 studies imported')).toBeVisible({ timeout: 30_000 });

// Alternative: poll API when UI doesn't have a visible indicator
await expect.poll(async () => {
  const res = await apiClient.get(`/api/projects/${projectId}/searches/${searchId}`);
  return (await res.json()).status;
}, { timeout: 30_000, intervals: [1000, 2000, 5000] }).toBe('completed');

Database Assertions¶

Verify backend state directly when UI assertions aren't sufficient:

// Custom fixture provides CSUUID-aware MongoDB client
const study = await db.collection('pmStudy').findOne({
  SystematicSearchId: searchId,
});
expect(study.ScreeningInfo.ProjectAgreementThresholds).toBeDefined();

Multiple User Roles¶

Playwright setup project authenticates each test user and caches storageState:

// Multi-user test
test('two reviewers can screen the same study @full', async ({ browser }) => {
  const reviewer1 = await browser.newContext({
    storageState: 'playwright/.auth/reviewer1.json',
  });
  const reviewer2 = await browser.newContext({
    storageState: 'playwright/.auth/reviewer2.json',
  });
  // Both navigate to same project, screen same study
  // Verify agreement/disagreement logic
});

Angular Considerations¶

Disable Angular animations in E2E mode to avoid timing issues with Material components
Use data-testid attributes for stable Playwright selectors where role/text locators are insufficient
Playwright's auto-waiting handles Angular zone stability automatically

Risks and Mitigations¶

Risk	Mitigation
LocalStack S3→Lambda notifications unreliable	Pin LocalStack version; verify notification chain in global setup before tests run
Flaky tests from shared mutable data	Per-test data isolation with unique identifiers
Slow CI from 4 Docker containers + 2 .NET services	Tier 1 is fast (~3 min); Tier 2 is label/merge-gated
`@auth0/auth0-angular` incompatible with mock OIDC	New `OidcAuthProvider` using generic OIDC library behind existing `IAuthProvider` interface
LocalStack .NET 10 runtime issues	Fallback: build Lambda as container image, deploy to LocalStack as container Lambda
Port conflicts with local dev	E2E uses offset ports (5080/5081) distinct from dev ports (8080/8081)
LocalStack image size (~1.9 GB)	Cached in GitHub Actions; one-time pull per runner
LocalStack requires Docker socket mount	GitHub Actions ubuntu-latest supports this; document in setup guide
Mock OIDC port 9000 protocol ambiguity	Use HTTP on port 9000 (non-HTTPS port); avoids self-signed cert handling
.NET service health check not specified	Services expose `/health` endpoint; `wait-for-services.ts` polls with 2s interval, 60s timeout, fail-fast on process exit

Future Iterations¶

Quartz integration: Add SQL Server container, test background job scheduling flows
Visual regression: Add screenshot comparison for key UI states
Performance baseline: Measure response times for critical flows
Accessibility testing: Integrate axe-core via Playwright
Preview environment E2E: Run smoke suite against deployed PR preview environments