Operations and recovery qualification

Bounded candidate reference — 2026-10-20: 0.2.0-rc.1 is committed locally through 84f415b3098af5653577b2c65eea31fca21df79c, unpublished and not production-approved; stable PHP >=8.5.11 <8.6.0 is required. This is documentation, not adoption: this knowledge base retains its installed 0.1.0-rc.1 dependency and PHP 8.2.11 local profile. Dated PHP 8.2 evidence is historical; earlier dirty-source reports retain their original test provenance, not retroactive clean-commit proof. Candidate examples are illustrative: API signatures were reviewed, but candidate code was not executed by this editorial update. See current checkpoint, open criteria and adoption boundary.

Later isolated OPS and quota-refusal evidence supplements dated narrative below. Named synthetic profiles retain restore <=5s / <10s and RPO <=5s; HTTP readiness detection <5s is measured separately, not a production SLO. Preserve damaged history and UNKNOWN; no catch-to-empty or blind retries.

Use the final worker/UNKNOWN runbook for exact claim ownership, dead-letter versus UNKNOWN distinctions, authoritative reconciliation and safe restart. Host qualification is explicitly deferred; these procedures authorize no service/database mutation.

The current file-audit guide documents the owner-approved batched scan ceilings and strict private-path profile. Existing unsafe modes, owners or aliases now refuse rather than being normalized. This applies only to the optional file-audit adapter; no consumer history, PHP-FPM user or host permissions were changed. Missing-file [] compatibility remains unchanged pending its separate Phase B policy decision.

The OPS-F01 audit correction is locally verified against actual bounded ENOSPC. FileAuditSink now rejects an incomplete/corrupt final record under its append lock; searches raise a safe integrity error for damaged records encountered instead of silently skipping them. It preserves every existing byte, even after capacity returns: there is no automatic history repair. Quarantine/recovery/rotation of real damaged history requires separately authorized operator procedures. Read/write access to the private audit file is required; per-record reads are bounded to 32 KiB. Append checks only the final record; search stops at the query result limit, so neither certifies all unseen history. No fsync/power-loss, cryptographic integrity or multi-node guarantee is added. SQL mutation/audit atomicity still requires the application-owned SQL transaction. Phase C subsequently reached C-020 local completion, including actual SQL atomic/rollback, Apache/FPM opaque failure, required-store readiness and fresh synthetic reference recovery. See the literal predicate disposition: OPS-06/OPS-10 remain TO VERIFY for owner/deployment qualification, not automatically promoted by a green disposable fixture.

What is supplied

The source-bound ownership review separates framework mechanics from TEST-only SQL/audit/HTTP/restore composition. The fixtures do not install a watchdog, SQL-audit daemon, backup product, deployment server, log-forwarder or live alert transport in an application.

ErrorLogLogger emits bounded one-line JSON: schema_version, UTC timestamp, environment, level, stable event code in message and sanitized context. Bind ContextLogger with the trusted environment/version/release, then add opaque request/job/correlation/causation IDs, enumerated subsystem/status, duration and safe error fingerprint. Never use email, full paths, raw errors, provider bodies, SQL values or secrets as event codes, labels, IDs or alert dimensions. Unrecognized environments are UNSPECIFIED, not silently inferred as PROD.

HealthRegistry separates liveness and readiness. Register only bounded process checks for liveness; required DB/queue/storage/scheduler conditions belong in readiness. Optional provider outages must not turn liveness red. An empty mode is unhealthy, not an implicit success. Public health responses should expose aggregate status only; keep diagnostic categories behind an explicit authenticated developer/operator policy. No HTTP debug route is installed by this framework.

Durable alert composition

// Composition only: inject a dedicated operations DB connection and clock.
// Apply the existing documented outbox schema before enabling this module.
$sink = new \Clean\Framework\Observability\DatabaseAlertSink(
    $operationsConnection,
    $clock,
    ['info' => 'operations', 'warning' => 'operations', 'critical' => 'pager'],
    deduplicationSeconds: 300,
    quietHoursUtc: [0, 1, 2, 3, 4, 5]
);
$sink->send(new \Clean\Framework\Observability\Alert(
    'worker.stale', 'critical', 'queue.worker_stale', $clock->now(),
    ['queue' => 'critical'] // Enumerated dimensions; never a payload/recipient.
));
// Configure OutboxAlertTransport($reviewedDestinationAdapter) under 'pager'
// in OutboxDispatcher. The application supplies its AlertTransport; no live
// service, credential, recipient or enabled worker is supplied by default.

The SQL unique message ID deduplicates the same key/severity within a fixed admission bucket, shared across processes. A critical incident is independent of a warning with the same key and is never delayed by quiet hours. Configure critical routing to the escalation destination; the operator owns staffing, acknowledgement deadlines, fallback and escalation after non-acknowledgement. Bucket boundaries can admit adjacent alerts; this is not a rolling suppression window. The process-local AlertManager alone is not shared durable admission.

The outbox retains status, attempt count, timestamps, last safe error and opaque delivery proof; it is not an immutable per-attempt delivery-history ledger. Transport SENT means destination delivery acknowledgement, not that a human read/responded. Only definitely-undelivered transient failures return RETRY; exceptions, lost acknowledgements and expired sends hold UNKNOWN until an authoritative destination lookup resolves them. Never blindly replay pager actions. Database admission errors propagate; the DB-failure alert path must also have an independent local watchdog/channel to avoid circular dependency.

Monitoring and data lifecycle

The Urgent operations runbook defines current stale/backlog/schedule signals, missing idle-worker pulse, explicit disabled/unqualified states, safe recovery decisions and separately authorized host-drill evidence. U-025/U-026 remain BLOCKED_EXTERNAL; local webhook/restore success does not close OPS-06/OPS-10.

The operator must bind actual metric/alert exporters. Use this bounded catalog:

Retention/export/rotation are deployment policy, not auto-installed daemons. Provisional recommendation: bounded local operational logs for 14 days, reviewed encrypted export for 30 days, incident/audit retention under the application's lawful schedule/holds. Owner must accept/change these before PROD. Size-cap/rotate outside public roots; exercise full-disk and permission failure on a disposable filesystem, not by filling a developer/production disk. Export only approved classifications; do not turn metrics into a user database.

Synthetic recovery rehearsal and host runbook

The guarded MySQL fixture now rehearses encrypted, authenticated synthetic rows plus one private object, wrong key/unknown key version/corrupt archive denials before SQL, distinct restore schema, matching code/schema version, reference/checksum verification, readiness and explicit post-cutoff loss. The owner selected two distinct named synthetic-only restore profiles on 2026-10-19. Neither profile is an application/host SLO or a speed limit:

http_readiness_response_v1 separately requires one local fault-time readiness response in <5,000 ms. It does not measure restoration, polling frequency, notification delivery or real incident detection latency. Detection, restore duration and checkpoint-to-simulated-loss are not interchangeable measurements.

Test-only tests/Support/SyntheticRecoveryProfiles.php owns the closed names, numeric budgets and exact inclusive/strict operators used by both declarations and assertions. Each plan is emitted before capture/faults/measurement; the observed receipt includes its profile, declared limits and actual duration. Unknown profiles and negative/nonfinite durations refuse. No caller or environment can choose a more lenient test budget after measurement. See the approved decision. Historical receipts retain their original thresholds and observations; they are not retroactively renamed or promoted by this prospective policy.

The encrypted drill's ephemeral key is never exported and is zeroed after its successful rehearsal. Neither format is a general backup product, concurrent whole-application snapshot or off-host backup service. Applications choose and qualify actual recovery targets and operational tooling separately.

Before host qualification assign incident commander, DB/storage operator, security reviewer, primary/backup maintainer and notification owner. Choose actual RPO/RTO/SLOs before a timed rehearsal; five-minute critical detection is a starting proposal, not a result. Then:

• Record incident ID, UTC detection/cutoff, code/artifact/schema versions and allowed isolated restore target; stop unsafe writes/workers as authorized.
• Verify encrypted off-host backup signature/hash, key availability, retention, access and restore authorization without copying it into source control.
• Restore data and private objects to the isolated target using the reviewed application backup tool; deny traversal/unexpected archive entries. Do not run an arbitrary SQL dump or archive extractor against a live target.
• Confirm object references/checksums, DB invariants, migration compatibility, smoke routes, least-privilege auth, one idempotent worker and alerts. Reconcile unknown effects before resuming. DDL rollback is not promised; use the documented corrective migration/forward recovery or compatible backup.
• Record actual downtime/RPO/RTO, missing transactions, safe rollback decision, restored scope, delivery proof and human acknowledgement; independent review signs the report. Keep missing/failed evidence open, not estimated as PASS.

Corrupt archive/expired key: stop before import, preserve evidence, obtain an authorized known-good checkpoint/key, never bypass verification. Storage denial or full disk: stop acceptance/dispatch when durability cannot be recorded, use independent monitoring and an opaque HTTP error, restore permissions/capacity under operator control. Stopped worker: check leases/unknowns before controlled restart; do not mass-replay. Failed restore: retain the isolated failed target for authorized diagnosis and keep the original system unchanged.

Phase C qualifies actual full-disk behavior only on its kernel-enforced8MiB disposable tmpfs and actual HTTP failure only in its owned Apache/FPM fixture. It verifies same-connection SQL audit/outbox rollback, explicit local rollback uncertainty reconciliation, independent local fallback, required-store readiness versus liveness and a quiesced two-operation checkpoint restore. It supplies no deployed monitoring daemon, SQL audit product, general backup tool or real recipient. Application/deployment owners must select and qualify that wiring.

Off-host export, actual on-call acknowledgement, production supervisor/cron, final-host full disk/restore and independent signoff remain TO VERIFY. See queue ownership, privacy and security boundary model.

Editorial provenance

Source: docs/OPERATIONS_AND_RECOVERY.md. Source SHA-256: 599014b73b7cbd56d64d33a35017f9eb680824dbbb593490228d170a7853512c.

Framework candidate reference; editorial checkpoint 2026-10-20. See candidate status and adoption boundary.