# PM081 complete fictional incident packet

This packet is a complete worked example for the OpenMax incident-postmortem guide. The company, service, people, records and numbers are fictional. It is not a customer incident, not a product test, not a benchmark and not a measured OpenMax result. All times are UTC. Evidence cutoff: `2026-09-04T12:00:00Z`.

## S01. Document control

| Field | Value |
|---|---|
| Postmortem ID | PM081 |
| Incident | Document-preview render timeouts and stale results |
| Fictional service | Northstar Preview, internal document-preview SaaS |
| Incident date | 2026-09-01 |
| Confirmed impact window | 09:07:11–10:18:00 UTC |
| Severity | SEV-2 under the fictional OPS-04 policy |
| Evidence cutoff | 2026-09-04T12:00:00Z |
| Document owner | IC, incident commander |
| Facilitator | PA, postmortem approver |
| Publication state | IN_REVIEW |
| Sensitivity | Internal; request bodies and account identifiers excluded |

Quick answer: a production configuration-schema change was followed by unbounded preview-worker concurrency and downstream pool exhaustion during a burst. That mechanism remains `PLAUSIBLE` pending independent service-lead review. The impact query identifies 279 affected attempts out of 3,200 eligible attempts (8.72% after rounding). The response disabled the new path at 09:41; the queue reached zero at 10:03 and a synthetic path passed at 10:18. Those observations indicate sampled recovery, not proof that every user result was correct.

## S02. Roles and authority

| Key | Responsibility | Approval boundary |
|---|---|---|
| IC | response chronology and decision log | may confirm response events, not causal mechanism alone |
| SL | service architecture and mechanism analysis | reviews C02 and technical actions |
| DL | impact query and category reconciliation | signs E09 and A05 evidence |
| OL | monitoring, alert and detection evidence | reviews E02/E05 and A03 |
| SE | security, privacy and disclosure screen | may require restricted handling or escalation |
| PA | facilitates and approves the postmortem state | accepts commitments after specialist reviews |
| AO | owner named on each action | supplies implementation and effectiveness evidence |

There are no personal names in this teaching packet. A real record should identify accountable people or durable teams according to internal policy.

## S03. Evidence register

| ID | Time UTC | Artifact | Supports | Limitation |
|---|---|---|---|---|
| E01 | 09:02:14 | deployment audit event, fictional version `v4.18.0` | schema change reached production | timing alone does not prove cause |
| E02 | 09:06:50–09:41:00 | queue-oldest-age and connection-pool series | queue age and pool use rose after the change | aggregate metrics contain no request-level outcome |
| E03 | 09:07:11 | first retained timeout trace in the review window | first confirmed impacted attempt for this clock | earlier unobserved impact remains possible |
| E04 | 09:20:03 | support intake timestamp with body redacted | a user-reported symptom existed | receipt time is not onset time; one report is not a denominator |
| E05 | 09:24:00 | alert event and routing receipt | automated detection time | threshold did not fire at first confirmed impact |
| E06 | 09:41:00 | feature-flag audit entry | new preview path was disabled | does not alone prove backlog clearance or correctness |
| E07 | 10:03:00 | queue metric snapshot | backlog returned to zero | empty queue does not prove rendered content was current |
| E08 | 10:18:00 | synthetic render result for a defined fixture | sampled render path passed | one synthetic fixture is not all-user proof |
| E09 | 09:07:11–10:18:00 | versioned impact-query export `impact-v3` | outcome counts reconcile to 3,200 eligible attempts | fictional query needs DL peer review; says nothing about unique people |

Access rule: E01–E03 and E05–E09 point to restricted internal systems. The postmortem contains stable references and redacted summaries, not raw tokens, request bodies or account identifiers. Evidence received after the cutoff enters an addendum and triggers review of affected claims.

## S04. Impact query and arithmetic

Population rule: include completed preview render attempts whose server receipt time falls from 09:07:11 through 10:18:00 UTC, production region `r1`, excluding retries marked as internal health checks. Assign exactly one precedence-ordered outcome: timeout, stale-preview checksum, or expected-preview checksum.

| Mutually exclusive outcome | Fictional query rule | Count |
|---|---|---:|
| Timed out | terminal status `TIMEOUT` | 248 |
| Stale preview | completed, checksum equals previous document version | 31 |
| Expected preview | completed, checksum equals submitted version | 2,921 |
| Eligible total | union of the three categories | 3,200 |

Reconciliation: `248 + 31 + 2,921 = 3,200`.

Impacted attempts: `248 + 31 = 279`.

Impact rate: `279 / 3,200 × 100 = 8.71875%`, displayed as **8.72%**.

The unit is an attempt. It is not a person, workspace, customer, contract, revenue amount or error budget. Retries may cause one actor to contribute multiple attempts; the packet therefore makes no unique-user claim.

## S05. Evidence-backed timeline

| Time UTC | Event or observation | Type | Evidence |
|---|---|---|---|
| 09:02:14 | `v4.18.0` production deployment completed | event | E01 |
| 09:06:50 | queue age and downstream pool use began a sustained rise | observation | E02 |
| 09:07:11 | first retained timeout trace in the chosen clock | observation | E03 |
| 09:20:03 | support received a report of a preview timeout | report | E04 |
| 09:24:00 | queue-age alert routed to on-call | observation | E05 |
| 09:29:00 | IC declared the fictional SEV-2 | decision | incident log linked from E05 |
| 09:34:00 | IC approved disabling the new path after status check | decision | incident log |
| 09:41:00 | feature flag showed the new path disabled | event | E06 |
| 10:03:00 | queue backlog returned to zero | observation | E07 |
| 10:18:00 | defined synthetic render fixture passed | observation | E08 |

Derived clocks:

- First confirmed impact → automated alert: `09:07:11–09:24:00 = 16m49s`.
- Incident declaration → mitigation action: `09:29:00–09:41:00 = 12m00s`.
- First confirmed impact → sampled recovery confirmation: `09:07:11–10:18:00 = 1h10m49s`.

These are explicitly named intervals, not automatically MTTA or MTTR. The fictional organization has not supplied approved metric definitions. The support timestamp is when a report arrived, not when the incident began.

## S06. Causal claims and uncertainty

| ID | Type | Claim | State | Evidence and limitation |
|---|---|---|---|---|
| C01 | trigger | the deployment introduced the configuration-schema change before impact | CONFIRMED | E01 plus schema diff; temporal order is not the whole mechanism |
| C02 | direct mechanism | omitted `max_inflight` was interpreted as unlimited, allowing a burst to exhaust the downstream pool | PLAUSIBLE | config diff, E02 correlation and a non-production replay; independent SL review pending |
| C03 | contributing condition | validator accepted a missing or zero concurrency value | CONFIRMED | isolated configuration unit test; does not measure production impact |
| C04 | contributing condition | canary plan did not include the burst shape associated with the incident | CONFIRMED | versioned canary plan and workload comparison |
| C05 | contributing condition | alerting lagged the first retained timeout by 16m49s | CONFIRMED | E03 and E05; earlier impact remains UNKNOWN |
| C06 | alternative | database content corruption caused stale results | DISPROVED for this incident | integrity scan found no persisted-content mutation; does not prove every possible database fault impossible |
| C07 | scope uncertainty | affected attempts occurred before E03 | UNKNOWN | no retained request-level trace establishing or rejecting it |

Blameless does not mean ownerless. The record examines how a permissive default, validation gap, incomplete canary shape and delayed signal made reasonable actions insufficient. Owners remain responsible for supplying evidence and closing actions.

## S07. Response review

What reduced impact: the feature flag allowed the team to stop the new path without a full deployment (E06). Backlog and synthetic checks then supplied two different recovery observations (E07/E08).

What delayed detection: the alert observed oldest queue age only after its configured threshold. Support reported the symptom earlier than the alert, but that report still arrived 12m52s after the first retained timeout. This comparison does not identify an individual failure.

What delayed causal confidence: the production config did not retain a normalized effective value for `max_inflight`, so the review combined a schema diff, aggregate metrics and a non-production replay. C02 remains `PLAUSIBLE` until independent review.

Where the team got lucky: the new path had a feature flag. The rollback runbook, however, did not require a backlog-clearance check or a correctness fixture; responders added those checks ad hoc. A04 turns that dependency on local knowledge into a versioned step.

Security/privacy screen: the example contains no evidence of unauthorized access, but that absence is not a security conclusion. SE must review any real incident involving access, disclosure, personal data or contractual notice obligations.

## S08. Corrective actions and verification

| ID | Change | Owner | Due | State at cutoff | Verification contract |
|---|---|---|---|---|---|
| A01 | reject missing, zero or negative concurrency config | Platform AO | 2026-09-09 | IN_PROGRESS | tests cover missing/zero/negative/valid; production config audit shows no invalid row |
| A02 | replay a 1,000-job staging burst with 40-worker ceiling | Reliability AO | 2026-09-12 | OPEN | queue drains; peak active workers <=40; zero stale results in defined fixture set |
| A03 | alert when oldest queue age is >120s for 3m | Observability AO | 2026-09-10 | IN_PROGRESS | injected-lag drill creates a correctly routed alert within the documented window |
| A04 | add feature-disable, backlog and correctness checks to rollback runbook | Incident Ops AO | 2026-09-08 | IMPLEMENTED | independent game-day operator completes versioned steps without oral help; not yet run |
| A05 | publish versioned mutually exclusive impact query and reconciliation check | Data AO | 2026-09-11 | IN_PROGRESS | category sum equals eligible population and DL peer reviewer signs the query version |

No action is `EFFECTIVENESS_VERIFIED` at the cutoff. A04 is written but its game-day observation is still absent. PA reviews action state weekly; failed verification reopens the action rather than rewriting the success criteria. `ACCEPTED_RISK` requires a named authority, rationale and review date; no PM081 row currently uses that state.

## S09. Review agenda and approval record

Proposed 60-minute review:

1. 0–5: restate blameless/evidence rules and scope.
2. 5–15: reconcile impact denominator and exclusions.
3. 15–27: correct the evidence-backed timeline.
4. 27–42: review C01–C07, including uncertainty and alternatives.
5. 42–54: accept, revise or reject A01–A05 and their verification contracts.
6. 54–60: read back owners, due dates, redaction needs and approval conditions.

Current approval: `IN_REVIEW`. PA has not marked the record approved because SL review of C02, DL review of E09/A05 and the SE disclosure screen are pending. This incomplete approval state is intentional; the example does not manufacture authority.

## S10. Follow-up ledger

| Follow-up | Due | Required evidence | Current observation |
|---|---|---|---|
| SL reviews C02 | 2026-09-08 | signed causal review | NOT_OBSERVED at cutoff |
| DL reviews impact-v3 | 2026-09-09 | query review and reconciliation | NOT_OBSERVED at cutoff |
| SE completes disclosure screen | 2026-09-08 | review decision | NOT_OBSERVED at cutoff |
| A04 game day | 2026-09-15 | operator log and result | PROSPECTIVE; not yet run |
| PA publication decision | after required reviews | approval record | NOT_OBSERVED at cutoff |

Late evidence belongs in an addendum with the affected claim IDs and a new approval decision. The original cutoff and prior state remain visible.