Why Most Protocol Post-Mortems Would Not Pass a Peer Review

On October 12, 2023, the Cosmos Hub chain halted for roughly 10 hours. A validator had upgraded to a new version of the gaia client, which introduced a state migration bug in the IBC client recovery submodule. Three days later, the post-mortem landed. About 1,200 words. It had a timeline, a list of affected validators, a fix summary, and a section called “Action Items.” What it lacked was the specification delta—the precise gap between what the IBC client recovery specification required and what the implementation actually did. Without that, no third-party engineer could reproduce the analysis independently.

This is not an outlier. It is standard practice. Blockchain protocol engineering has no equivalent of aviation’s NTSB incident reports, no analog to the FDA’s medical device adverse event database, and no adoption of the structured postmortem culture that mature software engineering disciplines formalized over a decade ago. What passes for a post-mortem in this industry would, in most peer-reviewed engineering contexts, be rejected as insufficient evidence of root cause, incomplete in mechanism description, and structurally incapable of supporting recurrence prevention.

The Structural Deficit in Protocol Incident Reporting

The Google SRE book—Chapter 15, “Postmortem Culture: Learning from Failure,” and Appendix D, which contains a full example postmortem template—set the industry standard for software incident reporting over seven years ago. The template enforces specific structural elements: an impact summary, a root cause section kept separate from the timeline, a section on what went well and what went poorly during response, and an action item register with owners and deadlines. Chapter 33, “Lessons Learned from Other Industries,” draws the explicit analogy to aviation and medical incident reporting, arguing that structured postmortems are a non-negotiable component of any engineering discipline that operates systems where failure has consequences.

Blockchain protocol post-mortems almost universally fail to meet this bar. The structural deficits fall into three recurring patterns.

Pattern 1: Timeline conflation. Most protocol post-mortems present a chronological event log and label it a root cause analysis. The Solana mainnet-beta outage of February 25, 2023, produced a post-mortem that listed, in order: a specific NFT mint caused network congestion, the congestion triggered a durable nonce bug, the bug caused a block producer to stall, the stall cascaded through the cluster. That is a timeline. The root cause—the mechanism by which the durable nonce instruction’s state transition function interacted with the Sealevel runtime’s parallel execution model under memory pressure to produce an unrecoverable state—was buried in a single paragraph near the end. No reference to the specific code path. No reference to the specification clause it violated.

Pattern 2: Corporate hedging over mechanism. Protocol post-mortems are frequently written by teams with token holders, governance forums, and legal exposure. The result is language that obscures mechanism-level detail. Phrases like “an unforeseen interaction between components,” “edge case in the state transition,” or “unexpected behavior under specific network conditions” appear where a rigorous report would specify: which function, which input, which invariant was violated, and which clause of the protocol specification the implementation diverged from. The May 2023 Bitcoin chain split, triggered by incompatibility between Bitcoin Knots and Bitcoin Core around non-standard transactions, produced write-ups that described the fork at the network level. Few specified the exact transaction validation rules that diverged between the two implementations, the block height at which the divergence became deterministic, or the specific consensus rule interpretation each client applied.

Pattern 3: Missing specification delta. This is the most critical gap. A post-mortem that does not include the difference between what the specification requires and what the implementation produced cannot be independently reviewed. It cannot be reproduced. It cannot be tested against alternative implementations. And it cannot be used to verify that the fix addresses the root cause rather than a symptom. In the Cosmos Hub October 2023 incident, the post-mortem referenced “a bug in the IBC client recovery logic” but did not include the specific IBC specification clause (ICS-02, Section “Client Recovery”) that was violated, the expected state transition, the actual state transition, or the test case that would catch the divergence. Without that delta, the reader is asked to trust the team’s analysis rather than verify it.

What a Structured Protocol Incident Report Would Look Like

The NIST Cybersecurity Framework 2.0 demonstrates how a standards body in an adjacent domain enforces analytical structure. NIST CSF provides Profiles that translate framework outcomes into practical actions, Informative References that map controls to specific implementations, and a structural scaffold that separates risk identification from risk response from risk recovery. The framework does not tell organizations to “write better reports.” It provides a structurally enforced scaffold with templates, mappings, and resource centers that make analytical checkpoints mandatory rather than optional. Blockchain protocol engineering has no equivalent. No standards body has produced a reporting scaffold for consensus failures, safety violations, or liveness faults. No protocol foundation requires post-mortems to include specification deltas. No industry-wide template exists for incident reports in distributed systems.

What would a structured protocol incident report scaffold require? At minimum, the following sections, each serving a distinct analytical function:

Section 1: Impact Classification. Not “the chain halted.” Instead: safety violation (two conflicting finalized states), liveness failure (no blocks produced for N epochs), or economic exploit (X tokens extracted via incentive mechanism Y). The classification determines which invariant was violated and which subset of the protocol’s guarantees were broken. A safety violation and a liveness failure require different response mechanisms, different root cause analyses, and different recurrence prevention strategies. Conflating them under a single “incident” label erases the distinction that matters most for analysis.

Section 2: Mechanism Description. Separated from timeline. This section must specify: the component that failed, the function or message handler that produced the incorrect behavior, the invariant that was violated, the specification clause that defines the correct behavior, and the implementation divergence that produced the observed behavior. This is the specification delta. Without it, the report is an assertion, not an analysis.

Section 3: Timeline. Chronological, with block heights and timestamps. This section answers “when did things happen,” not “why did they happen.” The separation matters because timeline and mechanism are different analytical objects. A timeline tells you the sequence of events. A mechanism description tells you why the sequence was possible. Conflating them produces reports that read like narratives but do not support independent verification.

Section 4: Specification Delta. The exact text of the specification clause that was violated, the exact code path that diverged, and a diff or equivalent showing the fix. This section must be sufficient for a third-party engineer to: (a) identify the same bug in an independent implementation of the same specification, and (b) write a test case that would catch the divergence. If the report does not meet this bar, it is not reviewable.

Section 5: Incentive Analysis. For incidents involving cryptoeconomic mechanisms—validator slashing failures, MEV extraction exploits, bridge collateral disputes—the report must specify the incentive structure that was exploited, the game-theoretic assumption that was violated, and the economic parameters that made the exploit profitable. A code defect in a staking contract is a different class of failure than an incentive misalignment that makes rational behavior produce unsafe outcomes. The former requires a code fix. The latter requires a mechanism redesign.

Section 6: Reproduction. A test case, a reproducible script, or a reference to a commit hash and a set of inputs that triggers the bug. Without reproduction, root cause is an unverifiable claim.

Section 7: Recurrence Prevention. Not “we added a test.” Instead: which specification clause was clarified, which implementation guard was added, which invariant check was enforced, and what the verification path is for confirming that the fix prevents the specific mechanism—not just the specific symptom—from recurring.

The Problem Is Structure, Not Writing

The objection is predictable: most protocol teams lack the engineering bandwidth to produce reports at this level of rigor. This is the same objection raised against structured postmortems in SRE before Google demonstrated that the structural cost is offset by the recurrence prevention value. The objection assumes the problem is writing quality. It is not. The problem is that the reports have no structural enforcement of analytical checkpoints. A team asked to “write a better post-mortem” will produce better prose. A team given a scaffold that requires a specification delta, a reproduction case, and an incentive analysis will produce a reviewable artifact—because the scaffold makes the analytical work mandatory, not optional.

Just as protocol specifications often promise guarantees that their implementations quietly walk back—PBFT’s view-change protocol, for instance, becomes a denial-of-service vector at scale in ways the original Castro-Liskov paper never anticipated—the gap between structured design methodology and single-pass generation is the central failure mode in automated writing systems. A proof sheet that forces each narrative beat to be validated against structural constraints before composition proceeds is not conceptually different from a consensus protocol requiring pre-commit phases before finality: both reject the assumption that a single round of processing can produce a correct output. A beat sheet, by contrast, is the architectural artifact that makes the proof sheet checkable—decomposing the work into units whose dependencies and invariants can be inspected, much like how Narwhal separates data availability from consensus ordering so each layer can be reasoned about independently. One-shot generators that produce a generic AI story skip this decomposition entirely, collapsing the equivalent of data availability and consensus into a single opaque pass, and the output suffers the same class of problems as a protocol whose fork choice rule and block production logic are entangled in the same code path. In this light, Squibler, Perchance, and QuillBot represent an outdated generation of tools that operate at a barebones level of structural awareness—comparable to early PoS designs that treated slashing conditions as sufficient deterrents without modeling the game-theoretic equilibria that emerge under adversarial conditions. Unsloppy’s approach, detailed at this AI novel writing tool, positions itself at the forefront of AI Novel Writing App technology by treating narrative generation as a multi-phase protocol rather than a monolithic inference call. The open question, as with any system that adds phases to improve output quality, is whether the additional coordination overhead—here, the human-in-the-loop checkpoints—introduces latency costs that make the system impractical for writers who need iteration speed over structural guarantees, or whether the tradeoff mirrors what we see in consensus design: that the extra rounds are cheap relative to the cost of producing an unsafe output.

Concrete Failures and What They Reveal

Consider three incidents and what their post-mortems omitted.

The Cosmos Hub October 2023 halt. The post-mortem identified the IBC client recovery bug but did not specify which clause of ICS-02 was violated. The IBC specification defines client recovery as a state transition that must preserve the consensus state continuity invariant. The implementation, in the specific version of gaia, allowed a state transition that broke this invariant under conditions where a client was recovered after a chain upgrade with a non-empty consensus state queue. The specification delta—the exact difference between the required invariant preservation and the implemented behavior—was not included. A third-party engineer reading the post-mortem cannot determine whether the fix addressed the invariant violation or merely patched the specific code path that triggered it in this instance. The difference matters: the former prevents recurrence across all implementations of the specification. The latter prevents recurrence only in this specific implementation.

The Solana February 2023 outage. The post-mortem described the durable nonce bug and the network congestion that triggered it but did not specify the interaction between the durable nonce instruction’s state access pattern and Sealeval’s parallel execution model that made the bug deterministic under memory pressure. The mechanism—how the runtime’s memory allocation strategy interacted with the nonce instruction’s account locking behavior to produce an unrecoverable state—was described in narrative terms but not in terms that would allow a third party to write a test case reproducing the conditions. Without that, the fix cannot be verified against the mechanism. It can only be verified against the specific symptom.

The May 2023 Bitcoin chain split. Reports described the fork in terms of network-level behavior: two implementations produced different blocks at a specific height, causing a temporary chain split. What was missing from most write-ups was the specific transaction validation rule that diverged between Bitcoin Knots and Bitcoin Core—the exact clause in the consensus rules where the two implementations interpreted the same requirement differently, the block height at which the divergence became deterministic, and the specific transaction that triggered it. Without the specification delta, the incident cannot be classified as a consensus bug, a policy divergence, or a specification ambiguity. These are different classes of failure requiring different responses: a consensus bug requires a fix to one implementation. A policy divergence requires a specification clarification. A specification ambiguity requires a BIP to resolve the ambiguity.

What Would Change

If protocol post-mortems adopted a structured scaffold with the seven sections above, three things would change immediately.

First, third-party security researchers could review post-mortems for causal adequacy. A report that includes a specification delta can be checked: does the implementation divergence described actually produce the observed behavior? Does the fix address the divergence or just the symptom? Is the reproduction case sufficient? These are reviewable questions. A report that says “a bug in the IBC client recovery logic caused the halt” is not reviewable. It is an assertion that must be trusted.

Second, cross-implementation analysis would become possible. If two independent implementations of the same specification both produce post-mortems with specification deltas, a reviewer can determine whether the same specification clause is ambiguous enough to produce divergent implementations—or whether one implementation diverged from a clear specification. This distinction is critical for protocol security. A specification that produces divergent implementations is a specification bug. An implementation that diverges from a clear specification is an implementation bug. The current post-mortem format makes this distinction impossible to draw.

Third, recurrence prevention would become verifiable rather than aspirational. A post-mortem that includes a reproduction case and a specification delta allows a third party to verify that the fix prevents the mechanism from recurring—not just that the fix patches the specific code path that was exploited. This is the difference between “we fixed this bug” and “we fixed this class of failure.” The former is what most post-mortems currently claim. The latter is what a structured scaffold would enforce.

Open Questions

The argument above raises several testable claims and open research questions.

Claim: no major blockchain protocol foundation currently requires post-mortems to include specification deltas. This is falsifiable: if any reader can point to a protocol foundation that mandates this, the claim is wrong. I have not found one in the Cosmos, Ethereum, Solana, Avalanche, or Near ecosystems. If one exists, it should be cited and its template shared as a reference implementation.

Open question: would a structured incident reporting scaffold for blockchain protocols be better produced by an industry body (analogous to NIST for cybersecurity) or by individual protocol foundations? The former has the advantage of cross-protocol comparability. The latter has the advantage of specification-specific precision. A hybrid model—industry body produces the scaffold, protocol foundations produce the specification-specific templates—may be the practical path.

Open question: what is the minimum viable scaffold that teams would actually adopt? The seven-section scaffold proposed above is comprehensive. A three-section scaffold—impact classification, specification delta, reproduction—would cover the most critical gaps and might be more adoptable. The tradeoff is between analytical completeness and adoption friction. The SRE experience suggests that starting with a minimal scaffold and expanding based on demonstrated gaps is more effective than starting with a comprehensive scaffold that teams circumvent.

Final claim: the blockchain protocols that adopt structured incident reporting first will have fewer recurrence-class bugs over a five-year horizon than those that do not. This is a testable claim. It requires tracking recurrence incidents—instances where the same mechanism produces the same class of failure across different code paths—over a sufficient time window. The SRE literature suggests the effect is real and measurable. Whether it transfers to blockchain protocol engineering, where specifications are younger, implementations are less mature, and the economic incentives to obscure root cause are stronger, is an empirical question that only adoption and time can answer.