Architecture That Can Recover

#Opening signal

A system that cannot recover is not robust; it is merely lucky. Every architecture will eventually reach a bad state — a failed deployment, a corrupt migration, a bug in production — and the only question is whether it can return to a known-good state or must be rebuilt from memory and hope. Recoverable architecture is architecture designed, from the start, so that recovery is a routine, rehearsed operation rather than a heroic improvisation. It is the reversibility of Chapter 18 built into the foundations, using the Recover and Track moves and the snapshot-first discipline.

#Mathematical core

FACT. The mathematical model of recoverable architecture is the event log with replay. Instead of storing only the current state, store the ordered sequence of events that produced it (the append-only ledger of Chapter 14). The current state becomes a derived artifact — the result of replaying all events from a known starting point — which means it can always be reconstructed, and any past state can be recovered by replaying up to that point. This is event sourcing, and it is the architectural form of the Fibonacci ledger: keep the derivations, and you can always rebuild the values.^[57] Versioned infrastructure and database migrations apply the same idea — each change is a recorded, reversible step from a known prior state, so the system can move forward and, crucially, backward. The mathematical guarantee is the same as the recurrence's reversibility: given the events and a starting point, every state is reconstructible.

#Odisena translation

METHOD. Build architecture so that state is derived from a preserved, ordered log of changes, and recovery is replay to a known-good point. Concretely: version everything (infrastructure as code, schema migrations, configuration); snapshot before every consequential change (Chapter 18); track changes as an append-only log (Chapter 14); and rehearse rollback (Chapter 18's exercise). The Odisena AVPT CI/CD dashboard is architecture of this kind — a durable ledger, snapshot adapters, and a comparison view that measures recovery time — so that the system's ability to recover is not asserted but measured.^[58] Recoverable architecture is the Recover move (Chapter 6) and the snapshot-first cadence, made structural.

#Boundary note

INTERPRETATION. Full event sourcing is powerful and genuinely expensive — replaying a long event log to reconstruct state is slow, and storing every event forever costs. The interpretation I hold: most systems do not need pure event sourcing; they need enough preserved, ordered change history to recover to recent known-good states, plus periodic snapshots so replay does not start from the beginning of time. The boundary is pragmatic: preserve enough history to recover from the failures you actually face, with snapshots to bound replay cost, rather than dogmatically logging everything forever. Recoverability is the goal; total event sourcing is one (heavy) means.

#Applied CASE

CASE. A production deployment goes wrong. In a forward-only architecture, the team scrambles: what changed? what was the previous config? can we even get back? In a recoverable architecture — the Odisena snapshot-first cadence — the answer is routine: the pre-deployment state was preserved before the canary rollout, the change is tracked with its drift, and rollback is a rehearsed operation that returns the system to the known-good snapshot in measured time.^[58] The governance work adds a further recoverable-architecture lesson: production promotion is kept fail-closed and separate from code merge, disabled until the preservation infrastructure (buckets, roles, ledgers) actually exists — so the architecture refuses to enable a forward step it could not reverse.^[59] Recovery is designed in, not bolted on.

#Failure mode

The failure mode is the state-only architecture: a system that stores its current state but not the ordered history that produced it, so recovery means guessing or rebuilding from scratch. Its signatures are the unanswerable questions of Chapter 14 ("what was the config before the change?") and the heroic, unrehearsed recovery that sometimes works and sometimes does not. A dangerous variant is enabling irreversible forward steps — turning on a capability before the machinery to reverse it exists. The fix is preserved, ordered change history with snapshots to bound replay, rehearsed rollback, and fail-closed gating of steps that cannot yet be reversed.

#Reusable protocol — The Recoverable-Architecture Checklist

1. Version everything. Infrastructure, schema, configuration — as code, in an ordered history.
2. Snapshot before change. Preserve the prior state before every consequential step (Chapter 18).
3. Log changes append-only. Track each change as a recorded, ordered event (Chapter 14).
4. Bound replay with snapshots. Periodic snapshots so recovery need not replay from the beginning.
5. Rehearse rollback. Test recovery deliberately; measure recovery time.
6. Fail closed on the irreversible. Do not enable a forward step whose reversal machinery does not yet exist.

#Validation questions

1. Can your architecture reconstruct a recent past state, or only show the current one?
2. Have you measured your recovery time, or only hoped it is short?
3. Is any irreversible capability enabled before its rollback machinery exists?
4. When did you last rehearse a rollback in production-like conditions?

#The pragmatic middle: snapshots plus recent log

INTERPRETATION. Pure event sourcing — storing every event forever and reconstructing all state by replay — is architecturally beautiful and, for most systems, impractical, and it is worth being concrete about the pragmatic middle that most real recoverable architectures actually occupy. That middle is periodic snapshots plus a recent change log: you take full snapshots of state at intervals, and you keep the ordered log of changes since the last snapshot. To recover to a recent point, you load the nearest prior snapshot and replay only the short log after it — fast, because the replay is bounded, and cheap, because you are not storing every event since the beginning of time. This is exactly the structure of the Fibonacci computation done right: you do not recompute from $F_0$ every time (pure replay), nor do you keep every term forever (unbounded storage); you keep a recent checkpoint (the adjacent pair) and advance from it, and if you need an older state you go back to an older checkpoint. The snapshot bounds the replay; the log provides the fine-grained recent history that snapshots alone would miss.

The design decision, then, is not "event sourcing: yes or no?" but "how frequent should the snapshots be, and how long should the log be retained?" — and both answers follow from the failures you actually face. Snapshot frequency should be tight enough that replay from the nearest snapshot is fast enough for your recovery-time target; log retention should be long enough to cover the window in which you might need fine-grained recovery. Answering these questions requires knowing your recovery-time target and your realistic failure horizon, which is itself a valuable forcing function: a team that can state how frequently it snapshots and how long it retains its change log is a team that has thought concretely about recovery, and a team that cannot is a team that is hoping. The pragmatic middle is not a compromise of the ideal; for most systems it is the ideal, correctly sized to the failures that actually occur.

#Bridge

Recoverable architecture preserves the history of changes made by people and machines. But increasingly, some of those changes are proposed by systems that generate — models that write, suggest, and decide. The next chapter applies the full RFPA/AVPT discipline to AI-assisted work, treating every generated output as a candidate in an evidence loop.