Multi-region is a failure model — Rhys

Putting servers in several regions does not define what the service can survive. Regional resilience depends on which failures are independent, where authority lives and what capacity remains after a loss. I would start with explicit failure scenarios and derive the topology, rather than treating a multi-region diagram as an availability guarantee.

Name the failure and the promise

A region can become unreachable to some users while remaining reachable to others. Its compute can be healthy while a shared identity service or global deployment makes the application unusable. A database can remain available for reads while losing the ability to accept safe writes. I would describe these as separate scenarios with separate expected outcomes. The promise might be continued writes after any one regional loss, bounded data loss during manual promotion or read-only degradation during a partition. Each promise implies different placement, replication and operating procedures.

AWS's fault-isolation documentation describes regions as isolation boundaries in its infrastructure model. That is a useful starting point, not proof that an application's dependencies are independent. A shared account policy, encryption-key service, DNS change or bad release can correlate failures across otherwise separated infrastructure. I would draw the dependency graph as well as the deployment map and identify components used by every region. The architecture is only as independent as the paths required to serve and recover the promised operations.

References: [1] AWS Fault Isolation Boundaries: Regions

Place voting authority according to the failure domain

For an illustrative consensus system with five voting replicas and majority commitment, three votes are required. Placing them across three regions as two, two and one leaves at least three voters after any single regional loss. Placing three in one region and two in another cannot preserve a majority after losing the first region. This is basic placement arithmetic, not a complete availability proof: the remaining voters must still communicate, storage must remain healthy and the protocol must elect or retain suitable authority.

Read-only replicas should not be counted as voting authority unless the actual system gives them that role. Spanner's replication documentation explicitly distinguishes read-write, read-only and witness replicas, and explains that read-only replicas do not participate in voting. The general lesson is to use the product's real topology rather than count every box on a diagram equally. The calculator below checks only regional majority survival for a fixed illustrative membership. It does not model witness constraints, correlated failures, replication lag or the product-specific rules needed to serve a request.

Illustrative fixed-majority placement arithmetic, not a model of any vendor's complete regional configuration. python

def surviving_votes(placement):
    total = sum(placement.values())
    majority = total // 2 + 1
    return majority, {region: total - count
                      for region, count in placement.items()}

majority, survivors = surviving_votes({'A': 2, 'B': 2, 'C': 1})
assert majority == 3
assert all(count >= majority for count in survivors.values())
majority, survivors = surviving_votes({'A': 3, 'B': 2})
assert survivors['A'] < majority
print(survivors, 'majority required:', majority)

References: [2] Google Cloud Spanner replication

Synchronous replication spends distance on the write path

A write that requires acknowledgements from another region must wait for communication across that distance as part of its protocol. Faster application servers cannot remove the network's contribution. The precise latency depends on leader placement, quorum choice, batching and the database's implementation. Spanner documents synchronous replication and the roles of replicas in reads and writes. I would use the actual product's topology and measured inter-region behaviour when setting latency expectations, rather than applying a generic local-database benchmark to a geographically distributed deployment.

Different operations can have different locality needs. A stale read may be served near the user under a documented staleness bound, while a strong read may need additional coordination. A write-heavy workflow may benefit from locating its application near the relevant authority even if some users are distant. The product decision is whether the consistency and failure-survival guarantees justify the extra latency for each operation. I would avoid promising every user local latency and globally current state without identifying the mechanism and assumptions that make the particular access path possible.

References: [2] Google Cloud Spanner replication

Asynchronous failover needs a data-loss and authority policy

Asynchronous replication can keep the ordinary write path local, but a remote replica may not contain the newest acknowledged writes when the primary region is lost. The recovery-point contract must account for that lag. Promotion also needs an authority decision: the old primary may still be alive behind a partition. If both sides accept incompatible writes without a reconciliation model, restoring connectivity creates a split-brain problem. I would define how the old authority is fenced, how promotion is authorised and what evidence the new region requires before serving writes.

Some applications can merge independent regional writes because their data model explicitly supports conflicts or partitions ownership. Others have invariants, such as a unique scarce allocation, that require stronger coordination. Neither approach is universally superior. The counterargument to synchronous coordination is its latency and reduced availability during some partitions; the counterargument to independent writes is the complexity and business meaning of conflicts. A multi-region design should choose this tradeoff per invariant rather than describe eventual consistency as a universal recovery plan or assume that replication automatically supplies conflict resolution.

References: [2] Google Cloud Spanner replication

Failover capacity must exist before the failure

Suppose three equal regions each normally run at sixty percent of their usable capacity and traffic can be redistributed evenly. Total demand is 1.8 regions of capacity. Losing one region leaves two, so each would run at ninety percent under the simplified model. With only two equal regions at sixty percent, losing one leaves demand equal to 120 percent of the survivor's capacity. Those calculations ignore cache coldness, retry amplification and differences in workload mix, so they are optimistic planning examples rather than predictions of stable operation.

Autoscaling may help, but it relies on available quotas, control planes, images and regional capacity during the incident. I would distinguish capacity already running from capacity expected to appear after a request. Failover tests should include cold caches, connection establishment and the surge created by retried requests. A routing system can move traffic faster than the destination can absorb it. Gradual admission, load shedding and prioritisation may therefore be part of the regional failure contract, especially when the service cannot afford enough idle headroom to preserve every operation at its normal quality level.

References: [1] AWS Fault Isolation Boundaries: Regions

Traffic movement is not the whole cutover

DNS or load-balancer changes do not instantly move every existing connection. Clients cache answers, keep sockets open and apply their own retry policies. Background workers and scheduled jobs may use different endpoints from interactive traffic. I would list every path that creates work and define how it discovers current authority. The failover procedure also needs to handle requests whose outcomes were ambiguous at the moment of movement. Stable operation identifiers and reconciliation can prevent a new region from treating every retried request as entirely new business intent.

Failback deserves equal attention. After the failed region returns, it may contain stale state, old configuration and queued work from before the incident. Sending traffic back immediately can reintroduce the original inconsistency or duplicate effects. The region should rejoin through the system's supported synchronisation and authority protocol, with validation before exposure. A useful counterargument is that a simpler single-region system with strong backups may be easier to operate correctly. That can be true when the required recovery objective allows it. Regional complexity is justified by a concrete promise, not by the appearance of the diagram.

References: [1] AWS Fault Isolation Boundaries: Regions [2] Google Cloud Spanner replication

Exercise the model at the boundaries

A meaningful exercise removes a region's relevant dependencies and checks the promised behaviour, including database authority, traffic routing and capacity. Another partitions regions without killing their processes, because partial reachability exposes different risks from clean shutdown. Test a shared bad configuration and unavailable control-plane credentials to reveal correlations the map hides. Record which operations continue, which degrade and which stop safely, then compare that evidence with the stated contract. A successful health check in the surviving region is not enough to prove useful service.

I would leave the exercise with an updated failure model: assumptions confirmed, dependencies discovered, measured recovery times and capacity limits. The goal is not to claim immunity from every possible regional event. It is to make a specific set of failures predictable and survivable within a defined consistency, latency and data-loss budget. Multi-region architecture becomes valuable when those budgets drive placement and procedures. Without them, geographical duplication can add cost and coordination paths while leaving the application's most important shared failure untouched.

References: [1] AWS Fault Isolation Boundaries: Regions [2] Google Cloud Spanner replication

Sources and further reading

AWS Fault Isolation Boundaries: Regions
Primary infrastructure guidance describes regional isolation boundaries. Application dependencies, capacity arithmetic and failure exercises are independent illustrative analysis.
Google Cloud Spanner replication
Primary documentation explains synchronous replication and replica roles. The five-voter placement example is explicitly generic and is not a claim about a particular Spanner configuration.

Name the failure and the promise

Place voting authority according to the failure domain

Synchronous replication spends distance on the write path

Asynchronous failover needs a data-loss and authority policy

Failover capacity must exist before the failure

Traffic movement is not the whole cutover

Exercise the model at the boundaries

Sources and further reading

RELATED NOTES

Disaster recovery is a replay budget

Raft is a replicated state machine contract

WebSocket reconnection is state reconciliation