SYSTEMSENGINEERING ESSAY · 7 MIN READ

Memory ordering without mysticism

Notebook dates are an editorial chronology, separate from publication dates.

I find memory ordering easier to review when every ordering choice answers a specific question: which operation publishes this data, and which observation permits another thread to use it?

Atomicity answers only one question

Suppose one thread prepares a value and then raises a ready flag. Another thread sees the flag and reads the value. Making the flag atomic prevents a data race on that flag, but it does not automatically turn every surrounding access into a correctly synchronized operation. There are two questions: whether the flag itself is accessed atomically, and whether observing it establishes the required relationship with the prepared value. I would write these questions separately before choosing an ordering or attempting to remove a lock.

An atomic variable also has its own modification order. That does not mean unrelated atomic variables collectively describe one transactional snapshot. Reading a count and a pointer separately can observe a combination the producer never intended as a unit. Stronger ordering may constrain which executions are permitted, but it does not magically combine the reads. If the invariant involves several fields moving together, a mutex, immutable published object or carefully designed version protocol may express the requirement more directly than independently strengthening every operation.

References: [1] Rust: atomic Ordering

Publication needs a matching observation

A release store can publish earlier writes to a thread whose acquire load observes that store's value. The observation is the crucial link. An acquire load is not a general request to refresh all memory, and a release store is not a broadcast that forces every reader to advance. In the illustrative program below, the consumer waits until it observes true from the producer's release. Only then does it read the payload. The payload is itself atomic so the example requires no unsafe code, while its relaxed accesses make the synchronization responsibility visible.

The reasoning forms a chain: the producer writes 42 before publishing ready; the consumer's successful acquire observes that publication; the payload read follows the acquire. With this one producer, one publication and no subsequent payload mutation, the assertion follows. The example is deliberately narrower than a reusable mailbox. Resetting ready and overwriting the payload would require an additional acknowledgement or ownership protocol. A demonstration that safely publishes once should not be copied into a streaming channel without proving what happens during the next cycle.

Runnable standard-library example; one publication only, with no reset or cancellation protocol. rust
use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering};
use std::thread;
fn main() {
    let payload = AtomicUsize::new(0);
    let ready = AtomicBool::new(false);
    thread::scope(|scope| {
        scope.spawn(|| {
            payload.store(42, Ordering::Relaxed);
            ready.store(true, Ordering::Release);
        });
        while !ready.load(Ordering::Acquire) { thread::yield_now(); }
        assert_eq!(payload.load(Ordering::Relaxed), 42);
    });
}

References: [1] Rust: atomic Ordering

Read-modify-write has two halves

An atomic read-modify-write operation both observes a previous value and installs a new one. Acquire constrains the observing side; release constrains publication through the storing side. A successful compare-and-exchange may need both. A failed compare-and-exchange does not perform the proposed store, so its failure ordering cannot publish anything. That distinction is easy to hide inside a helper named update. I would document separately what successful replacement establishes and what the returned failure value allows the caller to inspect on the next iteration.

For example, a pointer replacement algorithm may initialize a new node before installing it, while also following links reachable from the old pointer. Publishing the initialized node and safely observing the old node are different obligations. Even selecting acquire-release correctly does not establish that the old node remains allocated. A reclamation scheme is still required. This is why reviewing the ordering argument in isolation is insufficient: the memory model can justify visibility while the pointer refers to an object whose lifetime has already ended.

Relaxed can be the exact contract

A telemetry counter often needs atomic increments without using the count to authorize another memory access. Relaxed ordering can fit that narrow contract. If several workers increment a completed-work counter, an observer can obtain an atomic count without thereby acquiring the workers' result buffers. That last clause matters. A dashboard approximate count and a readiness barrier are different abstractions, even if both use the same integer representation. Converting the former into the latter through a new caller silently changes the required proof.

I would keep such counters near documentation saying what they cannot certify. Imagine a consumer checks completed equals ten and immediately traverses ten ordinary result objects. The arithmetic may look plausible, but the synchronization needs a concrete publication protocol. A join operation, channel receive or appropriate lock may already provide it. Using that existing mechanism is often clearer than upgrading a metric into a custom barrier. Relaxed should communicate the absence of a cross-object dependency, rather than serve as an unexplained performance preference.

Sequential consistency is a useful starting point

The counterargument to fine-grained ordering is persuasive: sequential consistency is easier to reason about in many small algorithms. Its extra ordering guarantee can simplify the candidate executions a reviewer must consider. I would begin with the clearest correct algorithm, then weaken operations only when a measured requirement justifies the additional reasoning burden. The result should include a written happens-before argument for each publication path. Replacing every sequentially consistent operation with acquire-release because it sounds faster is not such an argument.

Conversely, sequential consistency does not repair every concurrent design. It does not prevent use after free, combine multiple atomic fields into a transaction, guarantee fairness or make a retry loop terminate for a particular thread. Those properties require separate mechanisms. Treating the strongest ordering as a universal safety switch creates a different kind of mysticism. The useful question is which executions the algorithm must exclude and which rule excludes them. If the answer instead depends on ownership, bounds or reclamation, ordering is the wrong place to look.

References: [2] The Rustonomicon: Atomics

Hardware observations cannot replace the language contract

A stress test that succeeds on one processor provides evidence about those executions, not proof that the language permits no failing execution. Compiler transformations participate in the model, and other architectures may enforce different relationships between instructions. I would therefore resist arguments based solely on a familiar assembly listing. Assembly inspection is valuable for explaining a performance difference after correctness is established. It is much weaker as an argument that a missing synchronization edge can never matter in a future build or deployment.

Tests still have a substantial role. Exercise concurrent initialization, delayed readers, shutdown during publication and reuse of objects. A model checker or concurrency testing library can explore selected interleavings that ordinary stress rarely reaches. Keep the model small enough to understand and make its assumptions explicit, especially around allocation and scheduling. Passing those tests complements the proof rather than replacing it. A useful review artifact is a minimal execution that would become illegal under the chosen ordering, together with the invariant it would otherwise violate.

References: [2] The Rustonomicon: Atomics

Review the protocol around the instruction

My review starts with ownership before publication, the exact atomic observation that transfers permission, and ownership after consumption. Then I ask whether the object can be reused, destroyed or replaced while a reader retains access. Finally I separate safety from progress: a consumer can wait forever if its producer never publishes, even when every memory access is correct. The runnable example intentionally allows that dependency. A production interface needs a failure and shutdown policy suited to the application rather than an increasingly elaborate spin loop.

The desired outcome is a short explanation that survives changes in compiler and machine. This producer initializes these bytes; this release publishes them; this acquire observes that publication; this lifetime rule keeps them valid. If the explanation expands into assumptions about accidental timing, undocumented cache behaviour or unrelated counters, I would simplify the design. Atomics are valuable when they expose the synchronization the program actually needs. They become hazardous when their compact syntax persuades us that the surrounding protocol is equally compact.

For reusable publication, I would also identify which generation a reader observed. A boolean can return to the same value while representing different payloads, so value equality alone may conceal a lifecycle transition. An acknowledgement, sequence number or ownership transfer must connect reuse to the consumers that still depend on the previous generation.

Sources and further reading

  1. Rust: atomic Ordering

    Defines the ordering guarantees used below. The publication example and review method are original illustrations.

  2. The Rustonomicon: Atomics

    Explains the relationship between compiler transformations, hardware and atomic synchronization. The failure scenarios are this essay's analysis.

FROM THE NOTEBOOK.

Back to all notes