https://rsw.lol/notes/ Published notes on systems engineering, performance, infrastructure and AI by Rhys. en-GB https://rsw.lol/notes/latency-budgets-need-critical-paths/https://rsw.lol/notes/latency-budgets-need-critical-paths/Budget user-visible latency through dependency graphs, queue boundaries and deadlines rather than adding every span or percentile.PERFORMANCEFri, 25 Sep 2026 00:00:00 GMT https://rsw.lol/notes/websocket-reconnection-is-state-reconciliation/https://rsw.lol/notes/websocket-reconnection-is-state-reconciliation/Reconnect a live application with explicit cursors, consistent snapshots, bounded replay and application acknowledgements rather than a socket retry alone.DISTRIBUTED SYSTEMSFri, 25 Sep 2026 00:00:00 GMT https://rsw.lol/notes/context-compression-can-delete-the-proof/https://rsw.lol/notes/context-compression-can-delete-the-proof/Token reduction is useful only when compressed context preserves the evidence, exceptions and uncertainty needed to justify the answer.AIFri, 25 Sep 2026 00:00:00 GMT https://rsw.lol/notes/false-sharing-is-a-layout-bug/https://rsw.lol/notes/false-sharing-is-a-layout-bug/Distinguish logical independence from coherence traffic, then change layout only when ownership and measurements support it.PERFORMANCEFri, 25 Sep 2026 00:00:00 GMT https://rsw.lol/notes/backpressure-for-streaming-ai/https://rsw.lol/notes/backpressure-for-streaming-ai/Bound token and audio streams with explicit queue budgets, cancellation, fair scheduling and honest completion semantics.AIFri, 25 Sep 2026 00:00:00 GMT https://rsw.lol/notes/fuzzing-the-parser-boundary/https://rsw.lol/notes/fuzzing-the-parser-boundary/Build a parser fuzz target around bounded resources, explicit rejection and semantic oracles, then preserve useful counterexamples.SYSTEMSFri, 25 Sep 2026 00:00:00 GMT https://rsw.lol/notes/disaster-recovery-is-a-replay-budget/https://rsw.lol/notes/disaster-recovery-is-a-replay-budget/Deriving recovery objectives from durable copies, restore throughput, replay capacity, dependency recovery and verified cutover.INFRASTRUCTUREFri, 25 Sep 2026 00:00:00 GMT https://rsw.lol/notes/cache-invalidation-is-a-versioning-problem/https://rsw.lol/notes/cache-invalidation-is-a-versioning-problem/Making cache freshness explicit with complete keys, source versions, guarded fills and recovery rules for missed invalidations.DISTRIBUTED SYSTEMSFri, 25 Sep 2026 00:00:00 GMT https://rsw.lol/notes/floating-point-is-an-error-budget/https://rsw.lol/notes/floating-point-is-an-error-budget/Choose numeric representations, tolerances and stable algorithms from the meaning of the result rather than a blanket rule about decimals.SYSTEMSFri, 25 Sep 2026 00:00:00 GMT https://rsw.lol/notes/arena-allocation-and-the-cost-of-reset/https://rsw.lol/notes/arena-allocation-and-the-cost-of-reset/Arena allocation simplifies allocation only when object lifetimes, cleanup and retention agree with the reset boundary.SYSTEMSFri, 25 Sep 2026 00:00:00 GMT https://rsw.lol/notes/io-uring-and-buffer-ownership/https://rsw.lol/notes/io-uring-and-buffer-ownership/Model submitted buffers, terminal completions, cancellation and zero-copy notifications as an explicit ownership protocol.SYSTEMSFri, 25 Sep 2026 00:00:00 GMT https://rsw.lol/notes/tool-permissions-need-capabilities/https://rsw.lol/notes/tool-permissions-need-capabilities/Agent tools need narrow, enforceable authority over operations and resources, with explicit limits for delegation, expiry and commitment.AIFri, 25 Sep 2026 00:00:00 GMT https://rsw.lol/notes/zero-copy-is-a-lifetime-decision/https://rsw.lol/notes/zero-copy-is-a-lifetime-decision/Avoiding a copy changes who owns memory, how long buffers stay alive and which boundaries can safely share them.PERFORMANCEFri, 25 Sep 2026 00:00:00 GMT https://rsw.lol/notes/unknown-is-a-valid-output/https://rsw.lol/notes/unknown-is-a-valid-output/Abstention as a decision with a cost, and a Rust contract that keeps unsupported conclusions separate from negative answers.SYSTEMSFri, 25 Sep 2026 00:00:00 GMT https://rsw.lol/notes/embeddings-are-a-versioned-interface/https://rsw.lol/notes/embeddings-are-a-versioned-interface/An embedding migration changes the meaning of distance. Treat the encoder, corpus and index as one versioned retrieval contract.AIFri, 25 Sep 2026 00:00:00 GMT https://rsw.lol/notes/evaluating-ai-without-grading-the-vibes/https://rsw.lol/notes/evaluating-ai-without-grading-the-vibes/Build evaluations around decisions, paired comparisons, error costs and reproducible conditions rather than polished answers.AIFri, 25 Sep 2026 00:00:00 GMT https://rsw.lol/notes/speculative-decoding-spends-parallelism/https://rsw.lol/notes/speculative-decoding-spends-parallelism/The exact sampling correction behind speculative decoding, with a worked acceptance example and realistic cost boundaries.AIFri, 25 Sep 2026 00:00:00 GMT https://rsw.lol/notes/lock-free-is-not-wait-free/https://rsw.lol/notes/lock-free-is-not-wait-free/Separate system-wide progress, individual completion and elapsed deadlines when evaluating concurrent algorithms.SYSTEMSFri, 25 Sep 2026 00:00:00 GMT https://rsw.lol/notes/indexes-are-a-write-path-decision/https://rsw.lol/notes/indexes-are-a-write-path-decision/Evaluating indexes as maintained data structures with consequences for updates, storage, concurrency and the full query workload.PERFORMANCEFri, 25 Sep 2026 00:00:00 GMT https://rsw.lol/notes/bounded-queues-are-a-product-decision/https://rsw.lol/notes/bounded-queues-are-a-product-decision/Queue capacity determines which promises a service makes under load. Backpressure, rejection and deadlines have different user-visible meanings.SYSTEMSFri, 25 Sep 2026 00:00:00 GMT https://rsw.lol/notes/synthetic-data-needs-a-failure-model/https://rsw.lol/notes/synthetic-data-needs-a-failure-model/Generated examples can expand coverage or erase it. Analyse selection bias, recursive replacement and verification before treating volume as new information.AIFri, 25 Sep 2026 00:00:00 GMT https://rsw.lol/notes/property-tests-need-a-model/https://rsw.lol/notes/property-tests-need-a-model/Generated inputs become valuable when an independent model, meaningful invariants and intentional generators define what should happen.SYSTEMSFri, 25 Sep 2026 00:00:00 GMT https://rsw.lol/notes/event-time-is-not-wall-clock-time/https://rsw.lol/notes/event-time-is-not-wall-clock-time/An analysis of the Dataflow model through late events, watermarks, revisions and the application decisions that a streaming engine cannot make.DISTRIBUTED SYSTEMSFri, 25 Sep 2026 00:00:00 GMT https://rsw.lol/notes/small-models-and-the-cost-of-a-decision/https://rsw.lol/notes/small-models-and-the-cost-of-a-decision/Reason about tiny learned runtimes through total decision latency, feature costs, precision, fallback behavior and honest benchmarks.PERFORMANCEFri, 25 Sep 2026 00:00:00 GMT https://rsw.lol/notes/the-tail-is-the-system/https://rsw.lol/notes/the-tail-is-the-system/Reading Dean and Barroso through fan-out, correlated slowdowns and the resource budget that makes hedging succeed or fail.PERFORMANCEFri, 25 Sep 2026 00:00:00 GMT https://rsw.lol/notes/cross-chain-funding-tracer/https://rsw.lol/notes/cross-chain-funding-tracer/A rigorous model for cross-chain traces that separates observations, bridge joins, conservation checks and uncertain attribution.INFRASTRUCTUREFri, 25 Sep 2026 00:00:00 GMT https://rsw.lol/notes/prompt-injection-is-a-data-boundary/https://rsw.lol/notes/prompt-injection-is-a-data-boundary/Defensive design for keeping retrieved text, tool output and model suggestions from acquiring application authority.AIFri, 25 Sep 2026 00:00:00 GMT https://rsw.lol/notes/rust-async-without-a-runtime-myth/https://rsw.lol/notes/rust-async-without-a-runtime-myth/Futures define a polling contract. Executors, wakeups, I/O drivers and scheduling policy are separate responsibilities that still need an owner.SYSTEMSFri, 25 Sep 2026 00:00:00 GMT https://rsw.lol/notes/ai-evaluation-needs-a-dataset-lifecycle/https://rsw.lol/notes/ai-evaluation-needs-a-dataset-lifecycle/A benchmark is a versioned measurement instrument. Track provenance, split related cases together and make corrections without silently rewriting the result.AIFri, 25 Sep 2026 00:00:00 GMT https://rsw.lol/notes/cpu-time-is-not-wall-time/https://rsw.lol/notes/cpu-time-is-not-wall-time/Separate execution, waiting and parallel work before turning elapsed time or CPU utilization into a performance diagnosis.PERFORMANCEFri, 25 Sep 2026 00:00:00 GMT https://rsw.lol/notes/chunking-is-an-information-boundary/https://rsw.lol/notes/chunking-is-an-information-boundary/A retrieval chunk decides which qualifiers, relationships and source identities remain available when a passage is read alone.AIFri, 25 Sep 2026 00:00:00 GMT https://rsw.lol/notes/voice-ai-should-not-wait/https://rsw.lol/notes/voice-ai-should-not-wait/A design for speculative voice responses that separates listening, preparation, commitment and cancellation.AIFri, 25 Sep 2026 00:00:00 GMT https://rsw.lol/notes/cache-locality-before-cleverness/https://rsw.lol/notes/cache-locality-before-cleverness/The representation that exposes useful bytes can matter more than a clever loop. Start with access patterns, working sets and ownership between cores.PERFORMANCEFri, 25 Sep 2026 00:00:00 GMT https://rsw.lol/notes/speech-interruption-is-a-state-machine/https://rsw.lol/notes/speech-interruption-is-a-state-machine/Stopping a voice assistant requires more than cancelling generation. Invalidate stale audio, control playback and reconcile what the user actually heard.AIFri, 25 Sep 2026 00:00:00 GMT https://rsw.lol/notes/when-go-is-the-right-rewrite/https://rsw.lol/notes/when-go-is-the-right-rewrite/A rewrite should remove a measured constraint while preserving the service's real contract. Go's runtime is a tradeoff to evaluate, not a verdict.SYSTEMSFri, 25 Sep 2026 00:00:00 GMT https://rsw.lol/notes/the-dual-write-problem/https://rsw.lol/notes/the-dual-write-problem/An outbox makes a database change and publication intent atomic; delivery, ordering, deduplication and consumer effects still need their own contracts.DISTRIBUTED SYSTEMSFri, 25 Sep 2026 00:00:00 GMT https://rsw.lol/notes/multi-region-is-a-failure-model/https://rsw.lol/notes/multi-region-is-a-failure-model/Designing regional resilience through quorum placement, replication lag, failover authority, capacity headroom and shared dependencies.DISTRIBUTED SYSTEMSFri, 25 Sep 2026 00:00:00 GMT https://rsw.lol/notes/profile-the-production-shape/https://rsw.lol/notes/profile-the-production-shape/A profile estimates where a particular workload spent resources. Make its population, sampling and attribution boundaries explicit.PERFORMANCEFri, 25 Sep 2026 00:00:00 GMT https://rsw.lol/notes/lora-is-a-capacity-choice/https://rsw.lol/notes/lora-is-a-capacity-choice/A reading of LoRA through rank, parameter counts, deployment identity and the limitations of low-rank adaptation.AIFri, 25 Sep 2026 00:00:00 GMT https://rsw.lol/notes/attention-is-a-memory-interface/https://rsw.lol/notes/attention-is-a-memory-interface/Reading Attention Is All You Need through addressing, data movement, masking and the limits of weighted retrieval.AIFri, 25 Sep 2026 00:00:00 GMT https://rsw.lol/notes/tracing-needs-context-not-just-spans/https://rsw.lol/notes/tracing-needs-context-not-just-spans/Building useful traces through causal propagation, asynchronous boundaries, sampling and controlled diagnostic context.INFRASTRUCTUREFri, 25 Sep 2026 00:00:00 GMT https://rsw.lol/notes/model-routing-needs-a-fallback-contract/https://rsw.lol/notes/model-routing-needs-a-fallback-contract/Routing between models needs explicit quality, latency, privacy and action semantics when the preferred path fails or escalates.AIFri, 25 Sep 2026 00:00:00 GMT https://rsw.lol/notes/raft-is-a-replicated-state-machine-contract/https://rsw.lol/notes/raft-is-a-replicated-state-machine-contract/Reading Raft through its application boundary: commitment, deterministic execution, retries, membership changes and trustworthy reads.DISTRIBUTED SYSTEMSFri, 25 Sep 2026 00:00:00 GMT https://rsw.lol/notes/cancellation-is-a-protocol/https://rsw.lol/notes/cancellation-is-a-protocol/Stopping local waiting does not undo remote work. A reliable cancellation design needs ownership, acknowledgements and a clear commit boundary.SYSTEMSFri, 25 Sep 2026 00:00:00 GMT https://rsw.lol/notes/leases-need-fencing-tokens/https://rsw.lol/notes/leases-need-fencing-tokens/Why an expired lease cannot stop an old worker, and how resource-enforced fencing turns ownership changes into safe writes.DISTRIBUTED SYSTEMSFri, 25 Sep 2026 00:00:00 GMT https://rsw.lol/notes/retrieval-needs-an-abstention-policy/https://rsw.lol/notes/retrieval-needs-an-abstention-policy/Why retrieved text is not enough: evidence contracts, claim-level support and risk-versus-coverage decisions for RAG.AIFri, 25 Sep 2026 00:00:00 GMT https://rsw.lol/notes/simd-begins-with-a-data-contract/https://rsw.lol/notes/simd-begins-with-a-data-contract/Vectorization depends on aliasing, numeric semantics, tails and deployment capabilities before instruction width becomes useful.PERFORMANCEFri, 25 Sep 2026 00:00:00 GMT https://rsw.lol/notes/isolation-levels-are-observable-behaviour/https://rsw.lol/notes/isolation-levels-are-observable-behaviour/Understanding transaction isolation through concurrent histories, write skew, retries and the invariants users can observe.DISTRIBUTED SYSTEMSFri, 25 Sep 2026 00:00:00 GMT https://rsw.lol/notes/webhooks-are-a-distributed-systems-problem/https://rsw.lol/notes/webhooks-are-a-distributed-systems-problem/Treat webhook receipt, processing, ordering and reconciliation as separate contracts instead of assuming an HTTP callback is a reliable queue.DISTRIBUTED SYSTEMSFri, 25 Sep 2026 00:00:00 GMT https://rsw.lol/notes/prefix-caching-needs-an-identity/https://rsw.lol/notes/prefix-caching-needs-an-identity/A prefix cache is correct only when its key identifies the computation, its authorization scope and the inputs that produced the cached state.PERFORMANCEFri, 25 Sep 2026 00:00:00 GMT https://rsw.lol/notes/configuration-rollouts-are-distributed-writes/https://rsw.lol/notes/configuration-rollouts-are-distributed-writes/Treating configuration activation as versioned state replication with validation, compatibility, canaries and recovery from missed updates.DISTRIBUTED SYSTEMSFri, 25 Sep 2026 00:00:00 GMT https://rsw.lol/notes/state-machines-need-invalid-transitions/https://rsw.lol/notes/state-machines-need-invalid-transitions/A useful state machine specifies rejected events, guards, atomic transitions and recovery as carefully as its happy path.SYSTEMSFri, 25 Sep 2026 00:00:00 GMT https://rsw.lol/notes/ordering-events-without-a-global-clock/https://rsw.lol/notes/ordering-events-without-a-global-clock/Lamport clocks preserve causal order without measuring elapsed time; this analysis works through their guarantees, limits and practical alternatives.DISTRIBUTED SYSTEMSFri, 25 Sep 2026 00:00:00 GMT https://rsw.lol/notes/reranking-is-a-budget-allocation-problem/https://rsw.lol/notes/reranking-is-a-budget-allocation-problem/How candidate recall, cross-encoder cost and evidence coverage determine where a retrieval system should spend its reranking budget.AIFri, 25 Sep 2026 00:00:00 GMT https://rsw.lol/notes/decoding-is-part-of-the-product/https://rsw.lol/notes/decoding-is-part-of-the-product/Temperature, truncation and stopping rules change the behaviour users receive. Treat decoding as a versioned product policy, with task-specific evaluation.AIFri, 25 Sep 2026 00:00:00 GMT https://rsw.lol/notes/retry-budgets-prevent-amplification/https://rsw.lol/notes/retry-budgets-prevent-amplification/Treating retries as additional load with a finite budget, coordinated deadlines and explicit rules for ambiguous operations.INFRASTRUCTUREFri, 25 Sep 2026 00:00:00 GMT https://rsw.lol/notes/performance-regressions-need-experiments/https://rsw.lol/notes/performance-regressions-need-experiments/Design comparative CI experiments around pairing, practical effect sizes and controlled environments instead of brittle timing thresholds.PERFORMANCEFri, 25 Sep 2026 00:00:00 GMT https://rsw.lol/notes/kv-cache-is-a-systems-problem/https://rsw.lol/notes/kv-cache-is-a-systems-problem/A PagedAttention reading focused on memory accounting, fragmentation, ownership, scheduling and serving trade-offs.PERFORMANCEFri, 25 Sep 2026 00:00:00 GMT https://rsw.lol/notes/agents-need-transaction-boundaries/https://rsw.lol/notes/agents-need-transaction-boundaries/Design AI tool execution around proposals, durable intent, idempotency, stale state and uncertain external outcomes.AIFri, 25 Sep 2026 00:00:00 GMT https://rsw.lol/notes/cardinality-is-an-observability-budget/https://rsw.lol/notes/cardinality-is-an-observability-budget/Budgeting metric series through label combinations, histogram expansion, churn and the questions each dimension must answer.INFRASTRUCTUREFri, 25 Sep 2026 00:00:00 GMT https://rsw.lol/notes/7-4-microseconds/https://rsw.lol/notes/7-4-microseconds/A reported median is the beginning of a performance argument. Measurement boundaries, sampling and workload design determine what it means.PERFORMANCEFri, 25 Sep 2026 00:00:00 GMT https://rsw.lol/notes/rate-limits-need-a-fairness-policy/https://rsw.lol/notes/rate-limits-need-a-fairness-policy/Why a request counter cannot decide fair access, and how identities, cost weights, concurrency and reserved capacity change admission.INFRASTRUCTUREFri, 25 Sep 2026 00:00:00 GMT https://rsw.lol/notes/exactly-once-needs-a-boundary/https://rsw.lol/notes/exactly-once-needs-a-boundary/Exactly-once claims become useful when they specify the effect, transaction boundary, identity, retention window and recovery assumptions.DISTRIBUTED SYSTEMSFri, 25 Sep 2026 00:00:00 GMT https://rsw.lol/notes/structured-output-is-not-semantic-correctness/https://rsw.lol/notes/structured-output-is-not-semantic-correctness/A typed AI response can still be false, stale or unauthorized. Design validation around the decision it will drive.AIFri, 25 Sep 2026 00:00:00 GMT https://rsw.lol/notes/agent-memory-needs-forgetting/https://rsw.lol/notes/agent-memory-needs-forgetting/Durable agent memory needs retention, provenance, correction and deletion semantics that remain valid across summaries and indexes.AIFri, 25 Sep 2026 00:00:00 GMT https://rsw.lol/notes/open-loop-load-tests-tell-a-different-story/https://rsw.lol/notes/open-loop-load-tests-tell-a-different-story/Arrival schedules, coordinated omission and generator limits determine what a load test can honestly say about overload.PERFORMANCEFri, 25 Sep 2026 00:00:00 GMT https://rsw.lol/notes/backfills-that-do-not-fight-production/https://rsw.lol/notes/backfills-that-do-not-fight-production/Design resumable backfills around consistent cuts, conditional writes, adaptive resource budgets and verifiable completion.INFRASTRUCTUREFri, 25 Sep 2026 00:00:00 GMT https://rsw.lol/notes/mixture-of-experts-still-needs-a-scheduler/https://rsw.lol/notes/mixture-of-experts-still-needs-a-scheduler/Sparse expert activation saves arithmetic only within a system that can route tokens, balance capacity and control communication costs.PERFORMANCEFri, 25 Sep 2026 00:00:00 GMT https://rsw.lol/notes/designing-for-uncertainty/https://rsw.lol/notes/designing-for-uncertainty/Freshness, provenance and conflicting observations belong in the data contract, before a consumer decides what to do.SYSTEMSFri, 25 Sep 2026 00:00:00 GMT https://rsw.lol/notes/zero-downtime-migrations-need-expand-contract/https://rsw.lol/notes/zero-downtime-migrations-need-expand-contract/Planning database changes as overlapping application versions, bounded data work and evidence-based removal of old schema.INFRASTRUCTUREFri, 25 Sep 2026 00:00:00 GMT https://rsw.lol/notes/bloom-filters-spend-false-positives/https://rsw.lol/notes/bloom-filters-spend-false-positives/Sizing Bloom filters, pricing false positives and preserving the conditions under which a negative answer is trustworthy.PERFORMANCEFri, 25 Sep 2026 00:00:00 GMT https://rsw.lol/notes/syscalls-are-not-the-whole-io-cost/https://rsw.lol/notes/syscalls-are-not-the-whole-io-cost/Account for copies, cache state, queueing and durability before attributing I/O performance to system-call counts.PERFORMANCEFri, 25 Sep 2026 00:00:00 GMT https://rsw.lol/notes/lsm-trees-move-the-write-cost/https://rsw.lol/notes/lsm-trees-move-the-write-cost/Understanding log-structured storage through compaction debt, read and space amplification, tombstones and sustainable throughput.PERFORMANCEFri, 25 Sep 2026 00:00:00 GMT https://rsw.lol/notes/replay-is-an-architectural-feature/https://rsw.lol/notes/replay-is-an-architectural-feature/A retained event log becomes a recovery tool only when inputs, transformations, side effects, checkpoints and output comparison support replay.INFRASTRUCTUREFri, 25 Sep 2026 00:00:00 GMT https://rsw.lol/notes/pagination-needs-a-consistency-contract/https://rsw.lol/notes/pagination-needs-a-consistency-contract/Choosing between live traversal and stable snapshots, with deterministic keyset ordering and explicit cursor semantics.INFRASTRUCTUREFri, 25 Sep 2026 00:00:00 GMT https://rsw.lol/notes/memory-ordering-without-mysticism/https://rsw.lol/notes/memory-ordering-without-mysticism/Reason about atomic publication through happens-before edges, ownership and concrete failure cases rather than processor folklore.SYSTEMSFri, 25 Sep 2026 00:00:00 GMT https://rsw.lol/notes/schema-evolution-without-flag-days/https://rsw.lol/notes/schema-evolution-without-flag-days/Evolve distributed contracts with explicit compatibility directions, semantic migration, expand-and-contract rollout and archived replay fixtures.DISTRIBUTED SYSTEMSFri, 25 Sep 2026 00:00:00 GMT https://rsw.lol/notes/quantization-moves-the-error/https://rsw.lol/notes/quantization-moves-the-error/A worked quantization and bias-correction example shows why small weight error does not guarantee small decision error.PERFORMANCEFri, 25 Sep 2026 00:00:00 GMT