An LSM tree makes foreground writes efficient by arranging substantial work for later. The design is not a way to eliminate write cost; it is a way to schedule and shape that cost. I would judge it by the workload it can sustain after compaction, deletion and recovery have all become active.
Follow a write beyond its acknowledgement
In a typical log-structured storage path, a write enters a mutable in-memory structure and a durability log according to the engine's configured acknowledgement policy. Full memory structures become immutable sorted files, and compaction later merges files while discarding obsolete versions when safe. The exact durability boundary depends on WAL and synchronisation settings; the label LSM alone does not tell a caller whether a successful write survives power loss. RocksDB's compaction documentation describes the later organisation work, which is central to understanding the steady-state cost of this storage family.
The foreground path benefits from batching and sequential organisation, but later work still consumes disk bandwidth, CPU and temporary space. I would trace an acknowledged byte through flush, compaction, replication and backup before estimating capacity. A benchmark that stops shortly after loading an empty database can report excellent throughput while leaving a large amount of unfinished consolidation behind. The system may then slow down after the benchmark ends. Sustainable throughput means the background work keeps pace over a representative interval, rather than merely accepting writes faster than it can eventually organise them.
References: [1] RocksDB compaction
Keep amplification measurements explicit
Write amplification compares physical bytes written with logical bytes accepted, but the exact numerator must be stated. Does it include the WAL, replication traffic, filesystem metadata or only table-file writes? Read amplification can describe extra files consulted or extra bytes read, which are related but different measurements. Space amplification compares physical storage with live logical data and changes during compaction. I would publish the definitions beside any reported factor, because two honest measurements can look contradictory when one includes temporary output files and another counts only the final settled state.
The following ledger is an illustrative interval, not a claim about RocksDB performance. Suppose an application accepts 20 GB of logical changes and the engine writes 20 GB of WAL, 25 GB of flush output and 95 GB of compaction output. Counting those components gives 140 GB of physical writes, or a factor of seven. If a disk sustains only 700 MB per second for this workload, that accounting suggests at most 100 MB per second of logical writes before other bottlenecks and headroom. The calculation is useful precisely because its assumptions are visible.
| Component | GB written | Included in this example |
|---|---|---|
| Application changes | 20 | Denominator |
| WAL | 20 | Numerator |
| Flush output | 25 | Numerator |
| Compaction output | 95 | Numerator |
| Total physical writes | 140 | 7 times logical bytes |
References: [1] RocksDB compaction
Compaction policy chooses where to spend
RocksDB documents several compaction approaches with different tradeoffs. Leveled compaction organises files into levels and controls overlap below the initial level; other approaches can defer more merging at the cost of retaining more overlapping data. I read these choices as workload decisions rather than a contest with one universally superior algorithm. A write-heavy append workload, a latency-sensitive point-lookup service and a scan-heavy analytical workload place different values on background write traffic, lookup work and storage headroom.
File size and level-size ratios also influence the shape of the work. Larger consolidation steps may improve some steady-state ratios while producing longer bursts of resource consumption. More overlapping runs can make writes cheaper initially while requiring reads to inspect more candidates. Bloom filters and caches can reduce some point-lookup costs, but they do not make range scans or compaction free. I would choose candidate settings from the actual access pattern, then measure a complete mixed workload including updates and deletes. Tuning only a single throughput number hides the cost moved onto another operation.
References: [1] RocksDB compaction[2] RocksDB leveled compaction
Compaction debt is a queue with a stability condition
Imagine logical writes create compaction work at an average rate of 300 MB per second while available background capacity processes only 240 MB per second. Under those illustrative conditions, pending work grows by 60 MB per second, or 216 GB per hour in decimal units. A larger queue threshold delays the visible stall but does not change the long-run imbalance. Temporary bursts can be absorbed if later spare capacity repays the debt. A permanently positive arrival-minus-service rate eventually consumes either latency headroom, storage space or a configured safety limit.
This is why I would monitor pending compaction work, level pressure and write stalls alongside foreground latency. A healthy-looking median can coexist with increasing debt that predicts tomorrow's outage. Background parallelism is not a free fix: more compaction threads can compete with foreground reads, saturate storage or exhaust CPU on compression. The useful control is enough sustained service capacity to keep the debt bounded while preserving the application's latency objective. When that is impossible, admission control or reduced ingestion may be more honest than increasing thresholds until the disk fills.
References: [1] RocksDB compaction
Deletes and snapshots keep old state alive
A delete in a log-structured engine commonly creates a tombstone rather than immediately erasing every older copy. Removing that marker too early could allow an older value in another file to reappear. Compaction can discard obsolete versions only when the engine's visibility and overlap rules permit it. Long-lived snapshots can retain versions that current readers no longer need. I would therefore distinguish logical deletion, reclaimable storage and physically reclaimed space in operational dashboards and retention promises. They happen at different times and can respond differently to workload changes.
Consider a hypothetical batch deletion of half a large dataset. The application may immediately observe fewer live records while disk usage briefly rises because tombstones and compaction outputs coexist with old files. Provisioning based only on the expected final live size can make cleanup fail halfway through. The recovery plan needs temporary headroom and a policy for outstanding snapshots or readers. A claim that data is deleted from the serving view should also not be silently expanded into a claim that every backup or replica has already removed the bytes; those are separate lifecycle contracts.
References: [2] RocksDB leveled compaction
Read-heavy workloads may prefer another bargain
The strongest counterargument to an LSM design is that a different storage organisation may fit the workload with fewer background surprises. A B-tree can offer a more direct lookup path and different update locality, while paying its own costs in page management, random writes and fragmentation. I would not choose either structure from a slogan about write optimisation. The relevant comparison includes durability settings, working-set size, compression, skew, scan patterns and the hardware's behaviour. A tiny hot dataset can make both approaches look excellent for reasons unrelated to their large-scale tradeoffs.
There are also workloads where the application's schema dominates the result. Repeatedly updating a large value to change one small field can create substantial rewritten data regardless of the storage engine's name. Separating immutable blobs from mutable metadata may reduce more work than tuning a compaction ratio. Similarly, a badly distributed keyspace can concentrate contention or make scans expensive. My preference is to fix avoidable logical write volume first, then use engine-level tuning to handle the remaining workload rather than asking compaction policy to compensate for every modelling decision.
References: [1] RocksDB compaction
Benchmark the settled system and the recovery path
A meaningful evaluation preloads representative data, runs long enough for levels and caches to reach a useful operating state, and includes the expected mixture of inserts, updates, reads and deletes. Report latency distributions during compaction, not only between compaction bursts. Include disk occupancy and pending work at the start and end so that deferred cost is visible. If the end state contains much more debt, the measured ingest rate is a burst capacity result. It should not be advertised as the sustainable rate without a separate recovery interval.
I would also restart under load, restore from a checkpoint and test a constrained disk. Recovery can expose WAL replay costs, cache coldness and background work that an uninterrupted benchmark hides. The target is a service that maintains a clear durability and latency contract while paying for its stored history. LSM trees are powerful because they reshape work into batches and give the engine room to optimise. Understanding that advantage requires following the cost until it is actually paid, including the period after a successful client response has already been sent.
References: [1] RocksDB compaction[2] RocksDB leveled compaction
Sources and further reading
- RocksDB compaction
Primary engine documentation describes compaction strategies and tradeoffs. All capacity figures and accounting examples are hypothetical, not reported RocksDB benchmarks.
- RocksDB leveled compaction
Primary documentation explains level organisation and compaction behaviour. Queue models, deletion planning and workload recommendations are the essay's interpretation.