Syscalls are not the whole I/O cost

A crossing is one stage in a longer path

An I/O operation may involve preparing a buffer, entering the kernel, validating arguments, locating cached data, waiting for a device or network, copying bytes and interpreting the result. Reducing the number of entries into the kernel can help, especially for tiny operations. It does not eliminate the other stages. I would identify which stage dominates the representative workload before changing the interface. A benchmark that repeatedly reads cached bytes can say very little about a workload whose time is mostly spent waiting for storage.

The operation's meaning matters just as much as its size. Reading a byte already in the page cache, receiving a remote response and making a database record durable are all called I/O, but their completion boundaries differ. A successful call may mean bytes were copied into a buffer, accepted by a local subsystem or synchronized according to a documented durability contract. Comparing interfaces without holding that boundary constant can produce an apparent improvement that comes entirely from doing less work before reporting success.

Batching has an arithmetic benefit and a waiting cost

Assume an illustrative fixed overhead of one microsecond per call, solely to make the arithmetic visible. Sending 1,000 eight-byte records individually pays 1,000 microseconds of that overhead. Sending them in batches of 100 pays ten microseconds, before payload processing and other costs. Those invented numbers do not predict a real system, but they show why amortization can matter. The corresponding workload moves 8,000 payload bytes either way. The optimization removes repeated fixed work; it does not make the payload disappear.

A batch must form before it can be sent. If records arrive slowly, waiting for 100 of them can violate the first record's latency target. A practical policy may flush on either size or age, creating a tunable trade between efficiency and delay. I would include this waiting time in the end-to-end measurement and report the achieved batch-size distribution. A benchmark that begins with every record already available can exaggerate the benefit for an interactive stream whose arrivals are spread across time.

Copying competes with several other memory costs

Avoiding a copy can reduce bandwidth consumption, but the replacement may require pinning memory, maintaining scatter-gather descriptors or retaining a larger buffer longer. The surrounding computation still needs to inspect or transform bytes. A parser that touches every byte has a different opportunity from a relay that forwards opaque payloads. I would track copies and touches separately. Zero-copy terminology often describes one transfer boundary while leaving application reads, checksums, encryption or device-side movement outside the label.

Buffer placement can also matter on a NUMA machine. Memory allocation policy and first use influence which memory resources serve a worker, while a later worker or device path may operate elsewhere. A lower syscall count does not repair an unfavourable placement pattern. The appropriate experiment should preserve worker placement, allocation behaviour and buffer reuse. If a change simultaneously alters those factors, the explanation needs to account for them rather than crediting all of the gain to the fashionable I/O interface selected in the same patch.

References: [3] Linux kernel: NUMA Memory Policy

Completion counts must handle partial work

The read interface permits a successful result smaller than the requested count. Zero has a specific end-of-file meaning for ordinary file reads, and errors require their own handling. A loop that assumes one call fills an entire application record can fail under perfectly valid behaviour. I would define completion at the protocol level: accumulate the required bytes, detect a clean end condition or return a classified error. The number of calls follows from that contract. Minimizing calls cannot justify treating a partial result as a complete message.

The same discipline applies to writes and asynchronous completions according to their respective APIs. Progress may be reported in bytes, not logical records. Retry logic must preserve the remaining range and avoid duplicating already accepted bytes. An interrupted or failed operation may require consulting its documented semantics rather than blindly replaying the whole buffer. This is why a realistic I/O comparison includes the surrounding loop and error handling. A tiny benchmark of a single successful call leaves much of the production operation outside the measured boundary.

References: [1] Linux read(2) manual

Durability is a different finish line

An application can observe a fast write because the operating system accepted data into cached state. If the product requires durability, the relevant operation includes the synchronization mechanism and its error handling. The fsync documentation also distinguishes file synchronization from the directory work needed for some filesystem changes. I would write the crash-recovery invariant before benchmarking a storage path. Successfully returning bytes to a caller, persisting file contents and durably publishing a new filename are related but distinct promises.

Grouping durable updates can amortize synchronization costs, but it introduces a commit boundary and a waiting policy. Acknowledging each update before the group is durable changes the failure contract. Acknowledging after the group completes preserves a different contract while increasing some requests' latency. Neither choice should be hidden inside an optimization described only as fewer syscalls. The experiment must include when acknowledgements occur and what survives a crash. Otherwise a faster result may simply measure the removal of the guarantee the application was supposed to provide.

References: [2] Linux fsync(2) manual

Queues can dominate a more efficient interface

A submission mechanism that supports many outstanding operations can improve utilization when the device or network benefits from concurrency. It can also create a deeper queue and longer tail latency if admission is uncontrolled. I would vary queue depth and observe both throughput and response time, then choose a bound that matches the product. Maximum throughput is not automatically the useful operating point. The resource can remain busy while newly admitted requests wait so long that their eventual completion has little value.

The counterargument is that a simple synchronous path may leave expensive resources idle and waste available parallelism. That is a real reason to adopt batching or asynchronous I/O. The decision should be based on the workload's bottleneck and the cost of managing outstanding ownership. A straightforward blocking implementation can be preferable for low-rate maintenance work, while a high-throughput server may justify a more complex completion engine. I would avoid turning one interface into a universal recommendation across those very different requirements.

Compare equivalent operations from end to end

A useful comparison fixes payload distribution, cache state, concurrency, buffer lifecycle, error policy and completion meaning. It then reports elapsed latency, CPU, throughput and memory retained by outstanding operations. System-call counts remain a helpful diagnostic column, but they are not the verdict. If reducing calls also increases batch waiting or retained memory, the report should make that exchange visible. The goal is to understand why the complete operation improves and whether the improvement survives the conditions in which the application actually runs.

I would keep a simple baseline implementation as a correctness reference and operational fallback where practical. Complex I/O machinery has its own failure modes, including missed completions, leaked buffers and shutdown races. The performance benefit should justify that state space. Following bytes and ownership through the whole path produces a more durable design than optimizing crossings in isolation. A syscall is an observable boundary, which makes it easy to count. The system's cost and correctness boundaries are usually larger.

For the final experiment, include both small and large payloads and a deliberate slowdown in the completion path. That reveals whether the improvement depends on favourable batching or unlimited outstanding buffers. The slow case often explains the production risk more clearly than another run at the best attainable throughput.