Conviva, a company analyzing trillions of events daily to monitor user experience, recently experimented with replacing mmap with io_uring in their Rust query engine. Initially, mmap was ideal for reading large Arrow IPC files due to its zero-copy random access and alignment with Arrow’s memory layout. However, under heavy concurrent query loads in production, mmap caused severe performance degradation due to page cache thrashing and kernel lock contention.
Their workload involves reading multiple large Arrow IPC files (~3–5 GB each) locally from NVMe storage, with typical queries scanning about 13 GB of data daily. Tests on a 192-core server with high-speed NVMe arrays revealed that mmap’s shared page cache became a bottleneck as concurrency increased, leading to excessive page faults, shrinking OS page cache, and query latencies spiking from around 30 seconds to over 150 seconds at the 95th percentile.
A controlled benchmark comparing one pod to four pods on the same host showed that multiple pods competed for the shared mmap page cache, resulting in worse performance due to kernel-level lock contention. Analysis revealed that mmap’s implicit shared page cache and locking mechanisms do not scale well under high concurrency, causing a storm of major and minor page faults and millions of context switches per second.
To address these issues, Conviva turned to io_uring, which promises asynchronous I/O and the ability to bypass the page cache using O_DIRECT. Initial tests showed a significant reduction in major faults but a surprising eightfold increase in minor faults and an overall 60% slower query runtime compared to mmap. The slower performance was attributed to architectural choices, including a Batch Materialization Layer that serialized I/O coordination, Arrow decoding, and caching on a single thread, and Arrow’s default memory copying behavior that triggered many minor faults.
Efforts to optimize included enabling O_DIRECT to bypass the page cache and modifying Arrow buffer construction to avoid unnecessary memory copies. These improvements reduced runtime but still did not surpass mmap’s performance. The team concluded that proper io_uring integration requires careful design beyond simply switching I/O engines, especially for workloads involving many concurrent reads and complex data decoding.
This case underscores the challenges of replacing traditional memory-mapped I/O with newer asynchronous I/O mechanisms in high-performance data processing systems. While io_uring can reduce kernel-level contention and page cache thrashing, achieving better overall throughput demands architectural changes and detailed instrumentation to understand I/O behavior at a fine-grained level.
Conviva plans to share further insights and their architectural rethink in a follow-up report and will present a deep dive at the upcoming P99 CONF in October 2026.