Performance case study / Linux systems

rstat

From roughly 800 milliseconds to 0.16 milliseconds on average.

rstat is a long-running Linux monitor that supplies CPU, memory, task I/O, temperature, power-state and per-process data to Waybar, a desktop status bar. Its original shell path took roughly 800 milliseconds per update. Replacing the power-profile service call with a direct sysfs read and optimizing procfs cut that to 15 milliseconds. Moving recurring process accounting into the kernel cut the userspace refresh to 0.16 milliseconds on average.

Rust · eBPF · Linux kernel · 2026

Refresh latency by implementationlog scale · 5,000× faster
Shell + service call~800 ms
Direct sysfs + procfs~15 ms
eBPF userspace0.16 ms avg
0.1 ms1 ms10 ms100 ms1 s
Removing the blocking service call cut the update from roughly 800 to 15 milliseconds. Replacing recurring procfs reconstruction with eBPF accounting cut the userspace refresh to 0.16 milliseconds on average.

01 / Baseline

Service latency and per-update collection costs

Waybar requests one JSON status update on a fixed schedule. rstat's original Waybar module took roughly 800 milliseconds per update because it launched a shell script that invoked command-line tools including awk, bc and powerprofilesctl.

Service round trip ~800 ms with the D-Bus request in the update path
powerprofilesctl crossed process and D-Bus boundaries to retrieve the active power profile, even though Linux exposed the same value through a sysfs file.
Process creation Shell + at least 3 child processes / update
Each update started external utilities, incurring process creation, executable loading and teardown costs.
Text interfaces Per-process file reads and parsing
Linux's process filesystem, procfs, converted kernel counters into text. The command-line utilities then parsed that text and passed more text through the shell.
State lifetime One shell invocation per update
The script exited after each update, releasing its file descriptors and buffers. Persistent sampling state required a collector that remained running between updates.

02 / Procfs collector

Persistent collection and direct sysfs access

The first revision replaced the one-shot shell module with a long-running Rust collector that retained counters, buffers and previous samples between Waybar updates. The Rust collector removed nearly every external utility, but powerprofilesctl still contacted the system power-profile daemon over D-Bus inside every update.

Reading the active power profile directly from /sys/firmware/acpi/platform_profile removed that round trip. The Rust procfs collector then collapsed three process-directory walks into one, reused an 8 KB read buffer, took resident-set size (RSS) from /proc/[pid]/stat instead of opening statm, and identified kernel threads from the same record so it could skip their I/O file in the default view. Together, those changes cut each update from roughly 800 to 15 milliseconds.

Each refresh still enumerated /proc and opened per-process files, causing the kernel to format counters as text for userspace to parse. Collection cost therefore continued to grow with the number of live processes.

03 / Event-driven accounting

Scheduler tracepoints and task lifetime tracking

The optimized Rust procfs collector still scaled with the number of live processes. It opened stat for every process and io for each ordinary process. Across roughly 300 processes, those files can require about 1,800 open, read and close system calls per refresh, before directory enumeration and global metrics. The next revision moved recurring task accounting into an extended Berkeley Packet Filter (eBPF) program, verified by the Linux kernel before loading. Scheduler events supply the task being accounted for, allowing the program to accumulate its CPU residency and snapshot its RSS and I/O counters.

Four scheduler tracepoints maintain task state as processes fork, run, exit and are freed. Every record is keyed by its process identifier (PID) and kernel start timestamp, preventing a reused PID from joining two unrelated process lifetimes. A retired record remains in the shared eBPF statistics map until the Rust daemon consumes its final counters, so short-lived work remains visible. One procfs scan at startup seeds blocked threads and zombie process leaders that already existed when the eBPF program attached; that scan doesn't recur during refreshes.

KernelUserspace
Task lifecycle sched_process_fork sched_switch sched_process_exit sched_process_free Four tracepoints create, update and retire task state.
Account and snapshot CPU ← every switch RSS / I/O ← gated At most once per task every 10 ms; exit forces the final snapshot.
Retain counters stats[PID, start time] Final counters survive task exit until userspace consumes them.
Collect and emit batch map lookup reused buffers → Waybar JSON
The fork tracepoint establishes each task's I/O baseline. The scheduler-switch tracepoint charges CPU time and, at most once per task every 10 milliseconds, follows the kernel task record to its resident-memory and I/O-accounting fields. Exit and free events preserve the final counters for one last userspace collection. The Rust daemon then aggregates threads into processes, computes changes since the previous sample and formats Waybar's JSON response.

04 / Collection and output

Batch reads, retained storage and bounded output

With task accounting maintained in eBPF maps, each userspace refresh collects task records, reads global metrics, aggregates processes and constructs the Waybar response. Batch operations reduce system-call overhead, while retained buffers and bounded result sets limit allocation and sorting work.

Batch map collection
Linux's BPF_MAP_LOOKUP_BATCH operation transfers many task records into preallocated key and value arrays in one system call. The iterative fallback requires separate key-advance and lookup calls for each record.
RSS and I/O sampling interval
CPU residency is charged on every switch. RSS and I/O are sampled at most once per task every 10 milliseconds, with a forced final snapshot when the task exits.
Map capacity and userspace storage
The statistics, scheduler-start and PID-to-start-time hash maps have fixed capacities and are created before their tracepoints attach. Userspace reserves its batch arrays and process-aggregation tables once, then clears them without releasing their storage.
Current and previous snapshots
Two vectors retain capacity and swap roles between samples. Sorting records by PID and start time gives cache-friendly traversal and binary-search lookups against the previous sample.
Persistent system-file descriptors
sysinfo() returns all three load averages in one system call. The daemon discovers and opens its sysfs temperature, frequency, graphics-processor and throttling files once; pread() fills fixed byte buffers, and numeric values are parsed directly from those bytes.
Waybar JSON construction
write! appends the known Waybar schema to reusable tooltip and JSON strings. A small emitter escapes task names in place without a general serializer or temporary formatted strings.
Top-process selection
CPU, memory and I/O leaders are selected into fixed five-entry arrays, while the list of tasks blocked in Linux's uninterruptible D state is capped at ten. Only those small arrays are sorted for output.

05 / Measurement

Refresh latency and scheduler overhead

rstat --bench 1000 measures each userspace refresh from eBPF-map collection through system-file reads, process aggregation, retired-record cleanup and JSON construction.

0.16 msAverage map-to-JSON refresh
0.15 msMedian map-to-JSON refresh
0.22 ms95th-percentile map-to-JSON refresh

CPU cost at different refresh intervals

The procfs collector performs its accounting work during each refresh, so its collection cost per second scales with refresh frequency. The eBPF collector also runs handlers on scheduler events, making its total cost depend on both refresh frequency and scheduler activity. Under the measured ordinary load, eBPF uses less CPU at the three shorter intervals shown below. At a two-second interval, the procfs model is slightly cheaper. The deliberate context-switch storm makes scheduler accounting the dominant cost at every interval shown.

Monitoring CPU time per second at four Waybar refresh intervals, calculated from the 15-millisecond procfs refresh, the 0.16-millisecond eBPF userspace average and profiled scheduler accounting under ordinary load and a deliberate switch storm.
Refresh interval15 ms procfs modelOrdinary-load rstatSwitch-storm rstat
2000 ms7.5 ms/s8.1 ms/s1,812.5 ms/s
500 ms30.0 ms/s8.3 ms/s1,812.7 ms/s
250 ms60.0 ms/s8.6 ms/s1,813.1 ms/s
100 ms150.0 ms/s9.6 ms/s1,814.0 ms/s

rstat --profile measured 0.641 microseconds per scheduler-switch handler and 7.98 milliseconds of handler time per second under ordinary load. A deliberate 12-worker switch storm raised those figures to 0.767 microseconds per handler and 1,812 milliseconds per second. Production builds omit the profiling instrumentation.

Collection overhead

What changed in the collector

The surrounding machinery dominated the work: process startup, service calls, reopened files, text serialization and repeated allocations.

Replacing the D-Bus power-profile query with one sysfs read removed the blocking dependency used by the intermediate Rust collector. Moving recurring process accounting from procfs polling to scheduler events then made the measured map-to-JSON refresh typically sub-millisecond, while tying the background cost to scheduler activity.

View the rstat source

Have a performance problem that has survived the usual fixes?

Let’s speed things up. Sydney, Australia · Working worldwide