over|yonder Get in touch ↗
Menu

rstat: status updates in 0.16 milliseconds, down from roughly 800

rstat is a long-running Linux monitor that supplies CPU, memory, task I/O, temperature, power-state and per-process data to Waybar, a desktop status bar. Its original shell path spent most of each update starting tools and waiting on a power-profile service. Reading the power profile straight from sysfs and trimming the procfs work removed most of that time, and moving recurring process accounting into the kernel removed most of the rest.

Rust · eBPF · Linux kernel · 2026

Time per updatelog scale · 5,000× faster
Shell + service call
~800 ms
Rust procfs collector
~15 ms
eBPF collector
0.16 ms avg
0.1 ms1 ms10 ms100 ms1 s
Time per update for the original shell path, the Rust procfs collector and the eBPF collector's userspace refresh

Service latency and per-update collection costs

Waybar requests one JSON status update on a fixed schedule. rstat's original Waybar module answered each request by launching a shell script, which called command-line tools including awk, bc and powerprofilesctl.

Service round trip ~800 ms with the D-Bus request in the update path
powerprofilesctl crossed process and D-Bus boundaries to retrieve the active power profile, even though Linux exposed the same value through a sysfs file.
Process creation Shell + at least 3 child processes / update
Each update started external utilities, incurring process creation, executable loading and teardown costs.
Text interfaces Per-process file reads and parsing
Linux's process filesystem, procfs, converted kernel counters into text. The command-line utilities then parsed that text and passed more text through the shell.
State lifetime One shell invocation per update
The script exited after each update, releasing its file descriptors and buffers. Persistent sampling state required a collector that remained running between updates.

Persistent collection and direct sysfs access

The first revision replaced the one-shot shell module with a long-running Rust collector that retained counters, buffers and previous samples between Waybar updates. The Rust collector removed nearly every external utility, but powerprofilesctl still contacted the system power-profile daemon over D-Bus inside every update.

Reading the active power profile directly from sysfs removed that round trip. The procfs changes then went after the number of files opened per update. The collector had walked the process directories three times, opening files for every process on each walk, so collapsing the walks into one removed most of that work.

Each refresh still enumerated /proc and opened per-process files, and the kernel formatted their counters as text for userspace to parse. Collection cost therefore still grew with the number of live processes.

Scheduler tracepoints and task lifetime tracking

To stop the refresh cost growing with the process count, the next revision moved recurring task accounting into the kernel, as an extended Berkeley Packet Filter (eBPF) program. Each scheduler event tells the program which task it concerns, so the program accumulates that task’s CPU time and snapshots its resident memory (RSS) and I/O counters as the machine runs. A refresh then reads those counters instead of rebuilding them from files.

Four scheduler tracepoints maintain task state as processes fork, run, exit and are freed. Every record is keyed by its process identifier (PID) and kernel start timestamp, so a reused PID can't join two unrelated process lifetimes. A retired record stays in the statistics map until the Rust daemon reads its final counters, so short-lived work still shows up. One procfs scan at startup seeds the tasks that existed before the program attached.

KernelUserspace
Task lifecycle sched_process_fork sched_switch sched_process_exit sched_process_free Four tracepoints create, update and retire task state.
Account and snapshot CPU ← every switch RSS / I/O ← gated At most once per task every 10 ms, and exit forces a final snapshot.
Retain counters stats[PID, start time] Final counters survive task exit until userspace consumes them.
Collect and emit batch map lookup reused buffers → Waybar JSON
The kernel keeps per-task counters, and the Rust daemon aggregates threads into processes and computes changes since the previous sample

Batch reads, retained storage and bounded output

Each userspace refresh collects the task records, reads global metrics, aggregates processes and writes the Waybar response. Batch reads cut the system calls, while storage reserved once and small bounded result sets keep allocation and sorting off the refresh path.

Batch map collection
Linux's BPF_MAP_LOOKUP_BATCH copies many task records into preallocated arrays in one system call. Without it, each record takes its own key-advance and lookup calls.
Retained storage
The kernel maps have fixed capacities, and userspace reserves its batch arrays and aggregation tables once, then clears them between refreshes. The current and previous snapshots swap roles, sorted by PID and start time so each record finds its predecessor by binary search.
System files
The daemon opens its sysfs temperature, frequency, graphics and throttling files once and rereads them with pread() into fixed buffers, parsing numbers straight from the bytes.
Bounded output
CPU, memory and I/O leaders go into fixed five-entry arrays, and only those are sorted. The JSON is appended to reused strings, with task labels escaped in place.

Refresh latency and scheduler overhead

Each userspace refresh, from reading the eBPF maps to writing the JSON, timed over 1,000 refreshes.

0.16 msAverage map-to-JSON refresh
0.15 msMedian map-to-JSON refresh
0.22 ms95th-percentile map-to-JSON refresh

CPU cost at different refresh intervals

The procfs collector does all its accounting during each refresh, so its cost per second rises with the refresh rate. The eBPF collector's handlers run on every context switch, so its cost follows scheduler activity instead. At long refresh intervals the procfs model is slightly cheaper, because the handlers' background cost outweighs the rare refreshes. A deliberate context-switch storm shows the other extreme: each handler call costs about the same, but it runs so often that scheduler accounting dominates at every interval.

Monitoring CPU time per second at four Waybar refresh intervals, calculated from the 15-millisecond procfs refresh, the 0.16-millisecond eBPF userspace average and profiled scheduler accounting under ordinary load and a deliberate switch storm.
Refresh interval15 ms procfs modelOrdinary-load rstatSwitch-storm rstat
2000 ms7.5 ms/s8.1 ms/s1,812.5 ms/s
500 ms30.0 ms/s8.3 ms/s1,812.7 ms/s
250 ms60.0 ms/s8.6 ms/s1,813.1 ms/s
100 ms150.0 ms/s9.6 ms/s1,814.0 ms/s
View the rstat source ↗

If a performance problem has survived the usual fixes, send it our way.

Let’s speed things up. ↗ Sydney, Australia · Working worldwide