jitter_bench — how low can a mostly-Python 1 kHz control loop go on a stock kernel?¶

Date: 2026-07-12 · Env: pixi control (robostack-jazzy + conda-forge), cppyy 3.5.0, Python 3.12, linux-64 · ROS_DOMAIN_ID=63 · Machine: 16-core, kernel 6.17.0-1028-oem (Ubuntu 24.04, CONFIG_PREEMPT_DYNAMIC=y, CONFIG_HZ=1000).

The experiment. What loop-period jitter does a ~1 kHz control loop orchestrated from Python achieve on a stock Ubuntu kernel, and how far do successive tuning steps take it? This is a progressive-tuning story, not a pass/fail against a real-time kernel. CONFIG_PREEMPT_RT is not compiled in this image, and is not required for this goal — the stock PREEMPT_DYNAMIC kernel already ships every soft-real-time primitive that matters for a 1 kHz loop (SCHED_FIFO/RR/DEADLINE, PI futexes, high-res timers, runtime preemption-mode switching, CPU isolation, forced-threaded IRQs); PREEMPT_RT mainly buys hard worst-case bounds under adversarial load. This lane is Stage 0: the unprivileged-tuning reference numbers. Stage 1 (owner actions — sudo / GRUB / re-login) completes the tuning matrix and is fully documented at the end, not executed here.

Headline. One unprivileged call — prctl(PR_SET_TIMERSLACK, 1) — drops the median wakeup latency of a 1 kHz Python loop from ~52 µs to ~2.4 µs (the 52 µs was pure Linux default timer slack, nothing to do with Python). With that plus mlockall + CPU pinning, a 1 kHz loop driven entirely from Python holds 1000.0 Hz with a ~2 µs median and a ~0.7 % late-cycle rate (idle) on this stock kernel. The remaining story is the tail (p99.9 / max), which stays fat under load because Stage 0 cannot set SCHED_FIFO or change the preemption mode — that is exactly what Stage 1 addresses. Moving the loop into C++ (cppyy_kit nogil+cppdef_cached, variant b) tightens the tail without needing to change the median. Verdict: a stock kernel + unprivileged tuning already gives soft-real-time (prototyping / HIL / sim / teleop) with a µs-scale median; the tail is a tuning problem, not a Python problem, and Stage 1 is where it shrinks.

0. What was measured (and the honesty caveats up front)¶

Metric — wakeup latency (the headline). Each loop programs absolute wake deadlines base + i·period and records the actual wake time on the same clock used to program the deadline (CLOCK_MONOTONIC, which is exactly CPython's perf_counter clock — verified via time.get_clock_info). The wakeup latency wake[i] − deadline[i] (µs late relative to the ideal fixed grid) is the cyclictest-equivalent number, so the Stage-1 cyclictest reference will be directly comparable to variant a1. A secondary period jitter column (consecutive interval − target) is also reported because that is what control_kit's own bench (REPORT §4) reported — the two answer different questions and both are given.

The four loop bodies, one harness:

id	loop body	sleep mechanism	where the loop runs
a1	pure-Python timer + tiny compute	`clock_nanosleep(TIMER_ABSTIME)`	Python
a2	pure-Python timer + tiny compute	deadline-corrected `time.sleep`	Python
b	C++ wait+compute loop	`clock_nanosleep(TIMER_ABSTIME)` in C++	C++ (cppyy_kit `nogil`+`cppdef_cached`)
c	real ros2_control `read→update→write`, Python PD controller	`clock_nanosleep` (harness)	Python drives, CM in C++

Each ran 60 s at 1 kHz (60 000 cycles), idle and under load; the first 100 cycles are dropped from the stats (first-use JIT / cache warm / scheduler settling) and that drop is stated in every row (never silent). The per-cycle "control law" is a tiny fixed polynomial fold (matched in Python and C++), kept well under the period so the jitter reflects scheduling, not compute throughput — this experiment is about when the loop runs, not how fast the math is.

Honesty caveats — read these before the numbers:

The machine was NOT quiet. Several sibling agent lanes (perception pipeline, docs, recon) ran concurrently during this measurement, and the ROS stack itself has background threads. ROS_DOMAIN_ID=63 isolates DDS discovery, not CPU. So the "idle" column is really "no jitter_bench-induced load, but a busy developer machine" — these numbers are conservative (a genuinely quiet machine would show a thinner tail). This is the honest floor for "a 1 kHz Python loop on a working laptop," which is the realistic case.
Stock desktop kernel. CONFIG_PREEMPT_DYNAMIC=y, CONFIG_HZ=1000, NO_HZ_FULL=y. CONFIG_PREEMPT_RT is not compiled in this image (not required for this goal — see the framing above and the capability list in Stage 1). The live preemption mode is unreadable without sudo (recorded as unknown; the cmdline carries no preempt= override, so the boot mode is whatever PREEMPT_DYNAMIC defaults to on this image).
Unprivileged tuning only (Stage 0). mlockall and prctl(PR_SET_TIMERSLACK, 1) succeeded (no privilege needed). SCHED_FIFO needs an rtprio grant (ulimit -r is 0 here) so it is not available to Stage 0; the harness attempts it and records the denial, so the same command becomes a real FIFO run the moment the owner grants rtprio (Stage 1). nice also needs privilege here and was left unchanged.
Full-run stats, no window cherry-picking. Every percentile is over the whole 60 s run (minus the stated 100-cycle warmup). Outliers are reported, not trimmed.

RT knobs applied this run (all unprivileged, all succeeded): prctl(PR_SET_TIMERSLACK, 1) (see §1a — the big lever); mlockall(MCL_CURRENT|MCL_FUTURE) (the process fits under the ~8 GB memlock ulimit — no privilege needed); CPU affinity pinned the bench to cpu 2; the load condition pinned 8 busy-loop processes to cpus 8–15 (the measurement core is never oversubscribed — "loaded machine, dedicated core", the realistic embedded-style topology). nice / SCHED_FIFO need privilege and were recorded as unchanged / denied.

1a. The first tuning lever: `PR_SET_TIMERSLACK` (unprivileged, huge)¶

Linux' default timer slack is 50 µs — the kernel may defer any clock_nanosleep / futex / poll wakeup by up to that much to batch wakeups and save power. At 1 kHz (1000 µs period) that 50 µs is the median wakeup latency. Setting slack to 1 ns (prctl(PR_SET_TIMERSLACK, 1), no privilege) removes the batching. Measured, variant a1, idle, same machine (8 s each):

timer slack	p50 (µs)	mean (µs)	p99 (µs)	p99.9 (µs)	max (µs)
50 µs (OS default)	52.4	86.9	668.7	2196.8	3932.8
1 ns (tuned, `PR_SET_TIMERSLACK`)	2.4	22.0	413.6	1767.8	3135.5

A ~22× drop in median wakeup latency from one unprivileged syscall — and the ~52 µs it removes was timer slack, not Python overhead. This is Stage 0's single most important finding and the first step of the progressive-tuning story. The harness now sets slack to 1 ns by default (--timerslack-ns, default 1); every number in §1 below is with slack tuned. The tail (p99.9 / max) barely moves — the tail is a preemption problem (Stage 1), not a timer-slack one.

1. Results — the reference matrix (60 s/cell, 1 kHz, timer slack = 1 ns)¶

Wakeup latency (µs) — the headline metric¶

60 000 cycles/cell, first 100 dropped; SCHED_OTHER, timer slack 1 ns, mlockall on, bench pinned to cpu 2; load = 8 busy-loop processes pinned to cpus 8–15 (bench core not oversubscribed). All held 1000.0 Hz mean.

variant	cond	min	mean	p50	p99	p99.9	max	late %
a1 pure-Python `clock_nanosleep`	idle	1.3	29.2	2.4	460.9	2108.4	3138.4	0.65
	load	1.7	45.5	5.3	1137.0	2994.0	9306.7	2.05
a2 pure-Python `time.sleep`	idle	1.4	35.9	2.5	657.6	2426.0	8358.4	0.76
	load	1.4	45.5	5.7	1166.7	2647.3	3482.1	1.92
b cppyy_kit C++ loop (nogil+cached)	idle	1.1	30.3	2.0	461.8	2102.9	3174.3	0.67
	load	1.2	33.6	2.1	959.1	2426.4	6781.1	1.57
c real ros2_control loop (Python ctrl)	idle	1.2	18.5	2.3	500.9	2257.0	3454.7	0.70
	load	1.5	41.5	5.0	1052.4	2758.5	9805.2	1.75

a1 load is the re-measured value: the first attempt caught a single ~1.3 s machine-wide stall (mean → 14.5 ms, max → 1.30 s from one hiccup + its absolute-deadline catch-up); the re-run above is representative and consistent with a2/c load. Both are in build/jitter_bench.json / build/jitter_a1_load_rerun.json. See §1-reading for why that spike is the point, not a fault.

The one-line read of the table: median ~2 µs everywhere idle; under load the C++ loop (b) is the only one that keeps its ~2 µs median (Python loops rise to ~5 µs), and b's p99 under load is also the lowest of the four.

Reading the numbers¶

The median is µs-scale for everyone. With timer slack tuned, all four variants sit at p50 2.0–2.5 µs idle — Python, C++, and the real control loop alike. The orchestration language does not set the median; the kernel's high-res timer does, once slack is out of the way. All hold 1000.0 Hz mean with 0.65–0.76 % late cycles idle.
Under load, the C++ loop (variant b) is the standout. It holds p50 2.1 µs under load (essentially its 2.0 µs idle), while the pure-Python loops' median rises to ~5 µs (a2 5.7, c 5.0). b's p99 under load (959 µs) also beats a2 (1167) and c (1052). This is the cppyy_kit acceleration paying off exactly where intended — the determinism of the hot loop, not throughput.
Driving a REAL ros2_control loop from Python costs almost nothing in jitter. Variant c (real read→update→write, cross-inherited Python PD controller) sits right with the bare timer loops: p50 2.3 µs, p99 501 µs idle — its mean is actually the lowest of the four (18.5 µs). The ros2_control machinery + the cross-language update() call add negligible jitter at the median; the tail is the same preemption story as a bare loop. The one-time ~25 ms first-update() JIT (control_kit GAP #2) is warmed out (200 warmup updates + the 100-cycle drop), so it never appears in the stats.
clock_nanosleep vs time.sleep (a1 vs a2). Idle they are close (both now ride the tuned timer slack); the mechanism difference shows in the tail — a2's idle max is 8.4 ms vs a1's 3.1 ms, i.e. time.sleep's coarser wakeup occasionally overshoots more.
The tail is preemption, and this machine was busy. Idle p99.9 ≈ 2.1–2.4 ms, max ≈ 3–8 ms across variants — that is CFS preemption on a non-isolated core plus the concurrent agent lanes (§0 caveat 1). The load column's first a1 attempt caught a single ~1.3 s external stall (an entire cell's mean dragged to 14.5 ms by one machine-wide hiccup + its ~1300-cycle absolute-deadline catch-up); it was re-measured for the table above. That spike is precisely the "rare tail spike under adversarial load" that Stage 1's SCHED_FIFO + preempt=full + core isolation suppress and PREEMPT_RT would bound — the honest evidence for why Stage 1 exists, not a harness fault.

Per-cell latency histograms¶

a1 (pure-Python, clock_nanosleep) — idle. 81 % of cycles within 5 µs; the tail is a few % in the 100 µs–2 ms range.

  bucket(us)       count      %  histogram
  1-2               9812  16.4%  ##########
  2-5              38785  64.7%  ########################################
  5-10              4251   7.1%  ####
  10-20             1316   2.2%  #
  20-50              737   1.2%  #
  50-100            1244   2.1%  #
  100-200           1007   1.7%  #
  200-500           2208   3.7%  ##
  500-1000           264   0.4%
  1000-2000          211   0.4%
  2000-5000           65   0.1%
  total            59900 100.0%

b (cppyy_kit C++ loop, nogil+cached) — under load. Tighter than a1 is when idle: 91.6 % of cycles within 5 µs, and 42 % within 2 µs, despite 8 busy cores of background load. The nogil C++ loop never re-enters the interpreter between wake and next sleep, so the scheduler sees one long-running C++ thread rather than a Python thread cycling the interpreter — load perturbs it less.

  bucket(us)       count      %  histogram
  1-2              25271  42.2%  ##################################
  2-5              29567  49.4%  ########################################
  5-10               838   1.4%  #
  10-20              497   0.8%  #
  20-50              614   1.0%  #
  50-100             447   0.7%  #
  100-200            507   0.8%  #
  200-500            925   1.5%  #
  500-1000           674   1.1%  #
  1000-2000          422   0.7%  #
  2000-5000          136   0.2%
  >=5000               2   0.0%
  total            59900 100.0%

(Every cell's histogram is in build/jitter_bench.json via --json.)

2. What this says about Python control loops (the verdict)¶

A stock Ubuntu PREEMPT_DYNAMIC kernel + unprivileged tuning already gives a µs-scale median 1 kHz loop from Python. No sudo, no RT kernel, no boot changes: PR_SET_TIMERSLACK + mlockall + CPU pinning put a bare Python timer loop, a cppyy_kit C++ loop, and a real ros2_control loop all at ~2 µs median / 1000 Hz / <1 % late idle. That is soft-real-time grade — production-viable for prototyping, HIL, sim, and teleop. The C++ (cppyy_kit) loop additionally keeps that median under load, where the pure-Python loops degrade ~2×.

The open item is the tail under contention (idle p99.9 ≈ 2 ms; rare multi-hundred-ms spikes on this shared machine). That is a scheduling problem — SCHED_OTHER on a non-isolated core with a busy machine — not a Python or a cppyy_kit problem, and it is exactly what Stage 1 (SCHED_FIFO + preempt=full + isolcpus/nohz_full/rcu_nocbs + optional low-latency kernel) is set up to collapse, with CONFIG_PREEMPT_RT only needed if one wants bounded worst case under adversarial load. Verdict: soft-real-time now, on a stock kernel, from Python; hard-real-time is a Stage-1 tuning path on the same kernel, not a rewrite.

This extends control_kit's Stage-4 finding. control_kit REPORT §4 reported "100 Hz rock-solid, 1 kHz works on average with ~0.45 % late, GC is the RT hazard, prototyping-grade / soft-real-time." jitter_bench confirms the shape at 1 kHz with 60 s histograms and adds three things: (1) the median is not a Python cost — it is timer slack, fixed by one unprivileged call; (2) the mechanism comparison (a1 vs a2 vs b) — clock_nanosleep beats time.sleep, and the C++ loop tightens the tail; (3) the under-load column, which is the part Stage 1 most affects. The graduation path is unchanged: prototype the control law in Python against the real CM (control_kit), then lower update() to a native pluginlib controller for hard-RT deployment (control_kit REPORT §4, the L2 direct-compile path).

Where cppyy_kit helps and where it cannot. nogil+cppdef_cached (variant b) removes two Python-specific hazards from the hot path — the per-cycle interpreter/GIL bookkeeping between wake and next sleep, and the first-use call-wrapper JIT (cached away) — which thins the tail. It does not need to move the median (timer slack already did). The one thing no user-space trick reaches is the worst-case tail under contention — that is the preemption mode + scheduling class, i.e. Stage 1.

3. GAPS / honest boundaries¶

Not a quiet-machine measurement (§0 caveat 1) — conservative, not best-case.
No SCHED_FIFO, no preemption-mode change in Stage 0 — the tail is dominated by preemption this run cannot control; Stage 1 is where the tail should shrink.
No cyclictest reference yet — variant a1 is a cyclictest-equivalent (same clock, same clock_nanosleep mechanism, same mlockall), but the canonical cyclictest cross-check is a Stage-1 owner-action install.
mock hardware — variant c uses mock_components/GenericSystem; a real SystemInterface adds bus I/O latency not measured here.
1 kHz single rate — the harness takes --rate; 100 Hz / 2 kHz / 5 kHz sweeps are a --rate loop away but not run for this reference.

4. Reproduce / re-run¶

# The full reference matrix (what produced §1), 60 s/cell, timerslack+mlockall+cpu-pinned:
ROS_DOMAIN_ID=63 pixi run -e control bench-jitter          # -> build/jitter_bench.json

# The §1a timer-slack A/B (default vs tuned), fast:
pixi run -e control python jitter_bench/run_bench.py --variant a1 --condition idle \
    --duration 8 --timerslack-ns -1 --mlock --cpu 2 --no-hist    # default 50us
pixi run -e control python jitter_bench/run_bench.py --variant a1 --condition idle \
    --duration 8 --timerslack-ns 1  --mlock --cpu 2 --no-hist    # tuned 1ns

# Fast smoke (no ros2_control; a1/a2/b, 2 s, idle):
pixi run -e control bench-jitter-smoke

# One cell, one command (the Stage-1 rerun shape):
pixi run -e control bench-jitter-cell                      # a1 / idle / 60 s

# Tests (variant c runs in the control env; skips in the default env):
pixi run -e control test-jitter        # or: pixi run test  (jitter_bench/tests included)

Any single cell is one command: python jitter_bench/run_bench.py --variant <id> --condition <idle|load> --duration <s> --rate <hz> --timerslack-ns <ns> --mlock --cpu <n> [--sched fifo] [--preempt-label <mode>].

STAGE 1 — owner actions to complete the tuning matrix (DOCUMENT ONLY — not executed)¶

Stage 0 above is the unprivileged reference. The rest of the progressive-tuning story needs privileges this lane does not have (sudo / GRUB / re-login). These commands are for the machine owner (fact-checked against /boot/config-6.17.0-1028-oem); none were run here. No new kernel is required for any of them — the stock kernel already has the primitives.

The stock kernel already compiles in every soft-RT primitive (why no RT kernel is needed)¶

Verified present in this kernel's config: SCHED_FIFO/RR/DEADLINE, CONFIG_FUTEX_PI, CONFIG_HIGH_RES_TIMERS, CONFIG_SCHED_HRTICK, CONFIG_IRQ_FORCED_THREADING (so threadirqs works), CONFIG_NO_HZ_FULL, CONFIG_RCU_NOCB_CPU, CONFIG_PREEMPT_RCU, CONFIG_RT_MUTEXES, isolcpus, runtime preemption-mode switching (PREEMPT_DYNAMIC). The only thing CONFIG_PREEMPT_RT would add on top is bounded worst-case latency under adversarial load — not a capability gap for a 1 kHz soft-RT loop.

1. Grant rtprio + memlock so `SCHED_FIFO` becomes available¶

sudo bash -c 'printf "sapf - rtprio 98\nsapf - memlock unlimited\n" > /etc/security/limits.d/99-realtime.conf'
# log out and back in (or reboot) for the limits to take effect; verify:  ulimit -r   # -> 98

After this, --sched fifo in the harness stops reporting "DENIED" and runs a real SCHED_FIFO loop (the code path is already present; only the grant is missing).

2. Switch the runtime preemption mode (PREEMPT_DYNAMIC — live, no reboot)¶

cat /sys/kernel/debug/sched/preempt                      # read current (needs sudo)
echo none      | sudo tee /sys/kernel/debug/sched/preempt
echo voluntary | sudo tee /sys/kernel/debug/sched/preempt
echo full      | sudo tee /sys/kernel/debug/sched/preempt

3. GRUB cmdline on the SAME kernel (reboot, no new kernel) — isolate the control core¶

# add to GRUB_CMDLINE_LINUX_DEFAULT in /etc/default/grub, then: sudo update-grub && reboot
preempt=full threadirqs isolcpus=2 nohz_full=2 rcu_nocbs=2
# isolcpus/nohz_full/rcu_nocbs on the measurement core (cpu 2 here) remove the timer tick,
# RCU callbacks and other CPUs' schedulable load from it; threadirqs makes IRQ handlers
# preemptible; preempt=full sets the most aggressive preemption at boot.

4. Reference baseline + optional free middle step¶

sudo apt-get install -y rt-tests
# canonical wakeup-latency reference to sit beside variant a1 (same clock/mechanism):
cyclictest --mlockall --priority=80 --interval=1000 --distance=0 --duration=60 --histogram=2000
# OPTIONAL, no Ubuntu Pro: a low-latency HWE kernel (more aggressive preemption defaults,
# still NOT PREEMPT_RT) from the standard noble-updates archive:
sudo apt-get install -y linux-lowlatency-hwe-24.04     # 6.17.0-35; select at boot

5. The Stage-1 re-run matrix — {none/voluntary/full} × {SCHED_OTHER/SCHED_FIFO}¶

For each preemption mode (step 2 or 3) and each scheduling class, re-run one command per cell — the harness records the preempt mode you pass as a label and applies the scheduler:

echo full | sudo tee /sys/kernel/debug/sched/preempt
ROS_DOMAIN_ID=63 python jitter_bench/run_bench.py \
    --variant c --condition load --duration 60 --rate 1000 \
    --timerslack-ns 1 --mlock --cpu 2 --sched fifo --prio 80 --preempt-label full \
    --json build/jitter_c_full_fifo_load.json

Sweep --preempt-label {none,voluntary,full} × --sched {other,fifo} × --variant {a1,b,c}; each writes its own JSON, and the summary table + histogram print per run.

Expected finding (evidence-cited). With tuning, typical/p99 at 1 kHz on this class of kernel is double-digit µs; the known failure mode is rare tail spikes under adversarial load (GPU / storage contention), which is exactly what SCHED_FIFO + preempt=full + core isolation suppress and what CONFIG_PREEMPT_RT would bound. The under-load column of Stage 0 is therefore the most interesting part of this report — it shows the tail the Stage-1 knobs exist to collapse.

Possible follow-up (owner): the same harness on an embedded board (Raspberry Pi) — copy jitter_bench/, pip install numpy, run variant a1/b (no ROS needed); variant c needs a ROS install on the board.