ros2_pulse

The heartbeat of your ROS 2 graph. A low-overhead probe that measures per-topic message rate and active-node liveness for both inter-process and intra-process traffic, on stock ROS 2 binaries, with no rebuild, no privileges and no network traffic. The counting hot path costs under a nanosecond per message; the whole probe costs about 2 % of workload CPU on a harsh 4,900 msg/s stress and less on real graphs.

pulse-top --demo: the probe’s log, live. Sparklines per topic, intra-process rates, structured warnings with ages. Install: pip3 install ros2-pulse-top (details).

Forty seconds on what watching a topic costs and what the probe does instead. Watch the video (MP4, 10 MB) or read the docs.

Why

The question in production is simple: is every topic flowing at the rate it should, and which nodes are alive? The existing tools each fall short of answering it.

• ros2 topic hz and ros2 topic echo subscribe to one topic at a time. Watching a single 100 KB topic costs 7 % of a core with hz and 31 % with echo, they cannot see intra-process messages at all, and pointing hz at an intra-process topic makes the publisher start serializing every message, which raised the watched process’s CPU by 52 % in our measurements.
• Built-in topic statistics are bypassed by intra-process comms on Humble-class binaries, so composable nodes carrying point clouds lose all introspection. This is fixed on rolling and newer by rclcpp#3130 (merged April 2026); there is no public Humble backport as of this writing.
• ros2_tracing / LTTng is built for offline analysis: a session daemon plus post-processing of a CTF trace just to get a rate. On Humble it also needs ROS rebuilt with the lttng-ust backend (Jazzy and newer trace out of the box).
• CARET (Tier IV) uses the same hook layer as this probe, LD_PRELOAD over tracetools, which is a useful independent validation of the mechanism. It is built for deep offline latency and chain analysis and needs LTTng, a forked rclcpp and Jupyter post-processing. Complementary, not always-on.
• eBPF uprobes need CAP_SYS_ADMIN, debugfs and a kernel with BTF and uprobes, which is often a non-starter on Jetson and other embedded targets, and they pay a kernel trap per message.

ros2_pulse hooks the tracetools instrumentation layer that rclcpp already calls on every publish and every callback, counts in-process with a lock-free hot path, and writes ready-to-read Hz to a small rolling file.

What you get

# ts_ns=1782887153899445923 window_s=5.000
TOPIC /scan 20.000000                      # publish-side, inter-process
PUB   /points inter=0.000000 intra=30.000000   # publish-side incl. intra (Iron+)
RECV  /scan inter=20.000000 intra=0.000000 # receive-side, BOTH transports
RECV  /points inter=0.000000 intra=30.000000   # <- intra-process, invisible to other tools on Humble
JITTER /scan recv max_dt_ms=21.284             # largest inter-arrival gap (opt-in, see below)
NODE  /perception
NODE  /planner
WARN  TOPIC /scan hz=1.200000 expected=[18,22]  # only with an expected-rate spec (see below)

If a sidecar exporter or a log shipper is reading instead of a person, ROS_TOPIC_STATS_FORMAT=jsonl writes every window as one JSON object per line with the same gates and values. See JSON Lines output.

How it works

libros2_pulse.so is injected with LD_PRELOAD. It exports the same symbols as libtracetools.so’s tracepoint API (ros_trace_rcl_publish, ros_trace_callback_start, the init tracepoints and so on). The dynamic linker binds rclcpp’s calls to ours first; each interposer records a stat and forwards to the real function through dlsym(RTLD_NEXT, ...). rclcpp calls these functions unconditionally (the LTTng enable check is inside them), so the probe works with no tracing session and adds no DDS traffic.

• Intra-process visibility comes from callback_start(callback, is_intra_process), which fires for every subscription callback regardless of transport.
• The hot path is a per-endpoint relaxed atomic increment behind a 256-slot thread-local cache with a stride-breaking hash. No global lock, no per-message string hashing. Counting costs about 0.3 ns/op on a fixed endpoint and 0.6 to 1.2 ns/op alternating across a working set on the reference box, two orders of magnitude under a single LTTng-UST tracepoint (about 158 ns).
• A background timer snapshots and resets the counts every ROS_TOPIC_STATISTICS_PUBLISH_PERIOD seconds and appends the rates to ROS_TOPIC_STATS_OUTPUT_FILE.

The pure C++ core (core/) has no ROS dependency and is unit tested on its own; the probe layer (probe/) is a thin LD_PRELOAD shim.

Install

cd ~/ros2_ws/src && git clone https://github.com/TanayK07/ros2_pulse.git
cd ~/ros2_ws && colcon build --packages-select ros2_pulse && source install/setup.bash

The pulse-top dashboard installs separately, no ROS environment needed:

File truncated at 100 lines see the full file

Changelog

All notable changes to this project are documented here. Format follows Keep a Changelog; versions follow SemVer.

[Unreleased]

[0.4.1] - 2026-08-23

Packaging fix for the apt release; no behaviour change.

Fixed

• babeltrace is no longer a declared test_depend: it has no rosdep mapping for RHEL, and humble, jazzy and kilted all release on RHEL, so the declaration blocked bloom’s RPM generation and would have failed the build farm. The LTTng coexistence test already skips when no trace viewer is installed; CI installs babeltrace2 explicitly so the jazzy and kilted lanes keep running it.

Added

• Observer-effect bench (bench/run_observer_effect.sh, raw under bench/out/observer_effect/): what ros2 topic hz / echo cost and what they do to the topic. N=10 rotated arms on the stress farm, probe as the in-process ruler. Findings: the stock tools read the rate right (publisher held 50.000 Hz, hz within 0.3 %); they cost 7 % (hz) / 31 % (echo) of a core per watched 100 KB topic; and on an intra-process topic the watcher switches serialization on, +52 % CPU on the watched process, rcl_publish path lit in 10/10 trials. README “Why”, ALTERNATIVES and bench/RESULTS.md carry the numbers.
• Launch video source (video/): a 40 s Remotion composition: hook, the measured cost of ros2 topic hz / echo, the one-line probe, real pulse-top frames through a /scan stall, the Orin/x86 numbers, CTA. Every on-screen number lives in video/src/data.ts with the file it cites; capture_frames.py labels stall / Warns-tab frames by pixel colour so the cut does not depend on capture timing. Rendered MP4 ships as a GitHub Release asset, not in git.
• README demo GIF of pulse-top --demo (docs/assets/pulse-top-demo.gif, 30 frames through one scripted-incident loop, rendered headlessly from Textual screenshots).

Fixed

• pulse-top: a topic_rate warn for an in-range rate could read 20.04Hz > max 22.0Hz; the detail picked “> max” whenever the rate was not below min. It now names the bound actually crossed, or the bounds when neither is, and rounds the rate to one decimal. The demo’s /cmd_vel sag (18.4 ± 1.5 Hz against a 19 Hz min) overlapped the bound and fired such warns; it now sags to 14.5-17.5 Hz. Window title is pulse-top, not the class name.

[0.4.0] - 2026-08-23

Launch release: the live dashboard lands, and every performance claim now has an aarch64 row behind it: v0.3.0 was run on a production Jetson AGX Orin and the numbers, raw trials and a redacted production log are committed. pulse-top was then fixed against that very log.

Added

• pulse-top (tools/pulse-top/): live terminal dashboard over the probe’s jsonl log (pip install ./tools/pulse-top, command pulse-top). A pure log consumer (no ROS dependency, no node, no subscriptions), so watching costs the probed system nothing and works over ssh or on a dead log post-mortem, unlike graph-joining monitors (ros2top, ornis, …), and it sees the intra-process rates only the in-process probe can measure. Topics table with 60-window sparklines and warn-colored rows, namespace tree, node liveness, and a retained structured-warns view with ages (a one-window stall stays readable instead of blinking for one window period). Two rules carried into the UI: an absent jsonl key renders as — (never 0), and a topic absent from the current window renders as stale <age>, never as its old rate. Hardened by review (#32/#33): byte-exact tail-following (rotation/truncation-safe, partial multi-byte holds), tail-seek attach with a per-poll read cap, and log-derived text rendered without any markup parsing so a hostile or corrupt log cannot crash the viewer. pulse-top --demo runs a scripted-incident graph; 36 tests (model, byte-exact reader, headless pilot) plus a hostile-log smoke run in a dedicated no-ROS CI lane.
• On-Orin validation (test/orin/RESULTS.md, raw under test/orin/out/): v0.3.0 run on a production Jetson AGX Orin (L4T R36.4, Humble/CycloneDDS, 77 nodes). Hot-path bench on aarch64: 0.9 ns/op fixed, 2.1 ns/op alternating; the opt-in jitter clock read costs +51 ns/msg ± 0.6 (x86: +24) and the CLOCK_MONOTONIC vDSO works; the syscall-fallback risk from the R5 design is closed. Probe loaded into every process of the deployed stack on stock binaries, 0 sockets, all v0.3.0 features (JITTER, jsonl, QUIET) verified on target. Production log committed with operator node/topic names redacted. README, bench results and the R5 design note carry the aarch64 numbers.
• test/orin/run_orin_probe_test.sh honors ROS_TOPIC_STATS_OUTPUT_FILE and records process names with the CPU samples so ON/OFF runs can be matched across a stack restart. ORIN_RUNBOOK.md Phase 0 no longer needs git credentials on the robot.

Fixed

• pulse-top on a multi-process stack. Staleness was counted in windows, but windows from every probed process interleave (77 nodes ≈ 15 windows/s), so a healthy 5 Hz topic read stale 3w and every row re-rendered every tick; /tf_static lost its PUB rate whenever a subscriber’s window landed; the sparkline took one sample per process-window instead of per period; and the default path followed only the newest topic_freq.<pid>.log, one node of the graph. Now: staleness is wall time from the window timestamps (stale 12s past 1.5× the topic’s window), publish and receive sides merge per topic, one history sample per period, every matching per-pid file is followed (new ones picked up live), and only cells whose text or style changed are pushed to the table; Textual’s DataTable.update_cell invalidates and refreshes unconditionally, so the idle repaint cost over ssh is now zero. Found on the Orin run (2026-08-23).

[0.3.0] - 2026-08-20

Observability-formats + soundness release: gap visibility closes the stall blind spot in 0.2.0’s rate alerting, JSON Lines output lands for sidecar exporters, and the RMW matrix / LTTng coexistence results make the compatibility claims measured rather than assumed.

Added

• Gap visibility (ROADMAP R5): ROS_TOPIC_STATS_JITTER=1 measures the largest inter-arrival gap per endpoint per side and emits JITTER <topic> <side> max_dt_ms=…, plus a max_gap_ms: spec rule, WARN TOPIC <name> max_dt_ms=… expected_max_gap_ms=…, and pulse-check gating. This closes a soundness hole in the 0.2.0 rate alerting: a windowed mean cannot see a stall, so at 50 Hz a min_hz: 45 rule needs >0.5 s of dead time to fire and a 400 ms freeze passes at

File truncated at 100 lines see the full file