ros2_pulse

The heartbeat of your ROS 2 graph. A low-overhead probe that measures per-topic message rate and active-node liveness for both inter-process and intra-process traffic, on stock ROS 2 binaries, with no rebuild, no privileges and no network traffic. The counting hot path costs under a nanosecond per message; the whole probe costs about 2 % of workload CPU on a harsh 4,900 msg/s stress and less on real graphs.

pulse-top --demo: the probe’s log, live. Sparklines per topic, intra-process rates, structured warnings with ages. Install: pip3 install ros2-pulse-top (details).

Forty seconds on what watching a topic costs and what the probe does instead. Watch the video (40 s, YouTube) or read the docs.

The long version, for systems programmers: Watching a ROS 2 topic changes it, the observer-effect measurement and the symbol-interposition trick that avoids it.

Why

The question in production is simple: is every topic flowing at the rate it should, and which nodes are alive? The existing tools each fall short of answering it.

• ros2 topic hz and ros2 topic echo subscribe to one topic at a time. Watching a single 100 KB topic costs 7 % of a core with hz and 31 % with echo, they cannot see intra-process messages at all, and pointing hz at an intra-process topic makes the publisher start serializing every message, which raised the watched process’s CPU by 52 % in our measurements.
• Built-in topic statistics are bypassed by intra-process comms on Humble-class binaries, so composable nodes carrying point clouds lose all introspection. This is fixed on rolling and newer by rclcpp#3130 (merged April 2026); there is no public Humble backport as of this writing.
• ros2_tracing / LTTng is built for offline analysis: a session daemon plus post-processing of a CTF trace just to get a rate. On Humble it also needs ROS rebuilt with the lttng-ust backend (Jazzy and newer trace out of the box).
• CARET (Tier IV) uses the same hook layer as this probe, LD_PRELOAD over tracetools, which is a useful independent validation of the mechanism. It is built for deep offline latency and chain analysis and needs LTTng, a forked rclcpp and Jupyter post-processing. Complementary, not always-on.
• eBPF uprobes need CAP_SYS_ADMIN, debugfs and a kernel with BTF and uprobes, which is often a non-starter on Jetson and other embedded targets, and they pay a kernel trap per message.

ros2_pulse hooks the tracetools instrumentation layer that rclcpp already calls on every publish and every callback, counts in-process with a lock-free hot path, and writes ready-to-read Hz to a small rolling file.

What you get

# ts_ns=1782887153899445923 window_s=5.000
TOPIC /scan 20.000000                      # publish-side, inter-process
PUB   /points inter=0.000000 intra=30.000000   # publish-side incl. intra (Iron+)
RECV  /scan inter=20.000000 intra=0.000000 # receive-side, BOTH transports
RECV  /points inter=0.000000 intra=30.000000   # <- intra-process, invisible to other tools on Humble
JITTER /scan recv max_dt_ms=21.284             # largest inter-arrival gap (opt-in, see below)
NODE  /perception
NODE  /planner
WARN  TOPIC /scan hz=1.200000 expected=[18,22]  # only with an expected-rate spec (see below)

If a sidecar exporter or a log shipper is reading instead of a person, ROS_TOPIC_STATS_FORMAT=jsonl writes every window as one JSON object per line with the same gates and values. See JSON Lines output. Dashboards that already read rclcpp’s built-in topic statistics get the same numbers on /statistics from the pulse_bridge sidecar.

How it works

libros2_pulse.so is injected with LD_PRELOAD. It exports the same symbols as libtracetools.so’s tracepoint API (ros_trace_rcl_publish, ros_trace_callback_start, the init tracepoints and so on). The dynamic linker binds rclcpp’s calls to ours first; each interposer records a stat and forwards to the real function through dlsym(RTLD_NEXT, ...). rclcpp calls these functions unconditionally (the LTTng enable check is inside them), so the probe works with no tracing session and adds no DDS traffic.

• Intra-process visibility comes from callback_start(callback, is_intra_process), which fires for every subscription callback regardless of transport.
• The hot path is a per-endpoint relaxed atomic increment behind a 256-slot thread-local cache with a stride-breaking hash. No global lock, no per-message string hashing. Counting costs about 0.3 ns/op on a fixed endpoint and 0.6 to 1.2 ns/op alternating across a working set on the reference box, two orders of magnitude under a single LTTng-UST tracepoint (about 158 ns).
• A background timer snapshots and resets the counts every ROS_TOPIC_STATISTICS_PUBLISH_PERIOD seconds and appends the rates to ROS_TOPIC_STATS_OUTPUT_FILE.

The pure C++ core (core/) has no ROS dependency and is unit tested on its own; the probe layer (probe/) is a thin LD_PRELOAD shim.

Install

apt

Binary packages are built for Humble, Jazzy and Kilted:

File truncated at 100 lines see the full file

Changelog

All notable changes to this project are documented here. Format follows Keep a Changelog; versions follow SemVer.

[Unreleased]

Changed

• docs/ALTERNATIVES.md compares against NVIDIA’s greenwave_monitor, the packaged subscriber-based monitor a Discourse reader said they were switching from; section plus a positioning-table row.

Added

• pulse-top 0.3.0: recv_lag warn (issue #50). When a publisher’s process and a subscriber’s process are both probed, pulse-top pairs the topic’s publish rate (busier path) with its callback rate (inter + intra) and warns once the callbacks sit more than --lag-tol (default 10%) and more than two messages per window under the publish rate for --lag-windows (default 3) consecutive windows; it clears under half the tolerance and holds in between. A publish observation older than 1.5 periods cannot pair, so a dead or idle publisher (recv reads an explicit 0.0, KNOWN_ISSUES #12) is not lag; trackers are keyed by topic and source log, so a healthy subscriber process does not mask a lagging one, and a warn whose subscriber stopped flushing clears by time. Amber in the table (RECV cell), strip and Warns tab, with deficit, windows and source file in the sidebar; the text says the probe counts callbacks, not wire samples, so it cannot tell a drop from a backlog. Derived by the consumer: the probe, its output format, the spec grammar and pulse-check are unchanged. Test node mode slow_listener <ms> and test/integration/test_recv_lag.py prove the two logs carry the gap and the model derives the warn from them; 21 new pulse-top tests.
• Blog post, “Watching a ROS 2 topic changes it” (docs/blog/2026-09-18-watching-a-ros2-topic-changes-it.md, on the docs site under Blog). The observer-effect bench written up for a systems-programming audience: what a subscriber costs (hz 7 %, echo 31 % of a core per watched 100 KB topic), why hz on an intra-process topic switches serialization on in the watched process (+52 % CPU, 10/10 trials), how LD_PRELOAD interposition of the ros_trace_* symbols counts in-process instead, and what that costs (+1.9 % ± 0.7 % paired). Every number cites its file and line in a trailing comment. Two figures under docs/assets/blog/, regenerated from the committed CSV by bench/plot_observer_effect.py. site/build.py now rewrites links relative to the page’s own directory (pages under blog/), and folds ../ in link targets, so a results page linking ../../bench/RESULTS.md lands on the benchmarks page instead of GitHub.

[0.5.0] - 2026-09-09

Adds an optional /statistics sidecar. The probe itself is unchanged.

Added

• pulse_bridge (ros2 run ros2_pulse pulse_bridge): tails the probe’s jsonl log(s) and republishes every window on /statistics as statistics_msgs/MetricsMessage, in the shape rclcpp’s built-in topic statistics use (unit ms, AVERAGE = message period, SAMPLE_COUNT, MAXIMUM = largest gap when measured), so existing consumers of that topic read pulse windows, intra-process included, with no integration work. Parameters: files, glob (default $TMPDIR/topic_freq.*.log, re-expanded every poll), topic, unit (ms|Hz), poll_period_s, source_name. Runs as its own process so the probe stays out of the DDS graph and free of rclcpp; package.xml now declares rclcpp and statistics_msgs for this executable only. Rate is measured at the subscription when the process has one, at the publisher otherwise; inter/intra report the busier path, never the sum. Text-format logs are refused with one warning per file. Asked for on ROS Discourse (topic 57637).
• core metrics_mapper and line_follower: the window-to-sample mapping and the incremental line reader behind the bridge, ROS-free so both run in the standalone gtest lane (16 tests). test/integration/test_bridge.py covers the message shape, tail-follow of a window appended after startup, the Hz unit and text-log refusal (3 tests).
• apt install instructions. The rosdistro entries for humble, jazzy and kilted merged on 2026-08-30 and the build farm has published ros-<distro>-ros2-pulse 0.4.1-2 to ros2-testing; README now documents apt install plus how to enable the testing repository before the next sync to the main ROS 2 repository.
• Social preview card (docs/assets/social-preview.png). Used as the repository’s social preview and as og:image / twitter:image on the docs site, so links to either render a real card instead of GitHub’s default. The docs metadata lives in site/overrides/main.html, wired up with theme.custom_dir.

[0.4.1] - 2026-08-23

Packaging fix for the apt release; no behaviour change.

Fixed

• babeltrace is no longer a declared test_depend: it has no rosdep mapping for RHEL, and humble, jazzy and kilted all release on RHEL, so the declaration blocked bloom’s RPM generation and would have failed the build farm. The LTTng coexistence test already skips when no trace viewer is installed; CI installs babeltrace2 explicitly so the jazzy and kilted lanes keep running it.

Added

• Observer-effect bench (bench/run_observer_effect.sh, raw under bench/out/observer_effect/): what ros2 topic hz / echo cost and what they do to the topic. N=10 rotated arms on the stress farm, probe as the in-process ruler. Findings: the stock tools read the rate right (publisher held 50.000 Hz, hz within 0.3 %); they cost 7 % (hz) / 31 % (echo) of a core per watched 100 KB topic; and on an intra-process topic the watcher switches serialization on, +52 % CPU on the watched process, rcl_publish path lit in 10/10 trials. README “Why”, ALTERNATIVES and bench/RESULTS.md carry the numbers.
• Launch video source (video/): a 40 s Remotion composition: hook, the measured cost of ros2 topic hz / echo, the one-line probe, real pulse-top frames through a /scan stall, the Orin/x86 numbers, CTA. Every on-screen number lives in video/src/data.ts with the file it cites; capture_frames.py labels stall / Warns-tab frames by pixel colour so the cut does not depend on capture timing. Rendered MP4 ships as a GitHub Release asset, not in git.
• README demo GIF of pulse-top --demo (docs/assets/pulse-top-demo.gif, 30 frames through one scripted-incident loop, rendered headlessly from Textual screenshots).

Fixed

• pulse-top: a topic_rate warn for an in-range rate could read 20.04Hz > max 22.0Hz; the detail picked “> max” whenever the rate was not below min. It now names the bound actually crossed, or the bounds when neither is, and rounds the rate to one decimal. The demo’s /cmd_vel sag (18.4 ± 1.5 Hz against a 19 Hz min) overlapped the bound and fired such warns; it now sags to 14.5-17.5 Hz. Window title is pulse-top, not the class name.

File truncated at 100 lines see the full file