ROS Index
Package Summary
| Version | 0.7.0 |
| License | Apache-2.0 |
| Build type | AMENT_CMAKE |
| Use | RECOMMENDED |
Repository Summary
| Checkout URI | https://github.com/selfpatch/ros2_medkit.git |
| VCS Type | git |
| VCS Version | main |
| Last Updated | 2026-09-06 |
| Dev Status | DEVELOPED |
| Released | RELEASED |
| Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
Gateway plugin that detects silent faults in the ROS 2 graph: QoS mismatches, orphaned topics left by a name typo, dead nodes, parameter drift and unmet lifecycle expectations.
Maintainers
- bburda
Authors
No additional authors.
ros2_medkit_graph_watchdog
Gateway plugin that detects silent faults in the ROS 2 graph: failures where every node is up, nothing logs an error, and the robot is still broken.
Detectors read the graph and raise faults through a ReportFault service client on the
gateway node; the faults surface via FaultManager on the gateway /faults API. That
client, like the lifecycle subscriptions below, lives in the plugin’s own callback group
and is driven by the plugin’s own executor from the tick thread, never by the gateway’s
ROS executor.
This package carries the plugin skeleton, the central reliability gate that holds raises
until the graph has quiesced, and five detectors, qos_mismatch, orphan,
param_drift, lifecycle_expectation and node_death. Two silent-fault classes remain
undelivered, GRAPH_TF_STALE and GRAPH_LATENCY_BUDGET; they land in follow-up changes,
each against its own issue, and their fault codes are already reserved in the frozen
GRAPH_* namespace (see “Fault codes”).
Build
With the gateway built and sourced:
colcon build --packages-select ros2_medkit_graph_watchdog
colcon test --packages-select ros2_medkit_graph_watchdog && colcon test-result --verbose
Load into the gateway
ros2 run ros2_medkit_gateway gateway_node --ros-args \
-p plugins:="[graph_watchdog]" \
-p "plugins.graph_watchdog.path:=$(ros2 pkg prefix ros2_medkit_graph_watchdog)/lib/ros2_medkit_graph_watchdog/libros2_medkit_graph_watchdog.so"
Configuration (plugins.graph_watchdog.<key>)
| Key | Type | Default | Meaning |
|---|---|---|---|
tick_interval_ms |
int | 1000 |
Detector tick cadence. Accepted range 1..INT_MAX; the check runs on the wide integer, so a value past INT_MAX is rejected rather than truncated back into the band. |
warmup_cycles |
int | 5 |
An entity must be continuously present for this many ticks before it arms. A mid-run restart re-warms it. Enforced centrally, see “Reliability (bringup-quiesce)” below. Accepted range 0..INT_MAX, checked on the wide integer. |
prune_grace |
int | 60 |
Default injected into every detector’s own config before configure(); a per-detector detectors.<id>.prune_grace overrides it. Used by detectors that keep per-key bookkeeping. Accepted range 0..INT_MAX here, checked on the wide integer; each detector re-validates it against its own documented range before using it. |
detectors.<id>.mode |
string | bool | raise |
raise = push faults; advisory = observe, do not push; off = disabled. Bare off/no (YAML booleans) also disable; bare on/yes mean raise. |
detectors.<id>.<field> |
any | - | Per-detector thresholds, passed to that detector’s configure(). |
Nested delivery. Keys are written dotted in YAML or --ros-args, and the gateway
delivers them to the plugin as a nested object. A bare off is a YAML 1.1 boolean, so
mode: off arrives as false, not the string "off"; both forms disable the detector.
Unknown keys are reported. Anything under detectors.<id> that the detector does not
read produces one startup warning naming the key and listing the ones that exist, so a
typo like allow_list: for allowlist: cannot silently do nothing. A typo in the detector
id itself is warned about the same way, with the registered ids listed.
lifecycle_expectation keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. |
require_active |
string[] | [] |
Node names that must be in the active lifecycle state, each matched against a live node by its App::id, its full FQN (/ns/name), or the bare leaf of that FQN. A bare name matches that node in EVERY namespace (a fleet-wide “all controller_servers must be active”); use a full FQN to pin one robot’s. Both the bare and the FQN form match against the node’s stable FQN, so they survive the App::id renaming that a same-bare-name collision triggers; an entry written as an App::id also works, but can stop matching on a multi-robot graph for exactly that reason. Empty = the detector checks nothing and emits nothing. A require_active that is not a string array warns and is ignored; an empty-string entry warns and is skipped - never dropped silently, since the operator would go on believing the node is covered. |
grace |
int | 5 |
Consecutive not-active ticks a required node tolerates before being reported inactive - bringup time for a managed node to reach active (configure + activate) before the expectation is enforced. Counted per NODE, not per matching entry: a node named by both a bare-name and a full-FQN entry still advances its streak once per tick, so mixing the two documented forms cannot halve the grace that was configured. An ALREADY-CONFIRMED streak keeps its content while the node is ABSENT, so a node that leaves the graph after being confirmed stays confirmed; a streak that has not yet crossed grace on a node the presence detector could also report (it was armed at least once) is HELD rather than advanced while absent, so a node is never confirmed out of ticks gathered while nobody could observe it - only a PRESENT tick may still cross grace for that node. A node that was NEVER armed gets no such hold: node_death can never track it either, so absence keeps advancing its streak too, the same as a present tick would. Accepted range 0..300 - five minutes at the shipped 1 s cadence, and already an extravagant allowance for a managed node to reach active. The upper end is a real bound rather than a formality: grace decides how long a PRESENT node may go unreported, and how long a node returning from a departure still inactive takes to (re-)mature, so at the old maximum of INT_MAX - 1 that PRESENT-side silence lasted about 24 days at the shipped cadence with no warning. Whether it also bounds how long a node that leaves the graph BELOW grace sits unsettled depends on the same split: for an armed node that lasts for as long as the node stays gone, independent of grace (see “Bounded by evidence, not by age” below); for a never-armed one grace bounds that too, since absence matures it within grace + absence_grace + 1 ticks either way. The check runs on the wide integer, so a value that only fits after truncation is rejected rather than silently turned into a hair-trigger. Anything outside the range warns and keeps the default. |
prune_grace |
int | 60 |
Consecutive ticks an IDLE tracked node - both clocks at zero and no matured ownership, so nothing to lose - may stay ABSENT from the graph before its bookkeeping is reclaimed. A node carrying evidence is never reclaimed by age at all, however long it stays gone, so this key cannot erase a fault’s own state; the map is bounded instead by tracked_node_cap (see “Bounded by evidence, not by age”). Injected for every detector at plugin scope and overridable per detector; a value outside 0..3600 warns and this detector keeps its own default of 60. The range check runs on the wide integer, so an out-of-int-range value is rejected rather than truncated. The value is used as written - there is no grace + 1 clamp, because there is no longer anything for one to protect. |
tracked_node_cap |
int | 512 |
The most nodes this detector keeps bookkeeping for at once. It is what bounds the map, since evidence is never reclaimed by age (see “Bounded by evidence, not by age”): at the cap, idle entries are reclaimed first, then entries for DEPARTED nodes are collapsed into a count, and only if every tracked node is PRESENT and carrying evidence is a newly matched node refused - which withholds GRAPH_NODE_INACTIVE’s clear and is reported both in the log and on GET /x-medkit-watchdog. 512 is comfortably above every node in a realistic graph (a full Nav2 stack plus perception is roughly a hundred nodes; a ten-robot fleet sharing one domain a few hundred), so raising it is only needed where require_active legitimately matches more PRESENT nodes than that. Accepted range 1..16384 - 16384 is about 8 MB of bookkeeping at roughly 500 bytes per tracked node, and a deployment needing more distinct required-node identities alive at once has identity churn rather than a large fleet. Anything outside the range, including 0 (a cap of nothing would mean checking nothing), warns and keeps the default; the check runs on the wide integer, so an out-of-int-range value is rejected rather than truncated. |
node_death keys
Zero-config, unlike every detector above: there is no require_active-style list of nodes
to watch, because every armed App in the graph is a candidate.
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. |
miss_grace |
int | 2 |
Consecutive missed ticks a tracked node tolerates before being reported dead - a death is reported once misses EXCEED this, i.e. after miss_grace + 1 ticks. Accepted range 0..3600, and additionally floored to whatever wall-clock window the configured tick_interval_ms needs (see “The wall-clock floor” below); a value below that floor is silently raised to it, with a warning naming the ms window it actually spans. |
prune_grace |
int | 60 |
Plugin-injected default (overridable here); how long a key being DURABLY suppressed (see “Suppression” below) may sit unreported before its bookkeeping is reclaimed. Accepted range 0..3600. The value actually used is max(prune_grace, miss_grace + 1), a silent internal floor rather than a rejection: without it, a durably-suppressed key could be reclaimed the very tick it would first have become eligible to report, losing the report rather than merely suppressing it. |
allowlist |
string[] | [] |
Node identities never reported dead. A dead key matches if it - or a name the same node is also known by - is present verbatim: the key’s own fqn, its bare leaf, or (captured while the node was still present) its App::id. Has no effect unless named in suppress; naming it without a configured list is a no-op. |
suppress |
string[] | [] |
Suppression mechanisms to activate: "allowlist" and "lifecycle". Any other entry warns as unrecognized and is ignored; a non-string or non-array entry warns the same way. |
tracked_node_cap |
int | 512 |
The most DEPARTED (not-yet-confirmed-dead or already-collapsed) identities this detector keeps individual bookkeeping for at once - a present/armed node is never counted against it and never refused tracking. Accepted range 1..16384; 0 is refused rather than clamped, since a cap of nothing would mean tracking nothing. See “Bounded by evidence, not by age” below for what it actually bounds and why, unlike the sibling detector’s identical-looking cap, a newcomer here is never the thing refused. |
tick_interval_ms is not an own key of this detector, but this is the one detector that
reads the plugin-injected value directly, to compute miss_grace’s wall-clock floor below.
Liveness, and why membership in the snapshot is not enough. App::is_online is what
this detector tracks, never mere presence in the entity snapshot. In runtime-only discovery
the two happen to coincide - a dead node’s App leaves the snapshot entirely - but a manifest
keeps a bound App present with only is_online cleared once its ROS binding disappears, and
hybrid discovery inherits that same shape; counting snapshot membership alone would make a
manifest-declared node immortal. A managed lifecycle node that merely deactivates keeps
is_online: true - its process is still running, only its ROS 2 lifecycle state changed -
so a deactivation is never mistaken for a death either; that is lifecycle_expectation’s
concern, not this one’s (see “The boundary with lifecycle_expectation” below).
Tracking is keyed on the STABLE fqn (App::effective_fqn()), never App::id: an id is
recomputed every sweep and only gains a namespace prefix once a same-bare-name collision
currently exists anywhere in the graph, so a live node’s id can change out from under a key
built from it.
What is never tracked.
| Excluded | Why |
|---|---|
Peer-aggregated apps (app.source starting peer:) |
A peer app carries no ROS binding of its own, so effective_fqn() is empty for every one of them; tracking them would collapse an entire peer fleet onto one "" key, and one online peer app would mask every other peer app’s departure. |
_ros2cli_<pid> nodes |
Every ros2 CLI invocation (ros2 topic echo, ros2 param get, …) spins up a real, short-lived node under this prefix - rcl’s own hidden-node naming convention, matched structurally rather than guessed at. Letting these through would accumulate one permanently “dead” entry per CLI invocation for the life of the gateway. |
| An App that never comes online | A key is admitted to tracking only once the reliability gate has said this detector OWNS its departure - armed at least once, and on one of the two GROUNDS ownership can rest on (a lifecycle state read as active, or no lifecycle to read at all; the third case, a state nobody could read, is admitted PROVISIONALLY and is described in the row below). A manifest App whose binding never starts is never armed and so can never be falsely called dead; a node still inside its warmup_cycles window gets the identical protection. Once tracked, though, a node’s continued life-or-death judgement rests on PRESENCE alone, not on staying armed - a tracked node that later goes lifecycle-inactive is not thereby mistaken for dead. |
| A managed App whose lifecycle state has not been read YET | Not tracked, and only for as long as that is still true. The gate’s own raise permission is answered PERMISSIVELY for a managed node whose GetState has not answered (LifecycleWatcher::node_ok()) - without that, qos_mismatch, orphan and param_drift would all fall silent about a node whose lifecycle service is broken. Permission is not knowledge, though, so tracking is withheld while a measurement may still arrive. LifecycleWatcher charges a GetState re-seed budget per node, and only for a read that actually ran, so “still asking” is a fact rather than a guess. Once that budget is spent the node is tracked here after all: nothing will ASK again, and lifecycle_expectation looks at nothing unless an operator named the node in require_active (empty by default), so withholding it any longer would mean the death is reported by nobody. That admission is PROVISIONAL, not final. The ~/transition_event subscription outlives the seed budget, so a label can still arrive without being asked for, and one that reads non-active says the node was the lifecycle detector’s all along - so the presence detector hands it back on the spot, while it is still alive. Waiting for the budget alone would only have delayed the wrong report, not prevented it. |
File truncated at 100 lines see the full file
CHANGELOG
Changelog for package ros2_medkit_graph_watchdog
0.7.0 (2026-08-27)
- Initial release of the package: a gateway plugin that raises faults
for silent failures in the ROS 2 graph - problems that leave every
node alive and every topic present, so nothing in the stack reports
them today. The package carries the plugin skeleton, a central
reliability gate that holds raises until the graph has quiesced, and
a detector registry. Detectors are configured under
plugins.graph_watchdog.<key>, each with araise/advisory/offmode, and an unknown key underdetectors.<id>is reported rather than silently ignored, so a typo cannot quietly disable a check (#571) -
qos_mismatchdetector: raisesGRAPH_QOS_MISMATCHfor a publisher and subscriber whose QoS profiles cannot match, which leaves the connection silently unestablished (#571) -
orphandetector: raisesGRAPH_ORPHANfor a one-sided topic - a publisher with no subscriber, or the reverse - but only when a complementary near-miss counterpart exists: same message type, a namespace and leaf within the configured edit distance, and the opposite side. A lone unsubscribed topic raises nothing. The pair is the signature of a remap or a topic-name typo, which is what the detector is for (#578) -
param_driftdetector: raisesGRAPH_PARAM_DRIFTwhen a node's live parameter value diverges from the declared expectation (#580) -
lifecycle_expectationdetector: raisesGRAPH_NODE_INACTIVEfor a node named inrequire_activethat is not in theactivelifecycle state once itsgracewindow has passed. It raises two further codes for the cases where the state could not be established at all:GRAPH_NODE_UNREADABLEwhen the lifecycle service does not answer, andGRAPH_NODE_NOT_MANAGEDwhen the named node has no lifecycle to read. These are distinct codes on the wire, so a consumer filtering onGRAPH_NODE_INACTIVEalone sees neither unmeasured case. A node is matched by itsApp::id, its full FQN, or the bare leaf of that FQN, and the grace streak is counted per node, so naming one node in two documented forms cannot halve the grace that was configured (#587) -
node_deathdetector: raisesGRAPH_NODE_DISAPPEAREDfor a node that leaves the graph when nothing else is left to report it, with a suppression framework that decides ownership of a departure from knowledge rather than from silence. A node whose lifecycle state was never readable is admitted only provisionally, and is handed back the moment a label arrives saying the departure belonged to the lifecycle detector instead. Zero-config: there is no list of nodes to maintain (#625, #624) - Two silent-fault classes are not delivered in this release.
GRAPH_TF_STALEandGRAPH_LATENCY_BUDGEThave their fault codes reserved in the frozenGRAPH_*namespace and land in later changes, each against its own issue - The plugin keeps its ROS entities off the gateway executor, so an entity created and destroyed while the gateway runs cannot have its destructor run on an executor thread concurrently with a create on the same node
- Contributors: \@bburda
Package Dependencies
System Dependencies
Dependant Packages
No known dependants.
Launch files
No launch files found
Messages
No message files found.
Services
No service files found
Plugins
No plugins found.
Recent questions tagged ros2_medkit_graph_watchdog at Robotics Stack Exchange
Package Summary
| Version | 0.7.0 |
| License | Apache-2.0 |
| Build type | AMENT_CMAKE |
| Use | RECOMMENDED |
Repository Summary
| Checkout URI | https://github.com/selfpatch/ros2_medkit.git |
| VCS Type | git |
| VCS Version | main |
| Last Updated | 2026-09-06 |
| Dev Status | DEVELOPED |
| Released | RELEASED |
| Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
Gateway plugin that detects silent faults in the ROS 2 graph: QoS mismatches, orphaned topics left by a name typo, dead nodes, parameter drift and unmet lifecycle expectations.
Maintainers
- bburda
Authors
No additional authors.
ros2_medkit_graph_watchdog
Gateway plugin that detects silent faults in the ROS 2 graph: failures where every node is up, nothing logs an error, and the robot is still broken.
Detectors read the graph and raise faults through a ReportFault service client on the
gateway node; the faults surface via FaultManager on the gateway /faults API. That
client, like the lifecycle subscriptions below, lives in the plugin’s own callback group
and is driven by the plugin’s own executor from the tick thread, never by the gateway’s
ROS executor.
This package carries the plugin skeleton, the central reliability gate that holds raises
until the graph has quiesced, and five detectors, qos_mismatch, orphan,
param_drift, lifecycle_expectation and node_death. Two silent-fault classes remain
undelivered, GRAPH_TF_STALE and GRAPH_LATENCY_BUDGET; they land in follow-up changes,
each against its own issue, and their fault codes are already reserved in the frozen
GRAPH_* namespace (see “Fault codes”).
Build
With the gateway built and sourced:
colcon build --packages-select ros2_medkit_graph_watchdog
colcon test --packages-select ros2_medkit_graph_watchdog && colcon test-result --verbose
Load into the gateway
ros2 run ros2_medkit_gateway gateway_node --ros-args \
-p plugins:="[graph_watchdog]" \
-p "plugins.graph_watchdog.path:=$(ros2 pkg prefix ros2_medkit_graph_watchdog)/lib/ros2_medkit_graph_watchdog/libros2_medkit_graph_watchdog.so"
Configuration (plugins.graph_watchdog.<key>)
| Key | Type | Default | Meaning |
|---|---|---|---|
tick_interval_ms |
int | 1000 |
Detector tick cadence. Accepted range 1..INT_MAX; the check runs on the wide integer, so a value past INT_MAX is rejected rather than truncated back into the band. |
warmup_cycles |
int | 5 |
An entity must be continuously present for this many ticks before it arms. A mid-run restart re-warms it. Enforced centrally, see “Reliability (bringup-quiesce)” below. Accepted range 0..INT_MAX, checked on the wide integer. |
prune_grace |
int | 60 |
Default injected into every detector’s own config before configure(); a per-detector detectors.<id>.prune_grace overrides it. Used by detectors that keep per-key bookkeeping. Accepted range 0..INT_MAX here, checked on the wide integer; each detector re-validates it against its own documented range before using it. |
detectors.<id>.mode |
string | bool | raise |
raise = push faults; advisory = observe, do not push; off = disabled. Bare off/no (YAML booleans) also disable; bare on/yes mean raise. |
detectors.<id>.<field> |
any | - | Per-detector thresholds, passed to that detector’s configure(). |
Nested delivery. Keys are written dotted in YAML or --ros-args, and the gateway
delivers them to the plugin as a nested object. A bare off is a YAML 1.1 boolean, so
mode: off arrives as false, not the string "off"; both forms disable the detector.
Unknown keys are reported. Anything under detectors.<id> that the detector does not
read produces one startup warning naming the key and listing the ones that exist, so a
typo like allow_list: for allowlist: cannot silently do nothing. A typo in the detector
id itself is warned about the same way, with the registered ids listed.
lifecycle_expectation keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. |
require_active |
string[] | [] |
Node names that must be in the active lifecycle state, each matched against a live node by its App::id, its full FQN (/ns/name), or the bare leaf of that FQN. A bare name matches that node in EVERY namespace (a fleet-wide “all controller_servers must be active”); use a full FQN to pin one robot’s. Both the bare and the FQN form match against the node’s stable FQN, so they survive the App::id renaming that a same-bare-name collision triggers; an entry written as an App::id also works, but can stop matching on a multi-robot graph for exactly that reason. Empty = the detector checks nothing and emits nothing. A require_active that is not a string array warns and is ignored; an empty-string entry warns and is skipped - never dropped silently, since the operator would go on believing the node is covered. |
grace |
int | 5 |
Consecutive not-active ticks a required node tolerates before being reported inactive - bringup time for a managed node to reach active (configure + activate) before the expectation is enforced. Counted per NODE, not per matching entry: a node named by both a bare-name and a full-FQN entry still advances its streak once per tick, so mixing the two documented forms cannot halve the grace that was configured. An ALREADY-CONFIRMED streak keeps its content while the node is ABSENT, so a node that leaves the graph after being confirmed stays confirmed; a streak that has not yet crossed grace on a node the presence detector could also report (it was armed at least once) is HELD rather than advanced while absent, so a node is never confirmed out of ticks gathered while nobody could observe it - only a PRESENT tick may still cross grace for that node. A node that was NEVER armed gets no such hold: node_death can never track it either, so absence keeps advancing its streak too, the same as a present tick would. Accepted range 0..300 - five minutes at the shipped 1 s cadence, and already an extravagant allowance for a managed node to reach active. The upper end is a real bound rather than a formality: grace decides how long a PRESENT node may go unreported, and how long a node returning from a departure still inactive takes to (re-)mature, so at the old maximum of INT_MAX - 1 that PRESENT-side silence lasted about 24 days at the shipped cadence with no warning. Whether it also bounds how long a node that leaves the graph BELOW grace sits unsettled depends on the same split: for an armed node that lasts for as long as the node stays gone, independent of grace (see “Bounded by evidence, not by age” below); for a never-armed one grace bounds that too, since absence matures it within grace + absence_grace + 1 ticks either way. The check runs on the wide integer, so a value that only fits after truncation is rejected rather than silently turned into a hair-trigger. Anything outside the range warns and keeps the default. |
prune_grace |
int | 60 |
Consecutive ticks an IDLE tracked node - both clocks at zero and no matured ownership, so nothing to lose - may stay ABSENT from the graph before its bookkeeping is reclaimed. A node carrying evidence is never reclaimed by age at all, however long it stays gone, so this key cannot erase a fault’s own state; the map is bounded instead by tracked_node_cap (see “Bounded by evidence, not by age”). Injected for every detector at plugin scope and overridable per detector; a value outside 0..3600 warns and this detector keeps its own default of 60. The range check runs on the wide integer, so an out-of-int-range value is rejected rather than truncated. The value is used as written - there is no grace + 1 clamp, because there is no longer anything for one to protect. |
tracked_node_cap |
int | 512 |
The most nodes this detector keeps bookkeeping for at once. It is what bounds the map, since evidence is never reclaimed by age (see “Bounded by evidence, not by age”): at the cap, idle entries are reclaimed first, then entries for DEPARTED nodes are collapsed into a count, and only if every tracked node is PRESENT and carrying evidence is a newly matched node refused - which withholds GRAPH_NODE_INACTIVE’s clear and is reported both in the log and on GET /x-medkit-watchdog. 512 is comfortably above every node in a realistic graph (a full Nav2 stack plus perception is roughly a hundred nodes; a ten-robot fleet sharing one domain a few hundred), so raising it is only needed where require_active legitimately matches more PRESENT nodes than that. Accepted range 1..16384 - 16384 is about 8 MB of bookkeeping at roughly 500 bytes per tracked node, and a deployment needing more distinct required-node identities alive at once has identity churn rather than a large fleet. Anything outside the range, including 0 (a cap of nothing would mean checking nothing), warns and keeps the default; the check runs on the wide integer, so an out-of-int-range value is rejected rather than truncated. |
node_death keys
Zero-config, unlike every detector above: there is no require_active-style list of nodes
to watch, because every armed App in the graph is a candidate.
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. |
miss_grace |
int | 2 |
Consecutive missed ticks a tracked node tolerates before being reported dead - a death is reported once misses EXCEED this, i.e. after miss_grace + 1 ticks. Accepted range 0..3600, and additionally floored to whatever wall-clock window the configured tick_interval_ms needs (see “The wall-clock floor” below); a value below that floor is silently raised to it, with a warning naming the ms window it actually spans. |
prune_grace |
int | 60 |
Plugin-injected default (overridable here); how long a key being DURABLY suppressed (see “Suppression” below) may sit unreported before its bookkeeping is reclaimed. Accepted range 0..3600. The value actually used is max(prune_grace, miss_grace + 1), a silent internal floor rather than a rejection: without it, a durably-suppressed key could be reclaimed the very tick it would first have become eligible to report, losing the report rather than merely suppressing it. |
allowlist |
string[] | [] |
Node identities never reported dead. A dead key matches if it - or a name the same node is also known by - is present verbatim: the key’s own fqn, its bare leaf, or (captured while the node was still present) its App::id. Has no effect unless named in suppress; naming it without a configured list is a no-op. |
suppress |
string[] | [] |
Suppression mechanisms to activate: "allowlist" and "lifecycle". Any other entry warns as unrecognized and is ignored; a non-string or non-array entry warns the same way. |
tracked_node_cap |
int | 512 |
The most DEPARTED (not-yet-confirmed-dead or already-collapsed) identities this detector keeps individual bookkeeping for at once - a present/armed node is never counted against it and never refused tracking. Accepted range 1..16384; 0 is refused rather than clamped, since a cap of nothing would mean tracking nothing. See “Bounded by evidence, not by age” below for what it actually bounds and why, unlike the sibling detector’s identical-looking cap, a newcomer here is never the thing refused. |
tick_interval_ms is not an own key of this detector, but this is the one detector that
reads the plugin-injected value directly, to compute miss_grace’s wall-clock floor below.
Liveness, and why membership in the snapshot is not enough. App::is_online is what
this detector tracks, never mere presence in the entity snapshot. In runtime-only discovery
the two happen to coincide - a dead node’s App leaves the snapshot entirely - but a manifest
keeps a bound App present with only is_online cleared once its ROS binding disappears, and
hybrid discovery inherits that same shape; counting snapshot membership alone would make a
manifest-declared node immortal. A managed lifecycle node that merely deactivates keeps
is_online: true - its process is still running, only its ROS 2 lifecycle state changed -
so a deactivation is never mistaken for a death either; that is lifecycle_expectation’s
concern, not this one’s (see “The boundary with lifecycle_expectation” below).
Tracking is keyed on the STABLE fqn (App::effective_fqn()), never App::id: an id is
recomputed every sweep and only gains a namespace prefix once a same-bare-name collision
currently exists anywhere in the graph, so a live node’s id can change out from under a key
built from it.
What is never tracked.
| Excluded | Why |
|---|---|
Peer-aggregated apps (app.source starting peer:) |
A peer app carries no ROS binding of its own, so effective_fqn() is empty for every one of them; tracking them would collapse an entire peer fleet onto one "" key, and one online peer app would mask every other peer app’s departure. |
_ros2cli_<pid> nodes |
Every ros2 CLI invocation (ros2 topic echo, ros2 param get, …) spins up a real, short-lived node under this prefix - rcl’s own hidden-node naming convention, matched structurally rather than guessed at. Letting these through would accumulate one permanently “dead” entry per CLI invocation for the life of the gateway. |
| An App that never comes online | A key is admitted to tracking only once the reliability gate has said this detector OWNS its departure - armed at least once, and on one of the two GROUNDS ownership can rest on (a lifecycle state read as active, or no lifecycle to read at all; the third case, a state nobody could read, is admitted PROVISIONALLY and is described in the row below). A manifest App whose binding never starts is never armed and so can never be falsely called dead; a node still inside its warmup_cycles window gets the identical protection. Once tracked, though, a node’s continued life-or-death judgement rests on PRESENCE alone, not on staying armed - a tracked node that later goes lifecycle-inactive is not thereby mistaken for dead. |
| A managed App whose lifecycle state has not been read YET | Not tracked, and only for as long as that is still true. The gate’s own raise permission is answered PERMISSIVELY for a managed node whose GetState has not answered (LifecycleWatcher::node_ok()) - without that, qos_mismatch, orphan and param_drift would all fall silent about a node whose lifecycle service is broken. Permission is not knowledge, though, so tracking is withheld while a measurement may still arrive. LifecycleWatcher charges a GetState re-seed budget per node, and only for a read that actually ran, so “still asking” is a fact rather than a guess. Once that budget is spent the node is tracked here after all: nothing will ASK again, and lifecycle_expectation looks at nothing unless an operator named the node in require_active (empty by default), so withholding it any longer would mean the death is reported by nobody. That admission is PROVISIONAL, not final. The ~/transition_event subscription outlives the seed budget, so a label can still arrive without being asked for, and one that reads non-active says the node was the lifecycle detector’s all along - so the presence detector hands it back on the spot, while it is still alive. Waiting for the budget alone would only have delayed the wrong report, not prevented it. |
File truncated at 100 lines see the full file
CHANGELOG
Changelog for package ros2_medkit_graph_watchdog
0.7.0 (2026-08-27)
- Initial release of the package: a gateway plugin that raises faults
for silent failures in the ROS 2 graph - problems that leave every
node alive and every topic present, so nothing in the stack reports
them today. The package carries the plugin skeleton, a central
reliability gate that holds raises until the graph has quiesced, and
a detector registry. Detectors are configured under
plugins.graph_watchdog.<key>, each with araise/advisory/offmode, and an unknown key underdetectors.<id>is reported rather than silently ignored, so a typo cannot quietly disable a check (#571) -
qos_mismatchdetector: raisesGRAPH_QOS_MISMATCHfor a publisher and subscriber whose QoS profiles cannot match, which leaves the connection silently unestablished (#571) -
orphandetector: raisesGRAPH_ORPHANfor a one-sided topic - a publisher with no subscriber, or the reverse - but only when a complementary near-miss counterpart exists: same message type, a namespace and leaf within the configured edit distance, and the opposite side. A lone unsubscribed topic raises nothing. The pair is the signature of a remap or a topic-name typo, which is what the detector is for (#578) -
param_driftdetector: raisesGRAPH_PARAM_DRIFTwhen a node's live parameter value diverges from the declared expectation (#580) -
lifecycle_expectationdetector: raisesGRAPH_NODE_INACTIVEfor a node named inrequire_activethat is not in theactivelifecycle state once itsgracewindow has passed. It raises two further codes for the cases where the state could not be established at all:GRAPH_NODE_UNREADABLEwhen the lifecycle service does not answer, andGRAPH_NODE_NOT_MANAGEDwhen the named node has no lifecycle to read. These are distinct codes on the wire, so a consumer filtering onGRAPH_NODE_INACTIVEalone sees neither unmeasured case. A node is matched by itsApp::id, its full FQN, or the bare leaf of that FQN, and the grace streak is counted per node, so naming one node in two documented forms cannot halve the grace that was configured (#587) -
node_deathdetector: raisesGRAPH_NODE_DISAPPEAREDfor a node that leaves the graph when nothing else is left to report it, with a suppression framework that decides ownership of a departure from knowledge rather than from silence. A node whose lifecycle state was never readable is admitted only provisionally, and is handed back the moment a label arrives saying the departure belonged to the lifecycle detector instead. Zero-config: there is no list of nodes to maintain (#625, #624) - Two silent-fault classes are not delivered in this release.
GRAPH_TF_STALEandGRAPH_LATENCY_BUDGEThave their fault codes reserved in the frozenGRAPH_*namespace and land in later changes, each against its own issue - The plugin keeps its ROS entities off the gateway executor, so an entity created and destroyed while the gateway runs cannot have its destructor run on an executor thread concurrently with a create on the same node
- Contributors: \@bburda
Package Dependencies
System Dependencies
Dependant Packages
No known dependants.
Launch files
No launch files found
Messages
No message files found.
Services
No service files found
Plugins
No plugins found.
Recent questions tagged ros2_medkit_graph_watchdog at Robotics Stack Exchange
No version for distro kilted showing humble. Known supported distros are highlighted in the buttons above.
Package Summary
| Version | 0.7.0 |
| License | Apache-2.0 |
| Build type | AMENT_CMAKE |
| Use | RECOMMENDED |
Repository Summary
| Checkout URI | https://github.com/selfpatch/ros2_medkit.git |
| VCS Type | git |
| VCS Version | main |
| Last Updated | 2026-09-06 |
| Dev Status | DEVELOPED |
| Released | RELEASED |
| Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
Gateway plugin that detects silent faults in the ROS 2 graph: QoS mismatches, orphaned topics left by a name typo, dead nodes, parameter drift and unmet lifecycle expectations.
Maintainers
- bburda
Authors
No additional authors.
ros2_medkit_graph_watchdog
Gateway plugin that detects silent faults in the ROS 2 graph: failures where every node is up, nothing logs an error, and the robot is still broken.
Detectors read the graph and raise faults through a ReportFault service client on the
gateway node; the faults surface via FaultManager on the gateway /faults API. That
client, like the lifecycle subscriptions below, lives in the plugin’s own callback group
and is driven by the plugin’s own executor from the tick thread, never by the gateway’s
ROS executor.
This package carries the plugin skeleton, the central reliability gate that holds raises
until the graph has quiesced, and five detectors, qos_mismatch, orphan,
param_drift, lifecycle_expectation and node_death. Two silent-fault classes remain
undelivered, GRAPH_TF_STALE and GRAPH_LATENCY_BUDGET; they land in follow-up changes,
each against its own issue, and their fault codes are already reserved in the frozen
GRAPH_* namespace (see “Fault codes”).
Build
With the gateway built and sourced:
colcon build --packages-select ros2_medkit_graph_watchdog
colcon test --packages-select ros2_medkit_graph_watchdog && colcon test-result --verbose
Load into the gateway
ros2 run ros2_medkit_gateway gateway_node --ros-args \
-p plugins:="[graph_watchdog]" \
-p "plugins.graph_watchdog.path:=$(ros2 pkg prefix ros2_medkit_graph_watchdog)/lib/ros2_medkit_graph_watchdog/libros2_medkit_graph_watchdog.so"
Configuration (plugins.graph_watchdog.<key>)
| Key | Type | Default | Meaning |
|---|---|---|---|
tick_interval_ms |
int | 1000 |
Detector tick cadence. Accepted range 1..INT_MAX; the check runs on the wide integer, so a value past INT_MAX is rejected rather than truncated back into the band. |
warmup_cycles |
int | 5 |
An entity must be continuously present for this many ticks before it arms. A mid-run restart re-warms it. Enforced centrally, see “Reliability (bringup-quiesce)” below. Accepted range 0..INT_MAX, checked on the wide integer. |
prune_grace |
int | 60 |
Default injected into every detector’s own config before configure(); a per-detector detectors.<id>.prune_grace overrides it. Used by detectors that keep per-key bookkeeping. Accepted range 0..INT_MAX here, checked on the wide integer; each detector re-validates it against its own documented range before using it. |
detectors.<id>.mode |
string | bool | raise |
raise = push faults; advisory = observe, do not push; off = disabled. Bare off/no (YAML booleans) also disable; bare on/yes mean raise. |
detectors.<id>.<field> |
any | - | Per-detector thresholds, passed to that detector’s configure(). |
Nested delivery. Keys are written dotted in YAML or --ros-args, and the gateway
delivers them to the plugin as a nested object. A bare off is a YAML 1.1 boolean, so
mode: off arrives as false, not the string "off"; both forms disable the detector.
Unknown keys are reported. Anything under detectors.<id> that the detector does not
read produces one startup warning naming the key and listing the ones that exist, so a
typo like allow_list: for allowlist: cannot silently do nothing. A typo in the detector
id itself is warned about the same way, with the registered ids listed.
lifecycle_expectation keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. |
require_active |
string[] | [] |
Node names that must be in the active lifecycle state, each matched against a live node by its App::id, its full FQN (/ns/name), or the bare leaf of that FQN. A bare name matches that node in EVERY namespace (a fleet-wide “all controller_servers must be active”); use a full FQN to pin one robot’s. Both the bare and the FQN form match against the node’s stable FQN, so they survive the App::id renaming that a same-bare-name collision triggers; an entry written as an App::id also works, but can stop matching on a multi-robot graph for exactly that reason. Empty = the detector checks nothing and emits nothing. A require_active that is not a string array warns and is ignored; an empty-string entry warns and is skipped - never dropped silently, since the operator would go on believing the node is covered. |
grace |
int | 5 |
Consecutive not-active ticks a required node tolerates before being reported inactive - bringup time for a managed node to reach active (configure + activate) before the expectation is enforced. Counted per NODE, not per matching entry: a node named by both a bare-name and a full-FQN entry still advances its streak once per tick, so mixing the two documented forms cannot halve the grace that was configured. An ALREADY-CONFIRMED streak keeps its content while the node is ABSENT, so a node that leaves the graph after being confirmed stays confirmed; a streak that has not yet crossed grace on a node the presence detector could also report (it was armed at least once) is HELD rather than advanced while absent, so a node is never confirmed out of ticks gathered while nobody could observe it - only a PRESENT tick may still cross grace for that node. A node that was NEVER armed gets no such hold: node_death can never track it either, so absence keeps advancing its streak too, the same as a present tick would. Accepted range 0..300 - five minutes at the shipped 1 s cadence, and already an extravagant allowance for a managed node to reach active. The upper end is a real bound rather than a formality: grace decides how long a PRESENT node may go unreported, and how long a node returning from a departure still inactive takes to (re-)mature, so at the old maximum of INT_MAX - 1 that PRESENT-side silence lasted about 24 days at the shipped cadence with no warning. Whether it also bounds how long a node that leaves the graph BELOW grace sits unsettled depends on the same split: for an armed node that lasts for as long as the node stays gone, independent of grace (see “Bounded by evidence, not by age” below); for a never-armed one grace bounds that too, since absence matures it within grace + absence_grace + 1 ticks either way. The check runs on the wide integer, so a value that only fits after truncation is rejected rather than silently turned into a hair-trigger. Anything outside the range warns and keeps the default. |
prune_grace |
int | 60 |
Consecutive ticks an IDLE tracked node - both clocks at zero and no matured ownership, so nothing to lose - may stay ABSENT from the graph before its bookkeeping is reclaimed. A node carrying evidence is never reclaimed by age at all, however long it stays gone, so this key cannot erase a fault’s own state; the map is bounded instead by tracked_node_cap (see “Bounded by evidence, not by age”). Injected for every detector at plugin scope and overridable per detector; a value outside 0..3600 warns and this detector keeps its own default of 60. The range check runs on the wide integer, so an out-of-int-range value is rejected rather than truncated. The value is used as written - there is no grace + 1 clamp, because there is no longer anything for one to protect. |
tracked_node_cap |
int | 512 |
The most nodes this detector keeps bookkeeping for at once. It is what bounds the map, since evidence is never reclaimed by age (see “Bounded by evidence, not by age”): at the cap, idle entries are reclaimed first, then entries for DEPARTED nodes are collapsed into a count, and only if every tracked node is PRESENT and carrying evidence is a newly matched node refused - which withholds GRAPH_NODE_INACTIVE’s clear and is reported both in the log and on GET /x-medkit-watchdog. 512 is comfortably above every node in a realistic graph (a full Nav2 stack plus perception is roughly a hundred nodes; a ten-robot fleet sharing one domain a few hundred), so raising it is only needed where require_active legitimately matches more PRESENT nodes than that. Accepted range 1..16384 - 16384 is about 8 MB of bookkeeping at roughly 500 bytes per tracked node, and a deployment needing more distinct required-node identities alive at once has identity churn rather than a large fleet. Anything outside the range, including 0 (a cap of nothing would mean checking nothing), warns and keeps the default; the check runs on the wide integer, so an out-of-int-range value is rejected rather than truncated. |
node_death keys
Zero-config, unlike every detector above: there is no require_active-style list of nodes
to watch, because every armed App in the graph is a candidate.
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. |
miss_grace |
int | 2 |
Consecutive missed ticks a tracked node tolerates before being reported dead - a death is reported once misses EXCEED this, i.e. after miss_grace + 1 ticks. Accepted range 0..3600, and additionally floored to whatever wall-clock window the configured tick_interval_ms needs (see “The wall-clock floor” below); a value below that floor is silently raised to it, with a warning naming the ms window it actually spans. |
prune_grace |
int | 60 |
Plugin-injected default (overridable here); how long a key being DURABLY suppressed (see “Suppression” below) may sit unreported before its bookkeeping is reclaimed. Accepted range 0..3600. The value actually used is max(prune_grace, miss_grace + 1), a silent internal floor rather than a rejection: without it, a durably-suppressed key could be reclaimed the very tick it would first have become eligible to report, losing the report rather than merely suppressing it. |
allowlist |
string[] | [] |
Node identities never reported dead. A dead key matches if it - or a name the same node is also known by - is present verbatim: the key’s own fqn, its bare leaf, or (captured while the node was still present) its App::id. Has no effect unless named in suppress; naming it without a configured list is a no-op. |
suppress |
string[] | [] |
Suppression mechanisms to activate: "allowlist" and "lifecycle". Any other entry warns as unrecognized and is ignored; a non-string or non-array entry warns the same way. |
tracked_node_cap |
int | 512 |
The most DEPARTED (not-yet-confirmed-dead or already-collapsed) identities this detector keeps individual bookkeeping for at once - a present/armed node is never counted against it and never refused tracking. Accepted range 1..16384; 0 is refused rather than clamped, since a cap of nothing would mean tracking nothing. See “Bounded by evidence, not by age” below for what it actually bounds and why, unlike the sibling detector’s identical-looking cap, a newcomer here is never the thing refused. |
tick_interval_ms is not an own key of this detector, but this is the one detector that
reads the plugin-injected value directly, to compute miss_grace’s wall-clock floor below.
Liveness, and why membership in the snapshot is not enough. App::is_online is what
this detector tracks, never mere presence in the entity snapshot. In runtime-only discovery
the two happen to coincide - a dead node’s App leaves the snapshot entirely - but a manifest
keeps a bound App present with only is_online cleared once its ROS binding disappears, and
hybrid discovery inherits that same shape; counting snapshot membership alone would make a
manifest-declared node immortal. A managed lifecycle node that merely deactivates keeps
is_online: true - its process is still running, only its ROS 2 lifecycle state changed -
so a deactivation is never mistaken for a death either; that is lifecycle_expectation’s
concern, not this one’s (see “The boundary with lifecycle_expectation” below).
Tracking is keyed on the STABLE fqn (App::effective_fqn()), never App::id: an id is
recomputed every sweep and only gains a namespace prefix once a same-bare-name collision
currently exists anywhere in the graph, so a live node’s id can change out from under a key
built from it.
What is never tracked.
| Excluded | Why |
|---|---|
Peer-aggregated apps (app.source starting peer:) |
A peer app carries no ROS binding of its own, so effective_fqn() is empty for every one of them; tracking them would collapse an entire peer fleet onto one "" key, and one online peer app would mask every other peer app’s departure. |
_ros2cli_<pid> nodes |
Every ros2 CLI invocation (ros2 topic echo, ros2 param get, …) spins up a real, short-lived node under this prefix - rcl’s own hidden-node naming convention, matched structurally rather than guessed at. Letting these through would accumulate one permanently “dead” entry per CLI invocation for the life of the gateway. |
| An App that never comes online | A key is admitted to tracking only once the reliability gate has said this detector OWNS its departure - armed at least once, and on one of the two GROUNDS ownership can rest on (a lifecycle state read as active, or no lifecycle to read at all; the third case, a state nobody could read, is admitted PROVISIONALLY and is described in the row below). A manifest App whose binding never starts is never armed and so can never be falsely called dead; a node still inside its warmup_cycles window gets the identical protection. Once tracked, though, a node’s continued life-or-death judgement rests on PRESENCE alone, not on staying armed - a tracked node that later goes lifecycle-inactive is not thereby mistaken for dead. |
| A managed App whose lifecycle state has not been read YET | Not tracked, and only for as long as that is still true. The gate’s own raise permission is answered PERMISSIVELY for a managed node whose GetState has not answered (LifecycleWatcher::node_ok()) - without that, qos_mismatch, orphan and param_drift would all fall silent about a node whose lifecycle service is broken. Permission is not knowledge, though, so tracking is withheld while a measurement may still arrive. LifecycleWatcher charges a GetState re-seed budget per node, and only for a read that actually ran, so “still asking” is a fact rather than a guess. Once that budget is spent the node is tracked here after all: nothing will ASK again, and lifecycle_expectation looks at nothing unless an operator named the node in require_active (empty by default), so withholding it any longer would mean the death is reported by nobody. That admission is PROVISIONAL, not final. The ~/transition_event subscription outlives the seed budget, so a label can still arrive without being asked for, and one that reads non-active says the node was the lifecycle detector’s all along - so the presence detector hands it back on the spot, while it is still alive. Waiting for the budget alone would only have delayed the wrong report, not prevented it. |
File truncated at 100 lines see the full file
CHANGELOG
Changelog for package ros2_medkit_graph_watchdog
0.7.0 (2026-08-27)
- Initial release of the package: a gateway plugin that raises faults
for silent failures in the ROS 2 graph - problems that leave every
node alive and every topic present, so nothing in the stack reports
them today. The package carries the plugin skeleton, a central
reliability gate that holds raises until the graph has quiesced, and
a detector registry. Detectors are configured under
plugins.graph_watchdog.<key>, each with araise/advisory/offmode, and an unknown key underdetectors.<id>is reported rather than silently ignored, so a typo cannot quietly disable a check (#571) -
qos_mismatchdetector: raisesGRAPH_QOS_MISMATCHfor a publisher and subscriber whose QoS profiles cannot match, which leaves the connection silently unestablished (#571) -
orphandetector: raisesGRAPH_ORPHANfor a one-sided topic - a publisher with no subscriber, or the reverse - but only when a complementary near-miss counterpart exists: same message type, a namespace and leaf within the configured edit distance, and the opposite side. A lone unsubscribed topic raises nothing. The pair is the signature of a remap or a topic-name typo, which is what the detector is for (#578) -
param_driftdetector: raisesGRAPH_PARAM_DRIFTwhen a node's live parameter value diverges from the declared expectation (#580) -
lifecycle_expectationdetector: raisesGRAPH_NODE_INACTIVEfor a node named inrequire_activethat is not in theactivelifecycle state once itsgracewindow has passed. It raises two further codes for the cases where the state could not be established at all:GRAPH_NODE_UNREADABLEwhen the lifecycle service does not answer, andGRAPH_NODE_NOT_MANAGEDwhen the named node has no lifecycle to read. These are distinct codes on the wire, so a consumer filtering onGRAPH_NODE_INACTIVEalone sees neither unmeasured case. A node is matched by itsApp::id, its full FQN, or the bare leaf of that FQN, and the grace streak is counted per node, so naming one node in two documented forms cannot halve the grace that was configured (#587) -
node_deathdetector: raisesGRAPH_NODE_DISAPPEAREDfor a node that leaves the graph when nothing else is left to report it, with a suppression framework that decides ownership of a departure from knowledge rather than from silence. A node whose lifecycle state was never readable is admitted only provisionally, and is handed back the moment a label arrives saying the departure belonged to the lifecycle detector instead. Zero-config: there is no list of nodes to maintain (#625, #624) - Two silent-fault classes are not delivered in this release.
GRAPH_TF_STALEandGRAPH_LATENCY_BUDGEThave their fault codes reserved in the frozenGRAPH_*namespace and land in later changes, each against its own issue - The plugin keeps its ROS entities off the gateway executor, so an entity created and destroyed while the gateway runs cannot have its destructor run on an executor thread concurrently with a create on the same node
- Contributors: \@bburda
Package Dependencies
System Dependencies
Dependant Packages
No known dependants.
Launch files
No launch files found
Messages
No message files found.
Services
No service files found
Plugins
No plugins found.
Recent questions tagged ros2_medkit_graph_watchdog at Robotics Stack Exchange
Package Summary
| Version | 0.7.0 |
| License | Apache-2.0 |
| Build type | AMENT_CMAKE |
| Use | RECOMMENDED |
Repository Summary
| Checkout URI | https://github.com/selfpatch/ros2_medkit.git |
| VCS Type | git |
| VCS Version | main |
| Last Updated | 2026-09-06 |
| Dev Status | DEVELOPED |
| Released | RELEASED |
| Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
Gateway plugin that detects silent faults in the ROS 2 graph: QoS mismatches, orphaned topics left by a name typo, dead nodes, parameter drift and unmet lifecycle expectations.
Maintainers
- bburda
Authors
No additional authors.
ros2_medkit_graph_watchdog
Gateway plugin that detects silent faults in the ROS 2 graph: failures where every node is up, nothing logs an error, and the robot is still broken.
Detectors read the graph and raise faults through a ReportFault service client on the
gateway node; the faults surface via FaultManager on the gateway /faults API. That
client, like the lifecycle subscriptions below, lives in the plugin’s own callback group
and is driven by the plugin’s own executor from the tick thread, never by the gateway’s
ROS executor.
This package carries the plugin skeleton, the central reliability gate that holds raises
until the graph has quiesced, and five detectors, qos_mismatch, orphan,
param_drift, lifecycle_expectation and node_death. Two silent-fault classes remain
undelivered, GRAPH_TF_STALE and GRAPH_LATENCY_BUDGET; they land in follow-up changes,
each against its own issue, and their fault codes are already reserved in the frozen
GRAPH_* namespace (see “Fault codes”).
Build
With the gateway built and sourced:
colcon build --packages-select ros2_medkit_graph_watchdog
colcon test --packages-select ros2_medkit_graph_watchdog && colcon test-result --verbose
Load into the gateway
ros2 run ros2_medkit_gateway gateway_node --ros-args \
-p plugins:="[graph_watchdog]" \
-p "plugins.graph_watchdog.path:=$(ros2 pkg prefix ros2_medkit_graph_watchdog)/lib/ros2_medkit_graph_watchdog/libros2_medkit_graph_watchdog.so"
Configuration (plugins.graph_watchdog.<key>)
| Key | Type | Default | Meaning |
|---|---|---|---|
tick_interval_ms |
int | 1000 |
Detector tick cadence. Accepted range 1..INT_MAX; the check runs on the wide integer, so a value past INT_MAX is rejected rather than truncated back into the band. |
warmup_cycles |
int | 5 |
An entity must be continuously present for this many ticks before it arms. A mid-run restart re-warms it. Enforced centrally, see “Reliability (bringup-quiesce)” below. Accepted range 0..INT_MAX, checked on the wide integer. |
prune_grace |
int | 60 |
Default injected into every detector’s own config before configure(); a per-detector detectors.<id>.prune_grace overrides it. Used by detectors that keep per-key bookkeeping. Accepted range 0..INT_MAX here, checked on the wide integer; each detector re-validates it against its own documented range before using it. |
detectors.<id>.mode |
string | bool | raise |
raise = push faults; advisory = observe, do not push; off = disabled. Bare off/no (YAML booleans) also disable; bare on/yes mean raise. |
detectors.<id>.<field> |
any | - | Per-detector thresholds, passed to that detector’s configure(). |
Nested delivery. Keys are written dotted in YAML or --ros-args, and the gateway
delivers them to the plugin as a nested object. A bare off is a YAML 1.1 boolean, so
mode: off arrives as false, not the string "off"; both forms disable the detector.
Unknown keys are reported. Anything under detectors.<id> that the detector does not
read produces one startup warning naming the key and listing the ones that exist, so a
typo like allow_list: for allowlist: cannot silently do nothing. A typo in the detector
id itself is warned about the same way, with the registered ids listed.
lifecycle_expectation keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. |
require_active |
string[] | [] |
Node names that must be in the active lifecycle state, each matched against a live node by its App::id, its full FQN (/ns/name), or the bare leaf of that FQN. A bare name matches that node in EVERY namespace (a fleet-wide “all controller_servers must be active”); use a full FQN to pin one robot’s. Both the bare and the FQN form match against the node’s stable FQN, so they survive the App::id renaming that a same-bare-name collision triggers; an entry written as an App::id also works, but can stop matching on a multi-robot graph for exactly that reason. Empty = the detector checks nothing and emits nothing. A require_active that is not a string array warns and is ignored; an empty-string entry warns and is skipped - never dropped silently, since the operator would go on believing the node is covered. |
grace |
int | 5 |
Consecutive not-active ticks a required node tolerates before being reported inactive - bringup time for a managed node to reach active (configure + activate) before the expectation is enforced. Counted per NODE, not per matching entry: a node named by both a bare-name and a full-FQN entry still advances its streak once per tick, so mixing the two documented forms cannot halve the grace that was configured. An ALREADY-CONFIRMED streak keeps its content while the node is ABSENT, so a node that leaves the graph after being confirmed stays confirmed; a streak that has not yet crossed grace on a node the presence detector could also report (it was armed at least once) is HELD rather than advanced while absent, so a node is never confirmed out of ticks gathered while nobody could observe it - only a PRESENT tick may still cross grace for that node. A node that was NEVER armed gets no such hold: node_death can never track it either, so absence keeps advancing its streak too, the same as a present tick would. Accepted range 0..300 - five minutes at the shipped 1 s cadence, and already an extravagant allowance for a managed node to reach active. The upper end is a real bound rather than a formality: grace decides how long a PRESENT node may go unreported, and how long a node returning from a departure still inactive takes to (re-)mature, so at the old maximum of INT_MAX - 1 that PRESENT-side silence lasted about 24 days at the shipped cadence with no warning. Whether it also bounds how long a node that leaves the graph BELOW grace sits unsettled depends on the same split: for an armed node that lasts for as long as the node stays gone, independent of grace (see “Bounded by evidence, not by age” below); for a never-armed one grace bounds that too, since absence matures it within grace + absence_grace + 1 ticks either way. The check runs on the wide integer, so a value that only fits after truncation is rejected rather than silently turned into a hair-trigger. Anything outside the range warns and keeps the default. |
prune_grace |
int | 60 |
Consecutive ticks an IDLE tracked node - both clocks at zero and no matured ownership, so nothing to lose - may stay ABSENT from the graph before its bookkeeping is reclaimed. A node carrying evidence is never reclaimed by age at all, however long it stays gone, so this key cannot erase a fault’s own state; the map is bounded instead by tracked_node_cap (see “Bounded by evidence, not by age”). Injected for every detector at plugin scope and overridable per detector; a value outside 0..3600 warns and this detector keeps its own default of 60. The range check runs on the wide integer, so an out-of-int-range value is rejected rather than truncated. The value is used as written - there is no grace + 1 clamp, because there is no longer anything for one to protect. |
tracked_node_cap |
int | 512 |
The most nodes this detector keeps bookkeeping for at once. It is what bounds the map, since evidence is never reclaimed by age (see “Bounded by evidence, not by age”): at the cap, idle entries are reclaimed first, then entries for DEPARTED nodes are collapsed into a count, and only if every tracked node is PRESENT and carrying evidence is a newly matched node refused - which withholds GRAPH_NODE_INACTIVE’s clear and is reported both in the log and on GET /x-medkit-watchdog. 512 is comfortably above every node in a realistic graph (a full Nav2 stack plus perception is roughly a hundred nodes; a ten-robot fleet sharing one domain a few hundred), so raising it is only needed where require_active legitimately matches more PRESENT nodes than that. Accepted range 1..16384 - 16384 is about 8 MB of bookkeeping at roughly 500 bytes per tracked node, and a deployment needing more distinct required-node identities alive at once has identity churn rather than a large fleet. Anything outside the range, including 0 (a cap of nothing would mean checking nothing), warns and keeps the default; the check runs on the wide integer, so an out-of-int-range value is rejected rather than truncated. |
node_death keys
Zero-config, unlike every detector above: there is no require_active-style list of nodes
to watch, because every armed App in the graph is a candidate.
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. |
miss_grace |
int | 2 |
Consecutive missed ticks a tracked node tolerates before being reported dead - a death is reported once misses EXCEED this, i.e. after miss_grace + 1 ticks. Accepted range 0..3600, and additionally floored to whatever wall-clock window the configured tick_interval_ms needs (see “The wall-clock floor” below); a value below that floor is silently raised to it, with a warning naming the ms window it actually spans. |
prune_grace |
int | 60 |
Plugin-injected default (overridable here); how long a key being DURABLY suppressed (see “Suppression” below) may sit unreported before its bookkeeping is reclaimed. Accepted range 0..3600. The value actually used is max(prune_grace, miss_grace + 1), a silent internal floor rather than a rejection: without it, a durably-suppressed key could be reclaimed the very tick it would first have become eligible to report, losing the report rather than merely suppressing it. |
allowlist |
string[] | [] |
Node identities never reported dead. A dead key matches if it - or a name the same node is also known by - is present verbatim: the key’s own fqn, its bare leaf, or (captured while the node was still present) its App::id. Has no effect unless named in suppress; naming it without a configured list is a no-op. |
suppress |
string[] | [] |
Suppression mechanisms to activate: "allowlist" and "lifecycle". Any other entry warns as unrecognized and is ignored; a non-string or non-array entry warns the same way. |
tracked_node_cap |
int | 512 |
The most DEPARTED (not-yet-confirmed-dead or already-collapsed) identities this detector keeps individual bookkeeping for at once - a present/armed node is never counted against it and never refused tracking. Accepted range 1..16384; 0 is refused rather than clamped, since a cap of nothing would mean tracking nothing. See “Bounded by evidence, not by age” below for what it actually bounds and why, unlike the sibling detector’s identical-looking cap, a newcomer here is never the thing refused. |
tick_interval_ms is not an own key of this detector, but this is the one detector that
reads the plugin-injected value directly, to compute miss_grace’s wall-clock floor below.
Liveness, and why membership in the snapshot is not enough. App::is_online is what
this detector tracks, never mere presence in the entity snapshot. In runtime-only discovery
the two happen to coincide - a dead node’s App leaves the snapshot entirely - but a manifest
keeps a bound App present with only is_online cleared once its ROS binding disappears, and
hybrid discovery inherits that same shape; counting snapshot membership alone would make a
manifest-declared node immortal. A managed lifecycle node that merely deactivates keeps
is_online: true - its process is still running, only its ROS 2 lifecycle state changed -
so a deactivation is never mistaken for a death either; that is lifecycle_expectation’s
concern, not this one’s (see “The boundary with lifecycle_expectation” below).
Tracking is keyed on the STABLE fqn (App::effective_fqn()), never App::id: an id is
recomputed every sweep and only gains a namespace prefix once a same-bare-name collision
currently exists anywhere in the graph, so a live node’s id can change out from under a key
built from it.
What is never tracked.
| Excluded | Why |
|---|---|
Peer-aggregated apps (app.source starting peer:) |
A peer app carries no ROS binding of its own, so effective_fqn() is empty for every one of them; tracking them would collapse an entire peer fleet onto one "" key, and one online peer app would mask every other peer app’s departure. |
_ros2cli_<pid> nodes |
Every ros2 CLI invocation (ros2 topic echo, ros2 param get, …) spins up a real, short-lived node under this prefix - rcl’s own hidden-node naming convention, matched structurally rather than guessed at. Letting these through would accumulate one permanently “dead” entry per CLI invocation for the life of the gateway. |
| An App that never comes online | A key is admitted to tracking only once the reliability gate has said this detector OWNS its departure - armed at least once, and on one of the two GROUNDS ownership can rest on (a lifecycle state read as active, or no lifecycle to read at all; the third case, a state nobody could read, is admitted PROVISIONALLY and is described in the row below). A manifest App whose binding never starts is never armed and so can never be falsely called dead; a node still inside its warmup_cycles window gets the identical protection. Once tracked, though, a node’s continued life-or-death judgement rests on PRESENCE alone, not on staying armed - a tracked node that later goes lifecycle-inactive is not thereby mistaken for dead. |
| A managed App whose lifecycle state has not been read YET | Not tracked, and only for as long as that is still true. The gate’s own raise permission is answered PERMISSIVELY for a managed node whose GetState has not answered (LifecycleWatcher::node_ok()) - without that, qos_mismatch, orphan and param_drift would all fall silent about a node whose lifecycle service is broken. Permission is not knowledge, though, so tracking is withheld while a measurement may still arrive. LifecycleWatcher charges a GetState re-seed budget per node, and only for a read that actually ran, so “still asking” is a fact rather than a guess. Once that budget is spent the node is tracked here after all: nothing will ASK again, and lifecycle_expectation looks at nothing unless an operator named the node in require_active (empty by default), so withholding it any longer would mean the death is reported by nobody. That admission is PROVISIONAL, not final. The ~/transition_event subscription outlives the seed budget, so a label can still arrive without being asked for, and one that reads non-active says the node was the lifecycle detector’s all along - so the presence detector hands it back on the spot, while it is still alive. Waiting for the budget alone would only have delayed the wrong report, not prevented it. |
File truncated at 100 lines see the full file
CHANGELOG
Changelog for package ros2_medkit_graph_watchdog
0.7.0 (2026-08-27)
- Initial release of the package: a gateway plugin that raises faults
for silent failures in the ROS 2 graph - problems that leave every
node alive and every topic present, so nothing in the stack reports
them today. The package carries the plugin skeleton, a central
reliability gate that holds raises until the graph has quiesced, and
a detector registry. Detectors are configured under
plugins.graph_watchdog.<key>, each with araise/advisory/offmode, and an unknown key underdetectors.<id>is reported rather than silently ignored, so a typo cannot quietly disable a check (#571) -
qos_mismatchdetector: raisesGRAPH_QOS_MISMATCHfor a publisher and subscriber whose QoS profiles cannot match, which leaves the connection silently unestablished (#571) -
orphandetector: raisesGRAPH_ORPHANfor a one-sided topic - a publisher with no subscriber, or the reverse - but only when a complementary near-miss counterpart exists: same message type, a namespace and leaf within the configured edit distance, and the opposite side. A lone unsubscribed topic raises nothing. The pair is the signature of a remap or a topic-name typo, which is what the detector is for (#578) -
param_driftdetector: raisesGRAPH_PARAM_DRIFTwhen a node's live parameter value diverges from the declared expectation (#580) -
lifecycle_expectationdetector: raisesGRAPH_NODE_INACTIVEfor a node named inrequire_activethat is not in theactivelifecycle state once itsgracewindow has passed. It raises two further codes for the cases where the state could not be established at all:GRAPH_NODE_UNREADABLEwhen the lifecycle service does not answer, andGRAPH_NODE_NOT_MANAGEDwhen the named node has no lifecycle to read. These are distinct codes on the wire, so a consumer filtering onGRAPH_NODE_INACTIVEalone sees neither unmeasured case. A node is matched by itsApp::id, its full FQN, or the bare leaf of that FQN, and the grace streak is counted per node, so naming one node in two documented forms cannot halve the grace that was configured (#587) -
node_deathdetector: raisesGRAPH_NODE_DISAPPEAREDfor a node that leaves the graph when nothing else is left to report it, with a suppression framework that decides ownership of a departure from knowledge rather than from silence. A node whose lifecycle state was never readable is admitted only provisionally, and is handed back the moment a label arrives saying the departure belonged to the lifecycle detector instead. Zero-config: there is no list of nodes to maintain (#625, #624) - Two silent-fault classes are not delivered in this release.
GRAPH_TF_STALEandGRAPH_LATENCY_BUDGEThave their fault codes reserved in the frozenGRAPH_*namespace and land in later changes, each against its own issue - The plugin keeps its ROS entities off the gateway executor, so an entity created and destroyed while the gateway runs cannot have its destructor run on an executor thread concurrently with a create on the same node
- Contributors: \@bburda
Package Dependencies
System Dependencies
Dependant Packages
No known dependants.
Launch files
No launch files found
Messages
No message files found.
Services
No service files found
Plugins
No plugins found.
Recent questions tagged ros2_medkit_graph_watchdog at Robotics Stack Exchange
No version for distro rolling showing humble. Known supported distros are highlighted in the buttons above.
Package Summary
| Version | 0.7.0 |
| License | Apache-2.0 |
| Build type | AMENT_CMAKE |
| Use | RECOMMENDED |
Repository Summary
| Checkout URI | https://github.com/selfpatch/ros2_medkit.git |
| VCS Type | git |
| VCS Version | main |
| Last Updated | 2026-09-06 |
| Dev Status | DEVELOPED |
| Released | RELEASED |
| Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
Gateway plugin that detects silent faults in the ROS 2 graph: QoS mismatches, orphaned topics left by a name typo, dead nodes, parameter drift and unmet lifecycle expectations.
Maintainers
- bburda
Authors
No additional authors.
ros2_medkit_graph_watchdog
Gateway plugin that detects silent faults in the ROS 2 graph: failures where every node is up, nothing logs an error, and the robot is still broken.
Detectors read the graph and raise faults through a ReportFault service client on the
gateway node; the faults surface via FaultManager on the gateway /faults API. That
client, like the lifecycle subscriptions below, lives in the plugin’s own callback group
and is driven by the plugin’s own executor from the tick thread, never by the gateway’s
ROS executor.
This package carries the plugin skeleton, the central reliability gate that holds raises
until the graph has quiesced, and five detectors, qos_mismatch, orphan,
param_drift, lifecycle_expectation and node_death. Two silent-fault classes remain
undelivered, GRAPH_TF_STALE and GRAPH_LATENCY_BUDGET; they land in follow-up changes,
each against its own issue, and their fault codes are already reserved in the frozen
GRAPH_* namespace (see “Fault codes”).
Build
With the gateway built and sourced:
colcon build --packages-select ros2_medkit_graph_watchdog
colcon test --packages-select ros2_medkit_graph_watchdog && colcon test-result --verbose
Load into the gateway
ros2 run ros2_medkit_gateway gateway_node --ros-args \
-p plugins:="[graph_watchdog]" \
-p "plugins.graph_watchdog.path:=$(ros2 pkg prefix ros2_medkit_graph_watchdog)/lib/ros2_medkit_graph_watchdog/libros2_medkit_graph_watchdog.so"
Configuration (plugins.graph_watchdog.<key>)
| Key | Type | Default | Meaning |
|---|---|---|---|
tick_interval_ms |
int | 1000 |
Detector tick cadence. Accepted range 1..INT_MAX; the check runs on the wide integer, so a value past INT_MAX is rejected rather than truncated back into the band. |
warmup_cycles |
int | 5 |
An entity must be continuously present for this many ticks before it arms. A mid-run restart re-warms it. Enforced centrally, see “Reliability (bringup-quiesce)” below. Accepted range 0..INT_MAX, checked on the wide integer. |
prune_grace |
int | 60 |
Default injected into every detector’s own config before configure(); a per-detector detectors.<id>.prune_grace overrides it. Used by detectors that keep per-key bookkeeping. Accepted range 0..INT_MAX here, checked on the wide integer; each detector re-validates it against its own documented range before using it. |
detectors.<id>.mode |
string | bool | raise |
raise = push faults; advisory = observe, do not push; off = disabled. Bare off/no (YAML booleans) also disable; bare on/yes mean raise. |
detectors.<id>.<field> |
any | - | Per-detector thresholds, passed to that detector’s configure(). |
Nested delivery. Keys are written dotted in YAML or --ros-args, and the gateway
delivers them to the plugin as a nested object. A bare off is a YAML 1.1 boolean, so
mode: off arrives as false, not the string "off"; both forms disable the detector.
Unknown keys are reported. Anything under detectors.<id> that the detector does not
read produces one startup warning naming the key and listing the ones that exist, so a
typo like allow_list: for allowlist: cannot silently do nothing. A typo in the detector
id itself is warned about the same way, with the registered ids listed.
lifecycle_expectation keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. |
require_active |
string[] | [] |
Node names that must be in the active lifecycle state, each matched against a live node by its App::id, its full FQN (/ns/name), or the bare leaf of that FQN. A bare name matches that node in EVERY namespace (a fleet-wide “all controller_servers must be active”); use a full FQN to pin one robot’s. Both the bare and the FQN form match against the node’s stable FQN, so they survive the App::id renaming that a same-bare-name collision triggers; an entry written as an App::id also works, but can stop matching on a multi-robot graph for exactly that reason. Empty = the detector checks nothing and emits nothing. A require_active that is not a string array warns and is ignored; an empty-string entry warns and is skipped - never dropped silently, since the operator would go on believing the node is covered. |
grace |
int | 5 |
Consecutive not-active ticks a required node tolerates before being reported inactive - bringup time for a managed node to reach active (configure + activate) before the expectation is enforced. Counted per NODE, not per matching entry: a node named by both a bare-name and a full-FQN entry still advances its streak once per tick, so mixing the two documented forms cannot halve the grace that was configured. An ALREADY-CONFIRMED streak keeps its content while the node is ABSENT, so a node that leaves the graph after being confirmed stays confirmed; a streak that has not yet crossed grace on a node the presence detector could also report (it was armed at least once) is HELD rather than advanced while absent, so a node is never confirmed out of ticks gathered while nobody could observe it - only a PRESENT tick may still cross grace for that node. A node that was NEVER armed gets no such hold: node_death can never track it either, so absence keeps advancing its streak too, the same as a present tick would. Accepted range 0..300 - five minutes at the shipped 1 s cadence, and already an extravagant allowance for a managed node to reach active. The upper end is a real bound rather than a formality: grace decides how long a PRESENT node may go unreported, and how long a node returning from a departure still inactive takes to (re-)mature, so at the old maximum of INT_MAX - 1 that PRESENT-side silence lasted about 24 days at the shipped cadence with no warning. Whether it also bounds how long a node that leaves the graph BELOW grace sits unsettled depends on the same split: for an armed node that lasts for as long as the node stays gone, independent of grace (see “Bounded by evidence, not by age” below); for a never-armed one grace bounds that too, since absence matures it within grace + absence_grace + 1 ticks either way. The check runs on the wide integer, so a value that only fits after truncation is rejected rather than silently turned into a hair-trigger. Anything outside the range warns and keeps the default. |
prune_grace |
int | 60 |
Consecutive ticks an IDLE tracked node - both clocks at zero and no matured ownership, so nothing to lose - may stay ABSENT from the graph before its bookkeeping is reclaimed. A node carrying evidence is never reclaimed by age at all, however long it stays gone, so this key cannot erase a fault’s own state; the map is bounded instead by tracked_node_cap (see “Bounded by evidence, not by age”). Injected for every detector at plugin scope and overridable per detector; a value outside 0..3600 warns and this detector keeps its own default of 60. The range check runs on the wide integer, so an out-of-int-range value is rejected rather than truncated. The value is used as written - there is no grace + 1 clamp, because there is no longer anything for one to protect. |
tracked_node_cap |
int | 512 |
The most nodes this detector keeps bookkeeping for at once. It is what bounds the map, since evidence is never reclaimed by age (see “Bounded by evidence, not by age”): at the cap, idle entries are reclaimed first, then entries for DEPARTED nodes are collapsed into a count, and only if every tracked node is PRESENT and carrying evidence is a newly matched node refused - which withholds GRAPH_NODE_INACTIVE’s clear and is reported both in the log and on GET /x-medkit-watchdog. 512 is comfortably above every node in a realistic graph (a full Nav2 stack plus perception is roughly a hundred nodes; a ten-robot fleet sharing one domain a few hundred), so raising it is only needed where require_active legitimately matches more PRESENT nodes than that. Accepted range 1..16384 - 16384 is about 8 MB of bookkeeping at roughly 500 bytes per tracked node, and a deployment needing more distinct required-node identities alive at once has identity churn rather than a large fleet. Anything outside the range, including 0 (a cap of nothing would mean checking nothing), warns and keeps the default; the check runs on the wide integer, so an out-of-int-range value is rejected rather than truncated. |
node_death keys
Zero-config, unlike every detector above: there is no require_active-style list of nodes
to watch, because every armed App in the graph is a candidate.
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. |
miss_grace |
int | 2 |
Consecutive missed ticks a tracked node tolerates before being reported dead - a death is reported once misses EXCEED this, i.e. after miss_grace + 1 ticks. Accepted range 0..3600, and additionally floored to whatever wall-clock window the configured tick_interval_ms needs (see “The wall-clock floor” below); a value below that floor is silently raised to it, with a warning naming the ms window it actually spans. |
prune_grace |
int | 60 |
Plugin-injected default (overridable here); how long a key being DURABLY suppressed (see “Suppression” below) may sit unreported before its bookkeeping is reclaimed. Accepted range 0..3600. The value actually used is max(prune_grace, miss_grace + 1), a silent internal floor rather than a rejection: without it, a durably-suppressed key could be reclaimed the very tick it would first have become eligible to report, losing the report rather than merely suppressing it. |
allowlist |
string[] | [] |
Node identities never reported dead. A dead key matches if it - or a name the same node is also known by - is present verbatim: the key’s own fqn, its bare leaf, or (captured while the node was still present) its App::id. Has no effect unless named in suppress; naming it without a configured list is a no-op. |
suppress |
string[] | [] |
Suppression mechanisms to activate: "allowlist" and "lifecycle". Any other entry warns as unrecognized and is ignored; a non-string or non-array entry warns the same way. |
tracked_node_cap |
int | 512 |
The most DEPARTED (not-yet-confirmed-dead or already-collapsed) identities this detector keeps individual bookkeeping for at once - a present/armed node is never counted against it and never refused tracking. Accepted range 1..16384; 0 is refused rather than clamped, since a cap of nothing would mean tracking nothing. See “Bounded by evidence, not by age” below for what it actually bounds and why, unlike the sibling detector’s identical-looking cap, a newcomer here is never the thing refused. |
tick_interval_ms is not an own key of this detector, but this is the one detector that
reads the plugin-injected value directly, to compute miss_grace’s wall-clock floor below.
Liveness, and why membership in the snapshot is not enough. App::is_online is what
this detector tracks, never mere presence in the entity snapshot. In runtime-only discovery
the two happen to coincide - a dead node’s App leaves the snapshot entirely - but a manifest
keeps a bound App present with only is_online cleared once its ROS binding disappears, and
hybrid discovery inherits that same shape; counting snapshot membership alone would make a
manifest-declared node immortal. A managed lifecycle node that merely deactivates keeps
is_online: true - its process is still running, only its ROS 2 lifecycle state changed -
so a deactivation is never mistaken for a death either; that is lifecycle_expectation’s
concern, not this one’s (see “The boundary with lifecycle_expectation” below).
Tracking is keyed on the STABLE fqn (App::effective_fqn()), never App::id: an id is
recomputed every sweep and only gains a namespace prefix once a same-bare-name collision
currently exists anywhere in the graph, so a live node’s id can change out from under a key
built from it.
What is never tracked.
| Excluded | Why |
|---|---|
Peer-aggregated apps (app.source starting peer:) |
A peer app carries no ROS binding of its own, so effective_fqn() is empty for every one of them; tracking them would collapse an entire peer fleet onto one "" key, and one online peer app would mask every other peer app’s departure. |
_ros2cli_<pid> nodes |
Every ros2 CLI invocation (ros2 topic echo, ros2 param get, …) spins up a real, short-lived node under this prefix - rcl’s own hidden-node naming convention, matched structurally rather than guessed at. Letting these through would accumulate one permanently “dead” entry per CLI invocation for the life of the gateway. |
| An App that never comes online | A key is admitted to tracking only once the reliability gate has said this detector OWNS its departure - armed at least once, and on one of the two GROUNDS ownership can rest on (a lifecycle state read as active, or no lifecycle to read at all; the third case, a state nobody could read, is admitted PROVISIONALLY and is described in the row below). A manifest App whose binding never starts is never armed and so can never be falsely called dead; a node still inside its warmup_cycles window gets the identical protection. Once tracked, though, a node’s continued life-or-death judgement rests on PRESENCE alone, not on staying armed - a tracked node that later goes lifecycle-inactive is not thereby mistaken for dead. |
| A managed App whose lifecycle state has not been read YET | Not tracked, and only for as long as that is still true. The gate’s own raise permission is answered PERMISSIVELY for a managed node whose GetState has not answered (LifecycleWatcher::node_ok()) - without that, qos_mismatch, orphan and param_drift would all fall silent about a node whose lifecycle service is broken. Permission is not knowledge, though, so tracking is withheld while a measurement may still arrive. LifecycleWatcher charges a GetState re-seed budget per node, and only for a read that actually ran, so “still asking” is a fact rather than a guess. Once that budget is spent the node is tracked here after all: nothing will ASK again, and lifecycle_expectation looks at nothing unless an operator named the node in require_active (empty by default), so withholding it any longer would mean the death is reported by nobody. That admission is PROVISIONAL, not final. The ~/transition_event subscription outlives the seed budget, so a label can still arrive without being asked for, and one that reads non-active says the node was the lifecycle detector’s all along - so the presence detector hands it back on the spot, while it is still alive. Waiting for the budget alone would only have delayed the wrong report, not prevented it. |
File truncated at 100 lines see the full file
CHANGELOG
Changelog for package ros2_medkit_graph_watchdog
0.7.0 (2026-08-27)
- Initial release of the package: a gateway plugin that raises faults
for silent failures in the ROS 2 graph - problems that leave every
node alive and every topic present, so nothing in the stack reports
them today. The package carries the plugin skeleton, a central
reliability gate that holds raises until the graph has quiesced, and
a detector registry. Detectors are configured under
plugins.graph_watchdog.<key>, each with araise/advisory/offmode, and an unknown key underdetectors.<id>is reported rather than silently ignored, so a typo cannot quietly disable a check (#571) -
qos_mismatchdetector: raisesGRAPH_QOS_MISMATCHfor a publisher and subscriber whose QoS profiles cannot match, which leaves the connection silently unestablished (#571) -
orphandetector: raisesGRAPH_ORPHANfor a one-sided topic - a publisher with no subscriber, or the reverse - but only when a complementary near-miss counterpart exists: same message type, a namespace and leaf within the configured edit distance, and the opposite side. A lone unsubscribed topic raises nothing. The pair is the signature of a remap or a topic-name typo, which is what the detector is for (#578) -
param_driftdetector: raisesGRAPH_PARAM_DRIFTwhen a node's live parameter value diverges from the declared expectation (#580) -
lifecycle_expectationdetector: raisesGRAPH_NODE_INACTIVEfor a node named inrequire_activethat is not in theactivelifecycle state once itsgracewindow has passed. It raises two further codes for the cases where the state could not be established at all:GRAPH_NODE_UNREADABLEwhen the lifecycle service does not answer, andGRAPH_NODE_NOT_MANAGEDwhen the named node has no lifecycle to read. These are distinct codes on the wire, so a consumer filtering onGRAPH_NODE_INACTIVEalone sees neither unmeasured case. A node is matched by itsApp::id, its full FQN, or the bare leaf of that FQN, and the grace streak is counted per node, so naming one node in two documented forms cannot halve the grace that was configured (#587) -
node_deathdetector: raisesGRAPH_NODE_DISAPPEAREDfor a node that leaves the graph when nothing else is left to report it, with a suppression framework that decides ownership of a departure from knowledge rather than from silence. A node whose lifecycle state was never readable is admitted only provisionally, and is handed back the moment a label arrives saying the departure belonged to the lifecycle detector instead. Zero-config: there is no list of nodes to maintain (#625, #624) - Two silent-fault classes are not delivered in this release.
GRAPH_TF_STALEandGRAPH_LATENCY_BUDGEThave their fault codes reserved in the frozenGRAPH_*namespace and land in later changes, each against its own issue - The plugin keeps its ROS entities off the gateway executor, so an entity created and destroyed while the gateway runs cannot have its destructor run on an executor thread concurrently with a create on the same node
- Contributors: \@bburda
Package Dependencies
System Dependencies
Dependant Packages
No known dependants.
Launch files
No launch files found
Messages
No message files found.
Services
No service files found
Plugins
No plugins found.
Recent questions tagged ros2_medkit_graph_watchdog at Robotics Stack Exchange
No version for distro ardent showing humble. Known supported distros are highlighted in the buttons above.
Package Summary
| Version | 0.7.0 |
| License | Apache-2.0 |
| Build type | AMENT_CMAKE |
| Use | RECOMMENDED |
Repository Summary
| Checkout URI | https://github.com/selfpatch/ros2_medkit.git |
| VCS Type | git |
| VCS Version | main |
| Last Updated | 2026-09-06 |
| Dev Status | DEVELOPED |
| Released | RELEASED |
| Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
Gateway plugin that detects silent faults in the ROS 2 graph: QoS mismatches, orphaned topics left by a name typo, dead nodes, parameter drift and unmet lifecycle expectations.
Maintainers
- bburda
Authors
No additional authors.
ros2_medkit_graph_watchdog
Gateway plugin that detects silent faults in the ROS 2 graph: failures where every node is up, nothing logs an error, and the robot is still broken.
Detectors read the graph and raise faults through a ReportFault service client on the
gateway node; the faults surface via FaultManager on the gateway /faults API. That
client, like the lifecycle subscriptions below, lives in the plugin’s own callback group
and is driven by the plugin’s own executor from the tick thread, never by the gateway’s
ROS executor.
This package carries the plugin skeleton, the central reliability gate that holds raises
until the graph has quiesced, and five detectors, qos_mismatch, orphan,
param_drift, lifecycle_expectation and node_death. Two silent-fault classes remain
undelivered, GRAPH_TF_STALE and GRAPH_LATENCY_BUDGET; they land in follow-up changes,
each against its own issue, and their fault codes are already reserved in the frozen
GRAPH_* namespace (see “Fault codes”).
Build
With the gateway built and sourced:
colcon build --packages-select ros2_medkit_graph_watchdog
colcon test --packages-select ros2_medkit_graph_watchdog && colcon test-result --verbose
Load into the gateway
ros2 run ros2_medkit_gateway gateway_node --ros-args \
-p plugins:="[graph_watchdog]" \
-p "plugins.graph_watchdog.path:=$(ros2 pkg prefix ros2_medkit_graph_watchdog)/lib/ros2_medkit_graph_watchdog/libros2_medkit_graph_watchdog.so"
Configuration (plugins.graph_watchdog.<key>)
| Key | Type | Default | Meaning |
|---|---|---|---|
tick_interval_ms |
int | 1000 |
Detector tick cadence. Accepted range 1..INT_MAX; the check runs on the wide integer, so a value past INT_MAX is rejected rather than truncated back into the band. |
warmup_cycles |
int | 5 |
An entity must be continuously present for this many ticks before it arms. A mid-run restart re-warms it. Enforced centrally, see “Reliability (bringup-quiesce)” below. Accepted range 0..INT_MAX, checked on the wide integer. |
prune_grace |
int | 60 |
Default injected into every detector’s own config before configure(); a per-detector detectors.<id>.prune_grace overrides it. Used by detectors that keep per-key bookkeeping. Accepted range 0..INT_MAX here, checked on the wide integer; each detector re-validates it against its own documented range before using it. |
detectors.<id>.mode |
string | bool | raise |
raise = push faults; advisory = observe, do not push; off = disabled. Bare off/no (YAML booleans) also disable; bare on/yes mean raise. |
detectors.<id>.<field> |
any | - | Per-detector thresholds, passed to that detector’s configure(). |
Nested delivery. Keys are written dotted in YAML or --ros-args, and the gateway
delivers them to the plugin as a nested object. A bare off is a YAML 1.1 boolean, so
mode: off arrives as false, not the string "off"; both forms disable the detector.
Unknown keys are reported. Anything under detectors.<id> that the detector does not
read produces one startup warning naming the key and listing the ones that exist, so a
typo like allow_list: for allowlist: cannot silently do nothing. A typo in the detector
id itself is warned about the same way, with the registered ids listed.
lifecycle_expectation keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. |
require_active |
string[] | [] |
Node names that must be in the active lifecycle state, each matched against a live node by its App::id, its full FQN (/ns/name), or the bare leaf of that FQN. A bare name matches that node in EVERY namespace (a fleet-wide “all controller_servers must be active”); use a full FQN to pin one robot’s. Both the bare and the FQN form match against the node’s stable FQN, so they survive the App::id renaming that a same-bare-name collision triggers; an entry written as an App::id also works, but can stop matching on a multi-robot graph for exactly that reason. Empty = the detector checks nothing and emits nothing. A require_active that is not a string array warns and is ignored; an empty-string entry warns and is skipped - never dropped silently, since the operator would go on believing the node is covered. |
grace |
int | 5 |
Consecutive not-active ticks a required node tolerates before being reported inactive - bringup time for a managed node to reach active (configure + activate) before the expectation is enforced. Counted per NODE, not per matching entry: a node named by both a bare-name and a full-FQN entry still advances its streak once per tick, so mixing the two documented forms cannot halve the grace that was configured. An ALREADY-CONFIRMED streak keeps its content while the node is ABSENT, so a node that leaves the graph after being confirmed stays confirmed; a streak that has not yet crossed grace on a node the presence detector could also report (it was armed at least once) is HELD rather than advanced while absent, so a node is never confirmed out of ticks gathered while nobody could observe it - only a PRESENT tick may still cross grace for that node. A node that was NEVER armed gets no such hold: node_death can never track it either, so absence keeps advancing its streak too, the same as a present tick would. Accepted range 0..300 - five minutes at the shipped 1 s cadence, and already an extravagant allowance for a managed node to reach active. The upper end is a real bound rather than a formality: grace decides how long a PRESENT node may go unreported, and how long a node returning from a departure still inactive takes to (re-)mature, so at the old maximum of INT_MAX - 1 that PRESENT-side silence lasted about 24 days at the shipped cadence with no warning. Whether it also bounds how long a node that leaves the graph BELOW grace sits unsettled depends on the same split: for an armed node that lasts for as long as the node stays gone, independent of grace (see “Bounded by evidence, not by age” below); for a never-armed one grace bounds that too, since absence matures it within grace + absence_grace + 1 ticks either way. The check runs on the wide integer, so a value that only fits after truncation is rejected rather than silently turned into a hair-trigger. Anything outside the range warns and keeps the default. |
prune_grace |
int | 60 |
Consecutive ticks an IDLE tracked node - both clocks at zero and no matured ownership, so nothing to lose - may stay ABSENT from the graph before its bookkeeping is reclaimed. A node carrying evidence is never reclaimed by age at all, however long it stays gone, so this key cannot erase a fault’s own state; the map is bounded instead by tracked_node_cap (see “Bounded by evidence, not by age”). Injected for every detector at plugin scope and overridable per detector; a value outside 0..3600 warns and this detector keeps its own default of 60. The range check runs on the wide integer, so an out-of-int-range value is rejected rather than truncated. The value is used as written - there is no grace + 1 clamp, because there is no longer anything for one to protect. |
tracked_node_cap |
int | 512 |
The most nodes this detector keeps bookkeeping for at once. It is what bounds the map, since evidence is never reclaimed by age (see “Bounded by evidence, not by age”): at the cap, idle entries are reclaimed first, then entries for DEPARTED nodes are collapsed into a count, and only if every tracked node is PRESENT and carrying evidence is a newly matched node refused - which withholds GRAPH_NODE_INACTIVE’s clear and is reported both in the log and on GET /x-medkit-watchdog. 512 is comfortably above every node in a realistic graph (a full Nav2 stack plus perception is roughly a hundred nodes; a ten-robot fleet sharing one domain a few hundred), so raising it is only needed where require_active legitimately matches more PRESENT nodes than that. Accepted range 1..16384 - 16384 is about 8 MB of bookkeeping at roughly 500 bytes per tracked node, and a deployment needing more distinct required-node identities alive at once has identity churn rather than a large fleet. Anything outside the range, including 0 (a cap of nothing would mean checking nothing), warns and keeps the default; the check runs on the wide integer, so an out-of-int-range value is rejected rather than truncated. |
node_death keys
Zero-config, unlike every detector above: there is no require_active-style list of nodes
to watch, because every armed App in the graph is a candidate.
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. |
miss_grace |
int | 2 |
Consecutive missed ticks a tracked node tolerates before being reported dead - a death is reported once misses EXCEED this, i.e. after miss_grace + 1 ticks. Accepted range 0..3600, and additionally floored to whatever wall-clock window the configured tick_interval_ms needs (see “The wall-clock floor” below); a value below that floor is silently raised to it, with a warning naming the ms window it actually spans. |
prune_grace |
int | 60 |
Plugin-injected default (overridable here); how long a key being DURABLY suppressed (see “Suppression” below) may sit unreported before its bookkeeping is reclaimed. Accepted range 0..3600. The value actually used is max(prune_grace, miss_grace + 1), a silent internal floor rather than a rejection: without it, a durably-suppressed key could be reclaimed the very tick it would first have become eligible to report, losing the report rather than merely suppressing it. |
allowlist |
string[] | [] |
Node identities never reported dead. A dead key matches if it - or a name the same node is also known by - is present verbatim: the key’s own fqn, its bare leaf, or (captured while the node was still present) its App::id. Has no effect unless named in suppress; naming it without a configured list is a no-op. |
suppress |
string[] | [] |
Suppression mechanisms to activate: "allowlist" and "lifecycle". Any other entry warns as unrecognized and is ignored; a non-string or non-array entry warns the same way. |
tracked_node_cap |
int | 512 |
The most DEPARTED (not-yet-confirmed-dead or already-collapsed) identities this detector keeps individual bookkeeping for at once - a present/armed node is never counted against it and never refused tracking. Accepted range 1..16384; 0 is refused rather than clamped, since a cap of nothing would mean tracking nothing. See “Bounded by evidence, not by age” below for what it actually bounds and why, unlike the sibling detector’s identical-looking cap, a newcomer here is never the thing refused. |
tick_interval_ms is not an own key of this detector, but this is the one detector that
reads the plugin-injected value directly, to compute miss_grace’s wall-clock floor below.
Liveness, and why membership in the snapshot is not enough. App::is_online is what
this detector tracks, never mere presence in the entity snapshot. In runtime-only discovery
the two happen to coincide - a dead node’s App leaves the snapshot entirely - but a manifest
keeps a bound App present with only is_online cleared once its ROS binding disappears, and
hybrid discovery inherits that same shape; counting snapshot membership alone would make a
manifest-declared node immortal. A managed lifecycle node that merely deactivates keeps
is_online: true - its process is still running, only its ROS 2 lifecycle state changed -
so a deactivation is never mistaken for a death either; that is lifecycle_expectation’s
concern, not this one’s (see “The boundary with lifecycle_expectation” below).
Tracking is keyed on the STABLE fqn (App::effective_fqn()), never App::id: an id is
recomputed every sweep and only gains a namespace prefix once a same-bare-name collision
currently exists anywhere in the graph, so a live node’s id can change out from under a key
built from it.
What is never tracked.
| Excluded | Why |
|---|---|
Peer-aggregated apps (app.source starting peer:) |
A peer app carries no ROS binding of its own, so effective_fqn() is empty for every one of them; tracking them would collapse an entire peer fleet onto one "" key, and one online peer app would mask every other peer app’s departure. |
_ros2cli_<pid> nodes |
Every ros2 CLI invocation (ros2 topic echo, ros2 param get, …) spins up a real, short-lived node under this prefix - rcl’s own hidden-node naming convention, matched structurally rather than guessed at. Letting these through would accumulate one permanently “dead” entry per CLI invocation for the life of the gateway. |
| An App that never comes online | A key is admitted to tracking only once the reliability gate has said this detector OWNS its departure - armed at least once, and on one of the two GROUNDS ownership can rest on (a lifecycle state read as active, or no lifecycle to read at all; the third case, a state nobody could read, is admitted PROVISIONALLY and is described in the row below). A manifest App whose binding never starts is never armed and so can never be falsely called dead; a node still inside its warmup_cycles window gets the identical protection. Once tracked, though, a node’s continued life-or-death judgement rests on PRESENCE alone, not on staying armed - a tracked node that later goes lifecycle-inactive is not thereby mistaken for dead. |
| A managed App whose lifecycle state has not been read YET | Not tracked, and only for as long as that is still true. The gate’s own raise permission is answered PERMISSIVELY for a managed node whose GetState has not answered (LifecycleWatcher::node_ok()) - without that, qos_mismatch, orphan and param_drift would all fall silent about a node whose lifecycle service is broken. Permission is not knowledge, though, so tracking is withheld while a measurement may still arrive. LifecycleWatcher charges a GetState re-seed budget per node, and only for a read that actually ran, so “still asking” is a fact rather than a guess. Once that budget is spent the node is tracked here after all: nothing will ASK again, and lifecycle_expectation looks at nothing unless an operator named the node in require_active (empty by default), so withholding it any longer would mean the death is reported by nobody. That admission is PROVISIONAL, not final. The ~/transition_event subscription outlives the seed budget, so a label can still arrive without being asked for, and one that reads non-active says the node was the lifecycle detector’s all along - so the presence detector hands it back on the spot, while it is still alive. Waiting for the budget alone would only have delayed the wrong report, not prevented it. |
File truncated at 100 lines see the full file
CHANGELOG
Changelog for package ros2_medkit_graph_watchdog
0.7.0 (2026-08-27)
- Initial release of the package: a gateway plugin that raises faults
for silent failures in the ROS 2 graph - problems that leave every
node alive and every topic present, so nothing in the stack reports
them today. The package carries the plugin skeleton, a central
reliability gate that holds raises until the graph has quiesced, and
a detector registry. Detectors are configured under
plugins.graph_watchdog.<key>, each with araise/advisory/offmode, and an unknown key underdetectors.<id>is reported rather than silently ignored, so a typo cannot quietly disable a check (#571) -
qos_mismatchdetector: raisesGRAPH_QOS_MISMATCHfor a publisher and subscriber whose QoS profiles cannot match, which leaves the connection silently unestablished (#571) -
orphandetector: raisesGRAPH_ORPHANfor a one-sided topic - a publisher with no subscriber, or the reverse - but only when a complementary near-miss counterpart exists: same message type, a namespace and leaf within the configured edit distance, and the opposite side. A lone unsubscribed topic raises nothing. The pair is the signature of a remap or a topic-name typo, which is what the detector is for (#578) -
param_driftdetector: raisesGRAPH_PARAM_DRIFTwhen a node's live parameter value diverges from the declared expectation (#580) -
lifecycle_expectationdetector: raisesGRAPH_NODE_INACTIVEfor a node named inrequire_activethat is not in theactivelifecycle state once itsgracewindow has passed. It raises two further codes for the cases where the state could not be established at all:GRAPH_NODE_UNREADABLEwhen the lifecycle service does not answer, andGRAPH_NODE_NOT_MANAGEDwhen the named node has no lifecycle to read. These are distinct codes on the wire, so a consumer filtering onGRAPH_NODE_INACTIVEalone sees neither unmeasured case. A node is matched by itsApp::id, its full FQN, or the bare leaf of that FQN, and the grace streak is counted per node, so naming one node in two documented forms cannot halve the grace that was configured (#587) -
node_deathdetector: raisesGRAPH_NODE_DISAPPEAREDfor a node that leaves the graph when nothing else is left to report it, with a suppression framework that decides ownership of a departure from knowledge rather than from silence. A node whose lifecycle state was never readable is admitted only provisionally, and is handed back the moment a label arrives saying the departure belonged to the lifecycle detector instead. Zero-config: there is no list of nodes to maintain (#625, #624) - Two silent-fault classes are not delivered in this release.
GRAPH_TF_STALEandGRAPH_LATENCY_BUDGEThave their fault codes reserved in the frozenGRAPH_*namespace and land in later changes, each against its own issue - The plugin keeps its ROS entities off the gateway executor, so an entity created and destroyed while the gateway runs cannot have its destructor run on an executor thread concurrently with a create on the same node
- Contributors: \@bburda
Package Dependencies
System Dependencies
Dependant Packages
No known dependants.
Launch files
No launch files found
Messages
No message files found.
Services
No service files found
Plugins
No plugins found.
Recent questions tagged ros2_medkit_graph_watchdog at Robotics Stack Exchange
No version for distro bouncy showing humble. Known supported distros are highlighted in the buttons above.
Package Summary
| Version | 0.7.0 |
| License | Apache-2.0 |
| Build type | AMENT_CMAKE |
| Use | RECOMMENDED |
Repository Summary
| Checkout URI | https://github.com/selfpatch/ros2_medkit.git |
| VCS Type | git |
| VCS Version | main |
| Last Updated | 2026-09-06 |
| Dev Status | DEVELOPED |
| Released | RELEASED |
| Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
Gateway plugin that detects silent faults in the ROS 2 graph: QoS mismatches, orphaned topics left by a name typo, dead nodes, parameter drift and unmet lifecycle expectations.
Maintainers
- bburda
Authors
No additional authors.
ros2_medkit_graph_watchdog
Gateway plugin that detects silent faults in the ROS 2 graph: failures where every node is up, nothing logs an error, and the robot is still broken.
Detectors read the graph and raise faults through a ReportFault service client on the
gateway node; the faults surface via FaultManager on the gateway /faults API. That
client, like the lifecycle subscriptions below, lives in the plugin’s own callback group
and is driven by the plugin’s own executor from the tick thread, never by the gateway’s
ROS executor.
This package carries the plugin skeleton, the central reliability gate that holds raises
until the graph has quiesced, and five detectors, qos_mismatch, orphan,
param_drift, lifecycle_expectation and node_death. Two silent-fault classes remain
undelivered, GRAPH_TF_STALE and GRAPH_LATENCY_BUDGET; they land in follow-up changes,
each against its own issue, and their fault codes are already reserved in the frozen
GRAPH_* namespace (see “Fault codes”).
Build
With the gateway built and sourced:
colcon build --packages-select ros2_medkit_graph_watchdog
colcon test --packages-select ros2_medkit_graph_watchdog && colcon test-result --verbose
Load into the gateway
ros2 run ros2_medkit_gateway gateway_node --ros-args \
-p plugins:="[graph_watchdog]" \
-p "plugins.graph_watchdog.path:=$(ros2 pkg prefix ros2_medkit_graph_watchdog)/lib/ros2_medkit_graph_watchdog/libros2_medkit_graph_watchdog.so"
Configuration (plugins.graph_watchdog.<key>)
| Key | Type | Default | Meaning |
|---|---|---|---|
tick_interval_ms |
int | 1000 |
Detector tick cadence. Accepted range 1..INT_MAX; the check runs on the wide integer, so a value past INT_MAX is rejected rather than truncated back into the band. |
warmup_cycles |
int | 5 |
An entity must be continuously present for this many ticks before it arms. A mid-run restart re-warms it. Enforced centrally, see “Reliability (bringup-quiesce)” below. Accepted range 0..INT_MAX, checked on the wide integer. |
prune_grace |
int | 60 |
Default injected into every detector’s own config before configure(); a per-detector detectors.<id>.prune_grace overrides it. Used by detectors that keep per-key bookkeeping. Accepted range 0..INT_MAX here, checked on the wide integer; each detector re-validates it against its own documented range before using it. |
detectors.<id>.mode |
string | bool | raise |
raise = push faults; advisory = observe, do not push; off = disabled. Bare off/no (YAML booleans) also disable; bare on/yes mean raise. |
detectors.<id>.<field> |
any | - | Per-detector thresholds, passed to that detector’s configure(). |
Nested delivery. Keys are written dotted in YAML or --ros-args, and the gateway
delivers them to the plugin as a nested object. A bare off is a YAML 1.1 boolean, so
mode: off arrives as false, not the string "off"; both forms disable the detector.
Unknown keys are reported. Anything under detectors.<id> that the detector does not
read produces one startup warning naming the key and listing the ones that exist, so a
typo like allow_list: for allowlist: cannot silently do nothing. A typo in the detector
id itself is warned about the same way, with the registered ids listed.
lifecycle_expectation keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. |
require_active |
string[] | [] |
Node names that must be in the active lifecycle state, each matched against a live node by its App::id, its full FQN (/ns/name), or the bare leaf of that FQN. A bare name matches that node in EVERY namespace (a fleet-wide “all controller_servers must be active”); use a full FQN to pin one robot’s. Both the bare and the FQN form match against the node’s stable FQN, so they survive the App::id renaming that a same-bare-name collision triggers; an entry written as an App::id also works, but can stop matching on a multi-robot graph for exactly that reason. Empty = the detector checks nothing and emits nothing. A require_active that is not a string array warns and is ignored; an empty-string entry warns and is skipped - never dropped silently, since the operator would go on believing the node is covered. |
grace |
int | 5 |
Consecutive not-active ticks a required node tolerates before being reported inactive - bringup time for a managed node to reach active (configure + activate) before the expectation is enforced. Counted per NODE, not per matching entry: a node named by both a bare-name and a full-FQN entry still advances its streak once per tick, so mixing the two documented forms cannot halve the grace that was configured. An ALREADY-CONFIRMED streak keeps its content while the node is ABSENT, so a node that leaves the graph after being confirmed stays confirmed; a streak that has not yet crossed grace on a node the presence detector could also report (it was armed at least once) is HELD rather than advanced while absent, so a node is never confirmed out of ticks gathered while nobody could observe it - only a PRESENT tick may still cross grace for that node. A node that was NEVER armed gets no such hold: node_death can never track it either, so absence keeps advancing its streak too, the same as a present tick would. Accepted range 0..300 - five minutes at the shipped 1 s cadence, and already an extravagant allowance for a managed node to reach active. The upper end is a real bound rather than a formality: grace decides how long a PRESENT node may go unreported, and how long a node returning from a departure still inactive takes to (re-)mature, so at the old maximum of INT_MAX - 1 that PRESENT-side silence lasted about 24 days at the shipped cadence with no warning. Whether it also bounds how long a node that leaves the graph BELOW grace sits unsettled depends on the same split: for an armed node that lasts for as long as the node stays gone, independent of grace (see “Bounded by evidence, not by age” below); for a never-armed one grace bounds that too, since absence matures it within grace + absence_grace + 1 ticks either way. The check runs on the wide integer, so a value that only fits after truncation is rejected rather than silently turned into a hair-trigger. Anything outside the range warns and keeps the default. |
prune_grace |
int | 60 |
Consecutive ticks an IDLE tracked node - both clocks at zero and no matured ownership, so nothing to lose - may stay ABSENT from the graph before its bookkeeping is reclaimed. A node carrying evidence is never reclaimed by age at all, however long it stays gone, so this key cannot erase a fault’s own state; the map is bounded instead by tracked_node_cap (see “Bounded by evidence, not by age”). Injected for every detector at plugin scope and overridable per detector; a value outside 0..3600 warns and this detector keeps its own default of 60. The range check runs on the wide integer, so an out-of-int-range value is rejected rather than truncated. The value is used as written - there is no grace + 1 clamp, because there is no longer anything for one to protect. |
tracked_node_cap |
int | 512 |
The most nodes this detector keeps bookkeeping for at once. It is what bounds the map, since evidence is never reclaimed by age (see “Bounded by evidence, not by age”): at the cap, idle entries are reclaimed first, then entries for DEPARTED nodes are collapsed into a count, and only if every tracked node is PRESENT and carrying evidence is a newly matched node refused - which withholds GRAPH_NODE_INACTIVE’s clear and is reported both in the log and on GET /x-medkit-watchdog. 512 is comfortably above every node in a realistic graph (a full Nav2 stack plus perception is roughly a hundred nodes; a ten-robot fleet sharing one domain a few hundred), so raising it is only needed where require_active legitimately matches more PRESENT nodes than that. Accepted range 1..16384 - 16384 is about 8 MB of bookkeeping at roughly 500 bytes per tracked node, and a deployment needing more distinct required-node identities alive at once has identity churn rather than a large fleet. Anything outside the range, including 0 (a cap of nothing would mean checking nothing), warns and keeps the default; the check runs on the wide integer, so an out-of-int-range value is rejected rather than truncated. |
node_death keys
Zero-config, unlike every detector above: there is no require_active-style list of nodes
to watch, because every armed App in the graph is a candidate.
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. |
miss_grace |
int | 2 |
Consecutive missed ticks a tracked node tolerates before being reported dead - a death is reported once misses EXCEED this, i.e. after miss_grace + 1 ticks. Accepted range 0..3600, and additionally floored to whatever wall-clock window the configured tick_interval_ms needs (see “The wall-clock floor” below); a value below that floor is silently raised to it, with a warning naming the ms window it actually spans. |
prune_grace |
int | 60 |
Plugin-injected default (overridable here); how long a key being DURABLY suppressed (see “Suppression” below) may sit unreported before its bookkeeping is reclaimed. Accepted range 0..3600. The value actually used is max(prune_grace, miss_grace + 1), a silent internal floor rather than a rejection: without it, a durably-suppressed key could be reclaimed the very tick it would first have become eligible to report, losing the report rather than merely suppressing it. |
allowlist |
string[] | [] |
Node identities never reported dead. A dead key matches if it - or a name the same node is also known by - is present verbatim: the key’s own fqn, its bare leaf, or (captured while the node was still present) its App::id. Has no effect unless named in suppress; naming it without a configured list is a no-op. |
suppress |
string[] | [] |
Suppression mechanisms to activate: "allowlist" and "lifecycle". Any other entry warns as unrecognized and is ignored; a non-string or non-array entry warns the same way. |
tracked_node_cap |
int | 512 |
The most DEPARTED (not-yet-confirmed-dead or already-collapsed) identities this detector keeps individual bookkeeping for at once - a present/armed node is never counted against it and never refused tracking. Accepted range 1..16384; 0 is refused rather than clamped, since a cap of nothing would mean tracking nothing. See “Bounded by evidence, not by age” below for what it actually bounds and why, unlike the sibling detector’s identical-looking cap, a newcomer here is never the thing refused. |
tick_interval_ms is not an own key of this detector, but this is the one detector that
reads the plugin-injected value directly, to compute miss_grace’s wall-clock floor below.
Liveness, and why membership in the snapshot is not enough. App::is_online is what
this detector tracks, never mere presence in the entity snapshot. In runtime-only discovery
the two happen to coincide - a dead node’s App leaves the snapshot entirely - but a manifest
keeps a bound App present with only is_online cleared once its ROS binding disappears, and
hybrid discovery inherits that same shape; counting snapshot membership alone would make a
manifest-declared node immortal. A managed lifecycle node that merely deactivates keeps
is_online: true - its process is still running, only its ROS 2 lifecycle state changed -
so a deactivation is never mistaken for a death either; that is lifecycle_expectation’s
concern, not this one’s (see “The boundary with lifecycle_expectation” below).
Tracking is keyed on the STABLE fqn (App::effective_fqn()), never App::id: an id is
recomputed every sweep and only gains a namespace prefix once a same-bare-name collision
currently exists anywhere in the graph, so a live node’s id can change out from under a key
built from it.
What is never tracked.
| Excluded | Why |
|---|---|
Peer-aggregated apps (app.source starting peer:) |
A peer app carries no ROS binding of its own, so effective_fqn() is empty for every one of them; tracking them would collapse an entire peer fleet onto one "" key, and one online peer app would mask every other peer app’s departure. |
_ros2cli_<pid> nodes |
Every ros2 CLI invocation (ros2 topic echo, ros2 param get, …) spins up a real, short-lived node under this prefix - rcl’s own hidden-node naming convention, matched structurally rather than guessed at. Letting these through would accumulate one permanently “dead” entry per CLI invocation for the life of the gateway. |
| An App that never comes online | A key is admitted to tracking only once the reliability gate has said this detector OWNS its departure - armed at least once, and on one of the two GROUNDS ownership can rest on (a lifecycle state read as active, or no lifecycle to read at all; the third case, a state nobody could read, is admitted PROVISIONALLY and is described in the row below). A manifest App whose binding never starts is never armed and so can never be falsely called dead; a node still inside its warmup_cycles window gets the identical protection. Once tracked, though, a node’s continued life-or-death judgement rests on PRESENCE alone, not on staying armed - a tracked node that later goes lifecycle-inactive is not thereby mistaken for dead. |
| A managed App whose lifecycle state has not been read YET | Not tracked, and only for as long as that is still true. The gate’s own raise permission is answered PERMISSIVELY for a managed node whose GetState has not answered (LifecycleWatcher::node_ok()) - without that, qos_mismatch, orphan and param_drift would all fall silent about a node whose lifecycle service is broken. Permission is not knowledge, though, so tracking is withheld while a measurement may still arrive. LifecycleWatcher charges a GetState re-seed budget per node, and only for a read that actually ran, so “still asking” is a fact rather than a guess. Once that budget is spent the node is tracked here after all: nothing will ASK again, and lifecycle_expectation looks at nothing unless an operator named the node in require_active (empty by default), so withholding it any longer would mean the death is reported by nobody. That admission is PROVISIONAL, not final. The ~/transition_event subscription outlives the seed budget, so a label can still arrive without being asked for, and one that reads non-active says the node was the lifecycle detector’s all along - so the presence detector hands it back on the spot, while it is still alive. Waiting for the budget alone would only have delayed the wrong report, not prevented it. |
File truncated at 100 lines see the full file
CHANGELOG
Changelog for package ros2_medkit_graph_watchdog
0.7.0 (2026-08-27)
- Initial release of the package: a gateway plugin that raises faults
for silent failures in the ROS 2 graph - problems that leave every
node alive and every topic present, so nothing in the stack reports
them today. The package carries the plugin skeleton, a central
reliability gate that holds raises until the graph has quiesced, and
a detector registry. Detectors are configured under
plugins.graph_watchdog.<key>, each with araise/advisory/offmode, and an unknown key underdetectors.<id>is reported rather than silently ignored, so a typo cannot quietly disable a check (#571) -
qos_mismatchdetector: raisesGRAPH_QOS_MISMATCHfor a publisher and subscriber whose QoS profiles cannot match, which leaves the connection silently unestablished (#571) -
orphandetector: raisesGRAPH_ORPHANfor a one-sided topic - a publisher with no subscriber, or the reverse - but only when a complementary near-miss counterpart exists: same message type, a namespace and leaf within the configured edit distance, and the opposite side. A lone unsubscribed topic raises nothing. The pair is the signature of a remap or a topic-name typo, which is what the detector is for (#578) -
param_driftdetector: raisesGRAPH_PARAM_DRIFTwhen a node's live parameter value diverges from the declared expectation (#580) -
lifecycle_expectationdetector: raisesGRAPH_NODE_INACTIVEfor a node named inrequire_activethat is not in theactivelifecycle state once itsgracewindow has passed. It raises two further codes for the cases where the state could not be established at all:GRAPH_NODE_UNREADABLEwhen the lifecycle service does not answer, andGRAPH_NODE_NOT_MANAGEDwhen the named node has no lifecycle to read. These are distinct codes on the wire, so a consumer filtering onGRAPH_NODE_INACTIVEalone sees neither unmeasured case. A node is matched by itsApp::id, its full FQN, or the bare leaf of that FQN, and the grace streak is counted per node, so naming one node in two documented forms cannot halve the grace that was configured (#587) -
node_deathdetector: raisesGRAPH_NODE_DISAPPEAREDfor a node that leaves the graph when nothing else is left to report it, with a suppression framework that decides ownership of a departure from knowledge rather than from silence. A node whose lifecycle state was never readable is admitted only provisionally, and is handed back the moment a label arrives saying the departure belonged to the lifecycle detector instead. Zero-config: there is no list of nodes to maintain (#625, #624) - Two silent-fault classes are not delivered in this release.
GRAPH_TF_STALEandGRAPH_LATENCY_BUDGEThave their fault codes reserved in the frozenGRAPH_*namespace and land in later changes, each against its own issue - The plugin keeps its ROS entities off the gateway executor, so an entity created and destroyed while the gateway runs cannot have its destructor run on an executor thread concurrently with a create on the same node
- Contributors: \@bburda
Package Dependencies
System Dependencies
Dependant Packages
No known dependants.
Launch files
No launch files found
Messages
No message files found.
Services
No service files found
Plugins
No plugins found.
Recent questions tagged ros2_medkit_graph_watchdog at Robotics Stack Exchange
No version for distro crystal showing humble. Known supported distros are highlighted in the buttons above.
Package Summary
| Version | 0.7.0 |
| License | Apache-2.0 |
| Build type | AMENT_CMAKE |
| Use | RECOMMENDED |
Repository Summary
| Checkout URI | https://github.com/selfpatch/ros2_medkit.git |
| VCS Type | git |
| VCS Version | main |
| Last Updated | 2026-09-06 |
| Dev Status | DEVELOPED |
| Released | RELEASED |
| Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
Gateway plugin that detects silent faults in the ROS 2 graph: QoS mismatches, orphaned topics left by a name typo, dead nodes, parameter drift and unmet lifecycle expectations.
Maintainers
- bburda
Authors
No additional authors.
ros2_medkit_graph_watchdog
Gateway plugin that detects silent faults in the ROS 2 graph: failures where every node is up, nothing logs an error, and the robot is still broken.
Detectors read the graph and raise faults through a ReportFault service client on the
gateway node; the faults surface via FaultManager on the gateway /faults API. That
client, like the lifecycle subscriptions below, lives in the plugin’s own callback group
and is driven by the plugin’s own executor from the tick thread, never by the gateway’s
ROS executor.
This package carries the plugin skeleton, the central reliability gate that holds raises
until the graph has quiesced, and five detectors, qos_mismatch, orphan,
param_drift, lifecycle_expectation and node_death. Two silent-fault classes remain
undelivered, GRAPH_TF_STALE and GRAPH_LATENCY_BUDGET; they land in follow-up changes,
each against its own issue, and their fault codes are already reserved in the frozen
GRAPH_* namespace (see “Fault codes”).
Build
With the gateway built and sourced:
colcon build --packages-select ros2_medkit_graph_watchdog
colcon test --packages-select ros2_medkit_graph_watchdog && colcon test-result --verbose
Load into the gateway
ros2 run ros2_medkit_gateway gateway_node --ros-args \
-p plugins:="[graph_watchdog]" \
-p "plugins.graph_watchdog.path:=$(ros2 pkg prefix ros2_medkit_graph_watchdog)/lib/ros2_medkit_graph_watchdog/libros2_medkit_graph_watchdog.so"
Configuration (plugins.graph_watchdog.<key>)
| Key | Type | Default | Meaning |
|---|---|---|---|
tick_interval_ms |
int | 1000 |
Detector tick cadence. Accepted range 1..INT_MAX; the check runs on the wide integer, so a value past INT_MAX is rejected rather than truncated back into the band. |
warmup_cycles |
int | 5 |
An entity must be continuously present for this many ticks before it arms. A mid-run restart re-warms it. Enforced centrally, see “Reliability (bringup-quiesce)” below. Accepted range 0..INT_MAX, checked on the wide integer. |
prune_grace |
int | 60 |
Default injected into every detector’s own config before configure(); a per-detector detectors.<id>.prune_grace overrides it. Used by detectors that keep per-key bookkeeping. Accepted range 0..INT_MAX here, checked on the wide integer; each detector re-validates it against its own documented range before using it. |
detectors.<id>.mode |
string | bool | raise |
raise = push faults; advisory = observe, do not push; off = disabled. Bare off/no (YAML booleans) also disable; bare on/yes mean raise. |
detectors.<id>.<field> |
any | - | Per-detector thresholds, passed to that detector’s configure(). |
Nested delivery. Keys are written dotted in YAML or --ros-args, and the gateway
delivers them to the plugin as a nested object. A bare off is a YAML 1.1 boolean, so
mode: off arrives as false, not the string "off"; both forms disable the detector.
Unknown keys are reported. Anything under detectors.<id> that the detector does not
read produces one startup warning naming the key and listing the ones that exist, so a
typo like allow_list: for allowlist: cannot silently do nothing. A typo in the detector
id itself is warned about the same way, with the registered ids listed.
lifecycle_expectation keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. |
require_active |
string[] | [] |
Node names that must be in the active lifecycle state, each matched against a live node by its App::id, its full FQN (/ns/name), or the bare leaf of that FQN. A bare name matches that node in EVERY namespace (a fleet-wide “all controller_servers must be active”); use a full FQN to pin one robot’s. Both the bare and the FQN form match against the node’s stable FQN, so they survive the App::id renaming that a same-bare-name collision triggers; an entry written as an App::id also works, but can stop matching on a multi-robot graph for exactly that reason. Empty = the detector checks nothing and emits nothing. A require_active that is not a string array warns and is ignored; an empty-string entry warns and is skipped - never dropped silently, since the operator would go on believing the node is covered. |
grace |
int | 5 |
Consecutive not-active ticks a required node tolerates before being reported inactive - bringup time for a managed node to reach active (configure + activate) before the expectation is enforced. Counted per NODE, not per matching entry: a node named by both a bare-name and a full-FQN entry still advances its streak once per tick, so mixing the two documented forms cannot halve the grace that was configured. An ALREADY-CONFIRMED streak keeps its content while the node is ABSENT, so a node that leaves the graph after being confirmed stays confirmed; a streak that has not yet crossed grace on a node the presence detector could also report (it was armed at least once) is HELD rather than advanced while absent, so a node is never confirmed out of ticks gathered while nobody could observe it - only a PRESENT tick may still cross grace for that node. A node that was NEVER armed gets no such hold: node_death can never track it either, so absence keeps advancing its streak too, the same as a present tick would. Accepted range 0..300 - five minutes at the shipped 1 s cadence, and already an extravagant allowance for a managed node to reach active. The upper end is a real bound rather than a formality: grace decides how long a PRESENT node may go unreported, and how long a node returning from a departure still inactive takes to (re-)mature, so at the old maximum of INT_MAX - 1 that PRESENT-side silence lasted about 24 days at the shipped cadence with no warning. Whether it also bounds how long a node that leaves the graph BELOW grace sits unsettled depends on the same split: for an armed node that lasts for as long as the node stays gone, independent of grace (see “Bounded by evidence, not by age” below); for a never-armed one grace bounds that too, since absence matures it within grace + absence_grace + 1 ticks either way. The check runs on the wide integer, so a value that only fits after truncation is rejected rather than silently turned into a hair-trigger. Anything outside the range warns and keeps the default. |
prune_grace |
int | 60 |
Consecutive ticks an IDLE tracked node - both clocks at zero and no matured ownership, so nothing to lose - may stay ABSENT from the graph before its bookkeeping is reclaimed. A node carrying evidence is never reclaimed by age at all, however long it stays gone, so this key cannot erase a fault’s own state; the map is bounded instead by tracked_node_cap (see “Bounded by evidence, not by age”). Injected for every detector at plugin scope and overridable per detector; a value outside 0..3600 warns and this detector keeps its own default of 60. The range check runs on the wide integer, so an out-of-int-range value is rejected rather than truncated. The value is used as written - there is no grace + 1 clamp, because there is no longer anything for one to protect. |
tracked_node_cap |
int | 512 |
The most nodes this detector keeps bookkeeping for at once. It is what bounds the map, since evidence is never reclaimed by age (see “Bounded by evidence, not by age”): at the cap, idle entries are reclaimed first, then entries for DEPARTED nodes are collapsed into a count, and only if every tracked node is PRESENT and carrying evidence is a newly matched node refused - which withholds GRAPH_NODE_INACTIVE’s clear and is reported both in the log and on GET /x-medkit-watchdog. 512 is comfortably above every node in a realistic graph (a full Nav2 stack plus perception is roughly a hundred nodes; a ten-robot fleet sharing one domain a few hundred), so raising it is only needed where require_active legitimately matches more PRESENT nodes than that. Accepted range 1..16384 - 16384 is about 8 MB of bookkeeping at roughly 500 bytes per tracked node, and a deployment needing more distinct required-node identities alive at once has identity churn rather than a large fleet. Anything outside the range, including 0 (a cap of nothing would mean checking nothing), warns and keeps the default; the check runs on the wide integer, so an out-of-int-range value is rejected rather than truncated. |
node_death keys
Zero-config, unlike every detector above: there is no require_active-style list of nodes
to watch, because every armed App in the graph is a candidate.
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. |
miss_grace |
int | 2 |
Consecutive missed ticks a tracked node tolerates before being reported dead - a death is reported once misses EXCEED this, i.e. after miss_grace + 1 ticks. Accepted range 0..3600, and additionally floored to whatever wall-clock window the configured tick_interval_ms needs (see “The wall-clock floor” below); a value below that floor is silently raised to it, with a warning naming the ms window it actually spans. |
prune_grace |
int | 60 |
Plugin-injected default (overridable here); how long a key being DURABLY suppressed (see “Suppression” below) may sit unreported before its bookkeeping is reclaimed. Accepted range 0..3600. The value actually used is max(prune_grace, miss_grace + 1), a silent internal floor rather than a rejection: without it, a durably-suppressed key could be reclaimed the very tick it would first have become eligible to report, losing the report rather than merely suppressing it. |
allowlist |
string[] | [] |
Node identities never reported dead. A dead key matches if it - or a name the same node is also known by - is present verbatim: the key’s own fqn, its bare leaf, or (captured while the node was still present) its App::id. Has no effect unless named in suppress; naming it without a configured list is a no-op. |
suppress |
string[] | [] |
Suppression mechanisms to activate: "allowlist" and "lifecycle". Any other entry warns as unrecognized and is ignored; a non-string or non-array entry warns the same way. |
tracked_node_cap |
int | 512 |
The most DEPARTED (not-yet-confirmed-dead or already-collapsed) identities this detector keeps individual bookkeeping for at once - a present/armed node is never counted against it and never refused tracking. Accepted range 1..16384; 0 is refused rather than clamped, since a cap of nothing would mean tracking nothing. See “Bounded by evidence, not by age” below for what it actually bounds and why, unlike the sibling detector’s identical-looking cap, a newcomer here is never the thing refused. |
tick_interval_ms is not an own key of this detector, but this is the one detector that
reads the plugin-injected value directly, to compute miss_grace’s wall-clock floor below.
Liveness, and why membership in the snapshot is not enough. App::is_online is what
this detector tracks, never mere presence in the entity snapshot. In runtime-only discovery
the two happen to coincide - a dead node’s App leaves the snapshot entirely - but a manifest
keeps a bound App present with only is_online cleared once its ROS binding disappears, and
hybrid discovery inherits that same shape; counting snapshot membership alone would make a
manifest-declared node immortal. A managed lifecycle node that merely deactivates keeps
is_online: true - its process is still running, only its ROS 2 lifecycle state changed -
so a deactivation is never mistaken for a death either; that is lifecycle_expectation’s
concern, not this one’s (see “The boundary with lifecycle_expectation” below).
Tracking is keyed on the STABLE fqn (App::effective_fqn()), never App::id: an id is
recomputed every sweep and only gains a namespace prefix once a same-bare-name collision
currently exists anywhere in the graph, so a live node’s id can change out from under a key
built from it.
What is never tracked.
| Excluded | Why |
|---|---|
Peer-aggregated apps (app.source starting peer:) |
A peer app carries no ROS binding of its own, so effective_fqn() is empty for every one of them; tracking them would collapse an entire peer fleet onto one "" key, and one online peer app would mask every other peer app’s departure. |
_ros2cli_<pid> nodes |
Every ros2 CLI invocation (ros2 topic echo, ros2 param get, …) spins up a real, short-lived node under this prefix - rcl’s own hidden-node naming convention, matched structurally rather than guessed at. Letting these through would accumulate one permanently “dead” entry per CLI invocation for the life of the gateway. |
| An App that never comes online | A key is admitted to tracking only once the reliability gate has said this detector OWNS its departure - armed at least once, and on one of the two GROUNDS ownership can rest on (a lifecycle state read as active, or no lifecycle to read at all; the third case, a state nobody could read, is admitted PROVISIONALLY and is described in the row below). A manifest App whose binding never starts is never armed and so can never be falsely called dead; a node still inside its warmup_cycles window gets the identical protection. Once tracked, though, a node’s continued life-or-death judgement rests on PRESENCE alone, not on staying armed - a tracked node that later goes lifecycle-inactive is not thereby mistaken for dead. |
| A managed App whose lifecycle state has not been read YET | Not tracked, and only for as long as that is still true. The gate’s own raise permission is answered PERMISSIVELY for a managed node whose GetState has not answered (LifecycleWatcher::node_ok()) - without that, qos_mismatch, orphan and param_drift would all fall silent about a node whose lifecycle service is broken. Permission is not knowledge, though, so tracking is withheld while a measurement may still arrive. LifecycleWatcher charges a GetState re-seed budget per node, and only for a read that actually ran, so “still asking” is a fact rather than a guess. Once that budget is spent the node is tracked here after all: nothing will ASK again, and lifecycle_expectation looks at nothing unless an operator named the node in require_active (empty by default), so withholding it any longer would mean the death is reported by nobody. That admission is PROVISIONAL, not final. The ~/transition_event subscription outlives the seed budget, so a label can still arrive without being asked for, and one that reads non-active says the node was the lifecycle detector’s all along - so the presence detector hands it back on the spot, while it is still alive. Waiting for the budget alone would only have delayed the wrong report, not prevented it. |
File truncated at 100 lines see the full file
CHANGELOG
Changelog for package ros2_medkit_graph_watchdog
0.7.0 (2026-08-27)
- Initial release of the package: a gateway plugin that raises faults
for silent failures in the ROS 2 graph - problems that leave every
node alive and every topic present, so nothing in the stack reports
them today. The package carries the plugin skeleton, a central
reliability gate that holds raises until the graph has quiesced, and
a detector registry. Detectors are configured under
plugins.graph_watchdog.<key>, each with araise/advisory/offmode, and an unknown key underdetectors.<id>is reported rather than silently ignored, so a typo cannot quietly disable a check (#571) -
qos_mismatchdetector: raisesGRAPH_QOS_MISMATCHfor a publisher and subscriber whose QoS profiles cannot match, which leaves the connection silently unestablished (#571) -
orphandetector: raisesGRAPH_ORPHANfor a one-sided topic - a publisher with no subscriber, or the reverse - but only when a complementary near-miss counterpart exists: same message type, a namespace and leaf within the configured edit distance, and the opposite side. A lone unsubscribed topic raises nothing. The pair is the signature of a remap or a topic-name typo, which is what the detector is for (#578) -
param_driftdetector: raisesGRAPH_PARAM_DRIFTwhen a node's live parameter value diverges from the declared expectation (#580) -
lifecycle_expectationdetector: raisesGRAPH_NODE_INACTIVEfor a node named inrequire_activethat is not in theactivelifecycle state once itsgracewindow has passed. It raises two further codes for the cases where the state could not be established at all:GRAPH_NODE_UNREADABLEwhen the lifecycle service does not answer, andGRAPH_NODE_NOT_MANAGEDwhen the named node has no lifecycle to read. These are distinct codes on the wire, so a consumer filtering onGRAPH_NODE_INACTIVEalone sees neither unmeasured case. A node is matched by itsApp::id, its full FQN, or the bare leaf of that FQN, and the grace streak is counted per node, so naming one node in two documented forms cannot halve the grace that was configured (#587) -
node_deathdetector: raisesGRAPH_NODE_DISAPPEAREDfor a node that leaves the graph when nothing else is left to report it, with a suppression framework that decides ownership of a departure from knowledge rather than from silence. A node whose lifecycle state was never readable is admitted only provisionally, and is handed back the moment a label arrives saying the departure belonged to the lifecycle detector instead. Zero-config: there is no list of nodes to maintain (#625, #624) - Two silent-fault classes are not delivered in this release.
GRAPH_TF_STALEandGRAPH_LATENCY_BUDGEThave their fault codes reserved in the frozenGRAPH_*namespace and land in later changes, each against its own issue - The plugin keeps its ROS entities off the gateway executor, so an entity created and destroyed while the gateway runs cannot have its destructor run on an executor thread concurrently with a create on the same node
- Contributors: \@bburda
Package Dependencies
System Dependencies
Dependant Packages
No known dependants.
Launch files
No launch files found
Messages
No message files found.
Services
No service files found
Plugins
No plugins found.
Recent questions tagged ros2_medkit_graph_watchdog at Robotics Stack Exchange
No version for distro eloquent showing humble. Known supported distros are highlighted in the buttons above.
Package Summary
| Version | 0.7.0 |
| License | Apache-2.0 |
| Build type | AMENT_CMAKE |
| Use | RECOMMENDED |
Repository Summary
| Checkout URI | https://github.com/selfpatch/ros2_medkit.git |
| VCS Type | git |
| VCS Version | main |
| Last Updated | 2026-09-06 |
| Dev Status | DEVELOPED |
| Released | RELEASED |
| Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
Gateway plugin that detects silent faults in the ROS 2 graph: QoS mismatches, orphaned topics left by a name typo, dead nodes, parameter drift and unmet lifecycle expectations.
Maintainers
- bburda
Authors
No additional authors.
ros2_medkit_graph_watchdog
Gateway plugin that detects silent faults in the ROS 2 graph: failures where every node is up, nothing logs an error, and the robot is still broken.
Detectors read the graph and raise faults through a ReportFault service client on the
gateway node; the faults surface via FaultManager on the gateway /faults API. That
client, like the lifecycle subscriptions below, lives in the plugin’s own callback group
and is driven by the plugin’s own executor from the tick thread, never by the gateway’s
ROS executor.
This package carries the plugin skeleton, the central reliability gate that holds raises
until the graph has quiesced, and five detectors, qos_mismatch, orphan,
param_drift, lifecycle_expectation and node_death. Two silent-fault classes remain
undelivered, GRAPH_TF_STALE and GRAPH_LATENCY_BUDGET; they land in follow-up changes,
each against its own issue, and their fault codes are already reserved in the frozen
GRAPH_* namespace (see “Fault codes”).
Build
With the gateway built and sourced:
colcon build --packages-select ros2_medkit_graph_watchdog
colcon test --packages-select ros2_medkit_graph_watchdog && colcon test-result --verbose
Load into the gateway
ros2 run ros2_medkit_gateway gateway_node --ros-args \
-p plugins:="[graph_watchdog]" \
-p "plugins.graph_watchdog.path:=$(ros2 pkg prefix ros2_medkit_graph_watchdog)/lib/ros2_medkit_graph_watchdog/libros2_medkit_graph_watchdog.so"
Configuration (plugins.graph_watchdog.<key>)
| Key | Type | Default | Meaning |
|---|---|---|---|
tick_interval_ms |
int | 1000 |
Detector tick cadence. Accepted range 1..INT_MAX; the check runs on the wide integer, so a value past INT_MAX is rejected rather than truncated back into the band. |
warmup_cycles |
int | 5 |
An entity must be continuously present for this many ticks before it arms. A mid-run restart re-warms it. Enforced centrally, see “Reliability (bringup-quiesce)” below. Accepted range 0..INT_MAX, checked on the wide integer. |
prune_grace |
int | 60 |
Default injected into every detector’s own config before configure(); a per-detector detectors.<id>.prune_grace overrides it. Used by detectors that keep per-key bookkeeping. Accepted range 0..INT_MAX here, checked on the wide integer; each detector re-validates it against its own documented range before using it. |
detectors.<id>.mode |
string | bool | raise |
raise = push faults; advisory = observe, do not push; off = disabled. Bare off/no (YAML booleans) also disable; bare on/yes mean raise. |
detectors.<id>.<field> |
any | - | Per-detector thresholds, passed to that detector’s configure(). |
Nested delivery. Keys are written dotted in YAML or --ros-args, and the gateway
delivers them to the plugin as a nested object. A bare off is a YAML 1.1 boolean, so
mode: off arrives as false, not the string "off"; both forms disable the detector.
Unknown keys are reported. Anything under detectors.<id> that the detector does not
read produces one startup warning naming the key and listing the ones that exist, so a
typo like allow_list: for allowlist: cannot silently do nothing. A typo in the detector
id itself is warned about the same way, with the registered ids listed.
lifecycle_expectation keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. |
require_active |
string[] | [] |
Node names that must be in the active lifecycle state, each matched against a live node by its App::id, its full FQN (/ns/name), or the bare leaf of that FQN. A bare name matches that node in EVERY namespace (a fleet-wide “all controller_servers must be active”); use a full FQN to pin one robot’s. Both the bare and the FQN form match against the node’s stable FQN, so they survive the App::id renaming that a same-bare-name collision triggers; an entry written as an App::id also works, but can stop matching on a multi-robot graph for exactly that reason. Empty = the detector checks nothing and emits nothing. A require_active that is not a string array warns and is ignored; an empty-string entry warns and is skipped - never dropped silently, since the operator would go on believing the node is covered. |
grace |
int | 5 |
Consecutive not-active ticks a required node tolerates before being reported inactive - bringup time for a managed node to reach active (configure + activate) before the expectation is enforced. Counted per NODE, not per matching entry: a node named by both a bare-name and a full-FQN entry still advances its streak once per tick, so mixing the two documented forms cannot halve the grace that was configured. An ALREADY-CONFIRMED streak keeps its content while the node is ABSENT, so a node that leaves the graph after being confirmed stays confirmed; a streak that has not yet crossed grace on a node the presence detector could also report (it was armed at least once) is HELD rather than advanced while absent, so a node is never confirmed out of ticks gathered while nobody could observe it - only a PRESENT tick may still cross grace for that node. A node that was NEVER armed gets no such hold: node_death can never track it either, so absence keeps advancing its streak too, the same as a present tick would. Accepted range 0..300 - five minutes at the shipped 1 s cadence, and already an extravagant allowance for a managed node to reach active. The upper end is a real bound rather than a formality: grace decides how long a PRESENT node may go unreported, and how long a node returning from a departure still inactive takes to (re-)mature, so at the old maximum of INT_MAX - 1 that PRESENT-side silence lasted about 24 days at the shipped cadence with no warning. Whether it also bounds how long a node that leaves the graph BELOW grace sits unsettled depends on the same split: for an armed node that lasts for as long as the node stays gone, independent of grace (see “Bounded by evidence, not by age” below); for a never-armed one grace bounds that too, since absence matures it within grace + absence_grace + 1 ticks either way. The check runs on the wide integer, so a value that only fits after truncation is rejected rather than silently turned into a hair-trigger. Anything outside the range warns and keeps the default. |
prune_grace |
int | 60 |
Consecutive ticks an IDLE tracked node - both clocks at zero and no matured ownership, so nothing to lose - may stay ABSENT from the graph before its bookkeeping is reclaimed. A node carrying evidence is never reclaimed by age at all, however long it stays gone, so this key cannot erase a fault’s own state; the map is bounded instead by tracked_node_cap (see “Bounded by evidence, not by age”). Injected for every detector at plugin scope and overridable per detector; a value outside 0..3600 warns and this detector keeps its own default of 60. The range check runs on the wide integer, so an out-of-int-range value is rejected rather than truncated. The value is used as written - there is no grace + 1 clamp, because there is no longer anything for one to protect. |
tracked_node_cap |
int | 512 |
The most nodes this detector keeps bookkeeping for at once. It is what bounds the map, since evidence is never reclaimed by age (see “Bounded by evidence, not by age”): at the cap, idle entries are reclaimed first, then entries for DEPARTED nodes are collapsed into a count, and only if every tracked node is PRESENT and carrying evidence is a newly matched node refused - which withholds GRAPH_NODE_INACTIVE’s clear and is reported both in the log and on GET /x-medkit-watchdog. 512 is comfortably above every node in a realistic graph (a full Nav2 stack plus perception is roughly a hundred nodes; a ten-robot fleet sharing one domain a few hundred), so raising it is only needed where require_active legitimately matches more PRESENT nodes than that. Accepted range 1..16384 - 16384 is about 8 MB of bookkeeping at roughly 500 bytes per tracked node, and a deployment needing more distinct required-node identities alive at once has identity churn rather than a large fleet. Anything outside the range, including 0 (a cap of nothing would mean checking nothing), warns and keeps the default; the check runs on the wide integer, so an out-of-int-range value is rejected rather than truncated. |
node_death keys
Zero-config, unlike every detector above: there is no require_active-style list of nodes
to watch, because every armed App in the graph is a candidate.
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. |
miss_grace |
int | 2 |
Consecutive missed ticks a tracked node tolerates before being reported dead - a death is reported once misses EXCEED this, i.e. after miss_grace + 1 ticks. Accepted range 0..3600, and additionally floored to whatever wall-clock window the configured tick_interval_ms needs (see “The wall-clock floor” below); a value below that floor is silently raised to it, with a warning naming the ms window it actually spans. |
prune_grace |
int | 60 |
Plugin-injected default (overridable here); how long a key being DURABLY suppressed (see “Suppression” below) may sit unreported before its bookkeeping is reclaimed. Accepted range 0..3600. The value actually used is max(prune_grace, miss_grace + 1), a silent internal floor rather than a rejection: without it, a durably-suppressed key could be reclaimed the very tick it would first have become eligible to report, losing the report rather than merely suppressing it. |
allowlist |
string[] | [] |
Node identities never reported dead. A dead key matches if it - or a name the same node is also known by - is present verbatim: the key’s own fqn, its bare leaf, or (captured while the node was still present) its App::id. Has no effect unless named in suppress; naming it without a configured list is a no-op. |
suppress |
string[] | [] |
Suppression mechanisms to activate: "allowlist" and "lifecycle". Any other entry warns as unrecognized and is ignored; a non-string or non-array entry warns the same way. |
tracked_node_cap |
int | 512 |
The most DEPARTED (not-yet-confirmed-dead or already-collapsed) identities this detector keeps individual bookkeeping for at once - a present/armed node is never counted against it and never refused tracking. Accepted range 1..16384; 0 is refused rather than clamped, since a cap of nothing would mean tracking nothing. See “Bounded by evidence, not by age” below for what it actually bounds and why, unlike the sibling detector’s identical-looking cap, a newcomer here is never the thing refused. |
tick_interval_ms is not an own key of this detector, but this is the one detector that
reads the plugin-injected value directly, to compute miss_grace’s wall-clock floor below.
Liveness, and why membership in the snapshot is not enough. App::is_online is what
this detector tracks, never mere presence in the entity snapshot. In runtime-only discovery
the two happen to coincide - a dead node’s App leaves the snapshot entirely - but a manifest
keeps a bound App present with only is_online cleared once its ROS binding disappears, and
hybrid discovery inherits that same shape; counting snapshot membership alone would make a
manifest-declared node immortal. A managed lifecycle node that merely deactivates keeps
is_online: true - its process is still running, only its ROS 2 lifecycle state changed -
so a deactivation is never mistaken for a death either; that is lifecycle_expectation’s
concern, not this one’s (see “The boundary with lifecycle_expectation” below).
Tracking is keyed on the STABLE fqn (App::effective_fqn()), never App::id: an id is
recomputed every sweep and only gains a namespace prefix once a same-bare-name collision
currently exists anywhere in the graph, so a live node’s id can change out from under a key
built from it.
What is never tracked.
| Excluded | Why |
|---|---|
Peer-aggregated apps (app.source starting peer:) |
A peer app carries no ROS binding of its own, so effective_fqn() is empty for every one of them; tracking them would collapse an entire peer fleet onto one "" key, and one online peer app would mask every other peer app’s departure. |
_ros2cli_<pid> nodes |
Every ros2 CLI invocation (ros2 topic echo, ros2 param get, …) spins up a real, short-lived node under this prefix - rcl’s own hidden-node naming convention, matched structurally rather than guessed at. Letting these through would accumulate one permanently “dead” entry per CLI invocation for the life of the gateway. |
| An App that never comes online | A key is admitted to tracking only once the reliability gate has said this detector OWNS its departure - armed at least once, and on one of the two GROUNDS ownership can rest on (a lifecycle state read as active, or no lifecycle to read at all; the third case, a state nobody could read, is admitted PROVISIONALLY and is described in the row below). A manifest App whose binding never starts is never armed and so can never be falsely called dead; a node still inside its warmup_cycles window gets the identical protection. Once tracked, though, a node’s continued life-or-death judgement rests on PRESENCE alone, not on staying armed - a tracked node that later goes lifecycle-inactive is not thereby mistaken for dead. |
| A managed App whose lifecycle state has not been read YET | Not tracked, and only for as long as that is still true. The gate’s own raise permission is answered PERMISSIVELY for a managed node whose GetState has not answered (LifecycleWatcher::node_ok()) - without that, qos_mismatch, orphan and param_drift would all fall silent about a node whose lifecycle service is broken. Permission is not knowledge, though, so tracking is withheld while a measurement may still arrive. LifecycleWatcher charges a GetState re-seed budget per node, and only for a read that actually ran, so “still asking” is a fact rather than a guess. Once that budget is spent the node is tracked here after all: nothing will ASK again, and lifecycle_expectation looks at nothing unless an operator named the node in require_active (empty by default), so withholding it any longer would mean the death is reported by nobody. That admission is PROVISIONAL, not final. The ~/transition_event subscription outlives the seed budget, so a label can still arrive without being asked for, and one that reads non-active says the node was the lifecycle detector’s all along - so the presence detector hands it back on the spot, while it is still alive. Waiting for the budget alone would only have delayed the wrong report, not prevented it. |
File truncated at 100 lines see the full file
CHANGELOG
Changelog for package ros2_medkit_graph_watchdog
0.7.0 (2026-08-27)
- Initial release of the package: a gateway plugin that raises faults
for silent failures in the ROS 2 graph - problems that leave every
node alive and every topic present, so nothing in the stack reports
them today. The package carries the plugin skeleton, a central
reliability gate that holds raises until the graph has quiesced, and
a detector registry. Detectors are configured under
plugins.graph_watchdog.<key>, each with araise/advisory/offmode, and an unknown key underdetectors.<id>is reported rather than silently ignored, so a typo cannot quietly disable a check (#571) -
qos_mismatchdetector: raisesGRAPH_QOS_MISMATCHfor a publisher and subscriber whose QoS profiles cannot match, which leaves the connection silently unestablished (#571) -
orphandetector: raisesGRAPH_ORPHANfor a one-sided topic - a publisher with no subscriber, or the reverse - but only when a complementary near-miss counterpart exists: same message type, a namespace and leaf within the configured edit distance, and the opposite side. A lone unsubscribed topic raises nothing. The pair is the signature of a remap or a topic-name typo, which is what the detector is for (#578) -
param_driftdetector: raisesGRAPH_PARAM_DRIFTwhen a node's live parameter value diverges from the declared expectation (#580) -
lifecycle_expectationdetector: raisesGRAPH_NODE_INACTIVEfor a node named inrequire_activethat is not in theactivelifecycle state once itsgracewindow has passed. It raises two further codes for the cases where the state could not be established at all:GRAPH_NODE_UNREADABLEwhen the lifecycle service does not answer, andGRAPH_NODE_NOT_MANAGEDwhen the named node has no lifecycle to read. These are distinct codes on the wire, so a consumer filtering onGRAPH_NODE_INACTIVEalone sees neither unmeasured case. A node is matched by itsApp::id, its full FQN, or the bare leaf of that FQN, and the grace streak is counted per node, so naming one node in two documented forms cannot halve the grace that was configured (#587) -
node_deathdetector: raisesGRAPH_NODE_DISAPPEAREDfor a node that leaves the graph when nothing else is left to report it, with a suppression framework that decides ownership of a departure from knowledge rather than from silence. A node whose lifecycle state was never readable is admitted only provisionally, and is handed back the moment a label arrives saying the departure belonged to the lifecycle detector instead. Zero-config: there is no list of nodes to maintain (#625, #624) - Two silent-fault classes are not delivered in this release.
GRAPH_TF_STALEandGRAPH_LATENCY_BUDGEThave their fault codes reserved in the frozenGRAPH_*namespace and land in later changes, each against its own issue - The plugin keeps its ROS entities off the gateway executor, so an entity created and destroyed while the gateway runs cannot have its destructor run on an executor thread concurrently with a create on the same node
- Contributors: \@bburda
Package Dependencies
System Dependencies
Dependant Packages
No known dependants.
Launch files
No launch files found
Messages
No message files found.
Services
No service files found
Plugins
No plugins found.
Recent questions tagged ros2_medkit_graph_watchdog at Robotics Stack Exchange
No version for distro dashing showing humble. Known supported distros are highlighted in the buttons above.
Package Summary
| Version | 0.7.0 |
| License | Apache-2.0 |
| Build type | AMENT_CMAKE |
| Use | RECOMMENDED |
Repository Summary
| Checkout URI | https://github.com/selfpatch/ros2_medkit.git |
| VCS Type | git |
| VCS Version | main |
| Last Updated | 2026-09-06 |
| Dev Status | DEVELOPED |
| Released | RELEASED |
| Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
Gateway plugin that detects silent faults in the ROS 2 graph: QoS mismatches, orphaned topics left by a name typo, dead nodes, parameter drift and unmet lifecycle expectations.
Maintainers
- bburda
Authors
No additional authors.
ros2_medkit_graph_watchdog
Gateway plugin that detects silent faults in the ROS 2 graph: failures where every node is up, nothing logs an error, and the robot is still broken.
Detectors read the graph and raise faults through a ReportFault service client on the
gateway node; the faults surface via FaultManager on the gateway /faults API. That
client, like the lifecycle subscriptions below, lives in the plugin’s own callback group
and is driven by the plugin’s own executor from the tick thread, never by the gateway’s
ROS executor.
This package carries the plugin skeleton, the central reliability gate that holds raises
until the graph has quiesced, and five detectors, qos_mismatch, orphan,
param_drift, lifecycle_expectation and node_death. Two silent-fault classes remain
undelivered, GRAPH_TF_STALE and GRAPH_LATENCY_BUDGET; they land in follow-up changes,
each against its own issue, and their fault codes are already reserved in the frozen
GRAPH_* namespace (see “Fault codes”).
Build
With the gateway built and sourced:
colcon build --packages-select ros2_medkit_graph_watchdog
colcon test --packages-select ros2_medkit_graph_watchdog && colcon test-result --verbose
Load into the gateway
ros2 run ros2_medkit_gateway gateway_node --ros-args \
-p plugins:="[graph_watchdog]" \
-p "plugins.graph_watchdog.path:=$(ros2 pkg prefix ros2_medkit_graph_watchdog)/lib/ros2_medkit_graph_watchdog/libros2_medkit_graph_watchdog.so"
Configuration (plugins.graph_watchdog.<key>)
| Key | Type | Default | Meaning |
|---|---|---|---|
tick_interval_ms |
int | 1000 |
Detector tick cadence. Accepted range 1..INT_MAX; the check runs on the wide integer, so a value past INT_MAX is rejected rather than truncated back into the band. |
warmup_cycles |
int | 5 |
An entity must be continuously present for this many ticks before it arms. A mid-run restart re-warms it. Enforced centrally, see “Reliability (bringup-quiesce)” below. Accepted range 0..INT_MAX, checked on the wide integer. |
prune_grace |
int | 60 |
Default injected into every detector’s own config before configure(); a per-detector detectors.<id>.prune_grace overrides it. Used by detectors that keep per-key bookkeeping. Accepted range 0..INT_MAX here, checked on the wide integer; each detector re-validates it against its own documented range before using it. |
detectors.<id>.mode |
string | bool | raise |
raise = push faults; advisory = observe, do not push; off = disabled. Bare off/no (YAML booleans) also disable; bare on/yes mean raise. |
detectors.<id>.<field> |
any | - | Per-detector thresholds, passed to that detector’s configure(). |
Nested delivery. Keys are written dotted in YAML or --ros-args, and the gateway
delivers them to the plugin as a nested object. A bare off is a YAML 1.1 boolean, so
mode: off arrives as false, not the string "off"; both forms disable the detector.
Unknown keys are reported. Anything under detectors.<id> that the detector does not
read produces one startup warning naming the key and listing the ones that exist, so a
typo like allow_list: for allowlist: cannot silently do nothing. A typo in the detector
id itself is warned about the same way, with the registered ids listed.
lifecycle_expectation keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. |
require_active |
string[] | [] |
Node names that must be in the active lifecycle state, each matched against a live node by its App::id, its full FQN (/ns/name), or the bare leaf of that FQN. A bare name matches that node in EVERY namespace (a fleet-wide “all controller_servers must be active”); use a full FQN to pin one robot’s. Both the bare and the FQN form match against the node’s stable FQN, so they survive the App::id renaming that a same-bare-name collision triggers; an entry written as an App::id also works, but can stop matching on a multi-robot graph for exactly that reason. Empty = the detector checks nothing and emits nothing. A require_active that is not a string array warns and is ignored; an empty-string entry warns and is skipped - never dropped silently, since the operator would go on believing the node is covered. |
grace |
int | 5 |
Consecutive not-active ticks a required node tolerates before being reported inactive - bringup time for a managed node to reach active (configure + activate) before the expectation is enforced. Counted per NODE, not per matching entry: a node named by both a bare-name and a full-FQN entry still advances its streak once per tick, so mixing the two documented forms cannot halve the grace that was configured. An ALREADY-CONFIRMED streak keeps its content while the node is ABSENT, so a node that leaves the graph after being confirmed stays confirmed; a streak that has not yet crossed grace on a node the presence detector could also report (it was armed at least once) is HELD rather than advanced while absent, so a node is never confirmed out of ticks gathered while nobody could observe it - only a PRESENT tick may still cross grace for that node. A node that was NEVER armed gets no such hold: node_death can never track it either, so absence keeps advancing its streak too, the same as a present tick would. Accepted range 0..300 - five minutes at the shipped 1 s cadence, and already an extravagant allowance for a managed node to reach active. The upper end is a real bound rather than a formality: grace decides how long a PRESENT node may go unreported, and how long a node returning from a departure still inactive takes to (re-)mature, so at the old maximum of INT_MAX - 1 that PRESENT-side silence lasted about 24 days at the shipped cadence with no warning. Whether it also bounds how long a node that leaves the graph BELOW grace sits unsettled depends on the same split: for an armed node that lasts for as long as the node stays gone, independent of grace (see “Bounded by evidence, not by age” below); for a never-armed one grace bounds that too, since absence matures it within grace + absence_grace + 1 ticks either way. The check runs on the wide integer, so a value that only fits after truncation is rejected rather than silently turned into a hair-trigger. Anything outside the range warns and keeps the default. |
prune_grace |
int | 60 |
Consecutive ticks an IDLE tracked node - both clocks at zero and no matured ownership, so nothing to lose - may stay ABSENT from the graph before its bookkeeping is reclaimed. A node carrying evidence is never reclaimed by age at all, however long it stays gone, so this key cannot erase a fault’s own state; the map is bounded instead by tracked_node_cap (see “Bounded by evidence, not by age”). Injected for every detector at plugin scope and overridable per detector; a value outside 0..3600 warns and this detector keeps its own default of 60. The range check runs on the wide integer, so an out-of-int-range value is rejected rather than truncated. The value is used as written - there is no grace + 1 clamp, because there is no longer anything for one to protect. |
tracked_node_cap |
int | 512 |
The most nodes this detector keeps bookkeeping for at once. It is what bounds the map, since evidence is never reclaimed by age (see “Bounded by evidence, not by age”): at the cap, idle entries are reclaimed first, then entries for DEPARTED nodes are collapsed into a count, and only if every tracked node is PRESENT and carrying evidence is a newly matched node refused - which withholds GRAPH_NODE_INACTIVE’s clear and is reported both in the log and on GET /x-medkit-watchdog. 512 is comfortably above every node in a realistic graph (a full Nav2 stack plus perception is roughly a hundred nodes; a ten-robot fleet sharing one domain a few hundred), so raising it is only needed where require_active legitimately matches more PRESENT nodes than that. Accepted range 1..16384 - 16384 is about 8 MB of bookkeeping at roughly 500 bytes per tracked node, and a deployment needing more distinct required-node identities alive at once has identity churn rather than a large fleet. Anything outside the range, including 0 (a cap of nothing would mean checking nothing), warns and keeps the default; the check runs on the wide integer, so an out-of-int-range value is rejected rather than truncated. |
node_death keys
Zero-config, unlike every detector above: there is no require_active-style list of nodes
to watch, because every armed App in the graph is a candidate.
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. |
miss_grace |
int | 2 |
Consecutive missed ticks a tracked node tolerates before being reported dead - a death is reported once misses EXCEED this, i.e. after miss_grace + 1 ticks. Accepted range 0..3600, and additionally floored to whatever wall-clock window the configured tick_interval_ms needs (see “The wall-clock floor” below); a value below that floor is silently raised to it, with a warning naming the ms window it actually spans. |
prune_grace |
int | 60 |
Plugin-injected default (overridable here); how long a key being DURABLY suppressed (see “Suppression” below) may sit unreported before its bookkeeping is reclaimed. Accepted range 0..3600. The value actually used is max(prune_grace, miss_grace + 1), a silent internal floor rather than a rejection: without it, a durably-suppressed key could be reclaimed the very tick it would first have become eligible to report, losing the report rather than merely suppressing it. |
allowlist |
string[] | [] |
Node identities never reported dead. A dead key matches if it - or a name the same node is also known by - is present verbatim: the key’s own fqn, its bare leaf, or (captured while the node was still present) its App::id. Has no effect unless named in suppress; naming it without a configured list is a no-op. |
suppress |
string[] | [] |
Suppression mechanisms to activate: "allowlist" and "lifecycle". Any other entry warns as unrecognized and is ignored; a non-string or non-array entry warns the same way. |
tracked_node_cap |
int | 512 |
The most DEPARTED (not-yet-confirmed-dead or already-collapsed) identities this detector keeps individual bookkeeping for at once - a present/armed node is never counted against it and never refused tracking. Accepted range 1..16384; 0 is refused rather than clamped, since a cap of nothing would mean tracking nothing. See “Bounded by evidence, not by age” below for what it actually bounds and why, unlike the sibling detector’s identical-looking cap, a newcomer here is never the thing refused. |
tick_interval_ms is not an own key of this detector, but this is the one detector that
reads the plugin-injected value directly, to compute miss_grace’s wall-clock floor below.
Liveness, and why membership in the snapshot is not enough. App::is_online is what
this detector tracks, never mere presence in the entity snapshot. In runtime-only discovery
the two happen to coincide - a dead node’s App leaves the snapshot entirely - but a manifest
keeps a bound App present with only is_online cleared once its ROS binding disappears, and
hybrid discovery inherits that same shape; counting snapshot membership alone would make a
manifest-declared node immortal. A managed lifecycle node that merely deactivates keeps
is_online: true - its process is still running, only its ROS 2 lifecycle state changed -
so a deactivation is never mistaken for a death either; that is lifecycle_expectation’s
concern, not this one’s (see “The boundary with lifecycle_expectation” below).
Tracking is keyed on the STABLE fqn (App::effective_fqn()), never App::id: an id is
recomputed every sweep and only gains a namespace prefix once a same-bare-name collision
currently exists anywhere in the graph, so a live node’s id can change out from under a key
built from it.
What is never tracked.
| Excluded | Why |
|---|---|
Peer-aggregated apps (app.source starting peer:) |
A peer app carries no ROS binding of its own, so effective_fqn() is empty for every one of them; tracking them would collapse an entire peer fleet onto one "" key, and one online peer app would mask every other peer app’s departure. |
_ros2cli_<pid> nodes |
Every ros2 CLI invocation (ros2 topic echo, ros2 param get, …) spins up a real, short-lived node under this prefix - rcl’s own hidden-node naming convention, matched structurally rather than guessed at. Letting these through would accumulate one permanently “dead” entry per CLI invocation for the life of the gateway. |
| An App that never comes online | A key is admitted to tracking only once the reliability gate has said this detector OWNS its departure - armed at least once, and on one of the two GROUNDS ownership can rest on (a lifecycle state read as active, or no lifecycle to read at all; the third case, a state nobody could read, is admitted PROVISIONALLY and is described in the row below). A manifest App whose binding never starts is never armed and so can never be falsely called dead; a node still inside its warmup_cycles window gets the identical protection. Once tracked, though, a node’s continued life-or-death judgement rests on PRESENCE alone, not on staying armed - a tracked node that later goes lifecycle-inactive is not thereby mistaken for dead. |
| A managed App whose lifecycle state has not been read YET | Not tracked, and only for as long as that is still true. The gate’s own raise permission is answered PERMISSIVELY for a managed node whose GetState has not answered (LifecycleWatcher::node_ok()) - without that, qos_mismatch, orphan and param_drift would all fall silent about a node whose lifecycle service is broken. Permission is not knowledge, though, so tracking is withheld while a measurement may still arrive. LifecycleWatcher charges a GetState re-seed budget per node, and only for a read that actually ran, so “still asking” is a fact rather than a guess. Once that budget is spent the node is tracked here after all: nothing will ASK again, and lifecycle_expectation looks at nothing unless an operator named the node in require_active (empty by default), so withholding it any longer would mean the death is reported by nobody. That admission is PROVISIONAL, not final. The ~/transition_event subscription outlives the seed budget, so a label can still arrive without being asked for, and one that reads non-active says the node was the lifecycle detector’s all along - so the presence detector hands it back on the spot, while it is still alive. Waiting for the budget alone would only have delayed the wrong report, not prevented it. |
File truncated at 100 lines see the full file
CHANGELOG
Changelog for package ros2_medkit_graph_watchdog
0.7.0 (2026-08-27)
- Initial release of the package: a gateway plugin that raises faults
for silent failures in the ROS 2 graph - problems that leave every
node alive and every topic present, so nothing in the stack reports
them today. The package carries the plugin skeleton, a central
reliability gate that holds raises until the graph has quiesced, and
a detector registry. Detectors are configured under
plugins.graph_watchdog.<key>, each with araise/advisory/offmode, and an unknown key underdetectors.<id>is reported rather than silently ignored, so a typo cannot quietly disable a check (#571) -
qos_mismatchdetector: raisesGRAPH_QOS_MISMATCHfor a publisher and subscriber whose QoS profiles cannot match, which leaves the connection silently unestablished (#571) -
orphandetector: raisesGRAPH_ORPHANfor a one-sided topic - a publisher with no subscriber, or the reverse - but only when a complementary near-miss counterpart exists: same message type, a namespace and leaf within the configured edit distance, and the opposite side. A lone unsubscribed topic raises nothing. The pair is the signature of a remap or a topic-name typo, which is what the detector is for (#578) -
param_driftdetector: raisesGRAPH_PARAM_DRIFTwhen a node's live parameter value diverges from the declared expectation (#580) -
lifecycle_expectationdetector: raisesGRAPH_NODE_INACTIVEfor a node named inrequire_activethat is not in theactivelifecycle state once itsgracewindow has passed. It raises two further codes for the cases where the state could not be established at all:GRAPH_NODE_UNREADABLEwhen the lifecycle service does not answer, andGRAPH_NODE_NOT_MANAGEDwhen the named node has no lifecycle to read. These are distinct codes on the wire, so a consumer filtering onGRAPH_NODE_INACTIVEalone sees neither unmeasured case. A node is matched by itsApp::id, its full FQN, or the bare leaf of that FQN, and the grace streak is counted per node, so naming one node in two documented forms cannot halve the grace that was configured (#587) -
node_deathdetector: raisesGRAPH_NODE_DISAPPEAREDfor a node that leaves the graph when nothing else is left to report it, with a suppression framework that decides ownership of a departure from knowledge rather than from silence. A node whose lifecycle state was never readable is admitted only provisionally, and is handed back the moment a label arrives saying the departure belonged to the lifecycle detector instead. Zero-config: there is no list of nodes to maintain (#625, #624) - Two silent-fault classes are not delivered in this release.
GRAPH_TF_STALEandGRAPH_LATENCY_BUDGEThave their fault codes reserved in the frozenGRAPH_*namespace and land in later changes, each against its own issue - The plugin keeps its ROS entities off the gateway executor, so an entity created and destroyed while the gateway runs cannot have its destructor run on an executor thread concurrently with a create on the same node
- Contributors: \@bburda
Package Dependencies
System Dependencies
Dependant Packages
No known dependants.
Launch files
No launch files found
Messages
No message files found.
Services
No service files found
Plugins
No plugins found.
Recent questions tagged ros2_medkit_graph_watchdog at Robotics Stack Exchange
No version for distro galactic showing humble. Known supported distros are highlighted in the buttons above.
Package Summary
| Version | 0.7.0 |
| License | Apache-2.0 |
| Build type | AMENT_CMAKE |
| Use | RECOMMENDED |
Repository Summary
| Checkout URI | https://github.com/selfpatch/ros2_medkit.git |
| VCS Type | git |
| VCS Version | main |
| Last Updated | 2026-09-06 |
| Dev Status | DEVELOPED |
| Released | RELEASED |
| Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
Gateway plugin that detects silent faults in the ROS 2 graph: QoS mismatches, orphaned topics left by a name typo, dead nodes, parameter drift and unmet lifecycle expectations.
Maintainers
- bburda
Authors
No additional authors.
ros2_medkit_graph_watchdog
Gateway plugin that detects silent faults in the ROS 2 graph: failures where every node is up, nothing logs an error, and the robot is still broken.
Detectors read the graph and raise faults through a ReportFault service client on the
gateway node; the faults surface via FaultManager on the gateway /faults API. That
client, like the lifecycle subscriptions below, lives in the plugin’s own callback group
and is driven by the plugin’s own executor from the tick thread, never by the gateway’s
ROS executor.
This package carries the plugin skeleton, the central reliability gate that holds raises
until the graph has quiesced, and five detectors, qos_mismatch, orphan,
param_drift, lifecycle_expectation and node_death. Two silent-fault classes remain
undelivered, GRAPH_TF_STALE and GRAPH_LATENCY_BUDGET; they land in follow-up changes,
each against its own issue, and their fault codes are already reserved in the frozen
GRAPH_* namespace (see “Fault codes”).
Build
With the gateway built and sourced:
colcon build --packages-select ros2_medkit_graph_watchdog
colcon test --packages-select ros2_medkit_graph_watchdog && colcon test-result --verbose
Load into the gateway
ros2 run ros2_medkit_gateway gateway_node --ros-args \
-p plugins:="[graph_watchdog]" \
-p "plugins.graph_watchdog.path:=$(ros2 pkg prefix ros2_medkit_graph_watchdog)/lib/ros2_medkit_graph_watchdog/libros2_medkit_graph_watchdog.so"
Configuration (plugins.graph_watchdog.<key>)
| Key | Type | Default | Meaning |
|---|---|---|---|
tick_interval_ms |
int | 1000 |
Detector tick cadence. Accepted range 1..INT_MAX; the check runs on the wide integer, so a value past INT_MAX is rejected rather than truncated back into the band. |
warmup_cycles |
int | 5 |
An entity must be continuously present for this many ticks before it arms. A mid-run restart re-warms it. Enforced centrally, see “Reliability (bringup-quiesce)” below. Accepted range 0..INT_MAX, checked on the wide integer. |
prune_grace |
int | 60 |
Default injected into every detector’s own config before configure(); a per-detector detectors.<id>.prune_grace overrides it. Used by detectors that keep per-key bookkeeping. Accepted range 0..INT_MAX here, checked on the wide integer; each detector re-validates it against its own documented range before using it. |
detectors.<id>.mode |
string | bool | raise |
raise = push faults; advisory = observe, do not push; off = disabled. Bare off/no (YAML booleans) also disable; bare on/yes mean raise. |
detectors.<id>.<field> |
any | - | Per-detector thresholds, passed to that detector’s configure(). |
Nested delivery. Keys are written dotted in YAML or --ros-args, and the gateway
delivers them to the plugin as a nested object. A bare off is a YAML 1.1 boolean, so
mode: off arrives as false, not the string "off"; both forms disable the detector.
Unknown keys are reported. Anything under detectors.<id> that the detector does not
read produces one startup warning naming the key and listing the ones that exist, so a
typo like allow_list: for allowlist: cannot silently do nothing. A typo in the detector
id itself is warned about the same way, with the registered ids listed.
lifecycle_expectation keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. |
require_active |
string[] | [] |
Node names that must be in the active lifecycle state, each matched against a live node by its App::id, its full FQN (/ns/name), or the bare leaf of that FQN. A bare name matches that node in EVERY namespace (a fleet-wide “all controller_servers must be active”); use a full FQN to pin one robot’s. Both the bare and the FQN form match against the node’s stable FQN, so they survive the App::id renaming that a same-bare-name collision triggers; an entry written as an App::id also works, but can stop matching on a multi-robot graph for exactly that reason. Empty = the detector checks nothing and emits nothing. A require_active that is not a string array warns and is ignored; an empty-string entry warns and is skipped - never dropped silently, since the operator would go on believing the node is covered. |
grace |
int | 5 |
Consecutive not-active ticks a required node tolerates before being reported inactive - bringup time for a managed node to reach active (configure + activate) before the expectation is enforced. Counted per NODE, not per matching entry: a node named by both a bare-name and a full-FQN entry still advances its streak once per tick, so mixing the two documented forms cannot halve the grace that was configured. An ALREADY-CONFIRMED streak keeps its content while the node is ABSENT, so a node that leaves the graph after being confirmed stays confirmed; a streak that has not yet crossed grace on a node the presence detector could also report (it was armed at least once) is HELD rather than advanced while absent, so a node is never confirmed out of ticks gathered while nobody could observe it - only a PRESENT tick may still cross grace for that node. A node that was NEVER armed gets no such hold: node_death can never track it either, so absence keeps advancing its streak too, the same as a present tick would. Accepted range 0..300 - five minutes at the shipped 1 s cadence, and already an extravagant allowance for a managed node to reach active. The upper end is a real bound rather than a formality: grace decides how long a PRESENT node may go unreported, and how long a node returning from a departure still inactive takes to (re-)mature, so at the old maximum of INT_MAX - 1 that PRESENT-side silence lasted about 24 days at the shipped cadence with no warning. Whether it also bounds how long a node that leaves the graph BELOW grace sits unsettled depends on the same split: for an armed node that lasts for as long as the node stays gone, independent of grace (see “Bounded by evidence, not by age” below); for a never-armed one grace bounds that too, since absence matures it within grace + absence_grace + 1 ticks either way. The check runs on the wide integer, so a value that only fits after truncation is rejected rather than silently turned into a hair-trigger. Anything outside the range warns and keeps the default. |
prune_grace |
int | 60 |
Consecutive ticks an IDLE tracked node - both clocks at zero and no matured ownership, so nothing to lose - may stay ABSENT from the graph before its bookkeeping is reclaimed. A node carrying evidence is never reclaimed by age at all, however long it stays gone, so this key cannot erase a fault’s own state; the map is bounded instead by tracked_node_cap (see “Bounded by evidence, not by age”). Injected for every detector at plugin scope and overridable per detector; a value outside 0..3600 warns and this detector keeps its own default of 60. The range check runs on the wide integer, so an out-of-int-range value is rejected rather than truncated. The value is used as written - there is no grace + 1 clamp, because there is no longer anything for one to protect. |
tracked_node_cap |
int | 512 |
The most nodes this detector keeps bookkeeping for at once. It is what bounds the map, since evidence is never reclaimed by age (see “Bounded by evidence, not by age”): at the cap, idle entries are reclaimed first, then entries for DEPARTED nodes are collapsed into a count, and only if every tracked node is PRESENT and carrying evidence is a newly matched node refused - which withholds GRAPH_NODE_INACTIVE’s clear and is reported both in the log and on GET /x-medkit-watchdog. 512 is comfortably above every node in a realistic graph (a full Nav2 stack plus perception is roughly a hundred nodes; a ten-robot fleet sharing one domain a few hundred), so raising it is only needed where require_active legitimately matches more PRESENT nodes than that. Accepted range 1..16384 - 16384 is about 8 MB of bookkeeping at roughly 500 bytes per tracked node, and a deployment needing more distinct required-node identities alive at once has identity churn rather than a large fleet. Anything outside the range, including 0 (a cap of nothing would mean checking nothing), warns and keeps the default; the check runs on the wide integer, so an out-of-int-range value is rejected rather than truncated. |
node_death keys
Zero-config, unlike every detector above: there is no require_active-style list of nodes
to watch, because every armed App in the graph is a candidate.
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. |
miss_grace |
int | 2 |
Consecutive missed ticks a tracked node tolerates before being reported dead - a death is reported once misses EXCEED this, i.e. after miss_grace + 1 ticks. Accepted range 0..3600, and additionally floored to whatever wall-clock window the configured tick_interval_ms needs (see “The wall-clock floor” below); a value below that floor is silently raised to it, with a warning naming the ms window it actually spans. |
prune_grace |
int | 60 |
Plugin-injected default (overridable here); how long a key being DURABLY suppressed (see “Suppression” below) may sit unreported before its bookkeeping is reclaimed. Accepted range 0..3600. The value actually used is max(prune_grace, miss_grace + 1), a silent internal floor rather than a rejection: without it, a durably-suppressed key could be reclaimed the very tick it would first have become eligible to report, losing the report rather than merely suppressing it. |
allowlist |
string[] | [] |
Node identities never reported dead. A dead key matches if it - or a name the same node is also known by - is present verbatim: the key’s own fqn, its bare leaf, or (captured while the node was still present) its App::id. Has no effect unless named in suppress; naming it without a configured list is a no-op. |
suppress |
string[] | [] |
Suppression mechanisms to activate: "allowlist" and "lifecycle". Any other entry warns as unrecognized and is ignored; a non-string or non-array entry warns the same way. |
tracked_node_cap |
int | 512 |
The most DEPARTED (not-yet-confirmed-dead or already-collapsed) identities this detector keeps individual bookkeeping for at once - a present/armed node is never counted against it and never refused tracking. Accepted range 1..16384; 0 is refused rather than clamped, since a cap of nothing would mean tracking nothing. See “Bounded by evidence, not by age” below for what it actually bounds and why, unlike the sibling detector’s identical-looking cap, a newcomer here is never the thing refused. |
tick_interval_ms is not an own key of this detector, but this is the one detector that
reads the plugin-injected value directly, to compute miss_grace’s wall-clock floor below.
Liveness, and why membership in the snapshot is not enough. App::is_online is what
this detector tracks, never mere presence in the entity snapshot. In runtime-only discovery
the two happen to coincide - a dead node’s App leaves the snapshot entirely - but a manifest
keeps a bound App present with only is_online cleared once its ROS binding disappears, and
hybrid discovery inherits that same shape; counting snapshot membership alone would make a
manifest-declared node immortal. A managed lifecycle node that merely deactivates keeps
is_online: true - its process is still running, only its ROS 2 lifecycle state changed -
so a deactivation is never mistaken for a death either; that is lifecycle_expectation’s
concern, not this one’s (see “The boundary with lifecycle_expectation” below).
Tracking is keyed on the STABLE fqn (App::effective_fqn()), never App::id: an id is
recomputed every sweep and only gains a namespace prefix once a same-bare-name collision
currently exists anywhere in the graph, so a live node’s id can change out from under a key
built from it.
What is never tracked.
| Excluded | Why |
|---|---|
Peer-aggregated apps (app.source starting peer:) |
A peer app carries no ROS binding of its own, so effective_fqn() is empty for every one of them; tracking them would collapse an entire peer fleet onto one "" key, and one online peer app would mask every other peer app’s departure. |
_ros2cli_<pid> nodes |
Every ros2 CLI invocation (ros2 topic echo, ros2 param get, …) spins up a real, short-lived node under this prefix - rcl’s own hidden-node naming convention, matched structurally rather than guessed at. Letting these through would accumulate one permanently “dead” entry per CLI invocation for the life of the gateway. |
| An App that never comes online | A key is admitted to tracking only once the reliability gate has said this detector OWNS its departure - armed at least once, and on one of the two GROUNDS ownership can rest on (a lifecycle state read as active, or no lifecycle to read at all; the third case, a state nobody could read, is admitted PROVISIONALLY and is described in the row below). A manifest App whose binding never starts is never armed and so can never be falsely called dead; a node still inside its warmup_cycles window gets the identical protection. Once tracked, though, a node’s continued life-or-death judgement rests on PRESENCE alone, not on staying armed - a tracked node that later goes lifecycle-inactive is not thereby mistaken for dead. |
| A managed App whose lifecycle state has not been read YET | Not tracked, and only for as long as that is still true. The gate’s own raise permission is answered PERMISSIVELY for a managed node whose GetState has not answered (LifecycleWatcher::node_ok()) - without that, qos_mismatch, orphan and param_drift would all fall silent about a node whose lifecycle service is broken. Permission is not knowledge, though, so tracking is withheld while a measurement may still arrive. LifecycleWatcher charges a GetState re-seed budget per node, and only for a read that actually ran, so “still asking” is a fact rather than a guess. Once that budget is spent the node is tracked here after all: nothing will ASK again, and lifecycle_expectation looks at nothing unless an operator named the node in require_active (empty by default), so withholding it any longer would mean the death is reported by nobody. That admission is PROVISIONAL, not final. The ~/transition_event subscription outlives the seed budget, so a label can still arrive without being asked for, and one that reads non-active says the node was the lifecycle detector’s all along - so the presence detector hands it back on the spot, while it is still alive. Waiting for the budget alone would only have delayed the wrong report, not prevented it. |
File truncated at 100 lines see the full file
CHANGELOG
Changelog for package ros2_medkit_graph_watchdog
0.7.0 (2026-08-27)
- Initial release of the package: a gateway plugin that raises faults
for silent failures in the ROS 2 graph - problems that leave every
node alive and every topic present, so nothing in the stack reports
them today. The package carries the plugin skeleton, a central
reliability gate that holds raises until the graph has quiesced, and
a detector registry. Detectors are configured under
plugins.graph_watchdog.<key>, each with araise/advisory/offmode, and an unknown key underdetectors.<id>is reported rather than silently ignored, so a typo cannot quietly disable a check (#571) -
qos_mismatchdetector: raisesGRAPH_QOS_MISMATCHfor a publisher and subscriber whose QoS profiles cannot match, which leaves the connection silently unestablished (#571) -
orphandetector: raisesGRAPH_ORPHANfor a one-sided topic - a publisher with no subscriber, or the reverse - but only when a complementary near-miss counterpart exists: same message type, a namespace and leaf within the configured edit distance, and the opposite side. A lone unsubscribed topic raises nothing. The pair is the signature of a remap or a topic-name typo, which is what the detector is for (#578) -
param_driftdetector: raisesGRAPH_PARAM_DRIFTwhen a node's live parameter value diverges from the declared expectation (#580) -
lifecycle_expectationdetector: raisesGRAPH_NODE_INACTIVEfor a node named inrequire_activethat is not in theactivelifecycle state once itsgracewindow has passed. It raises two further codes for the cases where the state could not be established at all:GRAPH_NODE_UNREADABLEwhen the lifecycle service does not answer, andGRAPH_NODE_NOT_MANAGEDwhen the named node has no lifecycle to read. These are distinct codes on the wire, so a consumer filtering onGRAPH_NODE_INACTIVEalone sees neither unmeasured case. A node is matched by itsApp::id, its full FQN, or the bare leaf of that FQN, and the grace streak is counted per node, so naming one node in two documented forms cannot halve the grace that was configured (#587) -
node_deathdetector: raisesGRAPH_NODE_DISAPPEAREDfor a node that leaves the graph when nothing else is left to report it, with a suppression framework that decides ownership of a departure from knowledge rather than from silence. A node whose lifecycle state was never readable is admitted only provisionally, and is handed back the moment a label arrives saying the departure belonged to the lifecycle detector instead. Zero-config: there is no list of nodes to maintain (#625, #624) - Two silent-fault classes are not delivered in this release.
GRAPH_TF_STALEandGRAPH_LATENCY_BUDGEThave their fault codes reserved in the frozenGRAPH_*namespace and land in later changes, each against its own issue - The plugin keeps its ROS entities off the gateway executor, so an entity created and destroyed while the gateway runs cannot have its destructor run on an executor thread concurrently with a create on the same node
- Contributors: \@bburda
Package Dependencies
System Dependencies
Dependant Packages
No known dependants.
Launch files
No launch files found
Messages
No message files found.
Services
No service files found
Plugins
No plugins found.
Recent questions tagged ros2_medkit_graph_watchdog at Robotics Stack Exchange
No version for distro foxy showing humble. Known supported distros are highlighted in the buttons above.
Package Summary
| Version | 0.7.0 |
| License | Apache-2.0 |
| Build type | AMENT_CMAKE |
| Use | RECOMMENDED |
Repository Summary
| Checkout URI | https://github.com/selfpatch/ros2_medkit.git |
| VCS Type | git |
| VCS Version | main |
| Last Updated | 2026-09-06 |
| Dev Status | DEVELOPED |
| Released | RELEASED |
| Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
Gateway plugin that detects silent faults in the ROS 2 graph: QoS mismatches, orphaned topics left by a name typo, dead nodes, parameter drift and unmet lifecycle expectations.
Maintainers
- bburda
Authors
No additional authors.
ros2_medkit_graph_watchdog
Gateway plugin that detects silent faults in the ROS 2 graph: failures where every node is up, nothing logs an error, and the robot is still broken.
Detectors read the graph and raise faults through a ReportFault service client on the
gateway node; the faults surface via FaultManager on the gateway /faults API. That
client, like the lifecycle subscriptions below, lives in the plugin’s own callback group
and is driven by the plugin’s own executor from the tick thread, never by the gateway’s
ROS executor.
This package carries the plugin skeleton, the central reliability gate that holds raises
until the graph has quiesced, and five detectors, qos_mismatch, orphan,
param_drift, lifecycle_expectation and node_death. Two silent-fault classes remain
undelivered, GRAPH_TF_STALE and GRAPH_LATENCY_BUDGET; they land in follow-up changes,
each against its own issue, and their fault codes are already reserved in the frozen
GRAPH_* namespace (see “Fault codes”).
Build
With the gateway built and sourced:
colcon build --packages-select ros2_medkit_graph_watchdog
colcon test --packages-select ros2_medkit_graph_watchdog && colcon test-result --verbose
Load into the gateway
ros2 run ros2_medkit_gateway gateway_node --ros-args \
-p plugins:="[graph_watchdog]" \
-p "plugins.graph_watchdog.path:=$(ros2 pkg prefix ros2_medkit_graph_watchdog)/lib/ros2_medkit_graph_watchdog/libros2_medkit_graph_watchdog.so"
Configuration (plugins.graph_watchdog.<key>)
| Key | Type | Default | Meaning |
|---|---|---|---|
tick_interval_ms |
int | 1000 |
Detector tick cadence. Accepted range 1..INT_MAX; the check runs on the wide integer, so a value past INT_MAX is rejected rather than truncated back into the band. |
warmup_cycles |
int | 5 |
An entity must be continuously present for this many ticks before it arms. A mid-run restart re-warms it. Enforced centrally, see “Reliability (bringup-quiesce)” below. Accepted range 0..INT_MAX, checked on the wide integer. |
prune_grace |
int | 60 |
Default injected into every detector’s own config before configure(); a per-detector detectors.<id>.prune_grace overrides it. Used by detectors that keep per-key bookkeeping. Accepted range 0..INT_MAX here, checked on the wide integer; each detector re-validates it against its own documented range before using it. |
detectors.<id>.mode |
string | bool | raise |
raise = push faults; advisory = observe, do not push; off = disabled. Bare off/no (YAML booleans) also disable; bare on/yes mean raise. |
detectors.<id>.<field> |
any | - | Per-detector thresholds, passed to that detector’s configure(). |
Nested delivery. Keys are written dotted in YAML or --ros-args, and the gateway
delivers them to the plugin as a nested object. A bare off is a YAML 1.1 boolean, so
mode: off arrives as false, not the string "off"; both forms disable the detector.
Unknown keys are reported. Anything under detectors.<id> that the detector does not
read produces one startup warning naming the key and listing the ones that exist, so a
typo like allow_list: for allowlist: cannot silently do nothing. A typo in the detector
id itself is warned about the same way, with the registered ids listed.
lifecycle_expectation keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. |
require_active |
string[] | [] |
Node names that must be in the active lifecycle state, each matched against a live node by its App::id, its full FQN (/ns/name), or the bare leaf of that FQN. A bare name matches that node in EVERY namespace (a fleet-wide “all controller_servers must be active”); use a full FQN to pin one robot’s. Both the bare and the FQN form match against the node’s stable FQN, so they survive the App::id renaming that a same-bare-name collision triggers; an entry written as an App::id also works, but can stop matching on a multi-robot graph for exactly that reason. Empty = the detector checks nothing and emits nothing. A require_active that is not a string array warns and is ignored; an empty-string entry warns and is skipped - never dropped silently, since the operator would go on believing the node is covered. |
grace |
int | 5 |
Consecutive not-active ticks a required node tolerates before being reported inactive - bringup time for a managed node to reach active (configure + activate) before the expectation is enforced. Counted per NODE, not per matching entry: a node named by both a bare-name and a full-FQN entry still advances its streak once per tick, so mixing the two documented forms cannot halve the grace that was configured. An ALREADY-CONFIRMED streak keeps its content while the node is ABSENT, so a node that leaves the graph after being confirmed stays confirmed; a streak that has not yet crossed grace on a node the presence detector could also report (it was armed at least once) is HELD rather than advanced while absent, so a node is never confirmed out of ticks gathered while nobody could observe it - only a PRESENT tick may still cross grace for that node. A node that was NEVER armed gets no such hold: node_death can never track it either, so absence keeps advancing its streak too, the same as a present tick would. Accepted range 0..300 - five minutes at the shipped 1 s cadence, and already an extravagant allowance for a managed node to reach active. The upper end is a real bound rather than a formality: grace decides how long a PRESENT node may go unreported, and how long a node returning from a departure still inactive takes to (re-)mature, so at the old maximum of INT_MAX - 1 that PRESENT-side silence lasted about 24 days at the shipped cadence with no warning. Whether it also bounds how long a node that leaves the graph BELOW grace sits unsettled depends on the same split: for an armed node that lasts for as long as the node stays gone, independent of grace (see “Bounded by evidence, not by age” below); for a never-armed one grace bounds that too, since absence matures it within grace + absence_grace + 1 ticks either way. The check runs on the wide integer, so a value that only fits after truncation is rejected rather than silently turned into a hair-trigger. Anything outside the range warns and keeps the default. |
prune_grace |
int | 60 |
Consecutive ticks an IDLE tracked node - both clocks at zero and no matured ownership, so nothing to lose - may stay ABSENT from the graph before its bookkeeping is reclaimed. A node carrying evidence is never reclaimed by age at all, however long it stays gone, so this key cannot erase a fault’s own state; the map is bounded instead by tracked_node_cap (see “Bounded by evidence, not by age”). Injected for every detector at plugin scope and overridable per detector; a value outside 0..3600 warns and this detector keeps its own default of 60. The range check runs on the wide integer, so an out-of-int-range value is rejected rather than truncated. The value is used as written - there is no grace + 1 clamp, because there is no longer anything for one to protect. |
tracked_node_cap |
int | 512 |
The most nodes this detector keeps bookkeeping for at once. It is what bounds the map, since evidence is never reclaimed by age (see “Bounded by evidence, not by age”): at the cap, idle entries are reclaimed first, then entries for DEPARTED nodes are collapsed into a count, and only if every tracked node is PRESENT and carrying evidence is a newly matched node refused - which withholds GRAPH_NODE_INACTIVE’s clear and is reported both in the log and on GET /x-medkit-watchdog. 512 is comfortably above every node in a realistic graph (a full Nav2 stack plus perception is roughly a hundred nodes; a ten-robot fleet sharing one domain a few hundred), so raising it is only needed where require_active legitimately matches more PRESENT nodes than that. Accepted range 1..16384 - 16384 is about 8 MB of bookkeeping at roughly 500 bytes per tracked node, and a deployment needing more distinct required-node identities alive at once has identity churn rather than a large fleet. Anything outside the range, including 0 (a cap of nothing would mean checking nothing), warns and keeps the default; the check runs on the wide integer, so an out-of-int-range value is rejected rather than truncated. |
node_death keys
Zero-config, unlike every detector above: there is no require_active-style list of nodes
to watch, because every armed App in the graph is a candidate.
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. |
miss_grace |
int | 2 |
Consecutive missed ticks a tracked node tolerates before being reported dead - a death is reported once misses EXCEED this, i.e. after miss_grace + 1 ticks. Accepted range 0..3600, and additionally floored to whatever wall-clock window the configured tick_interval_ms needs (see “The wall-clock floor” below); a value below that floor is silently raised to it, with a warning naming the ms window it actually spans. |
prune_grace |
int | 60 |
Plugin-injected default (overridable here); how long a key being DURABLY suppressed (see “Suppression” below) may sit unreported before its bookkeeping is reclaimed. Accepted range 0..3600. The value actually used is max(prune_grace, miss_grace + 1), a silent internal floor rather than a rejection: without it, a durably-suppressed key could be reclaimed the very tick it would first have become eligible to report, losing the report rather than merely suppressing it. |
allowlist |
string[] | [] |
Node identities never reported dead. A dead key matches if it - or a name the same node is also known by - is present verbatim: the key’s own fqn, its bare leaf, or (captured while the node was still present) its App::id. Has no effect unless named in suppress; naming it without a configured list is a no-op. |
suppress |
string[] | [] |
Suppression mechanisms to activate: "allowlist" and "lifecycle". Any other entry warns as unrecognized and is ignored; a non-string or non-array entry warns the same way. |
tracked_node_cap |
int | 512 |
The most DEPARTED (not-yet-confirmed-dead or already-collapsed) identities this detector keeps individual bookkeeping for at once - a present/armed node is never counted against it and never refused tracking. Accepted range 1..16384; 0 is refused rather than clamped, since a cap of nothing would mean tracking nothing. See “Bounded by evidence, not by age” below for what it actually bounds and why, unlike the sibling detector’s identical-looking cap, a newcomer here is never the thing refused. |
tick_interval_ms is not an own key of this detector, but this is the one detector that
reads the plugin-injected value directly, to compute miss_grace’s wall-clock floor below.
Liveness, and why membership in the snapshot is not enough. App::is_online is what
this detector tracks, never mere presence in the entity snapshot. In runtime-only discovery
the two happen to coincide - a dead node’s App leaves the snapshot entirely - but a manifest
keeps a bound App present with only is_online cleared once its ROS binding disappears, and
hybrid discovery inherits that same shape; counting snapshot membership alone would make a
manifest-declared node immortal. A managed lifecycle node that merely deactivates keeps
is_online: true - its process is still running, only its ROS 2 lifecycle state changed -
so a deactivation is never mistaken for a death either; that is lifecycle_expectation’s
concern, not this one’s (see “The boundary with lifecycle_expectation” below).
Tracking is keyed on the STABLE fqn (App::effective_fqn()), never App::id: an id is
recomputed every sweep and only gains a namespace prefix once a same-bare-name collision
currently exists anywhere in the graph, so a live node’s id can change out from under a key
built from it.
What is never tracked.
| Excluded | Why |
|---|---|
Peer-aggregated apps (app.source starting peer:) |
A peer app carries no ROS binding of its own, so effective_fqn() is empty for every one of them; tracking them would collapse an entire peer fleet onto one "" key, and one online peer app would mask every other peer app’s departure. |
_ros2cli_<pid> nodes |
Every ros2 CLI invocation (ros2 topic echo, ros2 param get, …) spins up a real, short-lived node under this prefix - rcl’s own hidden-node naming convention, matched structurally rather than guessed at. Letting these through would accumulate one permanently “dead” entry per CLI invocation for the life of the gateway. |
| An App that never comes online | A key is admitted to tracking only once the reliability gate has said this detector OWNS its departure - armed at least once, and on one of the two GROUNDS ownership can rest on (a lifecycle state read as active, or no lifecycle to read at all; the third case, a state nobody could read, is admitted PROVISIONALLY and is described in the row below). A manifest App whose binding never starts is never armed and so can never be falsely called dead; a node still inside its warmup_cycles window gets the identical protection. Once tracked, though, a node’s continued life-or-death judgement rests on PRESENCE alone, not on staying armed - a tracked node that later goes lifecycle-inactive is not thereby mistaken for dead. |
| A managed App whose lifecycle state has not been read YET | Not tracked, and only for as long as that is still true. The gate’s own raise permission is answered PERMISSIVELY for a managed node whose GetState has not answered (LifecycleWatcher::node_ok()) - without that, qos_mismatch, orphan and param_drift would all fall silent about a node whose lifecycle service is broken. Permission is not knowledge, though, so tracking is withheld while a measurement may still arrive. LifecycleWatcher charges a GetState re-seed budget per node, and only for a read that actually ran, so “still asking” is a fact rather than a guess. Once that budget is spent the node is tracked here after all: nothing will ASK again, and lifecycle_expectation looks at nothing unless an operator named the node in require_active (empty by default), so withholding it any longer would mean the death is reported by nobody. That admission is PROVISIONAL, not final. The ~/transition_event subscription outlives the seed budget, so a label can still arrive without being asked for, and one that reads non-active says the node was the lifecycle detector’s all along - so the presence detector hands it back on the spot, while it is still alive. Waiting for the budget alone would only have delayed the wrong report, not prevented it. |
File truncated at 100 lines see the full file
CHANGELOG
Changelog for package ros2_medkit_graph_watchdog
0.7.0 (2026-08-27)
- Initial release of the package: a gateway plugin that raises faults
for silent failures in the ROS 2 graph - problems that leave every
node alive and every topic present, so nothing in the stack reports
them today. The package carries the plugin skeleton, a central
reliability gate that holds raises until the graph has quiesced, and
a detector registry. Detectors are configured under
plugins.graph_watchdog.<key>, each with araise/advisory/offmode, and an unknown key underdetectors.<id>is reported rather than silently ignored, so a typo cannot quietly disable a check (#571) -
qos_mismatchdetector: raisesGRAPH_QOS_MISMATCHfor a publisher and subscriber whose QoS profiles cannot match, which leaves the connection silently unestablished (#571) -
orphandetector: raisesGRAPH_ORPHANfor a one-sided topic - a publisher with no subscriber, or the reverse - but only when a complementary near-miss counterpart exists: same message type, a namespace and leaf within the configured edit distance, and the opposite side. A lone unsubscribed topic raises nothing. The pair is the signature of a remap or a topic-name typo, which is what the detector is for (#578) -
param_driftdetector: raisesGRAPH_PARAM_DRIFTwhen a node's live parameter value diverges from the declared expectation (#580) -
lifecycle_expectationdetector: raisesGRAPH_NODE_INACTIVEfor a node named inrequire_activethat is not in theactivelifecycle state once itsgracewindow has passed. It raises two further codes for the cases where the state could not be established at all:GRAPH_NODE_UNREADABLEwhen the lifecycle service does not answer, andGRAPH_NODE_NOT_MANAGEDwhen the named node has no lifecycle to read. These are distinct codes on the wire, so a consumer filtering onGRAPH_NODE_INACTIVEalone sees neither unmeasured case. A node is matched by itsApp::id, its full FQN, or the bare leaf of that FQN, and the grace streak is counted per node, so naming one node in two documented forms cannot halve the grace that was configured (#587) -
node_deathdetector: raisesGRAPH_NODE_DISAPPEAREDfor a node that leaves the graph when nothing else is left to report it, with a suppression framework that decides ownership of a departure from knowledge rather than from silence. A node whose lifecycle state was never readable is admitted only provisionally, and is handed back the moment a label arrives saying the departure belonged to the lifecycle detector instead. Zero-config: there is no list of nodes to maintain (#625, #624) - Two silent-fault classes are not delivered in this release.
GRAPH_TF_STALEandGRAPH_LATENCY_BUDGEThave their fault codes reserved in the frozenGRAPH_*namespace and land in later changes, each against its own issue - The plugin keeps its ROS entities off the gateway executor, so an entity created and destroyed while the gateway runs cannot have its destructor run on an executor thread concurrently with a create on the same node
- Contributors: \@bburda
Package Dependencies
System Dependencies
Dependant Packages
No known dependants.
Launch files
No launch files found
Messages
No message files found.
Services
No service files found
Plugins
No plugins found.
Recent questions tagged ros2_medkit_graph_watchdog at Robotics Stack Exchange
No version for distro iron showing humble. Known supported distros are highlighted in the buttons above.
Package Summary
| Version | 0.7.0 |
| License | Apache-2.0 |
| Build type | AMENT_CMAKE |
| Use | RECOMMENDED |
Repository Summary
| Checkout URI | https://github.com/selfpatch/ros2_medkit.git |
| VCS Type | git |
| VCS Version | main |
| Last Updated | 2026-09-06 |
| Dev Status | DEVELOPED |
| Released | RELEASED |
| Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
Gateway plugin that detects silent faults in the ROS 2 graph: QoS mismatches, orphaned topics left by a name typo, dead nodes, parameter drift and unmet lifecycle expectations.
Maintainers
- bburda
Authors
No additional authors.
ros2_medkit_graph_watchdog
Gateway plugin that detects silent faults in the ROS 2 graph: failures where every node is up, nothing logs an error, and the robot is still broken.
Detectors read the graph and raise faults through a ReportFault service client on the
gateway node; the faults surface via FaultManager on the gateway /faults API. That
client, like the lifecycle subscriptions below, lives in the plugin’s own callback group
and is driven by the plugin’s own executor from the tick thread, never by the gateway’s
ROS executor.
This package carries the plugin skeleton, the central reliability gate that holds raises
until the graph has quiesced, and five detectors, qos_mismatch, orphan,
param_drift, lifecycle_expectation and node_death. Two silent-fault classes remain
undelivered, GRAPH_TF_STALE and GRAPH_LATENCY_BUDGET; they land in follow-up changes,
each against its own issue, and their fault codes are already reserved in the frozen
GRAPH_* namespace (see “Fault codes”).
Build
With the gateway built and sourced:
colcon build --packages-select ros2_medkit_graph_watchdog
colcon test --packages-select ros2_medkit_graph_watchdog && colcon test-result --verbose
Load into the gateway
ros2 run ros2_medkit_gateway gateway_node --ros-args \
-p plugins:="[graph_watchdog]" \
-p "plugins.graph_watchdog.path:=$(ros2 pkg prefix ros2_medkit_graph_watchdog)/lib/ros2_medkit_graph_watchdog/libros2_medkit_graph_watchdog.so"
Configuration (plugins.graph_watchdog.<key>)
| Key | Type | Default | Meaning |
|---|---|---|---|
tick_interval_ms |
int | 1000 |
Detector tick cadence. Accepted range 1..INT_MAX; the check runs on the wide integer, so a value past INT_MAX is rejected rather than truncated back into the band. |
warmup_cycles |
int | 5 |
An entity must be continuously present for this many ticks before it arms. A mid-run restart re-warms it. Enforced centrally, see “Reliability (bringup-quiesce)” below. Accepted range 0..INT_MAX, checked on the wide integer. |
prune_grace |
int | 60 |
Default injected into every detector’s own config before configure(); a per-detector detectors.<id>.prune_grace overrides it. Used by detectors that keep per-key bookkeeping. Accepted range 0..INT_MAX here, checked on the wide integer; each detector re-validates it against its own documented range before using it. |
detectors.<id>.mode |
string | bool | raise |
raise = push faults; advisory = observe, do not push; off = disabled. Bare off/no (YAML booleans) also disable; bare on/yes mean raise. |
detectors.<id>.<field> |
any | - | Per-detector thresholds, passed to that detector’s configure(). |
Nested delivery. Keys are written dotted in YAML or --ros-args, and the gateway
delivers them to the plugin as a nested object. A bare off is a YAML 1.1 boolean, so
mode: off arrives as false, not the string "off"; both forms disable the detector.
Unknown keys are reported. Anything under detectors.<id> that the detector does not
read produces one startup warning naming the key and listing the ones that exist, so a
typo like allow_list: for allowlist: cannot silently do nothing. A typo in the detector
id itself is warned about the same way, with the registered ids listed.
lifecycle_expectation keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. |
require_active |
string[] | [] |
Node names that must be in the active lifecycle state, each matched against a live node by its App::id, its full FQN (/ns/name), or the bare leaf of that FQN. A bare name matches that node in EVERY namespace (a fleet-wide “all controller_servers must be active”); use a full FQN to pin one robot’s. Both the bare and the FQN form match against the node’s stable FQN, so they survive the App::id renaming that a same-bare-name collision triggers; an entry written as an App::id also works, but can stop matching on a multi-robot graph for exactly that reason. Empty = the detector checks nothing and emits nothing. A require_active that is not a string array warns and is ignored; an empty-string entry warns and is skipped - never dropped silently, since the operator would go on believing the node is covered. |
grace |
int | 5 |
Consecutive not-active ticks a required node tolerates before being reported inactive - bringup time for a managed node to reach active (configure + activate) before the expectation is enforced. Counted per NODE, not per matching entry: a node named by both a bare-name and a full-FQN entry still advances its streak once per tick, so mixing the two documented forms cannot halve the grace that was configured. An ALREADY-CONFIRMED streak keeps its content while the node is ABSENT, so a node that leaves the graph after being confirmed stays confirmed; a streak that has not yet crossed grace on a node the presence detector could also report (it was armed at least once) is HELD rather than advanced while absent, so a node is never confirmed out of ticks gathered while nobody could observe it - only a PRESENT tick may still cross grace for that node. A node that was NEVER armed gets no such hold: node_death can never track it either, so absence keeps advancing its streak too, the same as a present tick would. Accepted range 0..300 - five minutes at the shipped 1 s cadence, and already an extravagant allowance for a managed node to reach active. The upper end is a real bound rather than a formality: grace decides how long a PRESENT node may go unreported, and how long a node returning from a departure still inactive takes to (re-)mature, so at the old maximum of INT_MAX - 1 that PRESENT-side silence lasted about 24 days at the shipped cadence with no warning. Whether it also bounds how long a node that leaves the graph BELOW grace sits unsettled depends on the same split: for an armed node that lasts for as long as the node stays gone, independent of grace (see “Bounded by evidence, not by age” below); for a never-armed one grace bounds that too, since absence matures it within grace + absence_grace + 1 ticks either way. The check runs on the wide integer, so a value that only fits after truncation is rejected rather than silently turned into a hair-trigger. Anything outside the range warns and keeps the default. |
prune_grace |
int | 60 |
Consecutive ticks an IDLE tracked node - both clocks at zero and no matured ownership, so nothing to lose - may stay ABSENT from the graph before its bookkeeping is reclaimed. A node carrying evidence is never reclaimed by age at all, however long it stays gone, so this key cannot erase a fault’s own state; the map is bounded instead by tracked_node_cap (see “Bounded by evidence, not by age”). Injected for every detector at plugin scope and overridable per detector; a value outside 0..3600 warns and this detector keeps its own default of 60. The range check runs on the wide integer, so an out-of-int-range value is rejected rather than truncated. The value is used as written - there is no grace + 1 clamp, because there is no longer anything for one to protect. |
tracked_node_cap |
int | 512 |
The most nodes this detector keeps bookkeeping for at once. It is what bounds the map, since evidence is never reclaimed by age (see “Bounded by evidence, not by age”): at the cap, idle entries are reclaimed first, then entries for DEPARTED nodes are collapsed into a count, and only if every tracked node is PRESENT and carrying evidence is a newly matched node refused - which withholds GRAPH_NODE_INACTIVE’s clear and is reported both in the log and on GET /x-medkit-watchdog. 512 is comfortably above every node in a realistic graph (a full Nav2 stack plus perception is roughly a hundred nodes; a ten-robot fleet sharing one domain a few hundred), so raising it is only needed where require_active legitimately matches more PRESENT nodes than that. Accepted range 1..16384 - 16384 is about 8 MB of bookkeeping at roughly 500 bytes per tracked node, and a deployment needing more distinct required-node identities alive at once has identity churn rather than a large fleet. Anything outside the range, including 0 (a cap of nothing would mean checking nothing), warns and keeps the default; the check runs on the wide integer, so an out-of-int-range value is rejected rather than truncated. |
node_death keys
Zero-config, unlike every detector above: there is no require_active-style list of nodes
to watch, because every armed App in the graph is a candidate.
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. |
miss_grace |
int | 2 |
Consecutive missed ticks a tracked node tolerates before being reported dead - a death is reported once misses EXCEED this, i.e. after miss_grace + 1 ticks. Accepted range 0..3600, and additionally floored to whatever wall-clock window the configured tick_interval_ms needs (see “The wall-clock floor” below); a value below that floor is silently raised to it, with a warning naming the ms window it actually spans. |
prune_grace |
int | 60 |
Plugin-injected default (overridable here); how long a key being DURABLY suppressed (see “Suppression” below) may sit unreported before its bookkeeping is reclaimed. Accepted range 0..3600. The value actually used is max(prune_grace, miss_grace + 1), a silent internal floor rather than a rejection: without it, a durably-suppressed key could be reclaimed the very tick it would first have become eligible to report, losing the report rather than merely suppressing it. |
allowlist |
string[] | [] |
Node identities never reported dead. A dead key matches if it - or a name the same node is also known by - is present verbatim: the key’s own fqn, its bare leaf, or (captured while the node was still present) its App::id. Has no effect unless named in suppress; naming it without a configured list is a no-op. |
suppress |
string[] | [] |
Suppression mechanisms to activate: "allowlist" and "lifecycle". Any other entry warns as unrecognized and is ignored; a non-string or non-array entry warns the same way. |
tracked_node_cap |
int | 512 |
The most DEPARTED (not-yet-confirmed-dead or already-collapsed) identities this detector keeps individual bookkeeping for at once - a present/armed node is never counted against it and never refused tracking. Accepted range 1..16384; 0 is refused rather than clamped, since a cap of nothing would mean tracking nothing. See “Bounded by evidence, not by age” below for what it actually bounds and why, unlike the sibling detector’s identical-looking cap, a newcomer here is never the thing refused. |
tick_interval_ms is not an own key of this detector, but this is the one detector that
reads the plugin-injected value directly, to compute miss_grace’s wall-clock floor below.
Liveness, and why membership in the snapshot is not enough. App::is_online is what
this detector tracks, never mere presence in the entity snapshot. In runtime-only discovery
the two happen to coincide - a dead node’s App leaves the snapshot entirely - but a manifest
keeps a bound App present with only is_online cleared once its ROS binding disappears, and
hybrid discovery inherits that same shape; counting snapshot membership alone would make a
manifest-declared node immortal. A managed lifecycle node that merely deactivates keeps
is_online: true - its process is still running, only its ROS 2 lifecycle state changed -
so a deactivation is never mistaken for a death either; that is lifecycle_expectation’s
concern, not this one’s (see “The boundary with lifecycle_expectation” below).
Tracking is keyed on the STABLE fqn (App::effective_fqn()), never App::id: an id is
recomputed every sweep and only gains a namespace prefix once a same-bare-name collision
currently exists anywhere in the graph, so a live node’s id can change out from under a key
built from it.
What is never tracked.
| Excluded | Why |
|---|---|
Peer-aggregated apps (app.source starting peer:) |
A peer app carries no ROS binding of its own, so effective_fqn() is empty for every one of them; tracking them would collapse an entire peer fleet onto one "" key, and one online peer app would mask every other peer app’s departure. |
_ros2cli_<pid> nodes |
Every ros2 CLI invocation (ros2 topic echo, ros2 param get, …) spins up a real, short-lived node under this prefix - rcl’s own hidden-node naming convention, matched structurally rather than guessed at. Letting these through would accumulate one permanently “dead” entry per CLI invocation for the life of the gateway. |
| An App that never comes online | A key is admitted to tracking only once the reliability gate has said this detector OWNS its departure - armed at least once, and on one of the two GROUNDS ownership can rest on (a lifecycle state read as active, or no lifecycle to read at all; the third case, a state nobody could read, is admitted PROVISIONALLY and is described in the row below). A manifest App whose binding never starts is never armed and so can never be falsely called dead; a node still inside its warmup_cycles window gets the identical protection. Once tracked, though, a node’s continued life-or-death judgement rests on PRESENCE alone, not on staying armed - a tracked node that later goes lifecycle-inactive is not thereby mistaken for dead. |
| A managed App whose lifecycle state has not been read YET | Not tracked, and only for as long as that is still true. The gate’s own raise permission is answered PERMISSIVELY for a managed node whose GetState has not answered (LifecycleWatcher::node_ok()) - without that, qos_mismatch, orphan and param_drift would all fall silent about a node whose lifecycle service is broken. Permission is not knowledge, though, so tracking is withheld while a measurement may still arrive. LifecycleWatcher charges a GetState re-seed budget per node, and only for a read that actually ran, so “still asking” is a fact rather than a guess. Once that budget is spent the node is tracked here after all: nothing will ASK again, and lifecycle_expectation looks at nothing unless an operator named the node in require_active (empty by default), so withholding it any longer would mean the death is reported by nobody. That admission is PROVISIONAL, not final. The ~/transition_event subscription outlives the seed budget, so a label can still arrive without being asked for, and one that reads non-active says the node was the lifecycle detector’s all along - so the presence detector hands it back on the spot, while it is still alive. Waiting for the budget alone would only have delayed the wrong report, not prevented it. |
File truncated at 100 lines see the full file
CHANGELOG
Changelog for package ros2_medkit_graph_watchdog
0.7.0 (2026-08-27)
- Initial release of the package: a gateway plugin that raises faults
for silent failures in the ROS 2 graph - problems that leave every
node alive and every topic present, so nothing in the stack reports
them today. The package carries the plugin skeleton, a central
reliability gate that holds raises until the graph has quiesced, and
a detector registry. Detectors are configured under
plugins.graph_watchdog.<key>, each with araise/advisory/offmode, and an unknown key underdetectors.<id>is reported rather than silently ignored, so a typo cannot quietly disable a check (#571) -
qos_mismatchdetector: raisesGRAPH_QOS_MISMATCHfor a publisher and subscriber whose QoS profiles cannot match, which leaves the connection silently unestablished (#571) -
orphandetector: raisesGRAPH_ORPHANfor a one-sided topic - a publisher with no subscriber, or the reverse - but only when a complementary near-miss counterpart exists: same message type, a namespace and leaf within the configured edit distance, and the opposite side. A lone unsubscribed topic raises nothing. The pair is the signature of a remap or a topic-name typo, which is what the detector is for (#578) -
param_driftdetector: raisesGRAPH_PARAM_DRIFTwhen a node's live parameter value diverges from the declared expectation (#580) -
lifecycle_expectationdetector: raisesGRAPH_NODE_INACTIVEfor a node named inrequire_activethat is not in theactivelifecycle state once itsgracewindow has passed. It raises two further codes for the cases where the state could not be established at all:GRAPH_NODE_UNREADABLEwhen the lifecycle service does not answer, andGRAPH_NODE_NOT_MANAGEDwhen the named node has no lifecycle to read. These are distinct codes on the wire, so a consumer filtering onGRAPH_NODE_INACTIVEalone sees neither unmeasured case. A node is matched by itsApp::id, its full FQN, or the bare leaf of that FQN, and the grace streak is counted per node, so naming one node in two documented forms cannot halve the grace that was configured (#587) -
node_deathdetector: raisesGRAPH_NODE_DISAPPEAREDfor a node that leaves the graph when nothing else is left to report it, with a suppression framework that decides ownership of a departure from knowledge rather than from silence. A node whose lifecycle state was never readable is admitted only provisionally, and is handed back the moment a label arrives saying the departure belonged to the lifecycle detector instead. Zero-config: there is no list of nodes to maintain (#625, #624) - Two silent-fault classes are not delivered in this release.
GRAPH_TF_STALEandGRAPH_LATENCY_BUDGEThave their fault codes reserved in the frozenGRAPH_*namespace and land in later changes, each against its own issue - The plugin keeps its ROS entities off the gateway executor, so an entity created and destroyed while the gateway runs cannot have its destructor run on an executor thread concurrently with a create on the same node
- Contributors: \@bburda
Package Dependencies
System Dependencies
Dependant Packages
No known dependants.
Launch files
No launch files found
Messages
No message files found.
Services
No service files found
Plugins
No plugins found.
Recent questions tagged ros2_medkit_graph_watchdog at Robotics Stack Exchange
No version for distro lunar showing humble. Known supported distros are highlighted in the buttons above.
Package Summary
| Version | 0.7.0 |
| License | Apache-2.0 |
| Build type | AMENT_CMAKE |
| Use | RECOMMENDED |
Repository Summary
| Checkout URI | https://github.com/selfpatch/ros2_medkit.git |
| VCS Type | git |
| VCS Version | main |
| Last Updated | 2026-09-06 |
| Dev Status | DEVELOPED |
| Released | RELEASED |
| Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
Gateway plugin that detects silent faults in the ROS 2 graph: QoS mismatches, orphaned topics left by a name typo, dead nodes, parameter drift and unmet lifecycle expectations.
Maintainers
- bburda
Authors
No additional authors.
ros2_medkit_graph_watchdog
Gateway plugin that detects silent faults in the ROS 2 graph: failures where every node is up, nothing logs an error, and the robot is still broken.
Detectors read the graph and raise faults through a ReportFault service client on the
gateway node; the faults surface via FaultManager on the gateway /faults API. That
client, like the lifecycle subscriptions below, lives in the plugin’s own callback group
and is driven by the plugin’s own executor from the tick thread, never by the gateway’s
ROS executor.
This package carries the plugin skeleton, the central reliability gate that holds raises
until the graph has quiesced, and five detectors, qos_mismatch, orphan,
param_drift, lifecycle_expectation and node_death. Two silent-fault classes remain
undelivered, GRAPH_TF_STALE and GRAPH_LATENCY_BUDGET; they land in follow-up changes,
each against its own issue, and their fault codes are already reserved in the frozen
GRAPH_* namespace (see “Fault codes”).
Build
With the gateway built and sourced:
colcon build --packages-select ros2_medkit_graph_watchdog
colcon test --packages-select ros2_medkit_graph_watchdog && colcon test-result --verbose
Load into the gateway
ros2 run ros2_medkit_gateway gateway_node --ros-args \
-p plugins:="[graph_watchdog]" \
-p "plugins.graph_watchdog.path:=$(ros2 pkg prefix ros2_medkit_graph_watchdog)/lib/ros2_medkit_graph_watchdog/libros2_medkit_graph_watchdog.so"
Configuration (plugins.graph_watchdog.<key>)
| Key | Type | Default | Meaning |
|---|---|---|---|
tick_interval_ms |
int | 1000 |
Detector tick cadence. Accepted range 1..INT_MAX; the check runs on the wide integer, so a value past INT_MAX is rejected rather than truncated back into the band. |
warmup_cycles |
int | 5 |
An entity must be continuously present for this many ticks before it arms. A mid-run restart re-warms it. Enforced centrally, see “Reliability (bringup-quiesce)” below. Accepted range 0..INT_MAX, checked on the wide integer. |
prune_grace |
int | 60 |
Default injected into every detector’s own config before configure(); a per-detector detectors.<id>.prune_grace overrides it. Used by detectors that keep per-key bookkeeping. Accepted range 0..INT_MAX here, checked on the wide integer; each detector re-validates it against its own documented range before using it. |
detectors.<id>.mode |
string | bool | raise |
raise = push faults; advisory = observe, do not push; off = disabled. Bare off/no (YAML booleans) also disable; bare on/yes mean raise. |
detectors.<id>.<field> |
any | - | Per-detector thresholds, passed to that detector’s configure(). |
Nested delivery. Keys are written dotted in YAML or --ros-args, and the gateway
delivers them to the plugin as a nested object. A bare off is a YAML 1.1 boolean, so
mode: off arrives as false, not the string "off"; both forms disable the detector.
Unknown keys are reported. Anything under detectors.<id> that the detector does not
read produces one startup warning naming the key and listing the ones that exist, so a
typo like allow_list: for allowlist: cannot silently do nothing. A typo in the detector
id itself is warned about the same way, with the registered ids listed.
lifecycle_expectation keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. |
require_active |
string[] | [] |
Node names that must be in the active lifecycle state, each matched against a live node by its App::id, its full FQN (/ns/name), or the bare leaf of that FQN. A bare name matches that node in EVERY namespace (a fleet-wide “all controller_servers must be active”); use a full FQN to pin one robot’s. Both the bare and the FQN form match against the node’s stable FQN, so they survive the App::id renaming that a same-bare-name collision triggers; an entry written as an App::id also works, but can stop matching on a multi-robot graph for exactly that reason. Empty = the detector checks nothing and emits nothing. A require_active that is not a string array warns and is ignored; an empty-string entry warns and is skipped - never dropped silently, since the operator would go on believing the node is covered. |
grace |
int | 5 |
Consecutive not-active ticks a required node tolerates before being reported inactive - bringup time for a managed node to reach active (configure + activate) before the expectation is enforced. Counted per NODE, not per matching entry: a node named by both a bare-name and a full-FQN entry still advances its streak once per tick, so mixing the two documented forms cannot halve the grace that was configured. An ALREADY-CONFIRMED streak keeps its content while the node is ABSENT, so a node that leaves the graph after being confirmed stays confirmed; a streak that has not yet crossed grace on a node the presence detector could also report (it was armed at least once) is HELD rather than advanced while absent, so a node is never confirmed out of ticks gathered while nobody could observe it - only a PRESENT tick may still cross grace for that node. A node that was NEVER armed gets no such hold: node_death can never track it either, so absence keeps advancing its streak too, the same as a present tick would. Accepted range 0..300 - five minutes at the shipped 1 s cadence, and already an extravagant allowance for a managed node to reach active. The upper end is a real bound rather than a formality: grace decides how long a PRESENT node may go unreported, and how long a node returning from a departure still inactive takes to (re-)mature, so at the old maximum of INT_MAX - 1 that PRESENT-side silence lasted about 24 days at the shipped cadence with no warning. Whether it also bounds how long a node that leaves the graph BELOW grace sits unsettled depends on the same split: for an armed node that lasts for as long as the node stays gone, independent of grace (see “Bounded by evidence, not by age” below); for a never-armed one grace bounds that too, since absence matures it within grace + absence_grace + 1 ticks either way. The check runs on the wide integer, so a value that only fits after truncation is rejected rather than silently turned into a hair-trigger. Anything outside the range warns and keeps the default. |
prune_grace |
int | 60 |
Consecutive ticks an IDLE tracked node - both clocks at zero and no matured ownership, so nothing to lose - may stay ABSENT from the graph before its bookkeeping is reclaimed. A node carrying evidence is never reclaimed by age at all, however long it stays gone, so this key cannot erase a fault’s own state; the map is bounded instead by tracked_node_cap (see “Bounded by evidence, not by age”). Injected for every detector at plugin scope and overridable per detector; a value outside 0..3600 warns and this detector keeps its own default of 60. The range check runs on the wide integer, so an out-of-int-range value is rejected rather than truncated. The value is used as written - there is no grace + 1 clamp, because there is no longer anything for one to protect. |
tracked_node_cap |
int | 512 |
The most nodes this detector keeps bookkeeping for at once. It is what bounds the map, since evidence is never reclaimed by age (see “Bounded by evidence, not by age”): at the cap, idle entries are reclaimed first, then entries for DEPARTED nodes are collapsed into a count, and only if every tracked node is PRESENT and carrying evidence is a newly matched node refused - which withholds GRAPH_NODE_INACTIVE’s clear and is reported both in the log and on GET /x-medkit-watchdog. 512 is comfortably above every node in a realistic graph (a full Nav2 stack plus perception is roughly a hundred nodes; a ten-robot fleet sharing one domain a few hundred), so raising it is only needed where require_active legitimately matches more PRESENT nodes than that. Accepted range 1..16384 - 16384 is about 8 MB of bookkeeping at roughly 500 bytes per tracked node, and a deployment needing more distinct required-node identities alive at once has identity churn rather than a large fleet. Anything outside the range, including 0 (a cap of nothing would mean checking nothing), warns and keeps the default; the check runs on the wide integer, so an out-of-int-range value is rejected rather than truncated. |
node_death keys
Zero-config, unlike every detector above: there is no require_active-style list of nodes
to watch, because every armed App in the graph is a candidate.
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. |
miss_grace |
int | 2 |
Consecutive missed ticks a tracked node tolerates before being reported dead - a death is reported once misses EXCEED this, i.e. after miss_grace + 1 ticks. Accepted range 0..3600, and additionally floored to whatever wall-clock window the configured tick_interval_ms needs (see “The wall-clock floor” below); a value below that floor is silently raised to it, with a warning naming the ms window it actually spans. |
prune_grace |
int | 60 |
Plugin-injected default (overridable here); how long a key being DURABLY suppressed (see “Suppression” below) may sit unreported before its bookkeeping is reclaimed. Accepted range 0..3600. The value actually used is max(prune_grace, miss_grace + 1), a silent internal floor rather than a rejection: without it, a durably-suppressed key could be reclaimed the very tick it would first have become eligible to report, losing the report rather than merely suppressing it. |
allowlist |
string[] | [] |
Node identities never reported dead. A dead key matches if it - or a name the same node is also known by - is present verbatim: the key’s own fqn, its bare leaf, or (captured while the node was still present) its App::id. Has no effect unless named in suppress; naming it without a configured list is a no-op. |
suppress |
string[] | [] |
Suppression mechanisms to activate: "allowlist" and "lifecycle". Any other entry warns as unrecognized and is ignored; a non-string or non-array entry warns the same way. |
tracked_node_cap |
int | 512 |
The most DEPARTED (not-yet-confirmed-dead or already-collapsed) identities this detector keeps individual bookkeeping for at once - a present/armed node is never counted against it and never refused tracking. Accepted range 1..16384; 0 is refused rather than clamped, since a cap of nothing would mean tracking nothing. See “Bounded by evidence, not by age” below for what it actually bounds and why, unlike the sibling detector’s identical-looking cap, a newcomer here is never the thing refused. |
tick_interval_ms is not an own key of this detector, but this is the one detector that
reads the plugin-injected value directly, to compute miss_grace’s wall-clock floor below.
Liveness, and why membership in the snapshot is not enough. App::is_online is what
this detector tracks, never mere presence in the entity snapshot. In runtime-only discovery
the two happen to coincide - a dead node’s App leaves the snapshot entirely - but a manifest
keeps a bound App present with only is_online cleared once its ROS binding disappears, and
hybrid discovery inherits that same shape; counting snapshot membership alone would make a
manifest-declared node immortal. A managed lifecycle node that merely deactivates keeps
is_online: true - its process is still running, only its ROS 2 lifecycle state changed -
so a deactivation is never mistaken for a death either; that is lifecycle_expectation’s
concern, not this one’s (see “The boundary with lifecycle_expectation” below).
Tracking is keyed on the STABLE fqn (App::effective_fqn()), never App::id: an id is
recomputed every sweep and only gains a namespace prefix once a same-bare-name collision
currently exists anywhere in the graph, so a live node’s id can change out from under a key
built from it.
What is never tracked.
| Excluded | Why |
|---|---|
Peer-aggregated apps (app.source starting peer:) |
A peer app carries no ROS binding of its own, so effective_fqn() is empty for every one of them; tracking them would collapse an entire peer fleet onto one "" key, and one online peer app would mask every other peer app’s departure. |
_ros2cli_<pid> nodes |
Every ros2 CLI invocation (ros2 topic echo, ros2 param get, …) spins up a real, short-lived node under this prefix - rcl’s own hidden-node naming convention, matched structurally rather than guessed at. Letting these through would accumulate one permanently “dead” entry per CLI invocation for the life of the gateway. |
| An App that never comes online | A key is admitted to tracking only once the reliability gate has said this detector OWNS its departure - armed at least once, and on one of the two GROUNDS ownership can rest on (a lifecycle state read as active, or no lifecycle to read at all; the third case, a state nobody could read, is admitted PROVISIONALLY and is described in the row below). A manifest App whose binding never starts is never armed and so can never be falsely called dead; a node still inside its warmup_cycles window gets the identical protection. Once tracked, though, a node’s continued life-or-death judgement rests on PRESENCE alone, not on staying armed - a tracked node that later goes lifecycle-inactive is not thereby mistaken for dead. |
| A managed App whose lifecycle state has not been read YET | Not tracked, and only for as long as that is still true. The gate’s own raise permission is answered PERMISSIVELY for a managed node whose GetState has not answered (LifecycleWatcher::node_ok()) - without that, qos_mismatch, orphan and param_drift would all fall silent about a node whose lifecycle service is broken. Permission is not knowledge, though, so tracking is withheld while a measurement may still arrive. LifecycleWatcher charges a GetState re-seed budget per node, and only for a read that actually ran, so “still asking” is a fact rather than a guess. Once that budget is spent the node is tracked here after all: nothing will ASK again, and lifecycle_expectation looks at nothing unless an operator named the node in require_active (empty by default), so withholding it any longer would mean the death is reported by nobody. That admission is PROVISIONAL, not final. The ~/transition_event subscription outlives the seed budget, so a label can still arrive without being asked for, and one that reads non-active says the node was the lifecycle detector’s all along - so the presence detector hands it back on the spot, while it is still alive. Waiting for the budget alone would only have delayed the wrong report, not prevented it. |
File truncated at 100 lines see the full file
CHANGELOG
Changelog for package ros2_medkit_graph_watchdog
0.7.0 (2026-08-27)
- Initial release of the package: a gateway plugin that raises faults
for silent failures in the ROS 2 graph - problems that leave every
node alive and every topic present, so nothing in the stack reports
them today. The package carries the plugin skeleton, a central
reliability gate that holds raises until the graph has quiesced, and
a detector registry. Detectors are configured under
plugins.graph_watchdog.<key>, each with araise/advisory/offmode, and an unknown key underdetectors.<id>is reported rather than silently ignored, so a typo cannot quietly disable a check (#571) -
qos_mismatchdetector: raisesGRAPH_QOS_MISMATCHfor a publisher and subscriber whose QoS profiles cannot match, which leaves the connection silently unestablished (#571) -
orphandetector: raisesGRAPH_ORPHANfor a one-sided topic - a publisher with no subscriber, or the reverse - but only when a complementary near-miss counterpart exists: same message type, a namespace and leaf within the configured edit distance, and the opposite side. A lone unsubscribed topic raises nothing. The pair is the signature of a remap or a topic-name typo, which is what the detector is for (#578) -
param_driftdetector: raisesGRAPH_PARAM_DRIFTwhen a node's live parameter value diverges from the declared expectation (#580) -
lifecycle_expectationdetector: raisesGRAPH_NODE_INACTIVEfor a node named inrequire_activethat is not in theactivelifecycle state once itsgracewindow has passed. It raises two further codes for the cases where the state could not be established at all:GRAPH_NODE_UNREADABLEwhen the lifecycle service does not answer, andGRAPH_NODE_NOT_MANAGEDwhen the named node has no lifecycle to read. These are distinct codes on the wire, so a consumer filtering onGRAPH_NODE_INACTIVEalone sees neither unmeasured case. A node is matched by itsApp::id, its full FQN, or the bare leaf of that FQN, and the grace streak is counted per node, so naming one node in two documented forms cannot halve the grace that was configured (#587) -
node_deathdetector: raisesGRAPH_NODE_DISAPPEAREDfor a node that leaves the graph when nothing else is left to report it, with a suppression framework that decides ownership of a departure from knowledge rather than from silence. A node whose lifecycle state was never readable is admitted only provisionally, and is handed back the moment a label arrives saying the departure belonged to the lifecycle detector instead. Zero-config: there is no list of nodes to maintain (#625, #624) - Two silent-fault classes are not delivered in this release.
GRAPH_TF_STALEandGRAPH_LATENCY_BUDGEThave their fault codes reserved in the frozenGRAPH_*namespace and land in later changes, each against its own issue - The plugin keeps its ROS entities off the gateway executor, so an entity created and destroyed while the gateway runs cannot have its destructor run on an executor thread concurrently with a create on the same node
- Contributors: \@bburda
Package Dependencies
System Dependencies
Dependant Packages
No known dependants.
Launch files
No launch files found
Messages
No message files found.
Services
No service files found
Plugins
No plugins found.
Recent questions tagged ros2_medkit_graph_watchdog at Robotics Stack Exchange
No version for distro jade showing humble. Known supported distros are highlighted in the buttons above.
Package Summary
| Version | 0.7.0 |
| License | Apache-2.0 |
| Build type | AMENT_CMAKE |
| Use | RECOMMENDED |
Repository Summary
| Checkout URI | https://github.com/selfpatch/ros2_medkit.git |
| VCS Type | git |
| VCS Version | main |
| Last Updated | 2026-09-06 |
| Dev Status | DEVELOPED |
| Released | RELEASED |
| Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
Gateway plugin that detects silent faults in the ROS 2 graph: QoS mismatches, orphaned topics left by a name typo, dead nodes, parameter drift and unmet lifecycle expectations.
Maintainers
- bburda
Authors
No additional authors.
ros2_medkit_graph_watchdog
Gateway plugin that detects silent faults in the ROS 2 graph: failures where every node is up, nothing logs an error, and the robot is still broken.
Detectors read the graph and raise faults through a ReportFault service client on the
gateway node; the faults surface via FaultManager on the gateway /faults API. That
client, like the lifecycle subscriptions below, lives in the plugin’s own callback group
and is driven by the plugin’s own executor from the tick thread, never by the gateway’s
ROS executor.
This package carries the plugin skeleton, the central reliability gate that holds raises
until the graph has quiesced, and five detectors, qos_mismatch, orphan,
param_drift, lifecycle_expectation and node_death. Two silent-fault classes remain
undelivered, GRAPH_TF_STALE and GRAPH_LATENCY_BUDGET; they land in follow-up changes,
each against its own issue, and their fault codes are already reserved in the frozen
GRAPH_* namespace (see “Fault codes”).
Build
With the gateway built and sourced:
colcon build --packages-select ros2_medkit_graph_watchdog
colcon test --packages-select ros2_medkit_graph_watchdog && colcon test-result --verbose
Load into the gateway
ros2 run ros2_medkit_gateway gateway_node --ros-args \
-p plugins:="[graph_watchdog]" \
-p "plugins.graph_watchdog.path:=$(ros2 pkg prefix ros2_medkit_graph_watchdog)/lib/ros2_medkit_graph_watchdog/libros2_medkit_graph_watchdog.so"
Configuration (plugins.graph_watchdog.<key>)
| Key | Type | Default | Meaning |
|---|---|---|---|
tick_interval_ms |
int | 1000 |
Detector tick cadence. Accepted range 1..INT_MAX; the check runs on the wide integer, so a value past INT_MAX is rejected rather than truncated back into the band. |
warmup_cycles |
int | 5 |
An entity must be continuously present for this many ticks before it arms. A mid-run restart re-warms it. Enforced centrally, see “Reliability (bringup-quiesce)” below. Accepted range 0..INT_MAX, checked on the wide integer. |
prune_grace |
int | 60 |
Default injected into every detector’s own config before configure(); a per-detector detectors.<id>.prune_grace overrides it. Used by detectors that keep per-key bookkeeping. Accepted range 0..INT_MAX here, checked on the wide integer; each detector re-validates it against its own documented range before using it. |
detectors.<id>.mode |
string | bool | raise |
raise = push faults; advisory = observe, do not push; off = disabled. Bare off/no (YAML booleans) also disable; bare on/yes mean raise. |
detectors.<id>.<field> |
any | - | Per-detector thresholds, passed to that detector’s configure(). |
Nested delivery. Keys are written dotted in YAML or --ros-args, and the gateway
delivers them to the plugin as a nested object. A bare off is a YAML 1.1 boolean, so
mode: off arrives as false, not the string "off"; both forms disable the detector.
Unknown keys are reported. Anything under detectors.<id> that the detector does not
read produces one startup warning naming the key and listing the ones that exist, so a
typo like allow_list: for allowlist: cannot silently do nothing. A typo in the detector
id itself is warned about the same way, with the registered ids listed.
lifecycle_expectation keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. |
require_active |
string[] | [] |
Node names that must be in the active lifecycle state, each matched against a live node by its App::id, its full FQN (/ns/name), or the bare leaf of that FQN. A bare name matches that node in EVERY namespace (a fleet-wide “all controller_servers must be active”); use a full FQN to pin one robot’s. Both the bare and the FQN form match against the node’s stable FQN, so they survive the App::id renaming that a same-bare-name collision triggers; an entry written as an App::id also works, but can stop matching on a multi-robot graph for exactly that reason. Empty = the detector checks nothing and emits nothing. A require_active that is not a string array warns and is ignored; an empty-string entry warns and is skipped - never dropped silently, since the operator would go on believing the node is covered. |
grace |
int | 5 |
Consecutive not-active ticks a required node tolerates before being reported inactive - bringup time for a managed node to reach active (configure + activate) before the expectation is enforced. Counted per NODE, not per matching entry: a node named by both a bare-name and a full-FQN entry still advances its streak once per tick, so mixing the two documented forms cannot halve the grace that was configured. An ALREADY-CONFIRMED streak keeps its content while the node is ABSENT, so a node that leaves the graph after being confirmed stays confirmed; a streak that has not yet crossed grace on a node the presence detector could also report (it was armed at least once) is HELD rather than advanced while absent, so a node is never confirmed out of ticks gathered while nobody could observe it - only a PRESENT tick may still cross grace for that node. A node that was NEVER armed gets no such hold: node_death can never track it either, so absence keeps advancing its streak too, the same as a present tick would. Accepted range 0..300 - five minutes at the shipped 1 s cadence, and already an extravagant allowance for a managed node to reach active. The upper end is a real bound rather than a formality: grace decides how long a PRESENT node may go unreported, and how long a node returning from a departure still inactive takes to (re-)mature, so at the old maximum of INT_MAX - 1 that PRESENT-side silence lasted about 24 days at the shipped cadence with no warning. Whether it also bounds how long a node that leaves the graph BELOW grace sits unsettled depends on the same split: for an armed node that lasts for as long as the node stays gone, independent of grace (see “Bounded by evidence, not by age” below); for a never-armed one grace bounds that too, since absence matures it within grace + absence_grace + 1 ticks either way. The check runs on the wide integer, so a value that only fits after truncation is rejected rather than silently turned into a hair-trigger. Anything outside the range warns and keeps the default. |
prune_grace |
int | 60 |
Consecutive ticks an IDLE tracked node - both clocks at zero and no matured ownership, so nothing to lose - may stay ABSENT from the graph before its bookkeeping is reclaimed. A node carrying evidence is never reclaimed by age at all, however long it stays gone, so this key cannot erase a fault’s own state; the map is bounded instead by tracked_node_cap (see “Bounded by evidence, not by age”). Injected for every detector at plugin scope and overridable per detector; a value outside 0..3600 warns and this detector keeps its own default of 60. The range check runs on the wide integer, so an out-of-int-range value is rejected rather than truncated. The value is used as written - there is no grace + 1 clamp, because there is no longer anything for one to protect. |
tracked_node_cap |
int | 512 |
The most nodes this detector keeps bookkeeping for at once. It is what bounds the map, since evidence is never reclaimed by age (see “Bounded by evidence, not by age”): at the cap, idle entries are reclaimed first, then entries for DEPARTED nodes are collapsed into a count, and only if every tracked node is PRESENT and carrying evidence is a newly matched node refused - which withholds GRAPH_NODE_INACTIVE’s clear and is reported both in the log and on GET /x-medkit-watchdog. 512 is comfortably above every node in a realistic graph (a full Nav2 stack plus perception is roughly a hundred nodes; a ten-robot fleet sharing one domain a few hundred), so raising it is only needed where require_active legitimately matches more PRESENT nodes than that. Accepted range 1..16384 - 16384 is about 8 MB of bookkeeping at roughly 500 bytes per tracked node, and a deployment needing more distinct required-node identities alive at once has identity churn rather than a large fleet. Anything outside the range, including 0 (a cap of nothing would mean checking nothing), warns and keeps the default; the check runs on the wide integer, so an out-of-int-range value is rejected rather than truncated. |
node_death keys
Zero-config, unlike every detector above: there is no require_active-style list of nodes
to watch, because every armed App in the graph is a candidate.
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. |
miss_grace |
int | 2 |
Consecutive missed ticks a tracked node tolerates before being reported dead - a death is reported once misses EXCEED this, i.e. after miss_grace + 1 ticks. Accepted range 0..3600, and additionally floored to whatever wall-clock window the configured tick_interval_ms needs (see “The wall-clock floor” below); a value below that floor is silently raised to it, with a warning naming the ms window it actually spans. |
prune_grace |
int | 60 |
Plugin-injected default (overridable here); how long a key being DURABLY suppressed (see “Suppression” below) may sit unreported before its bookkeeping is reclaimed. Accepted range 0..3600. The value actually used is max(prune_grace, miss_grace + 1), a silent internal floor rather than a rejection: without it, a durably-suppressed key could be reclaimed the very tick it would first have become eligible to report, losing the report rather than merely suppressing it. |
allowlist |
string[] | [] |
Node identities never reported dead. A dead key matches if it - or a name the same node is also known by - is present verbatim: the key’s own fqn, its bare leaf, or (captured while the node was still present) its App::id. Has no effect unless named in suppress; naming it without a configured list is a no-op. |
suppress |
string[] | [] |
Suppression mechanisms to activate: "allowlist" and "lifecycle". Any other entry warns as unrecognized and is ignored; a non-string or non-array entry warns the same way. |
tracked_node_cap |
int | 512 |
The most DEPARTED (not-yet-confirmed-dead or already-collapsed) identities this detector keeps individual bookkeeping for at once - a present/armed node is never counted against it and never refused tracking. Accepted range 1..16384; 0 is refused rather than clamped, since a cap of nothing would mean tracking nothing. See “Bounded by evidence, not by age” below for what it actually bounds and why, unlike the sibling detector’s identical-looking cap, a newcomer here is never the thing refused. |
tick_interval_ms is not an own key of this detector, but this is the one detector that
reads the plugin-injected value directly, to compute miss_grace’s wall-clock floor below.
Liveness, and why membership in the snapshot is not enough. App::is_online is what
this detector tracks, never mere presence in the entity snapshot. In runtime-only discovery
the two happen to coincide - a dead node’s App leaves the snapshot entirely - but a manifest
keeps a bound App present with only is_online cleared once its ROS binding disappears, and
hybrid discovery inherits that same shape; counting snapshot membership alone would make a
manifest-declared node immortal. A managed lifecycle node that merely deactivates keeps
is_online: true - its process is still running, only its ROS 2 lifecycle state changed -
so a deactivation is never mistaken for a death either; that is lifecycle_expectation’s
concern, not this one’s (see “The boundary with lifecycle_expectation” below).
Tracking is keyed on the STABLE fqn (App::effective_fqn()), never App::id: an id is
recomputed every sweep and only gains a namespace prefix once a same-bare-name collision
currently exists anywhere in the graph, so a live node’s id can change out from under a key
built from it.
What is never tracked.
| Excluded | Why |
|---|---|
Peer-aggregated apps (app.source starting peer:) |
A peer app carries no ROS binding of its own, so effective_fqn() is empty for every one of them; tracking them would collapse an entire peer fleet onto one "" key, and one online peer app would mask every other peer app’s departure. |
_ros2cli_<pid> nodes |
Every ros2 CLI invocation (ros2 topic echo, ros2 param get, …) spins up a real, short-lived node under this prefix - rcl’s own hidden-node naming convention, matched structurally rather than guessed at. Letting these through would accumulate one permanently “dead” entry per CLI invocation for the life of the gateway. |
| An App that never comes online | A key is admitted to tracking only once the reliability gate has said this detector OWNS its departure - armed at least once, and on one of the two GROUNDS ownership can rest on (a lifecycle state read as active, or no lifecycle to read at all; the third case, a state nobody could read, is admitted PROVISIONALLY and is described in the row below). A manifest App whose binding never starts is never armed and so can never be falsely called dead; a node still inside its warmup_cycles window gets the identical protection. Once tracked, though, a node’s continued life-or-death judgement rests on PRESENCE alone, not on staying armed - a tracked node that later goes lifecycle-inactive is not thereby mistaken for dead. |
| A managed App whose lifecycle state has not been read YET | Not tracked, and only for as long as that is still true. The gate’s own raise permission is answered PERMISSIVELY for a managed node whose GetState has not answered (LifecycleWatcher::node_ok()) - without that, qos_mismatch, orphan and param_drift would all fall silent about a node whose lifecycle service is broken. Permission is not knowledge, though, so tracking is withheld while a measurement may still arrive. LifecycleWatcher charges a GetState re-seed budget per node, and only for a read that actually ran, so “still asking” is a fact rather than a guess. Once that budget is spent the node is tracked here after all: nothing will ASK again, and lifecycle_expectation looks at nothing unless an operator named the node in require_active (empty by default), so withholding it any longer would mean the death is reported by nobody. That admission is PROVISIONAL, not final. The ~/transition_event subscription outlives the seed budget, so a label can still arrive without being asked for, and one that reads non-active says the node was the lifecycle detector’s all along - so the presence detector hands it back on the spot, while it is still alive. Waiting for the budget alone would only have delayed the wrong report, not prevented it. |
File truncated at 100 lines see the full file
CHANGELOG
Changelog for package ros2_medkit_graph_watchdog
0.7.0 (2026-08-27)
- Initial release of the package: a gateway plugin that raises faults
for silent failures in the ROS 2 graph - problems that leave every
node alive and every topic present, so nothing in the stack reports
them today. The package carries the plugin skeleton, a central
reliability gate that holds raises until the graph has quiesced, and
a detector registry. Detectors are configured under
plugins.graph_watchdog.<key>, each with araise/advisory/offmode, and an unknown key underdetectors.<id>is reported rather than silently ignored, so a typo cannot quietly disable a check (#571) -
qos_mismatchdetector: raisesGRAPH_QOS_MISMATCHfor a publisher and subscriber whose QoS profiles cannot match, which leaves the connection silently unestablished (#571) -
orphandetector: raisesGRAPH_ORPHANfor a one-sided topic - a publisher with no subscriber, or the reverse - but only when a complementary near-miss counterpart exists: same message type, a namespace and leaf within the configured edit distance, and the opposite side. A lone unsubscribed topic raises nothing. The pair is the signature of a remap or a topic-name typo, which is what the detector is for (#578) -
param_driftdetector: raisesGRAPH_PARAM_DRIFTwhen a node's live parameter value diverges from the declared expectation (#580) -
lifecycle_expectationdetector: raisesGRAPH_NODE_INACTIVEfor a node named inrequire_activethat is not in theactivelifecycle state once itsgracewindow has passed. It raises two further codes for the cases where the state could not be established at all:GRAPH_NODE_UNREADABLEwhen the lifecycle service does not answer, andGRAPH_NODE_NOT_MANAGEDwhen the named node has no lifecycle to read. These are distinct codes on the wire, so a consumer filtering onGRAPH_NODE_INACTIVEalone sees neither unmeasured case. A node is matched by itsApp::id, its full FQN, or the bare leaf of that FQN, and the grace streak is counted per node, so naming one node in two documented forms cannot halve the grace that was configured (#587) -
node_deathdetector: raisesGRAPH_NODE_DISAPPEAREDfor a node that leaves the graph when nothing else is left to report it, with a suppression framework that decides ownership of a departure from knowledge rather than from silence. A node whose lifecycle state was never readable is admitted only provisionally, and is handed back the moment a label arrives saying the departure belonged to the lifecycle detector instead. Zero-config: there is no list of nodes to maintain (#625, #624) - Two silent-fault classes are not delivered in this release.
GRAPH_TF_STALEandGRAPH_LATENCY_BUDGEThave their fault codes reserved in the frozenGRAPH_*namespace and land in later changes, each against its own issue - The plugin keeps its ROS entities off the gateway executor, so an entity created and destroyed while the gateway runs cannot have its destructor run on an executor thread concurrently with a create on the same node
- Contributors: \@bburda
Package Dependencies
System Dependencies
Dependant Packages
No known dependants.
Launch files
No launch files found
Messages
No message files found.
Services
No service files found
Plugins
No plugins found.
Recent questions tagged ros2_medkit_graph_watchdog at Robotics Stack Exchange
No version for distro indigo showing humble. Known supported distros are highlighted in the buttons above.
Package Summary
| Version | 0.7.0 |
| License | Apache-2.0 |
| Build type | AMENT_CMAKE |
| Use | RECOMMENDED |
Repository Summary
| Checkout URI | https://github.com/selfpatch/ros2_medkit.git |
| VCS Type | git |
| VCS Version | main |
| Last Updated | 2026-09-06 |
| Dev Status | DEVELOPED |
| Released | RELEASED |
| Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
Gateway plugin that detects silent faults in the ROS 2 graph: QoS mismatches, orphaned topics left by a name typo, dead nodes, parameter drift and unmet lifecycle expectations.
Maintainers
- bburda
Authors
No additional authors.
ros2_medkit_graph_watchdog
Gateway plugin that detects silent faults in the ROS 2 graph: failures where every node is up, nothing logs an error, and the robot is still broken.
Detectors read the graph and raise faults through a ReportFault service client on the
gateway node; the faults surface via FaultManager on the gateway /faults API. That
client, like the lifecycle subscriptions below, lives in the plugin’s own callback group
and is driven by the plugin’s own executor from the tick thread, never by the gateway’s
ROS executor.
This package carries the plugin skeleton, the central reliability gate that holds raises
until the graph has quiesced, and five detectors, qos_mismatch, orphan,
param_drift, lifecycle_expectation and node_death. Two silent-fault classes remain
undelivered, GRAPH_TF_STALE and GRAPH_LATENCY_BUDGET; they land in follow-up changes,
each against its own issue, and their fault codes are already reserved in the frozen
GRAPH_* namespace (see “Fault codes”).
Build
With the gateway built and sourced:
colcon build --packages-select ros2_medkit_graph_watchdog
colcon test --packages-select ros2_medkit_graph_watchdog && colcon test-result --verbose
Load into the gateway
ros2 run ros2_medkit_gateway gateway_node --ros-args \
-p plugins:="[graph_watchdog]" \
-p "plugins.graph_watchdog.path:=$(ros2 pkg prefix ros2_medkit_graph_watchdog)/lib/ros2_medkit_graph_watchdog/libros2_medkit_graph_watchdog.so"
Configuration (plugins.graph_watchdog.<key>)
| Key | Type | Default | Meaning |
|---|---|---|---|
tick_interval_ms |
int | 1000 |
Detector tick cadence. Accepted range 1..INT_MAX; the check runs on the wide integer, so a value past INT_MAX is rejected rather than truncated back into the band. |
warmup_cycles |
int | 5 |
An entity must be continuously present for this many ticks before it arms. A mid-run restart re-warms it. Enforced centrally, see “Reliability (bringup-quiesce)” below. Accepted range 0..INT_MAX, checked on the wide integer. |
prune_grace |
int | 60 |
Default injected into every detector’s own config before configure(); a per-detector detectors.<id>.prune_grace overrides it. Used by detectors that keep per-key bookkeeping. Accepted range 0..INT_MAX here, checked on the wide integer; each detector re-validates it against its own documented range before using it. |
detectors.<id>.mode |
string | bool | raise |
raise = push faults; advisory = observe, do not push; off = disabled. Bare off/no (YAML booleans) also disable; bare on/yes mean raise. |
detectors.<id>.<field> |
any | - | Per-detector thresholds, passed to that detector’s configure(). |
Nested delivery. Keys are written dotted in YAML or --ros-args, and the gateway
delivers them to the plugin as a nested object. A bare off is a YAML 1.1 boolean, so
mode: off arrives as false, not the string "off"; both forms disable the detector.
Unknown keys are reported. Anything under detectors.<id> that the detector does not
read produces one startup warning naming the key and listing the ones that exist, so a
typo like allow_list: for allowlist: cannot silently do nothing. A typo in the detector
id itself is warned about the same way, with the registered ids listed.
lifecycle_expectation keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. |
require_active |
string[] | [] |
Node names that must be in the active lifecycle state, each matched against a live node by its App::id, its full FQN (/ns/name), or the bare leaf of that FQN. A bare name matches that node in EVERY namespace (a fleet-wide “all controller_servers must be active”); use a full FQN to pin one robot’s. Both the bare and the FQN form match against the node’s stable FQN, so they survive the App::id renaming that a same-bare-name collision triggers; an entry written as an App::id also works, but can stop matching on a multi-robot graph for exactly that reason. Empty = the detector checks nothing and emits nothing. A require_active that is not a string array warns and is ignored; an empty-string entry warns and is skipped - never dropped silently, since the operator would go on believing the node is covered. |
grace |
int | 5 |
Consecutive not-active ticks a required node tolerates before being reported inactive - bringup time for a managed node to reach active (configure + activate) before the expectation is enforced. Counted per NODE, not per matching entry: a node named by both a bare-name and a full-FQN entry still advances its streak once per tick, so mixing the two documented forms cannot halve the grace that was configured. An ALREADY-CONFIRMED streak keeps its content while the node is ABSENT, so a node that leaves the graph after being confirmed stays confirmed; a streak that has not yet crossed grace on a node the presence detector could also report (it was armed at least once) is HELD rather than advanced while absent, so a node is never confirmed out of ticks gathered while nobody could observe it - only a PRESENT tick may still cross grace for that node. A node that was NEVER armed gets no such hold: node_death can never track it either, so absence keeps advancing its streak too, the same as a present tick would. Accepted range 0..300 - five minutes at the shipped 1 s cadence, and already an extravagant allowance for a managed node to reach active. The upper end is a real bound rather than a formality: grace decides how long a PRESENT node may go unreported, and how long a node returning from a departure still inactive takes to (re-)mature, so at the old maximum of INT_MAX - 1 that PRESENT-side silence lasted about 24 days at the shipped cadence with no warning. Whether it also bounds how long a node that leaves the graph BELOW grace sits unsettled depends on the same split: for an armed node that lasts for as long as the node stays gone, independent of grace (see “Bounded by evidence, not by age” below); for a never-armed one grace bounds that too, since absence matures it within grace + absence_grace + 1 ticks either way. The check runs on the wide integer, so a value that only fits after truncation is rejected rather than silently turned into a hair-trigger. Anything outside the range warns and keeps the default. |
prune_grace |
int | 60 |
Consecutive ticks an IDLE tracked node - both clocks at zero and no matured ownership, so nothing to lose - may stay ABSENT from the graph before its bookkeeping is reclaimed. A node carrying evidence is never reclaimed by age at all, however long it stays gone, so this key cannot erase a fault’s own state; the map is bounded instead by tracked_node_cap (see “Bounded by evidence, not by age”). Injected for every detector at plugin scope and overridable per detector; a value outside 0..3600 warns and this detector keeps its own default of 60. The range check runs on the wide integer, so an out-of-int-range value is rejected rather than truncated. The value is used as written - there is no grace + 1 clamp, because there is no longer anything for one to protect. |
tracked_node_cap |
int | 512 |
The most nodes this detector keeps bookkeeping for at once. It is what bounds the map, since evidence is never reclaimed by age (see “Bounded by evidence, not by age”): at the cap, idle entries are reclaimed first, then entries for DEPARTED nodes are collapsed into a count, and only if every tracked node is PRESENT and carrying evidence is a newly matched node refused - which withholds GRAPH_NODE_INACTIVE’s clear and is reported both in the log and on GET /x-medkit-watchdog. 512 is comfortably above every node in a realistic graph (a full Nav2 stack plus perception is roughly a hundred nodes; a ten-robot fleet sharing one domain a few hundred), so raising it is only needed where require_active legitimately matches more PRESENT nodes than that. Accepted range 1..16384 - 16384 is about 8 MB of bookkeeping at roughly 500 bytes per tracked node, and a deployment needing more distinct required-node identities alive at once has identity churn rather than a large fleet. Anything outside the range, including 0 (a cap of nothing would mean checking nothing), warns and keeps the default; the check runs on the wide integer, so an out-of-int-range value is rejected rather than truncated. |
node_death keys
Zero-config, unlike every detector above: there is no require_active-style list of nodes
to watch, because every armed App in the graph is a candidate.
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. |
miss_grace |
int | 2 |
Consecutive missed ticks a tracked node tolerates before being reported dead - a death is reported once misses EXCEED this, i.e. after miss_grace + 1 ticks. Accepted range 0..3600, and additionally floored to whatever wall-clock window the configured tick_interval_ms needs (see “The wall-clock floor” below); a value below that floor is silently raised to it, with a warning naming the ms window it actually spans. |
prune_grace |
int | 60 |
Plugin-injected default (overridable here); how long a key being DURABLY suppressed (see “Suppression” below) may sit unreported before its bookkeeping is reclaimed. Accepted range 0..3600. The value actually used is max(prune_grace, miss_grace + 1), a silent internal floor rather than a rejection: without it, a durably-suppressed key could be reclaimed the very tick it would first have become eligible to report, losing the report rather than merely suppressing it. |
allowlist |
string[] | [] |
Node identities never reported dead. A dead key matches if it - or a name the same node is also known by - is present verbatim: the key’s own fqn, its bare leaf, or (captured while the node was still present) its App::id. Has no effect unless named in suppress; naming it without a configured list is a no-op. |
suppress |
string[] | [] |
Suppression mechanisms to activate: "allowlist" and "lifecycle". Any other entry warns as unrecognized and is ignored; a non-string or non-array entry warns the same way. |
tracked_node_cap |
int | 512 |
The most DEPARTED (not-yet-confirmed-dead or already-collapsed) identities this detector keeps individual bookkeeping for at once - a present/armed node is never counted against it and never refused tracking. Accepted range 1..16384; 0 is refused rather than clamped, since a cap of nothing would mean tracking nothing. See “Bounded by evidence, not by age” below for what it actually bounds and why, unlike the sibling detector’s identical-looking cap, a newcomer here is never the thing refused. |
tick_interval_ms is not an own key of this detector, but this is the one detector that
reads the plugin-injected value directly, to compute miss_grace’s wall-clock floor below.
Liveness, and why membership in the snapshot is not enough. App::is_online is what
this detector tracks, never mere presence in the entity snapshot. In runtime-only discovery
the two happen to coincide - a dead node’s App leaves the snapshot entirely - but a manifest
keeps a bound App present with only is_online cleared once its ROS binding disappears, and
hybrid discovery inherits that same shape; counting snapshot membership alone would make a
manifest-declared node immortal. A managed lifecycle node that merely deactivates keeps
is_online: true - its process is still running, only its ROS 2 lifecycle state changed -
so a deactivation is never mistaken for a death either; that is lifecycle_expectation’s
concern, not this one’s (see “The boundary with lifecycle_expectation” below).
Tracking is keyed on the STABLE fqn (App::effective_fqn()), never App::id: an id is
recomputed every sweep and only gains a namespace prefix once a same-bare-name collision
currently exists anywhere in the graph, so a live node’s id can change out from under a key
built from it.
What is never tracked.
| Excluded | Why |
|---|---|
Peer-aggregated apps (app.source starting peer:) |
A peer app carries no ROS binding of its own, so effective_fqn() is empty for every one of them; tracking them would collapse an entire peer fleet onto one "" key, and one online peer app would mask every other peer app’s departure. |
_ros2cli_<pid> nodes |
Every ros2 CLI invocation (ros2 topic echo, ros2 param get, …) spins up a real, short-lived node under this prefix - rcl’s own hidden-node naming convention, matched structurally rather than guessed at. Letting these through would accumulate one permanently “dead” entry per CLI invocation for the life of the gateway. |
| An App that never comes online | A key is admitted to tracking only once the reliability gate has said this detector OWNS its departure - armed at least once, and on one of the two GROUNDS ownership can rest on (a lifecycle state read as active, or no lifecycle to read at all; the third case, a state nobody could read, is admitted PROVISIONALLY and is described in the row below). A manifest App whose binding never starts is never armed and so can never be falsely called dead; a node still inside its warmup_cycles window gets the identical protection. Once tracked, though, a node’s continued life-or-death judgement rests on PRESENCE alone, not on staying armed - a tracked node that later goes lifecycle-inactive is not thereby mistaken for dead. |
| A managed App whose lifecycle state has not been read YET | Not tracked, and only for as long as that is still true. The gate’s own raise permission is answered PERMISSIVELY for a managed node whose GetState has not answered (LifecycleWatcher::node_ok()) - without that, qos_mismatch, orphan and param_drift would all fall silent about a node whose lifecycle service is broken. Permission is not knowledge, though, so tracking is withheld while a measurement may still arrive. LifecycleWatcher charges a GetState re-seed budget per node, and only for a read that actually ran, so “still asking” is a fact rather than a guess. Once that budget is spent the node is tracked here after all: nothing will ASK again, and lifecycle_expectation looks at nothing unless an operator named the node in require_active (empty by default), so withholding it any longer would mean the death is reported by nobody. That admission is PROVISIONAL, not final. The ~/transition_event subscription outlives the seed budget, so a label can still arrive without being asked for, and one that reads non-active says the node was the lifecycle detector’s all along - so the presence detector hands it back on the spot, while it is still alive. Waiting for the budget alone would only have delayed the wrong report, not prevented it. |
File truncated at 100 lines see the full file
CHANGELOG
Changelog for package ros2_medkit_graph_watchdog
0.7.0 (2026-08-27)
- Initial release of the package: a gateway plugin that raises faults
for silent failures in the ROS 2 graph - problems that leave every
node alive and every topic present, so nothing in the stack reports
them today. The package carries the plugin skeleton, a central
reliability gate that holds raises until the graph has quiesced, and
a detector registry. Detectors are configured under
plugins.graph_watchdog.<key>, each with araise/advisory/offmode, and an unknown key underdetectors.<id>is reported rather than silently ignored, so a typo cannot quietly disable a check (#571) -
qos_mismatchdetector: raisesGRAPH_QOS_MISMATCHfor a publisher and subscriber whose QoS profiles cannot match, which leaves the connection silently unestablished (#571) -
orphandetector: raisesGRAPH_ORPHANfor a one-sided topic - a publisher with no subscriber, or the reverse - but only when a complementary near-miss counterpart exists: same message type, a namespace and leaf within the configured edit distance, and the opposite side. A lone unsubscribed topic raises nothing. The pair is the signature of a remap or a topic-name typo, which is what the detector is for (#578) -
param_driftdetector: raisesGRAPH_PARAM_DRIFTwhen a node's live parameter value diverges from the declared expectation (#580) -
lifecycle_expectationdetector: raisesGRAPH_NODE_INACTIVEfor a node named inrequire_activethat is not in theactivelifecycle state once itsgracewindow has passed. It raises two further codes for the cases where the state could not be established at all:GRAPH_NODE_UNREADABLEwhen the lifecycle service does not answer, andGRAPH_NODE_NOT_MANAGEDwhen the named node has no lifecycle to read. These are distinct codes on the wire, so a consumer filtering onGRAPH_NODE_INACTIVEalone sees neither unmeasured case. A node is matched by itsApp::id, its full FQN, or the bare leaf of that FQN, and the grace streak is counted per node, so naming one node in two documented forms cannot halve the grace that was configured (#587) -
node_deathdetector: raisesGRAPH_NODE_DISAPPEAREDfor a node that leaves the graph when nothing else is left to report it, with a suppression framework that decides ownership of a departure from knowledge rather than from silence. A node whose lifecycle state was never readable is admitted only provisionally, and is handed back the moment a label arrives saying the departure belonged to the lifecycle detector instead. Zero-config: there is no list of nodes to maintain (#625, #624) - Two silent-fault classes are not delivered in this release.
GRAPH_TF_STALEandGRAPH_LATENCY_BUDGEThave their fault codes reserved in the frozenGRAPH_*namespace and land in later changes, each against its own issue - The plugin keeps its ROS entities off the gateway executor, so an entity created and destroyed while the gateway runs cannot have its destructor run on an executor thread concurrently with a create on the same node
- Contributors: \@bburda
Package Dependencies
System Dependencies
Dependant Packages
No known dependants.
Launch files
No launch files found
Messages
No message files found.
Services
No service files found
Plugins
No plugins found.
Recent questions tagged ros2_medkit_graph_watchdog at Robotics Stack Exchange
No version for distro hydro showing humble. Known supported distros are highlighted in the buttons above.
Package Summary
| Version | 0.7.0 |
| License | Apache-2.0 |
| Build type | AMENT_CMAKE |
| Use | RECOMMENDED |
Repository Summary
| Checkout URI | https://github.com/selfpatch/ros2_medkit.git |
| VCS Type | git |
| VCS Version | main |
| Last Updated | 2026-09-06 |
| Dev Status | DEVELOPED |
| Released | RELEASED |
| Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
Gateway plugin that detects silent faults in the ROS 2 graph: QoS mismatches, orphaned topics left by a name typo, dead nodes, parameter drift and unmet lifecycle expectations.
Maintainers
- bburda
Authors
No additional authors.
ros2_medkit_graph_watchdog
Gateway plugin that detects silent faults in the ROS 2 graph: failures where every node is up, nothing logs an error, and the robot is still broken.
Detectors read the graph and raise faults through a ReportFault service client on the
gateway node; the faults surface via FaultManager on the gateway /faults API. That
client, like the lifecycle subscriptions below, lives in the plugin’s own callback group
and is driven by the plugin’s own executor from the tick thread, never by the gateway’s
ROS executor.
This package carries the plugin skeleton, the central reliability gate that holds raises
until the graph has quiesced, and five detectors, qos_mismatch, orphan,
param_drift, lifecycle_expectation and node_death. Two silent-fault classes remain
undelivered, GRAPH_TF_STALE and GRAPH_LATENCY_BUDGET; they land in follow-up changes,
each against its own issue, and their fault codes are already reserved in the frozen
GRAPH_* namespace (see “Fault codes”).
Build
With the gateway built and sourced:
colcon build --packages-select ros2_medkit_graph_watchdog
colcon test --packages-select ros2_medkit_graph_watchdog && colcon test-result --verbose
Load into the gateway
ros2 run ros2_medkit_gateway gateway_node --ros-args \
-p plugins:="[graph_watchdog]" \
-p "plugins.graph_watchdog.path:=$(ros2 pkg prefix ros2_medkit_graph_watchdog)/lib/ros2_medkit_graph_watchdog/libros2_medkit_graph_watchdog.so"
Configuration (plugins.graph_watchdog.<key>)
| Key | Type | Default | Meaning |
|---|---|---|---|
tick_interval_ms |
int | 1000 |
Detector tick cadence. Accepted range 1..INT_MAX; the check runs on the wide integer, so a value past INT_MAX is rejected rather than truncated back into the band. |
warmup_cycles |
int | 5 |
An entity must be continuously present for this many ticks before it arms. A mid-run restart re-warms it. Enforced centrally, see “Reliability (bringup-quiesce)” below. Accepted range 0..INT_MAX, checked on the wide integer. |
prune_grace |
int | 60 |
Default injected into every detector’s own config before configure(); a per-detector detectors.<id>.prune_grace overrides it. Used by detectors that keep per-key bookkeeping. Accepted range 0..INT_MAX here, checked on the wide integer; each detector re-validates it against its own documented range before using it. |
detectors.<id>.mode |
string | bool | raise |
raise = push faults; advisory = observe, do not push; off = disabled. Bare off/no (YAML booleans) also disable; bare on/yes mean raise. |
detectors.<id>.<field> |
any | - | Per-detector thresholds, passed to that detector’s configure(). |
Nested delivery. Keys are written dotted in YAML or --ros-args, and the gateway
delivers them to the plugin as a nested object. A bare off is a YAML 1.1 boolean, so
mode: off arrives as false, not the string "off"; both forms disable the detector.
Unknown keys are reported. Anything under detectors.<id> that the detector does not
read produces one startup warning naming the key and listing the ones that exist, so a
typo like allow_list: for allowlist: cannot silently do nothing. A typo in the detector
id itself is warned about the same way, with the registered ids listed.
lifecycle_expectation keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. |
require_active |
string[] | [] |
Node names that must be in the active lifecycle state, each matched against a live node by its App::id, its full FQN (/ns/name), or the bare leaf of that FQN. A bare name matches that node in EVERY namespace (a fleet-wide “all controller_servers must be active”); use a full FQN to pin one robot’s. Both the bare and the FQN form match against the node’s stable FQN, so they survive the App::id renaming that a same-bare-name collision triggers; an entry written as an App::id also works, but can stop matching on a multi-robot graph for exactly that reason. Empty = the detector checks nothing and emits nothing. A require_active that is not a string array warns and is ignored; an empty-string entry warns and is skipped - never dropped silently, since the operator would go on believing the node is covered. |
grace |
int | 5 |
Consecutive not-active ticks a required node tolerates before being reported inactive - bringup time for a managed node to reach active (configure + activate) before the expectation is enforced. Counted per NODE, not per matching entry: a node named by both a bare-name and a full-FQN entry still advances its streak once per tick, so mixing the two documented forms cannot halve the grace that was configured. An ALREADY-CONFIRMED streak keeps its content while the node is ABSENT, so a node that leaves the graph after being confirmed stays confirmed; a streak that has not yet crossed grace on a node the presence detector could also report (it was armed at least once) is HELD rather than advanced while absent, so a node is never confirmed out of ticks gathered while nobody could observe it - only a PRESENT tick may still cross grace for that node. A node that was NEVER armed gets no such hold: node_death can never track it either, so absence keeps advancing its streak too, the same as a present tick would. Accepted range 0..300 - five minutes at the shipped 1 s cadence, and already an extravagant allowance for a managed node to reach active. The upper end is a real bound rather than a formality: grace decides how long a PRESENT node may go unreported, and how long a node returning from a departure still inactive takes to (re-)mature, so at the old maximum of INT_MAX - 1 that PRESENT-side silence lasted about 24 days at the shipped cadence with no warning. Whether it also bounds how long a node that leaves the graph BELOW grace sits unsettled depends on the same split: for an armed node that lasts for as long as the node stays gone, independent of grace (see “Bounded by evidence, not by age” below); for a never-armed one grace bounds that too, since absence matures it within grace + absence_grace + 1 ticks either way. The check runs on the wide integer, so a value that only fits after truncation is rejected rather than silently turned into a hair-trigger. Anything outside the range warns and keeps the default. |
prune_grace |
int | 60 |
Consecutive ticks an IDLE tracked node - both clocks at zero and no matured ownership, so nothing to lose - may stay ABSENT from the graph before its bookkeeping is reclaimed. A node carrying evidence is never reclaimed by age at all, however long it stays gone, so this key cannot erase a fault’s own state; the map is bounded instead by tracked_node_cap (see “Bounded by evidence, not by age”). Injected for every detector at plugin scope and overridable per detector; a value outside 0..3600 warns and this detector keeps its own default of 60. The range check runs on the wide integer, so an out-of-int-range value is rejected rather than truncated. The value is used as written - there is no grace + 1 clamp, because there is no longer anything for one to protect. |
tracked_node_cap |
int | 512 |
The most nodes this detector keeps bookkeeping for at once. It is what bounds the map, since evidence is never reclaimed by age (see “Bounded by evidence, not by age”): at the cap, idle entries are reclaimed first, then entries for DEPARTED nodes are collapsed into a count, and only if every tracked node is PRESENT and carrying evidence is a newly matched node refused - which withholds GRAPH_NODE_INACTIVE’s clear and is reported both in the log and on GET /x-medkit-watchdog. 512 is comfortably above every node in a realistic graph (a full Nav2 stack plus perception is roughly a hundred nodes; a ten-robot fleet sharing one domain a few hundred), so raising it is only needed where require_active legitimately matches more PRESENT nodes than that. Accepted range 1..16384 - 16384 is about 8 MB of bookkeeping at roughly 500 bytes per tracked node, and a deployment needing more distinct required-node identities alive at once has identity churn rather than a large fleet. Anything outside the range, including 0 (a cap of nothing would mean checking nothing), warns and keeps the default; the check runs on the wide integer, so an out-of-int-range value is rejected rather than truncated. |
node_death keys
Zero-config, unlike every detector above: there is no require_active-style list of nodes
to watch, because every armed App in the graph is a candidate.
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. |
miss_grace |
int | 2 |
Consecutive missed ticks a tracked node tolerates before being reported dead - a death is reported once misses EXCEED this, i.e. after miss_grace + 1 ticks. Accepted range 0..3600, and additionally floored to whatever wall-clock window the configured tick_interval_ms needs (see “The wall-clock floor” below); a value below that floor is silently raised to it, with a warning naming the ms window it actually spans. |
prune_grace |
int | 60 |
Plugin-injected default (overridable here); how long a key being DURABLY suppressed (see “Suppression” below) may sit unreported before its bookkeeping is reclaimed. Accepted range 0..3600. The value actually used is max(prune_grace, miss_grace + 1), a silent internal floor rather than a rejection: without it, a durably-suppressed key could be reclaimed the very tick it would first have become eligible to report, losing the report rather than merely suppressing it. |
allowlist |
string[] | [] |
Node identities never reported dead. A dead key matches if it - or a name the same node is also known by - is present verbatim: the key’s own fqn, its bare leaf, or (captured while the node was still present) its App::id. Has no effect unless named in suppress; naming it without a configured list is a no-op. |
suppress |
string[] | [] |
Suppression mechanisms to activate: "allowlist" and "lifecycle". Any other entry warns as unrecognized and is ignored; a non-string or non-array entry warns the same way. |
tracked_node_cap |
int | 512 |
The most DEPARTED (not-yet-confirmed-dead or already-collapsed) identities this detector keeps individual bookkeeping for at once - a present/armed node is never counted against it and never refused tracking. Accepted range 1..16384; 0 is refused rather than clamped, since a cap of nothing would mean tracking nothing. See “Bounded by evidence, not by age” below for what it actually bounds and why, unlike the sibling detector’s identical-looking cap, a newcomer here is never the thing refused. |
tick_interval_ms is not an own key of this detector, but this is the one detector that
reads the plugin-injected value directly, to compute miss_grace’s wall-clock floor below.
Liveness, and why membership in the snapshot is not enough. App::is_online is what
this detector tracks, never mere presence in the entity snapshot. In runtime-only discovery
the two happen to coincide - a dead node’s App leaves the snapshot entirely - but a manifest
keeps a bound App present with only is_online cleared once its ROS binding disappears, and
hybrid discovery inherits that same shape; counting snapshot membership alone would make a
manifest-declared node immortal. A managed lifecycle node that merely deactivates keeps
is_online: true - its process is still running, only its ROS 2 lifecycle state changed -
so a deactivation is never mistaken for a death either; that is lifecycle_expectation’s
concern, not this one’s (see “The boundary with lifecycle_expectation” below).
Tracking is keyed on the STABLE fqn (App::effective_fqn()), never App::id: an id is
recomputed every sweep and only gains a namespace prefix once a same-bare-name collision
currently exists anywhere in the graph, so a live node’s id can change out from under a key
built from it.
What is never tracked.
| Excluded | Why |
|---|---|
Peer-aggregated apps (app.source starting peer:) |
A peer app carries no ROS binding of its own, so effective_fqn() is empty for every one of them; tracking them would collapse an entire peer fleet onto one "" key, and one online peer app would mask every other peer app’s departure. |
_ros2cli_<pid> nodes |
Every ros2 CLI invocation (ros2 topic echo, ros2 param get, …) spins up a real, short-lived node under this prefix - rcl’s own hidden-node naming convention, matched structurally rather than guessed at. Letting these through would accumulate one permanently “dead” entry per CLI invocation for the life of the gateway. |
| An App that never comes online | A key is admitted to tracking only once the reliability gate has said this detector OWNS its departure - armed at least once, and on one of the two GROUNDS ownership can rest on (a lifecycle state read as active, or no lifecycle to read at all; the third case, a state nobody could read, is admitted PROVISIONALLY and is described in the row below). A manifest App whose binding never starts is never armed and so can never be falsely called dead; a node still inside its warmup_cycles window gets the identical protection. Once tracked, though, a node’s continued life-or-death judgement rests on PRESENCE alone, not on staying armed - a tracked node that later goes lifecycle-inactive is not thereby mistaken for dead. |
| A managed App whose lifecycle state has not been read YET | Not tracked, and only for as long as that is still true. The gate’s own raise permission is answered PERMISSIVELY for a managed node whose GetState has not answered (LifecycleWatcher::node_ok()) - without that, qos_mismatch, orphan and param_drift would all fall silent about a node whose lifecycle service is broken. Permission is not knowledge, though, so tracking is withheld while a measurement may still arrive. LifecycleWatcher charges a GetState re-seed budget per node, and only for a read that actually ran, so “still asking” is a fact rather than a guess. Once that budget is spent the node is tracked here after all: nothing will ASK again, and lifecycle_expectation looks at nothing unless an operator named the node in require_active (empty by default), so withholding it any longer would mean the death is reported by nobody. That admission is PROVISIONAL, not final. The ~/transition_event subscription outlives the seed budget, so a label can still arrive without being asked for, and one that reads non-active says the node was the lifecycle detector’s all along - so the presence detector hands it back on the spot, while it is still alive. Waiting for the budget alone would only have delayed the wrong report, not prevented it. |
File truncated at 100 lines see the full file
CHANGELOG
Changelog for package ros2_medkit_graph_watchdog
0.7.0 (2026-08-27)
- Initial release of the package: a gateway plugin that raises faults
for silent failures in the ROS 2 graph - problems that leave every
node alive and every topic present, so nothing in the stack reports
them today. The package carries the plugin skeleton, a central
reliability gate that holds raises until the graph has quiesced, and
a detector registry. Detectors are configured under
plugins.graph_watchdog.<key>, each with araise/advisory/offmode, and an unknown key underdetectors.<id>is reported rather than silently ignored, so a typo cannot quietly disable a check (#571) -
qos_mismatchdetector: raisesGRAPH_QOS_MISMATCHfor a publisher and subscriber whose QoS profiles cannot match, which leaves the connection silently unestablished (#571) -
orphandetector: raisesGRAPH_ORPHANfor a one-sided topic - a publisher with no subscriber, or the reverse - but only when a complementary near-miss counterpart exists: same message type, a namespace and leaf within the configured edit distance, and the opposite side. A lone unsubscribed topic raises nothing. The pair is the signature of a remap or a topic-name typo, which is what the detector is for (#578) -
param_driftdetector: raisesGRAPH_PARAM_DRIFTwhen a node's live parameter value diverges from the declared expectation (#580) -
lifecycle_expectationdetector: raisesGRAPH_NODE_INACTIVEfor a node named inrequire_activethat is not in theactivelifecycle state once itsgracewindow has passed. It raises two further codes for the cases where the state could not be established at all:GRAPH_NODE_UNREADABLEwhen the lifecycle service does not answer, andGRAPH_NODE_NOT_MANAGEDwhen the named node has no lifecycle to read. These are distinct codes on the wire, so a consumer filtering onGRAPH_NODE_INACTIVEalone sees neither unmeasured case. A node is matched by itsApp::id, its full FQN, or the bare leaf of that FQN, and the grace streak is counted per node, so naming one node in two documented forms cannot halve the grace that was configured (#587) -
node_deathdetector: raisesGRAPH_NODE_DISAPPEAREDfor a node that leaves the graph when nothing else is left to report it, with a suppression framework that decides ownership of a departure from knowledge rather than from silence. A node whose lifecycle state was never readable is admitted only provisionally, and is handed back the moment a label arrives saying the departure belonged to the lifecycle detector instead. Zero-config: there is no list of nodes to maintain (#625, #624) - Two silent-fault classes are not delivered in this release.
GRAPH_TF_STALEandGRAPH_LATENCY_BUDGEThave their fault codes reserved in the frozenGRAPH_*namespace and land in later changes, each against its own issue - The plugin keeps its ROS entities off the gateway executor, so an entity created and destroyed while the gateway runs cannot have its destructor run on an executor thread concurrently with a create on the same node
- Contributors: \@bburda
Package Dependencies
System Dependencies
Dependant Packages
No known dependants.
Launch files
No launch files found
Messages
No message files found.
Services
No service files found
Plugins
No plugins found.
Recent questions tagged ros2_medkit_graph_watchdog at Robotics Stack Exchange
No version for distro kinetic showing humble. Known supported distros are highlighted in the buttons above.
Package Summary
| Version | 0.7.0 |
| License | Apache-2.0 |
| Build type | AMENT_CMAKE |
| Use | RECOMMENDED |
Repository Summary
| Checkout URI | https://github.com/selfpatch/ros2_medkit.git |
| VCS Type | git |
| VCS Version | main |
| Last Updated | 2026-09-06 |
| Dev Status | DEVELOPED |
| Released | RELEASED |
| Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
Gateway plugin that detects silent faults in the ROS 2 graph: QoS mismatches, orphaned topics left by a name typo, dead nodes, parameter drift and unmet lifecycle expectations.
Maintainers
- bburda
Authors
No additional authors.
ros2_medkit_graph_watchdog
Gateway plugin that detects silent faults in the ROS 2 graph: failures where every node is up, nothing logs an error, and the robot is still broken.
Detectors read the graph and raise faults through a ReportFault service client on the
gateway node; the faults surface via FaultManager on the gateway /faults API. That
client, like the lifecycle subscriptions below, lives in the plugin’s own callback group
and is driven by the plugin’s own executor from the tick thread, never by the gateway’s
ROS executor.
This package carries the plugin skeleton, the central reliability gate that holds raises
until the graph has quiesced, and five detectors, qos_mismatch, orphan,
param_drift, lifecycle_expectation and node_death. Two silent-fault classes remain
undelivered, GRAPH_TF_STALE and GRAPH_LATENCY_BUDGET; they land in follow-up changes,
each against its own issue, and their fault codes are already reserved in the frozen
GRAPH_* namespace (see “Fault codes”).
Build
With the gateway built and sourced:
colcon build --packages-select ros2_medkit_graph_watchdog
colcon test --packages-select ros2_medkit_graph_watchdog && colcon test-result --verbose
Load into the gateway
ros2 run ros2_medkit_gateway gateway_node --ros-args \
-p plugins:="[graph_watchdog]" \
-p "plugins.graph_watchdog.path:=$(ros2 pkg prefix ros2_medkit_graph_watchdog)/lib/ros2_medkit_graph_watchdog/libros2_medkit_graph_watchdog.so"
Configuration (plugins.graph_watchdog.<key>)
| Key | Type | Default | Meaning |
|---|---|---|---|
tick_interval_ms |
int | 1000 |
Detector tick cadence. Accepted range 1..INT_MAX; the check runs on the wide integer, so a value past INT_MAX is rejected rather than truncated back into the band. |
warmup_cycles |
int | 5 |
An entity must be continuously present for this many ticks before it arms. A mid-run restart re-warms it. Enforced centrally, see “Reliability (bringup-quiesce)” below. Accepted range 0..INT_MAX, checked on the wide integer. |
prune_grace |
int | 60 |
Default injected into every detector’s own config before configure(); a per-detector detectors.<id>.prune_grace overrides it. Used by detectors that keep per-key bookkeeping. Accepted range 0..INT_MAX here, checked on the wide integer; each detector re-validates it against its own documented range before using it. |
detectors.<id>.mode |
string | bool | raise |
raise = push faults; advisory = observe, do not push; off = disabled. Bare off/no (YAML booleans) also disable; bare on/yes mean raise. |
detectors.<id>.<field> |
any | - | Per-detector thresholds, passed to that detector’s configure(). |
Nested delivery. Keys are written dotted in YAML or --ros-args, and the gateway
delivers them to the plugin as a nested object. A bare off is a YAML 1.1 boolean, so
mode: off arrives as false, not the string "off"; both forms disable the detector.
Unknown keys are reported. Anything under detectors.<id> that the detector does not
read produces one startup warning naming the key and listing the ones that exist, so a
typo like allow_list: for allowlist: cannot silently do nothing. A typo in the detector
id itself is warned about the same way, with the registered ids listed.
lifecycle_expectation keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. |
require_active |
string[] | [] |
Node names that must be in the active lifecycle state, each matched against a live node by its App::id, its full FQN (/ns/name), or the bare leaf of that FQN. A bare name matches that node in EVERY namespace (a fleet-wide “all controller_servers must be active”); use a full FQN to pin one robot’s. Both the bare and the FQN form match against the node’s stable FQN, so they survive the App::id renaming that a same-bare-name collision triggers; an entry written as an App::id also works, but can stop matching on a multi-robot graph for exactly that reason. Empty = the detector checks nothing and emits nothing. A require_active that is not a string array warns and is ignored; an empty-string entry warns and is skipped - never dropped silently, since the operator would go on believing the node is covered. |
grace |
int | 5 |
Consecutive not-active ticks a required node tolerates before being reported inactive - bringup time for a managed node to reach active (configure + activate) before the expectation is enforced. Counted per NODE, not per matching entry: a node named by both a bare-name and a full-FQN entry still advances its streak once per tick, so mixing the two documented forms cannot halve the grace that was configured. An ALREADY-CONFIRMED streak keeps its content while the node is ABSENT, so a node that leaves the graph after being confirmed stays confirmed; a streak that has not yet crossed grace on a node the presence detector could also report (it was armed at least once) is HELD rather than advanced while absent, so a node is never confirmed out of ticks gathered while nobody could observe it - only a PRESENT tick may still cross grace for that node. A node that was NEVER armed gets no such hold: node_death can never track it either, so absence keeps advancing its streak too, the same as a present tick would. Accepted range 0..300 - five minutes at the shipped 1 s cadence, and already an extravagant allowance for a managed node to reach active. The upper end is a real bound rather than a formality: grace decides how long a PRESENT node may go unreported, and how long a node returning from a departure still inactive takes to (re-)mature, so at the old maximum of INT_MAX - 1 that PRESENT-side silence lasted about 24 days at the shipped cadence with no warning. Whether it also bounds how long a node that leaves the graph BELOW grace sits unsettled depends on the same split: for an armed node that lasts for as long as the node stays gone, independent of grace (see “Bounded by evidence, not by age” below); for a never-armed one grace bounds that too, since absence matures it within grace + absence_grace + 1 ticks either way. The check runs on the wide integer, so a value that only fits after truncation is rejected rather than silently turned into a hair-trigger. Anything outside the range warns and keeps the default. |
prune_grace |
int | 60 |
Consecutive ticks an IDLE tracked node - both clocks at zero and no matured ownership, so nothing to lose - may stay ABSENT from the graph before its bookkeeping is reclaimed. A node carrying evidence is never reclaimed by age at all, however long it stays gone, so this key cannot erase a fault’s own state; the map is bounded instead by tracked_node_cap (see “Bounded by evidence, not by age”). Injected for every detector at plugin scope and overridable per detector; a value outside 0..3600 warns and this detector keeps its own default of 60. The range check runs on the wide integer, so an out-of-int-range value is rejected rather than truncated. The value is used as written - there is no grace + 1 clamp, because there is no longer anything for one to protect. |
tracked_node_cap |
int | 512 |
The most nodes this detector keeps bookkeeping for at once. It is what bounds the map, since evidence is never reclaimed by age (see “Bounded by evidence, not by age”): at the cap, idle entries are reclaimed first, then entries for DEPARTED nodes are collapsed into a count, and only if every tracked node is PRESENT and carrying evidence is a newly matched node refused - which withholds GRAPH_NODE_INACTIVE’s clear and is reported both in the log and on GET /x-medkit-watchdog. 512 is comfortably above every node in a realistic graph (a full Nav2 stack plus perception is roughly a hundred nodes; a ten-robot fleet sharing one domain a few hundred), so raising it is only needed where require_active legitimately matches more PRESENT nodes than that. Accepted range 1..16384 - 16384 is about 8 MB of bookkeeping at roughly 500 bytes per tracked node, and a deployment needing more distinct required-node identities alive at once has identity churn rather than a large fleet. Anything outside the range, including 0 (a cap of nothing would mean checking nothing), warns and keeps the default; the check runs on the wide integer, so an out-of-int-range value is rejected rather than truncated. |
node_death keys
Zero-config, unlike every detector above: there is no require_active-style list of nodes
to watch, because every armed App in the graph is a candidate.
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. |
miss_grace |
int | 2 |
Consecutive missed ticks a tracked node tolerates before being reported dead - a death is reported once misses EXCEED this, i.e. after miss_grace + 1 ticks. Accepted range 0..3600, and additionally floored to whatever wall-clock window the configured tick_interval_ms needs (see “The wall-clock floor” below); a value below that floor is silently raised to it, with a warning naming the ms window it actually spans. |
prune_grace |
int | 60 |
Plugin-injected default (overridable here); how long a key being DURABLY suppressed (see “Suppression” below) may sit unreported before its bookkeeping is reclaimed. Accepted range 0..3600. The value actually used is max(prune_grace, miss_grace + 1), a silent internal floor rather than a rejection: without it, a durably-suppressed key could be reclaimed the very tick it would first have become eligible to report, losing the report rather than merely suppressing it. |
allowlist |
string[] | [] |
Node identities never reported dead. A dead key matches if it - or a name the same node is also known by - is present verbatim: the key’s own fqn, its bare leaf, or (captured while the node was still present) its App::id. Has no effect unless named in suppress; naming it without a configured list is a no-op. |
suppress |
string[] | [] |
Suppression mechanisms to activate: "allowlist" and "lifecycle". Any other entry warns as unrecognized and is ignored; a non-string or non-array entry warns the same way. |
tracked_node_cap |
int | 512 |
The most DEPARTED (not-yet-confirmed-dead or already-collapsed) identities this detector keeps individual bookkeeping for at once - a present/armed node is never counted against it and never refused tracking. Accepted range 1..16384; 0 is refused rather than clamped, since a cap of nothing would mean tracking nothing. See “Bounded by evidence, not by age” below for what it actually bounds and why, unlike the sibling detector’s identical-looking cap, a newcomer here is never the thing refused. |
tick_interval_ms is not an own key of this detector, but this is the one detector that
reads the plugin-injected value directly, to compute miss_grace’s wall-clock floor below.
Liveness, and why membership in the snapshot is not enough. App::is_online is what
this detector tracks, never mere presence in the entity snapshot. In runtime-only discovery
the two happen to coincide - a dead node’s App leaves the snapshot entirely - but a manifest
keeps a bound App present with only is_online cleared once its ROS binding disappears, and
hybrid discovery inherits that same shape; counting snapshot membership alone would make a
manifest-declared node immortal. A managed lifecycle node that merely deactivates keeps
is_online: true - its process is still running, only its ROS 2 lifecycle state changed -
so a deactivation is never mistaken for a death either; that is lifecycle_expectation’s
concern, not this one’s (see “The boundary with lifecycle_expectation” below).
Tracking is keyed on the STABLE fqn (App::effective_fqn()), never App::id: an id is
recomputed every sweep and only gains a namespace prefix once a same-bare-name collision
currently exists anywhere in the graph, so a live node’s id can change out from under a key
built from it.
What is never tracked.
| Excluded | Why |
|---|---|
Peer-aggregated apps (app.source starting peer:) |
A peer app carries no ROS binding of its own, so effective_fqn() is empty for every one of them; tracking them would collapse an entire peer fleet onto one "" key, and one online peer app would mask every other peer app’s departure. |
_ros2cli_<pid> nodes |
Every ros2 CLI invocation (ros2 topic echo, ros2 param get, …) spins up a real, short-lived node under this prefix - rcl’s own hidden-node naming convention, matched structurally rather than guessed at. Letting these through would accumulate one permanently “dead” entry per CLI invocation for the life of the gateway. |
| An App that never comes online | A key is admitted to tracking only once the reliability gate has said this detector OWNS its departure - armed at least once, and on one of the two GROUNDS ownership can rest on (a lifecycle state read as active, or no lifecycle to read at all; the third case, a state nobody could read, is admitted PROVISIONALLY and is described in the row below). A manifest App whose binding never starts is never armed and so can never be falsely called dead; a node still inside its warmup_cycles window gets the identical protection. Once tracked, though, a node’s continued life-or-death judgement rests on PRESENCE alone, not on staying armed - a tracked node that later goes lifecycle-inactive is not thereby mistaken for dead. |
| A managed App whose lifecycle state has not been read YET | Not tracked, and only for as long as that is still true. The gate’s own raise permission is answered PERMISSIVELY for a managed node whose GetState has not answered (LifecycleWatcher::node_ok()) - without that, qos_mismatch, orphan and param_drift would all fall silent about a node whose lifecycle service is broken. Permission is not knowledge, though, so tracking is withheld while a measurement may still arrive. LifecycleWatcher charges a GetState re-seed budget per node, and only for a read that actually ran, so “still asking” is a fact rather than a guess. Once that budget is spent the node is tracked here after all: nothing will ASK again, and lifecycle_expectation looks at nothing unless an operator named the node in require_active (empty by default), so withholding it any longer would mean the death is reported by nobody. That admission is PROVISIONAL, not final. The ~/transition_event subscription outlives the seed budget, so a label can still arrive without being asked for, and one that reads non-active says the node was the lifecycle detector’s all along - so the presence detector hands it back on the spot, while it is still alive. Waiting for the budget alone would only have delayed the wrong report, not prevented it. |
File truncated at 100 lines see the full file
CHANGELOG
Changelog for package ros2_medkit_graph_watchdog
0.7.0 (2026-08-27)
- Initial release of the package: a gateway plugin that raises faults
for silent failures in the ROS 2 graph - problems that leave every
node alive and every topic present, so nothing in the stack reports
them today. The package carries the plugin skeleton, a central
reliability gate that holds raises until the graph has quiesced, and
a detector registry. Detectors are configured under
plugins.graph_watchdog.<key>, each with araise/advisory/offmode, and an unknown key underdetectors.<id>is reported rather than silently ignored, so a typo cannot quietly disable a check (#571) -
qos_mismatchdetector: raisesGRAPH_QOS_MISMATCHfor a publisher and subscriber whose QoS profiles cannot match, which leaves the connection silently unestablished (#571) -
orphandetector: raisesGRAPH_ORPHANfor a one-sided topic - a publisher with no subscriber, or the reverse - but only when a complementary near-miss counterpart exists: same message type, a namespace and leaf within the configured edit distance, and the opposite side. A lone unsubscribed topic raises nothing. The pair is the signature of a remap or a topic-name typo, which is what the detector is for (#578) -
param_driftdetector: raisesGRAPH_PARAM_DRIFTwhen a node's live parameter value diverges from the declared expectation (#580) -
lifecycle_expectationdetector: raisesGRAPH_NODE_INACTIVEfor a node named inrequire_activethat is not in theactivelifecycle state once itsgracewindow has passed. It raises two further codes for the cases where the state could not be established at all:GRAPH_NODE_UNREADABLEwhen the lifecycle service does not answer, andGRAPH_NODE_NOT_MANAGEDwhen the named node has no lifecycle to read. These are distinct codes on the wire, so a consumer filtering onGRAPH_NODE_INACTIVEalone sees neither unmeasured case. A node is matched by itsApp::id, its full FQN, or the bare leaf of that FQN, and the grace streak is counted per node, so naming one node in two documented forms cannot halve the grace that was configured (#587) -
node_deathdetector: raisesGRAPH_NODE_DISAPPEAREDfor a node that leaves the graph when nothing else is left to report it, with a suppression framework that decides ownership of a departure from knowledge rather than from silence. A node whose lifecycle state was never readable is admitted only provisionally, and is handed back the moment a label arrives saying the departure belonged to the lifecycle detector instead. Zero-config: there is no list of nodes to maintain (#625, #624) - Two silent-fault classes are not delivered in this release.
GRAPH_TF_STALEandGRAPH_LATENCY_BUDGEThave their fault codes reserved in the frozenGRAPH_*namespace and land in later changes, each against its own issue - The plugin keeps its ROS entities off the gateway executor, so an entity created and destroyed while the gateway runs cannot have its destructor run on an executor thread concurrently with a create on the same node
- Contributors: \@bburda
Package Dependencies
System Dependencies
Dependant Packages
No known dependants.
Launch files
No launch files found
Messages
No message files found.
Services
No service files found
Plugins
No plugins found.
Recent questions tagged ros2_medkit_graph_watchdog at Robotics Stack Exchange
No version for distro melodic showing humble. Known supported distros are highlighted in the buttons above.
Package Summary
| Version | 0.7.0 |
| License | Apache-2.0 |
| Build type | AMENT_CMAKE |
| Use | RECOMMENDED |
Repository Summary
| Checkout URI | https://github.com/selfpatch/ros2_medkit.git |
| VCS Type | git |
| VCS Version | main |
| Last Updated | 2026-09-06 |
| Dev Status | DEVELOPED |
| Released | RELEASED |
| Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
Gateway plugin that detects silent faults in the ROS 2 graph: QoS mismatches, orphaned topics left by a name typo, dead nodes, parameter drift and unmet lifecycle expectations.
Maintainers
- bburda
Authors
No additional authors.
ros2_medkit_graph_watchdog
Gateway plugin that detects silent faults in the ROS 2 graph: failures where every node is up, nothing logs an error, and the robot is still broken.
Detectors read the graph and raise faults through a ReportFault service client on the
gateway node; the faults surface via FaultManager on the gateway /faults API. That
client, like the lifecycle subscriptions below, lives in the plugin’s own callback group
and is driven by the plugin’s own executor from the tick thread, never by the gateway’s
ROS executor.
This package carries the plugin skeleton, the central reliability gate that holds raises
until the graph has quiesced, and five detectors, qos_mismatch, orphan,
param_drift, lifecycle_expectation and node_death. Two silent-fault classes remain
undelivered, GRAPH_TF_STALE and GRAPH_LATENCY_BUDGET; they land in follow-up changes,
each against its own issue, and their fault codes are already reserved in the frozen
GRAPH_* namespace (see “Fault codes”).
Build
With the gateway built and sourced:
colcon build --packages-select ros2_medkit_graph_watchdog
colcon test --packages-select ros2_medkit_graph_watchdog && colcon test-result --verbose
Load into the gateway
ros2 run ros2_medkit_gateway gateway_node --ros-args \
-p plugins:="[graph_watchdog]" \
-p "plugins.graph_watchdog.path:=$(ros2 pkg prefix ros2_medkit_graph_watchdog)/lib/ros2_medkit_graph_watchdog/libros2_medkit_graph_watchdog.so"
Configuration (plugins.graph_watchdog.<key>)
| Key | Type | Default | Meaning |
|---|---|---|---|
tick_interval_ms |
int | 1000 |
Detector tick cadence. Accepted range 1..INT_MAX; the check runs on the wide integer, so a value past INT_MAX is rejected rather than truncated back into the band. |
warmup_cycles |
int | 5 |
An entity must be continuously present for this many ticks before it arms. A mid-run restart re-warms it. Enforced centrally, see “Reliability (bringup-quiesce)” below. Accepted range 0..INT_MAX, checked on the wide integer. |
prune_grace |
int | 60 |
Default injected into every detector’s own config before configure(); a per-detector detectors.<id>.prune_grace overrides it. Used by detectors that keep per-key bookkeeping. Accepted range 0..INT_MAX here, checked on the wide integer; each detector re-validates it against its own documented range before using it. |
detectors.<id>.mode |
string | bool | raise |
raise = push faults; advisory = observe, do not push; off = disabled. Bare off/no (YAML booleans) also disable; bare on/yes mean raise. |
detectors.<id>.<field> |
any | - | Per-detector thresholds, passed to that detector’s configure(). |
Nested delivery. Keys are written dotted in YAML or --ros-args, and the gateway
delivers them to the plugin as a nested object. A bare off is a YAML 1.1 boolean, so
mode: off arrives as false, not the string "off"; both forms disable the detector.
Unknown keys are reported. Anything under detectors.<id> that the detector does not
read produces one startup warning naming the key and listing the ones that exist, so a
typo like allow_list: for allowlist: cannot silently do nothing. A typo in the detector
id itself is warned about the same way, with the registered ids listed.
lifecycle_expectation keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. |
require_active |
string[] | [] |
Node names that must be in the active lifecycle state, each matched against a live node by its App::id, its full FQN (/ns/name), or the bare leaf of that FQN. A bare name matches that node in EVERY namespace (a fleet-wide “all controller_servers must be active”); use a full FQN to pin one robot’s. Both the bare and the FQN form match against the node’s stable FQN, so they survive the App::id renaming that a same-bare-name collision triggers; an entry written as an App::id also works, but can stop matching on a multi-robot graph for exactly that reason. Empty = the detector checks nothing and emits nothing. A require_active that is not a string array warns and is ignored; an empty-string entry warns and is skipped - never dropped silently, since the operator would go on believing the node is covered. |
grace |
int | 5 |
Consecutive not-active ticks a required node tolerates before being reported inactive - bringup time for a managed node to reach active (configure + activate) before the expectation is enforced. Counted per NODE, not per matching entry: a node named by both a bare-name and a full-FQN entry still advances its streak once per tick, so mixing the two documented forms cannot halve the grace that was configured. An ALREADY-CONFIRMED streak keeps its content while the node is ABSENT, so a node that leaves the graph after being confirmed stays confirmed; a streak that has not yet crossed grace on a node the presence detector could also report (it was armed at least once) is HELD rather than advanced while absent, so a node is never confirmed out of ticks gathered while nobody could observe it - only a PRESENT tick may still cross grace for that node. A node that was NEVER armed gets no such hold: node_death can never track it either, so absence keeps advancing its streak too, the same as a present tick would. Accepted range 0..300 - five minutes at the shipped 1 s cadence, and already an extravagant allowance for a managed node to reach active. The upper end is a real bound rather than a formality: grace decides how long a PRESENT node may go unreported, and how long a node returning from a departure still inactive takes to (re-)mature, so at the old maximum of INT_MAX - 1 that PRESENT-side silence lasted about 24 days at the shipped cadence with no warning. Whether it also bounds how long a node that leaves the graph BELOW grace sits unsettled depends on the same split: for an armed node that lasts for as long as the node stays gone, independent of grace (see “Bounded by evidence, not by age” below); for a never-armed one grace bounds that too, since absence matures it within grace + absence_grace + 1 ticks either way. The check runs on the wide integer, so a value that only fits after truncation is rejected rather than silently turned into a hair-trigger. Anything outside the range warns and keeps the default. |
prune_grace |
int | 60 |
Consecutive ticks an IDLE tracked node - both clocks at zero and no matured ownership, so nothing to lose - may stay ABSENT from the graph before its bookkeeping is reclaimed. A node carrying evidence is never reclaimed by age at all, however long it stays gone, so this key cannot erase a fault’s own state; the map is bounded instead by tracked_node_cap (see “Bounded by evidence, not by age”). Injected for every detector at plugin scope and overridable per detector; a value outside 0..3600 warns and this detector keeps its own default of 60. The range check runs on the wide integer, so an out-of-int-range value is rejected rather than truncated. The value is used as written - there is no grace + 1 clamp, because there is no longer anything for one to protect. |
tracked_node_cap |
int | 512 |
The most nodes this detector keeps bookkeeping for at once. It is what bounds the map, since evidence is never reclaimed by age (see “Bounded by evidence, not by age”): at the cap, idle entries are reclaimed first, then entries for DEPARTED nodes are collapsed into a count, and only if every tracked node is PRESENT and carrying evidence is a newly matched node refused - which withholds GRAPH_NODE_INACTIVE’s clear and is reported both in the log and on GET /x-medkit-watchdog. 512 is comfortably above every node in a realistic graph (a full Nav2 stack plus perception is roughly a hundred nodes; a ten-robot fleet sharing one domain a few hundred), so raising it is only needed where require_active legitimately matches more PRESENT nodes than that. Accepted range 1..16384 - 16384 is about 8 MB of bookkeeping at roughly 500 bytes per tracked node, and a deployment needing more distinct required-node identities alive at once has identity churn rather than a large fleet. Anything outside the range, including 0 (a cap of nothing would mean checking nothing), warns and keeps the default; the check runs on the wide integer, so an out-of-int-range value is rejected rather than truncated. |
node_death keys
Zero-config, unlike every detector above: there is no require_active-style list of nodes
to watch, because every armed App in the graph is a candidate.
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. |
miss_grace |
int | 2 |
Consecutive missed ticks a tracked node tolerates before being reported dead - a death is reported once misses EXCEED this, i.e. after miss_grace + 1 ticks. Accepted range 0..3600, and additionally floored to whatever wall-clock window the configured tick_interval_ms needs (see “The wall-clock floor” below); a value below that floor is silently raised to it, with a warning naming the ms window it actually spans. |
prune_grace |
int | 60 |
Plugin-injected default (overridable here); how long a key being DURABLY suppressed (see “Suppression” below) may sit unreported before its bookkeeping is reclaimed. Accepted range 0..3600. The value actually used is max(prune_grace, miss_grace + 1), a silent internal floor rather than a rejection: without it, a durably-suppressed key could be reclaimed the very tick it would first have become eligible to report, losing the report rather than merely suppressing it. |
allowlist |
string[] | [] |
Node identities never reported dead. A dead key matches if it - or a name the same node is also known by - is present verbatim: the key’s own fqn, its bare leaf, or (captured while the node was still present) its App::id. Has no effect unless named in suppress; naming it without a configured list is a no-op. |
suppress |
string[] | [] |
Suppression mechanisms to activate: "allowlist" and "lifecycle". Any other entry warns as unrecognized and is ignored; a non-string or non-array entry warns the same way. |
tracked_node_cap |
int | 512 |
The most DEPARTED (not-yet-confirmed-dead or already-collapsed) identities this detector keeps individual bookkeeping for at once - a present/armed node is never counted against it and never refused tracking. Accepted range 1..16384; 0 is refused rather than clamped, since a cap of nothing would mean tracking nothing. See “Bounded by evidence, not by age” below for what it actually bounds and why, unlike the sibling detector’s identical-looking cap, a newcomer here is never the thing refused. |
tick_interval_ms is not an own key of this detector, but this is the one detector that
reads the plugin-injected value directly, to compute miss_grace’s wall-clock floor below.
Liveness, and why membership in the snapshot is not enough. App::is_online is what
this detector tracks, never mere presence in the entity snapshot. In runtime-only discovery
the two happen to coincide - a dead node’s App leaves the snapshot entirely - but a manifest
keeps a bound App present with only is_online cleared once its ROS binding disappears, and
hybrid discovery inherits that same shape; counting snapshot membership alone would make a
manifest-declared node immortal. A managed lifecycle node that merely deactivates keeps
is_online: true - its process is still running, only its ROS 2 lifecycle state changed -
so a deactivation is never mistaken for a death either; that is lifecycle_expectation’s
concern, not this one’s (see “The boundary with lifecycle_expectation” below).
Tracking is keyed on the STABLE fqn (App::effective_fqn()), never App::id: an id is
recomputed every sweep and only gains a namespace prefix once a same-bare-name collision
currently exists anywhere in the graph, so a live node’s id can change out from under a key
built from it.
What is never tracked.
| Excluded | Why |
|---|---|
Peer-aggregated apps (app.source starting peer:) |
A peer app carries no ROS binding of its own, so effective_fqn() is empty for every one of them; tracking them would collapse an entire peer fleet onto one "" key, and one online peer app would mask every other peer app’s departure. |
_ros2cli_<pid> nodes |
Every ros2 CLI invocation (ros2 topic echo, ros2 param get, …) spins up a real, short-lived node under this prefix - rcl’s own hidden-node naming convention, matched structurally rather than guessed at. Letting these through would accumulate one permanently “dead” entry per CLI invocation for the life of the gateway. |
| An App that never comes online | A key is admitted to tracking only once the reliability gate has said this detector OWNS its departure - armed at least once, and on one of the two GROUNDS ownership can rest on (a lifecycle state read as active, or no lifecycle to read at all; the third case, a state nobody could read, is admitted PROVISIONALLY and is described in the row below). A manifest App whose binding never starts is never armed and so can never be falsely called dead; a node still inside its warmup_cycles window gets the identical protection. Once tracked, though, a node’s continued life-or-death judgement rests on PRESENCE alone, not on staying armed - a tracked node that later goes lifecycle-inactive is not thereby mistaken for dead. |
| A managed App whose lifecycle state has not been read YET | Not tracked, and only for as long as that is still true. The gate’s own raise permission is answered PERMISSIVELY for a managed node whose GetState has not answered (LifecycleWatcher::node_ok()) - without that, qos_mismatch, orphan and param_drift would all fall silent about a node whose lifecycle service is broken. Permission is not knowledge, though, so tracking is withheld while a measurement may still arrive. LifecycleWatcher charges a GetState re-seed budget per node, and only for a read that actually ran, so “still asking” is a fact rather than a guess. Once that budget is spent the node is tracked here after all: nothing will ASK again, and lifecycle_expectation looks at nothing unless an operator named the node in require_active (empty by default), so withholding it any longer would mean the death is reported by nobody. That admission is PROVISIONAL, not final. The ~/transition_event subscription outlives the seed budget, so a label can still arrive without being asked for, and one that reads non-active says the node was the lifecycle detector’s all along - so the presence detector hands it back on the spot, while it is still alive. Waiting for the budget alone would only have delayed the wrong report, not prevented it. |
File truncated at 100 lines see the full file
CHANGELOG
Changelog for package ros2_medkit_graph_watchdog
0.7.0 (2026-08-27)
- Initial release of the package: a gateway plugin that raises faults
for silent failures in the ROS 2 graph - problems that leave every
node alive and every topic present, so nothing in the stack reports
them today. The package carries the plugin skeleton, a central
reliability gate that holds raises until the graph has quiesced, and
a detector registry. Detectors are configured under
plugins.graph_watchdog.<key>, each with araise/advisory/offmode, and an unknown key underdetectors.<id>is reported rather than silently ignored, so a typo cannot quietly disable a check (#571) -
qos_mismatchdetector: raisesGRAPH_QOS_MISMATCHfor a publisher and subscriber whose QoS profiles cannot match, which leaves the connection silently unestablished (#571) -
orphandetector: raisesGRAPH_ORPHANfor a one-sided topic - a publisher with no subscriber, or the reverse - but only when a complementary near-miss counterpart exists: same message type, a namespace and leaf within the configured edit distance, and the opposite side. A lone unsubscribed topic raises nothing. The pair is the signature of a remap or a topic-name typo, which is what the detector is for (#578) -
param_driftdetector: raisesGRAPH_PARAM_DRIFTwhen a node's live parameter value diverges from the declared expectation (#580) -
lifecycle_expectationdetector: raisesGRAPH_NODE_INACTIVEfor a node named inrequire_activethat is not in theactivelifecycle state once itsgracewindow has passed. It raises two further codes for the cases where the state could not be established at all:GRAPH_NODE_UNREADABLEwhen the lifecycle service does not answer, andGRAPH_NODE_NOT_MANAGEDwhen the named node has no lifecycle to read. These are distinct codes on the wire, so a consumer filtering onGRAPH_NODE_INACTIVEalone sees neither unmeasured case. A node is matched by itsApp::id, its full FQN, or the bare leaf of that FQN, and the grace streak is counted per node, so naming one node in two documented forms cannot halve the grace that was configured (#587) -
node_deathdetector: raisesGRAPH_NODE_DISAPPEAREDfor a node that leaves the graph when nothing else is left to report it, with a suppression framework that decides ownership of a departure from knowledge rather than from silence. A node whose lifecycle state was never readable is admitted only provisionally, and is handed back the moment a label arrives saying the departure belonged to the lifecycle detector instead. Zero-config: there is no list of nodes to maintain (#625, #624) - Two silent-fault classes are not delivered in this release.
GRAPH_TF_STALEandGRAPH_LATENCY_BUDGEThave their fault codes reserved in the frozenGRAPH_*namespace and land in later changes, each against its own issue - The plugin keeps its ROS entities off the gateway executor, so an entity created and destroyed while the gateway runs cannot have its destructor run on an executor thread concurrently with a create on the same node
- Contributors: \@bburda
Package Dependencies
System Dependencies
Dependant Packages
No known dependants.
Launch files
No launch files found
Messages
No message files found.
Services
No service files found
Plugins
No plugins found.
Recent questions tagged ros2_medkit_graph_watchdog at Robotics Stack Exchange
No version for distro noetic showing humble. Known supported distros are highlighted in the buttons above.
Package Summary
| Version | 0.7.0 |
| License | Apache-2.0 |
| Build type | AMENT_CMAKE |
| Use | RECOMMENDED |
Repository Summary
| Checkout URI | https://github.com/selfpatch/ros2_medkit.git |
| VCS Type | git |
| VCS Version | main |
| Last Updated | 2026-09-06 |
| Dev Status | DEVELOPED |
| Released | RELEASED |
| Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
Gateway plugin that detects silent faults in the ROS 2 graph: QoS mismatches, orphaned topics left by a name typo, dead nodes, parameter drift and unmet lifecycle expectations.
Maintainers
- bburda
Authors
No additional authors.
ros2_medkit_graph_watchdog
Gateway plugin that detects silent faults in the ROS 2 graph: failures where every node is up, nothing logs an error, and the robot is still broken.
Detectors read the graph and raise faults through a ReportFault service client on the
gateway node; the faults surface via FaultManager on the gateway /faults API. That
client, like the lifecycle subscriptions below, lives in the plugin’s own callback group
and is driven by the plugin’s own executor from the tick thread, never by the gateway’s
ROS executor.
This package carries the plugin skeleton, the central reliability gate that holds raises
until the graph has quiesced, and five detectors, qos_mismatch, orphan,
param_drift, lifecycle_expectation and node_death. Two silent-fault classes remain
undelivered, GRAPH_TF_STALE and GRAPH_LATENCY_BUDGET; they land in follow-up changes,
each against its own issue, and their fault codes are already reserved in the frozen
GRAPH_* namespace (see “Fault codes”).
Build
With the gateway built and sourced:
colcon build --packages-select ros2_medkit_graph_watchdog
colcon test --packages-select ros2_medkit_graph_watchdog && colcon test-result --verbose
Load into the gateway
ros2 run ros2_medkit_gateway gateway_node --ros-args \
-p plugins:="[graph_watchdog]" \
-p "plugins.graph_watchdog.path:=$(ros2 pkg prefix ros2_medkit_graph_watchdog)/lib/ros2_medkit_graph_watchdog/libros2_medkit_graph_watchdog.so"
Configuration (plugins.graph_watchdog.<key>)
| Key | Type | Default | Meaning |
|---|---|---|---|
tick_interval_ms |
int | 1000 |
Detector tick cadence. Accepted range 1..INT_MAX; the check runs on the wide integer, so a value past INT_MAX is rejected rather than truncated back into the band. |
warmup_cycles |
int | 5 |
An entity must be continuously present for this many ticks before it arms. A mid-run restart re-warms it. Enforced centrally, see “Reliability (bringup-quiesce)” below. Accepted range 0..INT_MAX, checked on the wide integer. |
prune_grace |
int | 60 |
Default injected into every detector’s own config before configure(); a per-detector detectors.<id>.prune_grace overrides it. Used by detectors that keep per-key bookkeeping. Accepted range 0..INT_MAX here, checked on the wide integer; each detector re-validates it against its own documented range before using it. |
detectors.<id>.mode |
string | bool | raise |
raise = push faults; advisory = observe, do not push; off = disabled. Bare off/no (YAML booleans) also disable; bare on/yes mean raise. |
detectors.<id>.<field> |
any | - | Per-detector thresholds, passed to that detector’s configure(). |
Nested delivery. Keys are written dotted in YAML or --ros-args, and the gateway
delivers them to the plugin as a nested object. A bare off is a YAML 1.1 boolean, so
mode: off arrives as false, not the string "off"; both forms disable the detector.
Unknown keys are reported. Anything under detectors.<id> that the detector does not
read produces one startup warning naming the key and listing the ones that exist, so a
typo like allow_list: for allowlist: cannot silently do nothing. A typo in the detector
id itself is warned about the same way, with the registered ids listed.
lifecycle_expectation keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. |
require_active |
string[] | [] |
Node names that must be in the active lifecycle state, each matched against a live node by its App::id, its full FQN (/ns/name), or the bare leaf of that FQN. A bare name matches that node in EVERY namespace (a fleet-wide “all controller_servers must be active”); use a full FQN to pin one robot’s. Both the bare and the FQN form match against the node’s stable FQN, so they survive the App::id renaming that a same-bare-name collision triggers; an entry written as an App::id also works, but can stop matching on a multi-robot graph for exactly that reason. Empty = the detector checks nothing and emits nothing. A require_active that is not a string array warns and is ignored; an empty-string entry warns and is skipped - never dropped silently, since the operator would go on believing the node is covered. |
grace |
int | 5 |
Consecutive not-active ticks a required node tolerates before being reported inactive - bringup time for a managed node to reach active (configure + activate) before the expectation is enforced. Counted per NODE, not per matching entry: a node named by both a bare-name and a full-FQN entry still advances its streak once per tick, so mixing the two documented forms cannot halve the grace that was configured. An ALREADY-CONFIRMED streak keeps its content while the node is ABSENT, so a node that leaves the graph after being confirmed stays confirmed; a streak that has not yet crossed grace on a node the presence detector could also report (it was armed at least once) is HELD rather than advanced while absent, so a node is never confirmed out of ticks gathered while nobody could observe it - only a PRESENT tick may still cross grace for that node. A node that was NEVER armed gets no such hold: node_death can never track it either, so absence keeps advancing its streak too, the same as a present tick would. Accepted range 0..300 - five minutes at the shipped 1 s cadence, and already an extravagant allowance for a managed node to reach active. The upper end is a real bound rather than a formality: grace decides how long a PRESENT node may go unreported, and how long a node returning from a departure still inactive takes to (re-)mature, so at the old maximum of INT_MAX - 1 that PRESENT-side silence lasted about 24 days at the shipped cadence with no warning. Whether it also bounds how long a node that leaves the graph BELOW grace sits unsettled depends on the same split: for an armed node that lasts for as long as the node stays gone, independent of grace (see “Bounded by evidence, not by age” below); for a never-armed one grace bounds that too, since absence matures it within grace + absence_grace + 1 ticks either way. The check runs on the wide integer, so a value that only fits after truncation is rejected rather than silently turned into a hair-trigger. Anything outside the range warns and keeps the default. |
prune_grace |
int | 60 |
Consecutive ticks an IDLE tracked node - both clocks at zero and no matured ownership, so nothing to lose - may stay ABSENT from the graph before its bookkeeping is reclaimed. A node carrying evidence is never reclaimed by age at all, however long it stays gone, so this key cannot erase a fault’s own state; the map is bounded instead by tracked_node_cap (see “Bounded by evidence, not by age”). Injected for every detector at plugin scope and overridable per detector; a value outside 0..3600 warns and this detector keeps its own default of 60. The range check runs on the wide integer, so an out-of-int-range value is rejected rather than truncated. The value is used as written - there is no grace + 1 clamp, because there is no longer anything for one to protect. |
tracked_node_cap |
int | 512 |
The most nodes this detector keeps bookkeeping for at once. It is what bounds the map, since evidence is never reclaimed by age (see “Bounded by evidence, not by age”): at the cap, idle entries are reclaimed first, then entries for DEPARTED nodes are collapsed into a count, and only if every tracked node is PRESENT and carrying evidence is a newly matched node refused - which withholds GRAPH_NODE_INACTIVE’s clear and is reported both in the log and on GET /x-medkit-watchdog. 512 is comfortably above every node in a realistic graph (a full Nav2 stack plus perception is roughly a hundred nodes; a ten-robot fleet sharing one domain a few hundred), so raising it is only needed where require_active legitimately matches more PRESENT nodes than that. Accepted range 1..16384 - 16384 is about 8 MB of bookkeeping at roughly 500 bytes per tracked node, and a deployment needing more distinct required-node identities alive at once has identity churn rather than a large fleet. Anything outside the range, including 0 (a cap of nothing would mean checking nothing), warns and keeps the default; the check runs on the wide integer, so an out-of-int-range value is rejected rather than truncated. |
node_death keys
Zero-config, unlike every detector above: there is no require_active-style list of nodes
to watch, because every armed App in the graph is a candidate.
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. |
miss_grace |
int | 2 |
Consecutive missed ticks a tracked node tolerates before being reported dead - a death is reported once misses EXCEED this, i.e. after miss_grace + 1 ticks. Accepted range 0..3600, and additionally floored to whatever wall-clock window the configured tick_interval_ms needs (see “The wall-clock floor” below); a value below that floor is silently raised to it, with a warning naming the ms window it actually spans. |
prune_grace |
int | 60 |
Plugin-injected default (overridable here); how long a key being DURABLY suppressed (see “Suppression” below) may sit unreported before its bookkeeping is reclaimed. Accepted range 0..3600. The value actually used is max(prune_grace, miss_grace + 1), a silent internal floor rather than a rejection: without it, a durably-suppressed key could be reclaimed the very tick it would first have become eligible to report, losing the report rather than merely suppressing it. |
allowlist |
string[] | [] |
Node identities never reported dead. A dead key matches if it - or a name the same node is also known by - is present verbatim: the key’s own fqn, its bare leaf, or (captured while the node was still present) its App::id. Has no effect unless named in suppress; naming it without a configured list is a no-op. |
suppress |
string[] | [] |
Suppression mechanisms to activate: "allowlist" and "lifecycle". Any other entry warns as unrecognized and is ignored; a non-string or non-array entry warns the same way. |
tracked_node_cap |
int | 512 |
The most DEPARTED (not-yet-confirmed-dead or already-collapsed) identities this detector keeps individual bookkeeping for at once - a present/armed node is never counted against it and never refused tracking. Accepted range 1..16384; 0 is refused rather than clamped, since a cap of nothing would mean tracking nothing. See “Bounded by evidence, not by age” below for what it actually bounds and why, unlike the sibling detector’s identical-looking cap, a newcomer here is never the thing refused. |
tick_interval_ms is not an own key of this detector, but this is the one detector that
reads the plugin-injected value directly, to compute miss_grace’s wall-clock floor below.
Liveness, and why membership in the snapshot is not enough. App::is_online is what
this detector tracks, never mere presence in the entity snapshot. In runtime-only discovery
the two happen to coincide - a dead node’s App leaves the snapshot entirely - but a manifest
keeps a bound App present with only is_online cleared once its ROS binding disappears, and
hybrid discovery inherits that same shape; counting snapshot membership alone would make a
manifest-declared node immortal. A managed lifecycle node that merely deactivates keeps
is_online: true - its process is still running, only its ROS 2 lifecycle state changed -
so a deactivation is never mistaken for a death either; that is lifecycle_expectation’s
concern, not this one’s (see “The boundary with lifecycle_expectation” below).
Tracking is keyed on the STABLE fqn (App::effective_fqn()), never App::id: an id is
recomputed every sweep and only gains a namespace prefix once a same-bare-name collision
currently exists anywhere in the graph, so a live node’s id can change out from under a key
built from it.
What is never tracked.
| Excluded | Why |
|---|---|
Peer-aggregated apps (app.source starting peer:) |
A peer app carries no ROS binding of its own, so effective_fqn() is empty for every one of them; tracking them would collapse an entire peer fleet onto one "" key, and one online peer app would mask every other peer app’s departure. |
_ros2cli_<pid> nodes |
Every ros2 CLI invocation (ros2 topic echo, ros2 param get, …) spins up a real, short-lived node under this prefix - rcl’s own hidden-node naming convention, matched structurally rather than guessed at. Letting these through would accumulate one permanently “dead” entry per CLI invocation for the life of the gateway. |
| An App that never comes online | A key is admitted to tracking only once the reliability gate has said this detector OWNS its departure - armed at least once, and on one of the two GROUNDS ownership can rest on (a lifecycle state read as active, or no lifecycle to read at all; the third case, a state nobody could read, is admitted PROVISIONALLY and is described in the row below). A manifest App whose binding never starts is never armed and so can never be falsely called dead; a node still inside its warmup_cycles window gets the identical protection. Once tracked, though, a node’s continued life-or-death judgement rests on PRESENCE alone, not on staying armed - a tracked node that later goes lifecycle-inactive is not thereby mistaken for dead. |
| A managed App whose lifecycle state has not been read YET | Not tracked, and only for as long as that is still true. The gate’s own raise permission is answered PERMISSIVELY for a managed node whose GetState has not answered (LifecycleWatcher::node_ok()) - without that, qos_mismatch, orphan and param_drift would all fall silent about a node whose lifecycle service is broken. Permission is not knowledge, though, so tracking is withheld while a measurement may still arrive. LifecycleWatcher charges a GetState re-seed budget per node, and only for a read that actually ran, so “still asking” is a fact rather than a guess. Once that budget is spent the node is tracked here after all: nothing will ASK again, and lifecycle_expectation looks at nothing unless an operator named the node in require_active (empty by default), so withholding it any longer would mean the death is reported by nobody. That admission is PROVISIONAL, not final. The ~/transition_event subscription outlives the seed budget, so a label can still arrive without being asked for, and one that reads non-active says the node was the lifecycle detector’s all along - so the presence detector hands it back on the spot, while it is still alive. Waiting for the budget alone would only have delayed the wrong report, not prevented it. |
File truncated at 100 lines see the full file
CHANGELOG
Changelog for package ros2_medkit_graph_watchdog
0.7.0 (2026-08-27)
- Initial release of the package: a gateway plugin that raises faults
for silent failures in the ROS 2 graph - problems that leave every
node alive and every topic present, so nothing in the stack reports
them today. The package carries the plugin skeleton, a central
reliability gate that holds raises until the graph has quiesced, and
a detector registry. Detectors are configured under
plugins.graph_watchdog.<key>, each with araise/advisory/offmode, and an unknown key underdetectors.<id>is reported rather than silently ignored, so a typo cannot quietly disable a check (#571) -
qos_mismatchdetector: raisesGRAPH_QOS_MISMATCHfor a publisher and subscriber whose QoS profiles cannot match, which leaves the connection silently unestablished (#571) -
orphandetector: raisesGRAPH_ORPHANfor a one-sided topic - a publisher with no subscriber, or the reverse - but only when a complementary near-miss counterpart exists: same message type, a namespace and leaf within the configured edit distance, and the opposite side. A lone unsubscribed topic raises nothing. The pair is the signature of a remap or a topic-name typo, which is what the detector is for (#578) -
param_driftdetector: raisesGRAPH_PARAM_DRIFTwhen a node's live parameter value diverges from the declared expectation (#580) -
lifecycle_expectationdetector: raisesGRAPH_NODE_INACTIVEfor a node named inrequire_activethat is not in theactivelifecycle state once itsgracewindow has passed. It raises two further codes for the cases where the state could not be established at all:GRAPH_NODE_UNREADABLEwhen the lifecycle service does not answer, andGRAPH_NODE_NOT_MANAGEDwhen the named node has no lifecycle to read. These are distinct codes on the wire, so a consumer filtering onGRAPH_NODE_INACTIVEalone sees neither unmeasured case. A node is matched by itsApp::id, its full FQN, or the bare leaf of that FQN, and the grace streak is counted per node, so naming one node in two documented forms cannot halve the grace that was configured (#587) -
node_deathdetector: raisesGRAPH_NODE_DISAPPEAREDfor a node that leaves the graph when nothing else is left to report it, with a suppression framework that decides ownership of a departure from knowledge rather than from silence. A node whose lifecycle state was never readable is admitted only provisionally, and is handed back the moment a label arrives saying the departure belonged to the lifecycle detector instead. Zero-config: there is no list of nodes to maintain (#625, #624) - Two silent-fault classes are not delivered in this release.
GRAPH_TF_STALEandGRAPH_LATENCY_BUDGEThave their fault codes reserved in the frozenGRAPH_*namespace and land in later changes, each against its own issue - The plugin keeps its ROS entities off the gateway executor, so an entity created and destroyed while the gateway runs cannot have its destructor run on an executor thread concurrently with a create on the same node
- Contributors: \@bburda
Package Dependencies
System Dependencies
Dependant Packages
No known dependants.
Launch files
No launch files found
Messages
No message files found.
Services
No service files found
Plugins
No plugins found.