ROS Index
Package Summary
| Version | 0.6.0 |
| License | Apache-2.0 |
| Build type | AMENT_CMAKE |
| Use | RECOMMENDED |
Repository Summary
| Checkout URI | https://github.com/selfpatch/ros2_medkit.git |
| VCS Type | git |
| VCS Version | main |
| Last Updated | 2026-08-17 |
| Dev Status | DEVELOPED |
| Released | RELEASED |
| Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
Maintainers
- bburda
Authors
ros2_medkit_graph_watchdog
Gateway plugin that detects silent faults in the ROS 2 graph: failures where every node is up, nothing logs an error, and the robot is still broken.
Detectors read the graph and raise faults through a ReportFault service client on the
gateway node; the faults surface via FaultManager on the gateway /faults API. That
client, like the lifecycle subscriptions below, lives in the plugin’s own callback group
and is driven by the plugin’s own executor from the tick thread, never by the gateway’s
ROS executor.
This package carries the plugin skeleton, the central reliability gate that holds raises
until the graph has quiesced, and four detectors, qos_mismatch, orphan,
param_drift and lifecycle_expectation. The remaining
silent-fault classes land in follow-up changes, each against its own issue; their fault
codes are already reserved in the frozen GRAPH_* namespace (see “Fault codes”).
Build
With the gateway built and sourced:
colcon build --packages-select ros2_medkit_graph_watchdog
colcon test --packages-select ros2_medkit_graph_watchdog && colcon test-result --verbose
Load into the gateway
ros2 run ros2_medkit_gateway gateway_node --ros-args \
-p plugins:="[graph_watchdog]" \
-p "plugins.graph_watchdog.path:=$(ros2 pkg prefix ros2_medkit_graph_watchdog)/lib/ros2_medkit_graph_watchdog/libros2_medkit_graph_watchdog.so"
Configuration (plugins.graph_watchdog.<key>)
| Key | Type | Default | Meaning |
|---|---|---|---|
tick_interval_ms |
int | 1000 |
Detector tick cadence. |
warmup_cycles |
int | 5 |
An entity must be continuously present for this many ticks before it arms. A mid-run restart re-warms it. Enforced centrally, see “Reliability (bringup-quiesce)” below. |
prune_grace |
int | 60 |
Default injected into every detector’s own config before configure(); a per-detector detectors.<id>.prune_grace overrides it. Used by detectors that keep per-key bookkeeping. |
detectors.<id>.mode |
string | bool | raise |
raise = push faults; advisory = observe, do not push; off = disabled. Bare off/no (YAML booleans) also disable; bare on/yes mean raise. |
detectors.<id>.<field> |
any | - | Per-detector thresholds, passed to that detector’s configure(). |
Nested delivery. Keys are written dotted in YAML or --ros-args, and the gateway
delivers them to the plugin as a nested object. A bare off is a YAML 1.1 boolean, so
mode: off arrives as false, not the string "off"; both forms disable the detector.
Unknown keys are reported. Anything under detectors.<id> that the detector does not
read produces one startup warning naming the key and listing the ones that exist, so a
typo like allow_list: for allowlist: cannot silently do nothing. A typo in the detector
id itself is warned about the same way, with the registered ids listed.
lifecycle_expectation keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. |
require_active |
string[] | [] |
Node names that must be in the active lifecycle state, each matched against a live node by its App::id, its full FQN (/ns/name), or the bare leaf of that FQN. A bare name matches that node in EVERY namespace (a fleet-wide “all controller_servers must be active”); use a full FQN to pin one robot’s. Both the bare and the FQN form match against the node’s stable FQN, so they survive the App::id renaming that a same-bare-name collision triggers; an entry written as an App::id also works, but can stop matching on a multi-robot graph for exactly that reason. Empty = the detector checks nothing and emits nothing. A require_active that is not a string array warns and is ignored; an empty-string entry warns and is skipped - never dropped silently, since the operator would go on believing the node is covered. |
grace |
int | 5 |
Consecutive not-active ticks a required node tolerates before being reported inactive - bringup time for a managed node to reach active (configure + activate) before the expectation is enforced. Counted per NODE, not per matching entry: a node named by both a bare-name and a full-FQN entry still advances its streak once per tick, so mixing the two documented forms cannot halve the grace that was configured. Past the absence grace the streak keeps advancing while the node is ABSENT, so a node that leaves the graph while violating is still confirmed. Accepted range 0..300 - five minutes at the shipped 1 s cadence, and already an extravagant allowance for a managed node to reach active. The upper end is a real bound rather than a formality: grace also decides how long a node that LEFT the graph while not-active sits unsettled in the tracker, and GRAPH_NODE_INACTIVE’s clear is withheld for EVERY node while it does, so at the old maximum of INT_MAX - 1 the fault could neither raise nor heal for about 24 days. The check runs on the wide integer, so a value that only fits after truncation is rejected rather than silently turned into a hair-trigger. Anything outside the range warns and keeps the default. |
prune_grace |
int | 60 |
Consecutive ticks an IDLE tracked node - both clocks at zero and no matured ownership, so nothing to lose - may stay ABSENT from the graph before its bookkeeping is reclaimed. A node carrying evidence is never reclaimed by age at all, however long it stays gone, so this key cannot erase a fault’s own state; the map is bounded instead by tracked_node_cap (see “Bounded by evidence, not by age”). Injected for every detector at plugin scope and overridable per detector; a value outside 0..3600 warns and this detector keeps its own default of 60. The range check runs on the wide integer, so an out-of-int-range value is rejected rather than truncated. The value is used as written - there is no grace + 1 clamp, because there is no longer anything for one to protect. |
tracked_node_cap |
int | 512 |
The most nodes this detector keeps bookkeeping for at once. It is what bounds the map, since evidence is never reclaimed by age (see “Bounded by evidence, not by age”): at the cap, idle entries are reclaimed first, then entries for DEPARTED nodes are collapsed into a count, and only if every tracked node is PRESENT and carrying evidence is a newly matched node refused - which withholds GRAPH_NODE_INACTIVE’s clear and is reported both in the log and on GET /x-medkit-watchdog. 512 is comfortably above every node in a realistic graph (a full Nav2 stack plus perception is roughly a hundred nodes; a ten-robot fleet sharing one domain a few hundred), so raising it is only needed where require_active legitimately matches more PRESENT nodes than that. Accepted range 1..16384 - 16384 is about 8 MB of bookkeeping at roughly 500 bytes per tracked node, and a deployment needing more distinct required-node identities alive at once has identity churn rather than a large fleet. Anything outside the range, including 0 (a cap of nothing would mean checking nothing), warns and keeps the default; the check runs on the wide integer, so an out-of-int-range value is rejected rather than truncated. |
orphan keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | raise |
As above. |
grace |
int | 10 |
Consecutive sweeps a candidate pair must hold before it is reported. A staged bringup or a restart makes one side missing for a short time, which looks exactly like a typo until the other side appears. |
max_edit_distance |
int | 1 |
How many character edits the LEAF (the part after the last /) may differ by and still count as the same intended name. Values outside 1..8 keep the default: past a few edits a “near miss” starts matching unrelated topics. |
namespace_edit_distance |
int | 0 |
The same budget for the namespace, spent separately. 0 means the namespace must match to the character, which is what makes /robot1/scan and /robot2/scan two robots rather than a typo. Raise it to catch a misspelled namespace, and read the warning under the table first. Values outside 0..8 keep the default. |
allowlist |
string[] | [] |
Topic names never reported. Exact match only, never a prefix, so allowlisting /r1/scan does not also silence /r2/scan. |
What this detector will not catch, on purpose. A pair whose names match once every run of
digits is collapsed is treated as an enumeration of sensors, not a typo: /lidar_1 next to
/lidar_2, each one-sided, is an ordinary state on a multi-sensor robot and reporting it would
be a false alarm on every such machine. Collapsing runs rather than comparing character
positions is what also covers /lidar_9 next to /lidar_10. The cost is that a typo which
is a digit stops being reported. An appended digit is not that case: /scan1 collapses to
/scan#, which is not /scan, so it is still reported.
A misspelled namespace is opt-in, and it can be expensive. /robott/scan against
/robot/scan is one edit, but in the namespace, so by default it is not reported. Setting
namespace_edit_distance: 1 reports it. Check the naming scheme before doing that: on a fleet
named with letters, /amr_a and /amr_b are also one edit apart, and every robot in the
fleet will be reported as a typo. A numbered fleet is safe, since /robot1 against /robot2
differs only in its numeric field and the rule above already spares it.
What it cannot catch at all. A namespace added or dropped by mistake, /scan against
/robot/scan, is six edits apart, so no budget that is still specific will reach it. Do not
rely on this detector for that class.
param_drift keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. A bare off arrives as a YAML boolean; both forms disable the detector. |
baseline |
bool | true |
Self-capture the value of every parameter not listed in expect, then flag later changes to it. false checks only what expect pins, and skips reading the full parameter list. |
expect.<param> |
any | - | Absolute expected value for <param>, checked on every node that has it. This is the only rule that catches a value that was wrong from the first tick, which self-capture cannot see. A node that does not declare <param> answers NOT_FOUND, which is not a drift - so a misspelt name checks nothing at all. Once the sweep has covered every armed app - each one read, or given up on after repeated failed reads - and at least one of them answered, a pin that no app declares is warned about once, naming it. |
ignore |
string[] | [] |
Parameter-name globs never flagged. Applies to the self-captured set only, so it cannot silence an expect pin. A pattern of nothing but * matches every name and suppresses self-capture wholesale, which is indistinguishable at runtime from finding nothing; that combination warns at startup. |
max_reads_per_tick |
int | 8 |
Parameter-service ROUND TRIPS this detector may spend per tick, round-robin over nodes. Round trips, not calls: one call is several requests at the watched node, and the node pays for each of them. A baseline visit is charged 2 (a list, then one batched get) whatever the node’s parameter count; each expect pin is charged 3 (a list, a get and a descriptor read), or 2 on a node that does not declare it - see the table under “Coverage latency” for what each of those really costs the node. So a self-capture visit spends 2 of the budget and pinning N parameters spends up to 3N. It is an AMORTISED rate, not a per-tick cap: there is no pacing INSIDE a visit, so a node’s whole read set is issued back to back and the reader waits out the debt afterwards. Four pins against a budget of 8 fire 12 round trips inside one tick period and then idle for the rest of the next one; what the knob bounds is the sustained rate, not the burst. Accepted range 1..100000; anything else keeps the default and warns. |
prune_grace |
int | 60 |
Consecutive sweeps an app may be missing before its finding is dropped: the finding survives prune_grace absences and goes on the next one. Injected for every detector at plugin scope and overridable per detector; a value outside 0..3600 is rejected with a warning and this detector falls back to 2, not to the plugin default. The captured BASELINE has a shorter horizon of its own, and that one is NOT configurable - it is the compile-time kBaselineForgetGrace: the baseline is dropped on the third consecutive missed sweep, the two the reliability gate absorbs plus one, so an ordinary dropped discovery poll cannot re-capture a node on its already-drifted value. Setting prune_grace below 2 pulls that forward onto the sweep the finding itself is dropped on, so the two then go together. |
qos_mismatch keys
File truncated at 100 lines see the full file
Package Dependencies
System Dependencies
Dependant Packages
Launch files
Messages
Services
Plugins
Recent questions tagged ros2_medkit_graph_watchdog at Robotics Stack Exchange
Package Summary
| Version | 0.6.0 |
| License | Apache-2.0 |
| Build type | AMENT_CMAKE |
| Use | RECOMMENDED |
Repository Summary
| Checkout URI | https://github.com/selfpatch/ros2_medkit.git |
| VCS Type | git |
| VCS Version | main |
| Last Updated | 2026-08-17 |
| Dev Status | DEVELOPED |
| Released | RELEASED |
| Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
Maintainers
- bburda
Authors
ros2_medkit_graph_watchdog
Gateway plugin that detects silent faults in the ROS 2 graph: failures where every node is up, nothing logs an error, and the robot is still broken.
Detectors read the graph and raise faults through a ReportFault service client on the
gateway node; the faults surface via FaultManager on the gateway /faults API. That
client, like the lifecycle subscriptions below, lives in the plugin’s own callback group
and is driven by the plugin’s own executor from the tick thread, never by the gateway’s
ROS executor.
This package carries the plugin skeleton, the central reliability gate that holds raises
until the graph has quiesced, and four detectors, qos_mismatch, orphan,
param_drift and lifecycle_expectation. The remaining
silent-fault classes land in follow-up changes, each against its own issue; their fault
codes are already reserved in the frozen GRAPH_* namespace (see “Fault codes”).
Build
With the gateway built and sourced:
colcon build --packages-select ros2_medkit_graph_watchdog
colcon test --packages-select ros2_medkit_graph_watchdog && colcon test-result --verbose
Load into the gateway
ros2 run ros2_medkit_gateway gateway_node --ros-args \
-p plugins:="[graph_watchdog]" \
-p "plugins.graph_watchdog.path:=$(ros2 pkg prefix ros2_medkit_graph_watchdog)/lib/ros2_medkit_graph_watchdog/libros2_medkit_graph_watchdog.so"
Configuration (plugins.graph_watchdog.<key>)
| Key | Type | Default | Meaning |
|---|---|---|---|
tick_interval_ms |
int | 1000 |
Detector tick cadence. |
warmup_cycles |
int | 5 |
An entity must be continuously present for this many ticks before it arms. A mid-run restart re-warms it. Enforced centrally, see “Reliability (bringup-quiesce)” below. |
prune_grace |
int | 60 |
Default injected into every detector’s own config before configure(); a per-detector detectors.<id>.prune_grace overrides it. Used by detectors that keep per-key bookkeeping. |
detectors.<id>.mode |
string | bool | raise |
raise = push faults; advisory = observe, do not push; off = disabled. Bare off/no (YAML booleans) also disable; bare on/yes mean raise. |
detectors.<id>.<field> |
any | - | Per-detector thresholds, passed to that detector’s configure(). |
Nested delivery. Keys are written dotted in YAML or --ros-args, and the gateway
delivers them to the plugin as a nested object. A bare off is a YAML 1.1 boolean, so
mode: off arrives as false, not the string "off"; both forms disable the detector.
Unknown keys are reported. Anything under detectors.<id> that the detector does not
read produces one startup warning naming the key and listing the ones that exist, so a
typo like allow_list: for allowlist: cannot silently do nothing. A typo in the detector
id itself is warned about the same way, with the registered ids listed.
lifecycle_expectation keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. |
require_active |
string[] | [] |
Node names that must be in the active lifecycle state, each matched against a live node by its App::id, its full FQN (/ns/name), or the bare leaf of that FQN. A bare name matches that node in EVERY namespace (a fleet-wide “all controller_servers must be active”); use a full FQN to pin one robot’s. Both the bare and the FQN form match against the node’s stable FQN, so they survive the App::id renaming that a same-bare-name collision triggers; an entry written as an App::id also works, but can stop matching on a multi-robot graph for exactly that reason. Empty = the detector checks nothing and emits nothing. A require_active that is not a string array warns and is ignored; an empty-string entry warns and is skipped - never dropped silently, since the operator would go on believing the node is covered. |
grace |
int | 5 |
Consecutive not-active ticks a required node tolerates before being reported inactive - bringup time for a managed node to reach active (configure + activate) before the expectation is enforced. Counted per NODE, not per matching entry: a node named by both a bare-name and a full-FQN entry still advances its streak once per tick, so mixing the two documented forms cannot halve the grace that was configured. Past the absence grace the streak keeps advancing while the node is ABSENT, so a node that leaves the graph while violating is still confirmed. Accepted range 0..300 - five minutes at the shipped 1 s cadence, and already an extravagant allowance for a managed node to reach active. The upper end is a real bound rather than a formality: grace also decides how long a node that LEFT the graph while not-active sits unsettled in the tracker, and GRAPH_NODE_INACTIVE’s clear is withheld for EVERY node while it does, so at the old maximum of INT_MAX - 1 the fault could neither raise nor heal for about 24 days. The check runs on the wide integer, so a value that only fits after truncation is rejected rather than silently turned into a hair-trigger. Anything outside the range warns and keeps the default. |
prune_grace |
int | 60 |
Consecutive ticks an IDLE tracked node - both clocks at zero and no matured ownership, so nothing to lose - may stay ABSENT from the graph before its bookkeeping is reclaimed. A node carrying evidence is never reclaimed by age at all, however long it stays gone, so this key cannot erase a fault’s own state; the map is bounded instead by tracked_node_cap (see “Bounded by evidence, not by age”). Injected for every detector at plugin scope and overridable per detector; a value outside 0..3600 warns and this detector keeps its own default of 60. The range check runs on the wide integer, so an out-of-int-range value is rejected rather than truncated. The value is used as written - there is no grace + 1 clamp, because there is no longer anything for one to protect. |
tracked_node_cap |
int | 512 |
The most nodes this detector keeps bookkeeping for at once. It is what bounds the map, since evidence is never reclaimed by age (see “Bounded by evidence, not by age”): at the cap, idle entries are reclaimed first, then entries for DEPARTED nodes are collapsed into a count, and only if every tracked node is PRESENT and carrying evidence is a newly matched node refused - which withholds GRAPH_NODE_INACTIVE’s clear and is reported both in the log and on GET /x-medkit-watchdog. 512 is comfortably above every node in a realistic graph (a full Nav2 stack plus perception is roughly a hundred nodes; a ten-robot fleet sharing one domain a few hundred), so raising it is only needed where require_active legitimately matches more PRESENT nodes than that. Accepted range 1..16384 - 16384 is about 8 MB of bookkeeping at roughly 500 bytes per tracked node, and a deployment needing more distinct required-node identities alive at once has identity churn rather than a large fleet. Anything outside the range, including 0 (a cap of nothing would mean checking nothing), warns and keeps the default; the check runs on the wide integer, so an out-of-int-range value is rejected rather than truncated. |
orphan keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | raise |
As above. |
grace |
int | 10 |
Consecutive sweeps a candidate pair must hold before it is reported. A staged bringup or a restart makes one side missing for a short time, which looks exactly like a typo until the other side appears. |
max_edit_distance |
int | 1 |
How many character edits the LEAF (the part after the last /) may differ by and still count as the same intended name. Values outside 1..8 keep the default: past a few edits a “near miss” starts matching unrelated topics. |
namespace_edit_distance |
int | 0 |
The same budget for the namespace, spent separately. 0 means the namespace must match to the character, which is what makes /robot1/scan and /robot2/scan two robots rather than a typo. Raise it to catch a misspelled namespace, and read the warning under the table first. Values outside 0..8 keep the default. |
allowlist |
string[] | [] |
Topic names never reported. Exact match only, never a prefix, so allowlisting /r1/scan does not also silence /r2/scan. |
What this detector will not catch, on purpose. A pair whose names match once every run of
digits is collapsed is treated as an enumeration of sensors, not a typo: /lidar_1 next to
/lidar_2, each one-sided, is an ordinary state on a multi-sensor robot and reporting it would
be a false alarm on every such machine. Collapsing runs rather than comparing character
positions is what also covers /lidar_9 next to /lidar_10. The cost is that a typo which
is a digit stops being reported. An appended digit is not that case: /scan1 collapses to
/scan#, which is not /scan, so it is still reported.
A misspelled namespace is opt-in, and it can be expensive. /robott/scan against
/robot/scan is one edit, but in the namespace, so by default it is not reported. Setting
namespace_edit_distance: 1 reports it. Check the naming scheme before doing that: on a fleet
named with letters, /amr_a and /amr_b are also one edit apart, and every robot in the
fleet will be reported as a typo. A numbered fleet is safe, since /robot1 against /robot2
differs only in its numeric field and the rule above already spares it.
What it cannot catch at all. A namespace added or dropped by mistake, /scan against
/robot/scan, is six edits apart, so no budget that is still specific will reach it. Do not
rely on this detector for that class.
param_drift keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. A bare off arrives as a YAML boolean; both forms disable the detector. |
baseline |
bool | true |
Self-capture the value of every parameter not listed in expect, then flag later changes to it. false checks only what expect pins, and skips reading the full parameter list. |
expect.<param> |
any | - | Absolute expected value for <param>, checked on every node that has it. This is the only rule that catches a value that was wrong from the first tick, which self-capture cannot see. A node that does not declare <param> answers NOT_FOUND, which is not a drift - so a misspelt name checks nothing at all. Once the sweep has covered every armed app - each one read, or given up on after repeated failed reads - and at least one of them answered, a pin that no app declares is warned about once, naming it. |
ignore |
string[] | [] |
Parameter-name globs never flagged. Applies to the self-captured set only, so it cannot silence an expect pin. A pattern of nothing but * matches every name and suppresses self-capture wholesale, which is indistinguishable at runtime from finding nothing; that combination warns at startup. |
max_reads_per_tick |
int | 8 |
Parameter-service ROUND TRIPS this detector may spend per tick, round-robin over nodes. Round trips, not calls: one call is several requests at the watched node, and the node pays for each of them. A baseline visit is charged 2 (a list, then one batched get) whatever the node’s parameter count; each expect pin is charged 3 (a list, a get and a descriptor read), or 2 on a node that does not declare it - see the table under “Coverage latency” for what each of those really costs the node. So a self-capture visit spends 2 of the budget and pinning N parameters spends up to 3N. It is an AMORTISED rate, not a per-tick cap: there is no pacing INSIDE a visit, so a node’s whole read set is issued back to back and the reader waits out the debt afterwards. Four pins against a budget of 8 fire 12 round trips inside one tick period and then idle for the rest of the next one; what the knob bounds is the sustained rate, not the burst. Accepted range 1..100000; anything else keeps the default and warns. |
prune_grace |
int | 60 |
Consecutive sweeps an app may be missing before its finding is dropped: the finding survives prune_grace absences and goes on the next one. Injected for every detector at plugin scope and overridable per detector; a value outside 0..3600 is rejected with a warning and this detector falls back to 2, not to the plugin default. The captured BASELINE has a shorter horizon of its own, and that one is NOT configurable - it is the compile-time kBaselineForgetGrace: the baseline is dropped on the third consecutive missed sweep, the two the reliability gate absorbs plus one, so an ordinary dropped discovery poll cannot re-capture a node on its already-drifted value. Setting prune_grace below 2 pulls that forward onto the sweep the finding itself is dropped on, so the two then go together. |
qos_mismatch keys
File truncated at 100 lines see the full file
Package Dependencies
System Dependencies
Dependant Packages
Launch files
Messages
Services
Plugins
Recent questions tagged ros2_medkit_graph_watchdog at Robotics Stack Exchange
Package Summary
| Version | 0.6.0 |
| License | Apache-2.0 |
| Build type | AMENT_CMAKE |
| Use | RECOMMENDED |
Repository Summary
| Checkout URI | https://github.com/selfpatch/ros2_medkit.git |
| VCS Type | git |
| VCS Version | main |
| Last Updated | 2026-08-17 |
| Dev Status | DEVELOPED |
| Released | RELEASED |
| Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
Maintainers
- bburda
Authors
ros2_medkit_graph_watchdog
Gateway plugin that detects silent faults in the ROS 2 graph: failures where every node is up, nothing logs an error, and the robot is still broken.
Detectors read the graph and raise faults through a ReportFault service client on the
gateway node; the faults surface via FaultManager on the gateway /faults API. That
client, like the lifecycle subscriptions below, lives in the plugin’s own callback group
and is driven by the plugin’s own executor from the tick thread, never by the gateway’s
ROS executor.
This package carries the plugin skeleton, the central reliability gate that holds raises
until the graph has quiesced, and four detectors, qos_mismatch, orphan,
param_drift and lifecycle_expectation. The remaining
silent-fault classes land in follow-up changes, each against its own issue; their fault
codes are already reserved in the frozen GRAPH_* namespace (see “Fault codes”).
Build
With the gateway built and sourced:
colcon build --packages-select ros2_medkit_graph_watchdog
colcon test --packages-select ros2_medkit_graph_watchdog && colcon test-result --verbose
Load into the gateway
ros2 run ros2_medkit_gateway gateway_node --ros-args \
-p plugins:="[graph_watchdog]" \
-p "plugins.graph_watchdog.path:=$(ros2 pkg prefix ros2_medkit_graph_watchdog)/lib/ros2_medkit_graph_watchdog/libros2_medkit_graph_watchdog.so"
Configuration (plugins.graph_watchdog.<key>)
| Key | Type | Default | Meaning |
|---|---|---|---|
tick_interval_ms |
int | 1000 |
Detector tick cadence. |
warmup_cycles |
int | 5 |
An entity must be continuously present for this many ticks before it arms. A mid-run restart re-warms it. Enforced centrally, see “Reliability (bringup-quiesce)” below. |
prune_grace |
int | 60 |
Default injected into every detector’s own config before configure(); a per-detector detectors.<id>.prune_grace overrides it. Used by detectors that keep per-key bookkeeping. |
detectors.<id>.mode |
string | bool | raise |
raise = push faults; advisory = observe, do not push; off = disabled. Bare off/no (YAML booleans) also disable; bare on/yes mean raise. |
detectors.<id>.<field> |
any | - | Per-detector thresholds, passed to that detector’s configure(). |
Nested delivery. Keys are written dotted in YAML or --ros-args, and the gateway
delivers them to the plugin as a nested object. A bare off is a YAML 1.1 boolean, so
mode: off arrives as false, not the string "off"; both forms disable the detector.
Unknown keys are reported. Anything under detectors.<id> that the detector does not
read produces one startup warning naming the key and listing the ones that exist, so a
typo like allow_list: for allowlist: cannot silently do nothing. A typo in the detector
id itself is warned about the same way, with the registered ids listed.
lifecycle_expectation keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. |
require_active |
string[] | [] |
Node names that must be in the active lifecycle state, each matched against a live node by its App::id, its full FQN (/ns/name), or the bare leaf of that FQN. A bare name matches that node in EVERY namespace (a fleet-wide “all controller_servers must be active”); use a full FQN to pin one robot’s. Both the bare and the FQN form match against the node’s stable FQN, so they survive the App::id renaming that a same-bare-name collision triggers; an entry written as an App::id also works, but can stop matching on a multi-robot graph for exactly that reason. Empty = the detector checks nothing and emits nothing. A require_active that is not a string array warns and is ignored; an empty-string entry warns and is skipped - never dropped silently, since the operator would go on believing the node is covered. |
grace |
int | 5 |
Consecutive not-active ticks a required node tolerates before being reported inactive - bringup time for a managed node to reach active (configure + activate) before the expectation is enforced. Counted per NODE, not per matching entry: a node named by both a bare-name and a full-FQN entry still advances its streak once per tick, so mixing the two documented forms cannot halve the grace that was configured. Past the absence grace the streak keeps advancing while the node is ABSENT, so a node that leaves the graph while violating is still confirmed. Accepted range 0..300 - five minutes at the shipped 1 s cadence, and already an extravagant allowance for a managed node to reach active. The upper end is a real bound rather than a formality: grace also decides how long a node that LEFT the graph while not-active sits unsettled in the tracker, and GRAPH_NODE_INACTIVE’s clear is withheld for EVERY node while it does, so at the old maximum of INT_MAX - 1 the fault could neither raise nor heal for about 24 days. The check runs on the wide integer, so a value that only fits after truncation is rejected rather than silently turned into a hair-trigger. Anything outside the range warns and keeps the default. |
prune_grace |
int | 60 |
Consecutive ticks an IDLE tracked node - both clocks at zero and no matured ownership, so nothing to lose - may stay ABSENT from the graph before its bookkeeping is reclaimed. A node carrying evidence is never reclaimed by age at all, however long it stays gone, so this key cannot erase a fault’s own state; the map is bounded instead by tracked_node_cap (see “Bounded by evidence, not by age”). Injected for every detector at plugin scope and overridable per detector; a value outside 0..3600 warns and this detector keeps its own default of 60. The range check runs on the wide integer, so an out-of-int-range value is rejected rather than truncated. The value is used as written - there is no grace + 1 clamp, because there is no longer anything for one to protect. |
tracked_node_cap |
int | 512 |
The most nodes this detector keeps bookkeeping for at once. It is what bounds the map, since evidence is never reclaimed by age (see “Bounded by evidence, not by age”): at the cap, idle entries are reclaimed first, then entries for DEPARTED nodes are collapsed into a count, and only if every tracked node is PRESENT and carrying evidence is a newly matched node refused - which withholds GRAPH_NODE_INACTIVE’s clear and is reported both in the log and on GET /x-medkit-watchdog. 512 is comfortably above every node in a realistic graph (a full Nav2 stack plus perception is roughly a hundred nodes; a ten-robot fleet sharing one domain a few hundred), so raising it is only needed where require_active legitimately matches more PRESENT nodes than that. Accepted range 1..16384 - 16384 is about 8 MB of bookkeeping at roughly 500 bytes per tracked node, and a deployment needing more distinct required-node identities alive at once has identity churn rather than a large fleet. Anything outside the range, including 0 (a cap of nothing would mean checking nothing), warns and keeps the default; the check runs on the wide integer, so an out-of-int-range value is rejected rather than truncated. |
orphan keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | raise |
As above. |
grace |
int | 10 |
Consecutive sweeps a candidate pair must hold before it is reported. A staged bringup or a restart makes one side missing for a short time, which looks exactly like a typo until the other side appears. |
max_edit_distance |
int | 1 |
How many character edits the LEAF (the part after the last /) may differ by and still count as the same intended name. Values outside 1..8 keep the default: past a few edits a “near miss” starts matching unrelated topics. |
namespace_edit_distance |
int | 0 |
The same budget for the namespace, spent separately. 0 means the namespace must match to the character, which is what makes /robot1/scan and /robot2/scan two robots rather than a typo. Raise it to catch a misspelled namespace, and read the warning under the table first. Values outside 0..8 keep the default. |
allowlist |
string[] | [] |
Topic names never reported. Exact match only, never a prefix, so allowlisting /r1/scan does not also silence /r2/scan. |
What this detector will not catch, on purpose. A pair whose names match once every run of
digits is collapsed is treated as an enumeration of sensors, not a typo: /lidar_1 next to
/lidar_2, each one-sided, is an ordinary state on a multi-sensor robot and reporting it would
be a false alarm on every such machine. Collapsing runs rather than comparing character
positions is what also covers /lidar_9 next to /lidar_10. The cost is that a typo which
is a digit stops being reported. An appended digit is not that case: /scan1 collapses to
/scan#, which is not /scan, so it is still reported.
A misspelled namespace is opt-in, and it can be expensive. /robott/scan against
/robot/scan is one edit, but in the namespace, so by default it is not reported. Setting
namespace_edit_distance: 1 reports it. Check the naming scheme before doing that: on a fleet
named with letters, /amr_a and /amr_b are also one edit apart, and every robot in the
fleet will be reported as a typo. A numbered fleet is safe, since /robot1 against /robot2
differs only in its numeric field and the rule above already spares it.
What it cannot catch at all. A namespace added or dropped by mistake, /scan against
/robot/scan, is six edits apart, so no budget that is still specific will reach it. Do not
rely on this detector for that class.
param_drift keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. A bare off arrives as a YAML boolean; both forms disable the detector. |
baseline |
bool | true |
Self-capture the value of every parameter not listed in expect, then flag later changes to it. false checks only what expect pins, and skips reading the full parameter list. |
expect.<param> |
any | - | Absolute expected value for <param>, checked on every node that has it. This is the only rule that catches a value that was wrong from the first tick, which self-capture cannot see. A node that does not declare <param> answers NOT_FOUND, which is not a drift - so a misspelt name checks nothing at all. Once the sweep has covered every armed app - each one read, or given up on after repeated failed reads - and at least one of them answered, a pin that no app declares is warned about once, naming it. |
ignore |
string[] | [] |
Parameter-name globs never flagged. Applies to the self-captured set only, so it cannot silence an expect pin. A pattern of nothing but * matches every name and suppresses self-capture wholesale, which is indistinguishable at runtime from finding nothing; that combination warns at startup. |
max_reads_per_tick |
int | 8 |
Parameter-service ROUND TRIPS this detector may spend per tick, round-robin over nodes. Round trips, not calls: one call is several requests at the watched node, and the node pays for each of them. A baseline visit is charged 2 (a list, then one batched get) whatever the node’s parameter count; each expect pin is charged 3 (a list, a get and a descriptor read), or 2 on a node that does not declare it - see the table under “Coverage latency” for what each of those really costs the node. So a self-capture visit spends 2 of the budget and pinning N parameters spends up to 3N. It is an AMORTISED rate, not a per-tick cap: there is no pacing INSIDE a visit, so a node’s whole read set is issued back to back and the reader waits out the debt afterwards. Four pins against a budget of 8 fire 12 round trips inside one tick period and then idle for the rest of the next one; what the knob bounds is the sustained rate, not the burst. Accepted range 1..100000; anything else keeps the default and warns. |
prune_grace |
int | 60 |
Consecutive sweeps an app may be missing before its finding is dropped: the finding survives prune_grace absences and goes on the next one. Injected for every detector at plugin scope and overridable per detector; a value outside 0..3600 is rejected with a warning and this detector falls back to 2, not to the plugin default. The captured BASELINE has a shorter horizon of its own, and that one is NOT configurable - it is the compile-time kBaselineForgetGrace: the baseline is dropped on the third consecutive missed sweep, the two the reliability gate absorbs plus one, so an ordinary dropped discovery poll cannot re-capture a node on its already-drifted value. Setting prune_grace below 2 pulls that forward onto the sweep the finding itself is dropped on, so the two then go together. |
qos_mismatch keys
File truncated at 100 lines see the full file
Package Dependencies
System Dependencies
Dependant Packages
Launch files
Messages
Services
Plugins
Recent questions tagged ros2_medkit_graph_watchdog at Robotics Stack Exchange
Package Summary
| Version | 0.6.0 |
| License | Apache-2.0 |
| Build type | AMENT_CMAKE |
| Use | RECOMMENDED |
Repository Summary
| Checkout URI | https://github.com/selfpatch/ros2_medkit.git |
| VCS Type | git |
| VCS Version | main |
| Last Updated | 2026-08-17 |
| Dev Status | DEVELOPED |
| Released | RELEASED |
| Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
Maintainers
- bburda
Authors
ros2_medkit_graph_watchdog
Gateway plugin that detects silent faults in the ROS 2 graph: failures where every node is up, nothing logs an error, and the robot is still broken.
Detectors read the graph and raise faults through a ReportFault service client on the
gateway node; the faults surface via FaultManager on the gateway /faults API. That
client, like the lifecycle subscriptions below, lives in the plugin’s own callback group
and is driven by the plugin’s own executor from the tick thread, never by the gateway’s
ROS executor.
This package carries the plugin skeleton, the central reliability gate that holds raises
until the graph has quiesced, and four detectors, qos_mismatch, orphan,
param_drift and lifecycle_expectation. The remaining
silent-fault classes land in follow-up changes, each against its own issue; their fault
codes are already reserved in the frozen GRAPH_* namespace (see “Fault codes”).
Build
With the gateway built and sourced:
colcon build --packages-select ros2_medkit_graph_watchdog
colcon test --packages-select ros2_medkit_graph_watchdog && colcon test-result --verbose
Load into the gateway
ros2 run ros2_medkit_gateway gateway_node --ros-args \
-p plugins:="[graph_watchdog]" \
-p "plugins.graph_watchdog.path:=$(ros2 pkg prefix ros2_medkit_graph_watchdog)/lib/ros2_medkit_graph_watchdog/libros2_medkit_graph_watchdog.so"
Configuration (plugins.graph_watchdog.<key>)
| Key | Type | Default | Meaning |
|---|---|---|---|
tick_interval_ms |
int | 1000 |
Detector tick cadence. |
warmup_cycles |
int | 5 |
An entity must be continuously present for this many ticks before it arms. A mid-run restart re-warms it. Enforced centrally, see “Reliability (bringup-quiesce)” below. |
prune_grace |
int | 60 |
Default injected into every detector’s own config before configure(); a per-detector detectors.<id>.prune_grace overrides it. Used by detectors that keep per-key bookkeeping. |
detectors.<id>.mode |
string | bool | raise |
raise = push faults; advisory = observe, do not push; off = disabled. Bare off/no (YAML booleans) also disable; bare on/yes mean raise. |
detectors.<id>.<field> |
any | - | Per-detector thresholds, passed to that detector’s configure(). |
Nested delivery. Keys are written dotted in YAML or --ros-args, and the gateway
delivers them to the plugin as a nested object. A bare off is a YAML 1.1 boolean, so
mode: off arrives as false, not the string "off"; both forms disable the detector.
Unknown keys are reported. Anything under detectors.<id> that the detector does not
read produces one startup warning naming the key and listing the ones that exist, so a
typo like allow_list: for allowlist: cannot silently do nothing. A typo in the detector
id itself is warned about the same way, with the registered ids listed.
lifecycle_expectation keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. |
require_active |
string[] | [] |
Node names that must be in the active lifecycle state, each matched against a live node by its App::id, its full FQN (/ns/name), or the bare leaf of that FQN. A bare name matches that node in EVERY namespace (a fleet-wide “all controller_servers must be active”); use a full FQN to pin one robot’s. Both the bare and the FQN form match against the node’s stable FQN, so they survive the App::id renaming that a same-bare-name collision triggers; an entry written as an App::id also works, but can stop matching on a multi-robot graph for exactly that reason. Empty = the detector checks nothing and emits nothing. A require_active that is not a string array warns and is ignored; an empty-string entry warns and is skipped - never dropped silently, since the operator would go on believing the node is covered. |
grace |
int | 5 |
Consecutive not-active ticks a required node tolerates before being reported inactive - bringup time for a managed node to reach active (configure + activate) before the expectation is enforced. Counted per NODE, not per matching entry: a node named by both a bare-name and a full-FQN entry still advances its streak once per tick, so mixing the two documented forms cannot halve the grace that was configured. Past the absence grace the streak keeps advancing while the node is ABSENT, so a node that leaves the graph while violating is still confirmed. Accepted range 0..300 - five minutes at the shipped 1 s cadence, and already an extravagant allowance for a managed node to reach active. The upper end is a real bound rather than a formality: grace also decides how long a node that LEFT the graph while not-active sits unsettled in the tracker, and GRAPH_NODE_INACTIVE’s clear is withheld for EVERY node while it does, so at the old maximum of INT_MAX - 1 the fault could neither raise nor heal for about 24 days. The check runs on the wide integer, so a value that only fits after truncation is rejected rather than silently turned into a hair-trigger. Anything outside the range warns and keeps the default. |
prune_grace |
int | 60 |
Consecutive ticks an IDLE tracked node - both clocks at zero and no matured ownership, so nothing to lose - may stay ABSENT from the graph before its bookkeeping is reclaimed. A node carrying evidence is never reclaimed by age at all, however long it stays gone, so this key cannot erase a fault’s own state; the map is bounded instead by tracked_node_cap (see “Bounded by evidence, not by age”). Injected for every detector at plugin scope and overridable per detector; a value outside 0..3600 warns and this detector keeps its own default of 60. The range check runs on the wide integer, so an out-of-int-range value is rejected rather than truncated. The value is used as written - there is no grace + 1 clamp, because there is no longer anything for one to protect. |
tracked_node_cap |
int | 512 |
The most nodes this detector keeps bookkeeping for at once. It is what bounds the map, since evidence is never reclaimed by age (see “Bounded by evidence, not by age”): at the cap, idle entries are reclaimed first, then entries for DEPARTED nodes are collapsed into a count, and only if every tracked node is PRESENT and carrying evidence is a newly matched node refused - which withholds GRAPH_NODE_INACTIVE’s clear and is reported both in the log and on GET /x-medkit-watchdog. 512 is comfortably above every node in a realistic graph (a full Nav2 stack plus perception is roughly a hundred nodes; a ten-robot fleet sharing one domain a few hundred), so raising it is only needed where require_active legitimately matches more PRESENT nodes than that. Accepted range 1..16384 - 16384 is about 8 MB of bookkeeping at roughly 500 bytes per tracked node, and a deployment needing more distinct required-node identities alive at once has identity churn rather than a large fleet. Anything outside the range, including 0 (a cap of nothing would mean checking nothing), warns and keeps the default; the check runs on the wide integer, so an out-of-int-range value is rejected rather than truncated. |
orphan keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | raise |
As above. |
grace |
int | 10 |
Consecutive sweeps a candidate pair must hold before it is reported. A staged bringup or a restart makes one side missing for a short time, which looks exactly like a typo until the other side appears. |
max_edit_distance |
int | 1 |
How many character edits the LEAF (the part after the last /) may differ by and still count as the same intended name. Values outside 1..8 keep the default: past a few edits a “near miss” starts matching unrelated topics. |
namespace_edit_distance |
int | 0 |
The same budget for the namespace, spent separately. 0 means the namespace must match to the character, which is what makes /robot1/scan and /robot2/scan two robots rather than a typo. Raise it to catch a misspelled namespace, and read the warning under the table first. Values outside 0..8 keep the default. |
allowlist |
string[] | [] |
Topic names never reported. Exact match only, never a prefix, so allowlisting /r1/scan does not also silence /r2/scan. |
What this detector will not catch, on purpose. A pair whose names match once every run of
digits is collapsed is treated as an enumeration of sensors, not a typo: /lidar_1 next to
/lidar_2, each one-sided, is an ordinary state on a multi-sensor robot and reporting it would
be a false alarm on every such machine. Collapsing runs rather than comparing character
positions is what also covers /lidar_9 next to /lidar_10. The cost is that a typo which
is a digit stops being reported. An appended digit is not that case: /scan1 collapses to
/scan#, which is not /scan, so it is still reported.
A misspelled namespace is opt-in, and it can be expensive. /robott/scan against
/robot/scan is one edit, but in the namespace, so by default it is not reported. Setting
namespace_edit_distance: 1 reports it. Check the naming scheme before doing that: on a fleet
named with letters, /amr_a and /amr_b are also one edit apart, and every robot in the
fleet will be reported as a typo. A numbered fleet is safe, since /robot1 against /robot2
differs only in its numeric field and the rule above already spares it.
What it cannot catch at all. A namespace added or dropped by mistake, /scan against
/robot/scan, is six edits apart, so no budget that is still specific will reach it. Do not
rely on this detector for that class.
param_drift keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. A bare off arrives as a YAML boolean; both forms disable the detector. |
baseline |
bool | true |
Self-capture the value of every parameter not listed in expect, then flag later changes to it. false checks only what expect pins, and skips reading the full parameter list. |
expect.<param> |
any | - | Absolute expected value for <param>, checked on every node that has it. This is the only rule that catches a value that was wrong from the first tick, which self-capture cannot see. A node that does not declare <param> answers NOT_FOUND, which is not a drift - so a misspelt name checks nothing at all. Once the sweep has covered every armed app - each one read, or given up on after repeated failed reads - and at least one of them answered, a pin that no app declares is warned about once, naming it. |
ignore |
string[] | [] |
Parameter-name globs never flagged. Applies to the self-captured set only, so it cannot silence an expect pin. A pattern of nothing but * matches every name and suppresses self-capture wholesale, which is indistinguishable at runtime from finding nothing; that combination warns at startup. |
max_reads_per_tick |
int | 8 |
Parameter-service ROUND TRIPS this detector may spend per tick, round-robin over nodes. Round trips, not calls: one call is several requests at the watched node, and the node pays for each of them. A baseline visit is charged 2 (a list, then one batched get) whatever the node’s parameter count; each expect pin is charged 3 (a list, a get and a descriptor read), or 2 on a node that does not declare it - see the table under “Coverage latency” for what each of those really costs the node. So a self-capture visit spends 2 of the budget and pinning N parameters spends up to 3N. It is an AMORTISED rate, not a per-tick cap: there is no pacing INSIDE a visit, so a node’s whole read set is issued back to back and the reader waits out the debt afterwards. Four pins against a budget of 8 fire 12 round trips inside one tick period and then idle for the rest of the next one; what the knob bounds is the sustained rate, not the burst. Accepted range 1..100000; anything else keeps the default and warns. |
prune_grace |
int | 60 |
Consecutive sweeps an app may be missing before its finding is dropped: the finding survives prune_grace absences and goes on the next one. Injected for every detector at plugin scope and overridable per detector; a value outside 0..3600 is rejected with a warning and this detector falls back to 2, not to the plugin default. The captured BASELINE has a shorter horizon of its own, and that one is NOT configurable - it is the compile-time kBaselineForgetGrace: the baseline is dropped on the third consecutive missed sweep, the two the reliability gate absorbs plus one, so an ordinary dropped discovery poll cannot re-capture a node on its already-drifted value. Setting prune_grace below 2 pulls that forward onto the sweep the finding itself is dropped on, so the two then go together. |
qos_mismatch keys
File truncated at 100 lines see the full file
Package Dependencies
System Dependencies
Dependant Packages
Launch files
Messages
Services
Plugins
Recent questions tagged ros2_medkit_graph_watchdog at Robotics Stack Exchange
Package Summary
| Version | 0.6.0 |
| License | Apache-2.0 |
| Build type | AMENT_CMAKE |
| Use | RECOMMENDED |
Repository Summary
| Checkout URI | https://github.com/selfpatch/ros2_medkit.git |
| VCS Type | git |
| VCS Version | main |
| Last Updated | 2026-08-17 |
| Dev Status | DEVELOPED |
| Released | RELEASED |
| Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
Maintainers
- bburda
Authors
ros2_medkit_graph_watchdog
Gateway plugin that detects silent faults in the ROS 2 graph: failures where every node is up, nothing logs an error, and the robot is still broken.
Detectors read the graph and raise faults through a ReportFault service client on the
gateway node; the faults surface via FaultManager on the gateway /faults API. That
client, like the lifecycle subscriptions below, lives in the plugin’s own callback group
and is driven by the plugin’s own executor from the tick thread, never by the gateway’s
ROS executor.
This package carries the plugin skeleton, the central reliability gate that holds raises
until the graph has quiesced, and four detectors, qos_mismatch, orphan,
param_drift and lifecycle_expectation. The remaining
silent-fault classes land in follow-up changes, each against its own issue; their fault
codes are already reserved in the frozen GRAPH_* namespace (see “Fault codes”).
Build
With the gateway built and sourced:
colcon build --packages-select ros2_medkit_graph_watchdog
colcon test --packages-select ros2_medkit_graph_watchdog && colcon test-result --verbose
Load into the gateway
ros2 run ros2_medkit_gateway gateway_node --ros-args \
-p plugins:="[graph_watchdog]" \
-p "plugins.graph_watchdog.path:=$(ros2 pkg prefix ros2_medkit_graph_watchdog)/lib/ros2_medkit_graph_watchdog/libros2_medkit_graph_watchdog.so"
Configuration (plugins.graph_watchdog.<key>)
| Key | Type | Default | Meaning |
|---|---|---|---|
tick_interval_ms |
int | 1000 |
Detector tick cadence. |
warmup_cycles |
int | 5 |
An entity must be continuously present for this many ticks before it arms. A mid-run restart re-warms it. Enforced centrally, see “Reliability (bringup-quiesce)” below. |
prune_grace |
int | 60 |
Default injected into every detector’s own config before configure(); a per-detector detectors.<id>.prune_grace overrides it. Used by detectors that keep per-key bookkeeping. |
detectors.<id>.mode |
string | bool | raise |
raise = push faults; advisory = observe, do not push; off = disabled. Bare off/no (YAML booleans) also disable; bare on/yes mean raise. |
detectors.<id>.<field> |
any | - | Per-detector thresholds, passed to that detector’s configure(). |
Nested delivery. Keys are written dotted in YAML or --ros-args, and the gateway
delivers them to the plugin as a nested object. A bare off is a YAML 1.1 boolean, so
mode: off arrives as false, not the string "off"; both forms disable the detector.
Unknown keys are reported. Anything under detectors.<id> that the detector does not
read produces one startup warning naming the key and listing the ones that exist, so a
typo like allow_list: for allowlist: cannot silently do nothing. A typo in the detector
id itself is warned about the same way, with the registered ids listed.
lifecycle_expectation keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. |
require_active |
string[] | [] |
Node names that must be in the active lifecycle state, each matched against a live node by its App::id, its full FQN (/ns/name), or the bare leaf of that FQN. A bare name matches that node in EVERY namespace (a fleet-wide “all controller_servers must be active”); use a full FQN to pin one robot’s. Both the bare and the FQN form match against the node’s stable FQN, so they survive the App::id renaming that a same-bare-name collision triggers; an entry written as an App::id also works, but can stop matching on a multi-robot graph for exactly that reason. Empty = the detector checks nothing and emits nothing. A require_active that is not a string array warns and is ignored; an empty-string entry warns and is skipped - never dropped silently, since the operator would go on believing the node is covered. |
grace |
int | 5 |
Consecutive not-active ticks a required node tolerates before being reported inactive - bringup time for a managed node to reach active (configure + activate) before the expectation is enforced. Counted per NODE, not per matching entry: a node named by both a bare-name and a full-FQN entry still advances its streak once per tick, so mixing the two documented forms cannot halve the grace that was configured. Past the absence grace the streak keeps advancing while the node is ABSENT, so a node that leaves the graph while violating is still confirmed. Accepted range 0..300 - five minutes at the shipped 1 s cadence, and already an extravagant allowance for a managed node to reach active. The upper end is a real bound rather than a formality: grace also decides how long a node that LEFT the graph while not-active sits unsettled in the tracker, and GRAPH_NODE_INACTIVE’s clear is withheld for EVERY node while it does, so at the old maximum of INT_MAX - 1 the fault could neither raise nor heal for about 24 days. The check runs on the wide integer, so a value that only fits after truncation is rejected rather than silently turned into a hair-trigger. Anything outside the range warns and keeps the default. |
prune_grace |
int | 60 |
Consecutive ticks an IDLE tracked node - both clocks at zero and no matured ownership, so nothing to lose - may stay ABSENT from the graph before its bookkeeping is reclaimed. A node carrying evidence is never reclaimed by age at all, however long it stays gone, so this key cannot erase a fault’s own state; the map is bounded instead by tracked_node_cap (see “Bounded by evidence, not by age”). Injected for every detector at plugin scope and overridable per detector; a value outside 0..3600 warns and this detector keeps its own default of 60. The range check runs on the wide integer, so an out-of-int-range value is rejected rather than truncated. The value is used as written - there is no grace + 1 clamp, because there is no longer anything for one to protect. |
tracked_node_cap |
int | 512 |
The most nodes this detector keeps bookkeeping for at once. It is what bounds the map, since evidence is never reclaimed by age (see “Bounded by evidence, not by age”): at the cap, idle entries are reclaimed first, then entries for DEPARTED nodes are collapsed into a count, and only if every tracked node is PRESENT and carrying evidence is a newly matched node refused - which withholds GRAPH_NODE_INACTIVE’s clear and is reported both in the log and on GET /x-medkit-watchdog. 512 is comfortably above every node in a realistic graph (a full Nav2 stack plus perception is roughly a hundred nodes; a ten-robot fleet sharing one domain a few hundred), so raising it is only needed where require_active legitimately matches more PRESENT nodes than that. Accepted range 1..16384 - 16384 is about 8 MB of bookkeeping at roughly 500 bytes per tracked node, and a deployment needing more distinct required-node identities alive at once has identity churn rather than a large fleet. Anything outside the range, including 0 (a cap of nothing would mean checking nothing), warns and keeps the default; the check runs on the wide integer, so an out-of-int-range value is rejected rather than truncated. |
orphan keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | raise |
As above. |
grace |
int | 10 |
Consecutive sweeps a candidate pair must hold before it is reported. A staged bringup or a restart makes one side missing for a short time, which looks exactly like a typo until the other side appears. |
max_edit_distance |
int | 1 |
How many character edits the LEAF (the part after the last /) may differ by and still count as the same intended name. Values outside 1..8 keep the default: past a few edits a “near miss” starts matching unrelated topics. |
namespace_edit_distance |
int | 0 |
The same budget for the namespace, spent separately. 0 means the namespace must match to the character, which is what makes /robot1/scan and /robot2/scan two robots rather than a typo. Raise it to catch a misspelled namespace, and read the warning under the table first. Values outside 0..8 keep the default. |
allowlist |
string[] | [] |
Topic names never reported. Exact match only, never a prefix, so allowlisting /r1/scan does not also silence /r2/scan. |
What this detector will not catch, on purpose. A pair whose names match once every run of
digits is collapsed is treated as an enumeration of sensors, not a typo: /lidar_1 next to
/lidar_2, each one-sided, is an ordinary state on a multi-sensor robot and reporting it would
be a false alarm on every such machine. Collapsing runs rather than comparing character
positions is what also covers /lidar_9 next to /lidar_10. The cost is that a typo which
is a digit stops being reported. An appended digit is not that case: /scan1 collapses to
/scan#, which is not /scan, so it is still reported.
A misspelled namespace is opt-in, and it can be expensive. /robott/scan against
/robot/scan is one edit, but in the namespace, so by default it is not reported. Setting
namespace_edit_distance: 1 reports it. Check the naming scheme before doing that: on a fleet
named with letters, /amr_a and /amr_b are also one edit apart, and every robot in the
fleet will be reported as a typo. A numbered fleet is safe, since /robot1 against /robot2
differs only in its numeric field and the rule above already spares it.
What it cannot catch at all. A namespace added or dropped by mistake, /scan against
/robot/scan, is six edits apart, so no budget that is still specific will reach it. Do not
rely on this detector for that class.
param_drift keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. A bare off arrives as a YAML boolean; both forms disable the detector. |
baseline |
bool | true |
Self-capture the value of every parameter not listed in expect, then flag later changes to it. false checks only what expect pins, and skips reading the full parameter list. |
expect.<param> |
any | - | Absolute expected value for <param>, checked on every node that has it. This is the only rule that catches a value that was wrong from the first tick, which self-capture cannot see. A node that does not declare <param> answers NOT_FOUND, which is not a drift - so a misspelt name checks nothing at all. Once the sweep has covered every armed app - each one read, or given up on after repeated failed reads - and at least one of them answered, a pin that no app declares is warned about once, naming it. |
ignore |
string[] | [] |
Parameter-name globs never flagged. Applies to the self-captured set only, so it cannot silence an expect pin. A pattern of nothing but * matches every name and suppresses self-capture wholesale, which is indistinguishable at runtime from finding nothing; that combination warns at startup. |
max_reads_per_tick |
int | 8 |
Parameter-service ROUND TRIPS this detector may spend per tick, round-robin over nodes. Round trips, not calls: one call is several requests at the watched node, and the node pays for each of them. A baseline visit is charged 2 (a list, then one batched get) whatever the node’s parameter count; each expect pin is charged 3 (a list, a get and a descriptor read), or 2 on a node that does not declare it - see the table under “Coverage latency” for what each of those really costs the node. So a self-capture visit spends 2 of the budget and pinning N parameters spends up to 3N. It is an AMORTISED rate, not a per-tick cap: there is no pacing INSIDE a visit, so a node’s whole read set is issued back to back and the reader waits out the debt afterwards. Four pins against a budget of 8 fire 12 round trips inside one tick period and then idle for the rest of the next one; what the knob bounds is the sustained rate, not the burst. Accepted range 1..100000; anything else keeps the default and warns. |
prune_grace |
int | 60 |
Consecutive sweeps an app may be missing before its finding is dropped: the finding survives prune_grace absences and goes on the next one. Injected for every detector at plugin scope and overridable per detector; a value outside 0..3600 is rejected with a warning and this detector falls back to 2, not to the plugin default. The captured BASELINE has a shorter horizon of its own, and that one is NOT configurable - it is the compile-time kBaselineForgetGrace: the baseline is dropped on the third consecutive missed sweep, the two the reliability gate absorbs plus one, so an ordinary dropped discovery poll cannot re-capture a node on its already-drifted value. Setting prune_grace below 2 pulls that forward onto the sweep the finding itself is dropped on, so the two then go together. |
qos_mismatch keys
File truncated at 100 lines see the full file
Package Dependencies
System Dependencies
Dependant Packages
Launch files
Messages
Services
Plugins
Recent questions tagged ros2_medkit_graph_watchdog at Robotics Stack Exchange
Package Summary
| Version | 0.6.0 |
| License | Apache-2.0 |
| Build type | AMENT_CMAKE |
| Use | RECOMMENDED |
Repository Summary
| Checkout URI | https://github.com/selfpatch/ros2_medkit.git |
| VCS Type | git |
| VCS Version | main |
| Last Updated | 2026-08-17 |
| Dev Status | DEVELOPED |
| Released | RELEASED |
| Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
Maintainers
- bburda
Authors
ros2_medkit_graph_watchdog
Gateway plugin that detects silent faults in the ROS 2 graph: failures where every node is up, nothing logs an error, and the robot is still broken.
Detectors read the graph and raise faults through a ReportFault service client on the
gateway node; the faults surface via FaultManager on the gateway /faults API. That
client, like the lifecycle subscriptions below, lives in the plugin’s own callback group
and is driven by the plugin’s own executor from the tick thread, never by the gateway’s
ROS executor.
This package carries the plugin skeleton, the central reliability gate that holds raises
until the graph has quiesced, and four detectors, qos_mismatch, orphan,
param_drift and lifecycle_expectation. The remaining
silent-fault classes land in follow-up changes, each against its own issue; their fault
codes are already reserved in the frozen GRAPH_* namespace (see “Fault codes”).
Build
With the gateway built and sourced:
colcon build --packages-select ros2_medkit_graph_watchdog
colcon test --packages-select ros2_medkit_graph_watchdog && colcon test-result --verbose
Load into the gateway
ros2 run ros2_medkit_gateway gateway_node --ros-args \
-p plugins:="[graph_watchdog]" \
-p "plugins.graph_watchdog.path:=$(ros2 pkg prefix ros2_medkit_graph_watchdog)/lib/ros2_medkit_graph_watchdog/libros2_medkit_graph_watchdog.so"
Configuration (plugins.graph_watchdog.<key>)
| Key | Type | Default | Meaning |
|---|---|---|---|
tick_interval_ms |
int | 1000 |
Detector tick cadence. |
warmup_cycles |
int | 5 |
An entity must be continuously present for this many ticks before it arms. A mid-run restart re-warms it. Enforced centrally, see “Reliability (bringup-quiesce)” below. |
prune_grace |
int | 60 |
Default injected into every detector’s own config before configure(); a per-detector detectors.<id>.prune_grace overrides it. Used by detectors that keep per-key bookkeeping. |
detectors.<id>.mode |
string | bool | raise |
raise = push faults; advisory = observe, do not push; off = disabled. Bare off/no (YAML booleans) also disable; bare on/yes mean raise. |
detectors.<id>.<field> |
any | - | Per-detector thresholds, passed to that detector’s configure(). |
Nested delivery. Keys are written dotted in YAML or --ros-args, and the gateway
delivers them to the plugin as a nested object. A bare off is a YAML 1.1 boolean, so
mode: off arrives as false, not the string "off"; both forms disable the detector.
Unknown keys are reported. Anything under detectors.<id> that the detector does not
read produces one startup warning naming the key and listing the ones that exist, so a
typo like allow_list: for allowlist: cannot silently do nothing. A typo in the detector
id itself is warned about the same way, with the registered ids listed.
lifecycle_expectation keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. |
require_active |
string[] | [] |
Node names that must be in the active lifecycle state, each matched against a live node by its App::id, its full FQN (/ns/name), or the bare leaf of that FQN. A bare name matches that node in EVERY namespace (a fleet-wide “all controller_servers must be active”); use a full FQN to pin one robot’s. Both the bare and the FQN form match against the node’s stable FQN, so they survive the App::id renaming that a same-bare-name collision triggers; an entry written as an App::id also works, but can stop matching on a multi-robot graph for exactly that reason. Empty = the detector checks nothing and emits nothing. A require_active that is not a string array warns and is ignored; an empty-string entry warns and is skipped - never dropped silently, since the operator would go on believing the node is covered. |
grace |
int | 5 |
Consecutive not-active ticks a required node tolerates before being reported inactive - bringup time for a managed node to reach active (configure + activate) before the expectation is enforced. Counted per NODE, not per matching entry: a node named by both a bare-name and a full-FQN entry still advances its streak once per tick, so mixing the two documented forms cannot halve the grace that was configured. Past the absence grace the streak keeps advancing while the node is ABSENT, so a node that leaves the graph while violating is still confirmed. Accepted range 0..300 - five minutes at the shipped 1 s cadence, and already an extravagant allowance for a managed node to reach active. The upper end is a real bound rather than a formality: grace also decides how long a node that LEFT the graph while not-active sits unsettled in the tracker, and GRAPH_NODE_INACTIVE’s clear is withheld for EVERY node while it does, so at the old maximum of INT_MAX - 1 the fault could neither raise nor heal for about 24 days. The check runs on the wide integer, so a value that only fits after truncation is rejected rather than silently turned into a hair-trigger. Anything outside the range warns and keeps the default. |
prune_grace |
int | 60 |
Consecutive ticks an IDLE tracked node - both clocks at zero and no matured ownership, so nothing to lose - may stay ABSENT from the graph before its bookkeeping is reclaimed. A node carrying evidence is never reclaimed by age at all, however long it stays gone, so this key cannot erase a fault’s own state; the map is bounded instead by tracked_node_cap (see “Bounded by evidence, not by age”). Injected for every detector at plugin scope and overridable per detector; a value outside 0..3600 warns and this detector keeps its own default of 60. The range check runs on the wide integer, so an out-of-int-range value is rejected rather than truncated. The value is used as written - there is no grace + 1 clamp, because there is no longer anything for one to protect. |
tracked_node_cap |
int | 512 |
The most nodes this detector keeps bookkeeping for at once. It is what bounds the map, since evidence is never reclaimed by age (see “Bounded by evidence, not by age”): at the cap, idle entries are reclaimed first, then entries for DEPARTED nodes are collapsed into a count, and only if every tracked node is PRESENT and carrying evidence is a newly matched node refused - which withholds GRAPH_NODE_INACTIVE’s clear and is reported both in the log and on GET /x-medkit-watchdog. 512 is comfortably above every node in a realistic graph (a full Nav2 stack plus perception is roughly a hundred nodes; a ten-robot fleet sharing one domain a few hundred), so raising it is only needed where require_active legitimately matches more PRESENT nodes than that. Accepted range 1..16384 - 16384 is about 8 MB of bookkeeping at roughly 500 bytes per tracked node, and a deployment needing more distinct required-node identities alive at once has identity churn rather than a large fleet. Anything outside the range, including 0 (a cap of nothing would mean checking nothing), warns and keeps the default; the check runs on the wide integer, so an out-of-int-range value is rejected rather than truncated. |
orphan keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | raise |
As above. |
grace |
int | 10 |
Consecutive sweeps a candidate pair must hold before it is reported. A staged bringup or a restart makes one side missing for a short time, which looks exactly like a typo until the other side appears. |
max_edit_distance |
int | 1 |
How many character edits the LEAF (the part after the last /) may differ by and still count as the same intended name. Values outside 1..8 keep the default: past a few edits a “near miss” starts matching unrelated topics. |
namespace_edit_distance |
int | 0 |
The same budget for the namespace, spent separately. 0 means the namespace must match to the character, which is what makes /robot1/scan and /robot2/scan two robots rather than a typo. Raise it to catch a misspelled namespace, and read the warning under the table first. Values outside 0..8 keep the default. |
allowlist |
string[] | [] |
Topic names never reported. Exact match only, never a prefix, so allowlisting /r1/scan does not also silence /r2/scan. |
What this detector will not catch, on purpose. A pair whose names match once every run of
digits is collapsed is treated as an enumeration of sensors, not a typo: /lidar_1 next to
/lidar_2, each one-sided, is an ordinary state on a multi-sensor robot and reporting it would
be a false alarm on every such machine. Collapsing runs rather than comparing character
positions is what also covers /lidar_9 next to /lidar_10. The cost is that a typo which
is a digit stops being reported. An appended digit is not that case: /scan1 collapses to
/scan#, which is not /scan, so it is still reported.
A misspelled namespace is opt-in, and it can be expensive. /robott/scan against
/robot/scan is one edit, but in the namespace, so by default it is not reported. Setting
namespace_edit_distance: 1 reports it. Check the naming scheme before doing that: on a fleet
named with letters, /amr_a and /amr_b are also one edit apart, and every robot in the
fleet will be reported as a typo. A numbered fleet is safe, since /robot1 against /robot2
differs only in its numeric field and the rule above already spares it.
What it cannot catch at all. A namespace added or dropped by mistake, /scan against
/robot/scan, is six edits apart, so no budget that is still specific will reach it. Do not
rely on this detector for that class.
param_drift keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. A bare off arrives as a YAML boolean; both forms disable the detector. |
baseline |
bool | true |
Self-capture the value of every parameter not listed in expect, then flag later changes to it. false checks only what expect pins, and skips reading the full parameter list. |
expect.<param> |
any | - | Absolute expected value for <param>, checked on every node that has it. This is the only rule that catches a value that was wrong from the first tick, which self-capture cannot see. A node that does not declare <param> answers NOT_FOUND, which is not a drift - so a misspelt name checks nothing at all. Once the sweep has covered every armed app - each one read, or given up on after repeated failed reads - and at least one of them answered, a pin that no app declares is warned about once, naming it. |
ignore |
string[] | [] |
Parameter-name globs never flagged. Applies to the self-captured set only, so it cannot silence an expect pin. A pattern of nothing but * matches every name and suppresses self-capture wholesale, which is indistinguishable at runtime from finding nothing; that combination warns at startup. |
max_reads_per_tick |
int | 8 |
Parameter-service ROUND TRIPS this detector may spend per tick, round-robin over nodes. Round trips, not calls: one call is several requests at the watched node, and the node pays for each of them. A baseline visit is charged 2 (a list, then one batched get) whatever the node’s parameter count; each expect pin is charged 3 (a list, a get and a descriptor read), or 2 on a node that does not declare it - see the table under “Coverage latency” for what each of those really costs the node. So a self-capture visit spends 2 of the budget and pinning N parameters spends up to 3N. It is an AMORTISED rate, not a per-tick cap: there is no pacing INSIDE a visit, so a node’s whole read set is issued back to back and the reader waits out the debt afterwards. Four pins against a budget of 8 fire 12 round trips inside one tick period and then idle for the rest of the next one; what the knob bounds is the sustained rate, not the burst. Accepted range 1..100000; anything else keeps the default and warns. |
prune_grace |
int | 60 |
Consecutive sweeps an app may be missing before its finding is dropped: the finding survives prune_grace absences and goes on the next one. Injected for every detector at plugin scope and overridable per detector; a value outside 0..3600 is rejected with a warning and this detector falls back to 2, not to the plugin default. The captured BASELINE has a shorter horizon of its own, and that one is NOT configurable - it is the compile-time kBaselineForgetGrace: the baseline is dropped on the third consecutive missed sweep, the two the reliability gate absorbs plus one, so an ordinary dropped discovery poll cannot re-capture a node on its already-drifted value. Setting prune_grace below 2 pulls that forward onto the sweep the finding itself is dropped on, so the two then go together. |
qos_mismatch keys
File truncated at 100 lines see the full file
Package Dependencies
System Dependencies
Dependant Packages
Launch files
Messages
Services
Plugins
Recent questions tagged ros2_medkit_graph_watchdog at Robotics Stack Exchange
Package Summary
| Version | 0.6.0 |
| License | Apache-2.0 |
| Build type | AMENT_CMAKE |
| Use | RECOMMENDED |
Repository Summary
| Checkout URI | https://github.com/selfpatch/ros2_medkit.git |
| VCS Type | git |
| VCS Version | main |
| Last Updated | 2026-08-17 |
| Dev Status | DEVELOPED |
| Released | RELEASED |
| Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
Maintainers
- bburda
Authors
ros2_medkit_graph_watchdog
Gateway plugin that detects silent faults in the ROS 2 graph: failures where every node is up, nothing logs an error, and the robot is still broken.
Detectors read the graph and raise faults through a ReportFault service client on the
gateway node; the faults surface via FaultManager on the gateway /faults API. That
client, like the lifecycle subscriptions below, lives in the plugin’s own callback group
and is driven by the plugin’s own executor from the tick thread, never by the gateway’s
ROS executor.
This package carries the plugin skeleton, the central reliability gate that holds raises
until the graph has quiesced, and four detectors, qos_mismatch, orphan,
param_drift and lifecycle_expectation. The remaining
silent-fault classes land in follow-up changes, each against its own issue; their fault
codes are already reserved in the frozen GRAPH_* namespace (see “Fault codes”).
Build
With the gateway built and sourced:
colcon build --packages-select ros2_medkit_graph_watchdog
colcon test --packages-select ros2_medkit_graph_watchdog && colcon test-result --verbose
Load into the gateway
ros2 run ros2_medkit_gateway gateway_node --ros-args \
-p plugins:="[graph_watchdog]" \
-p "plugins.graph_watchdog.path:=$(ros2 pkg prefix ros2_medkit_graph_watchdog)/lib/ros2_medkit_graph_watchdog/libros2_medkit_graph_watchdog.so"
Configuration (plugins.graph_watchdog.<key>)
| Key | Type | Default | Meaning |
|---|---|---|---|
tick_interval_ms |
int | 1000 |
Detector tick cadence. |
warmup_cycles |
int | 5 |
An entity must be continuously present for this many ticks before it arms. A mid-run restart re-warms it. Enforced centrally, see “Reliability (bringup-quiesce)” below. |
prune_grace |
int | 60 |
Default injected into every detector’s own config before configure(); a per-detector detectors.<id>.prune_grace overrides it. Used by detectors that keep per-key bookkeeping. |
detectors.<id>.mode |
string | bool | raise |
raise = push faults; advisory = observe, do not push; off = disabled. Bare off/no (YAML booleans) also disable; bare on/yes mean raise. |
detectors.<id>.<field> |
any | - | Per-detector thresholds, passed to that detector’s configure(). |
Nested delivery. Keys are written dotted in YAML or --ros-args, and the gateway
delivers them to the plugin as a nested object. A bare off is a YAML 1.1 boolean, so
mode: off arrives as false, not the string "off"; both forms disable the detector.
Unknown keys are reported. Anything under detectors.<id> that the detector does not
read produces one startup warning naming the key and listing the ones that exist, so a
typo like allow_list: for allowlist: cannot silently do nothing. A typo in the detector
id itself is warned about the same way, with the registered ids listed.
lifecycle_expectation keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. |
require_active |
string[] | [] |
Node names that must be in the active lifecycle state, each matched against a live node by its App::id, its full FQN (/ns/name), or the bare leaf of that FQN. A bare name matches that node in EVERY namespace (a fleet-wide “all controller_servers must be active”); use a full FQN to pin one robot’s. Both the bare and the FQN form match against the node’s stable FQN, so they survive the App::id renaming that a same-bare-name collision triggers; an entry written as an App::id also works, but can stop matching on a multi-robot graph for exactly that reason. Empty = the detector checks nothing and emits nothing. A require_active that is not a string array warns and is ignored; an empty-string entry warns and is skipped - never dropped silently, since the operator would go on believing the node is covered. |
grace |
int | 5 |
Consecutive not-active ticks a required node tolerates before being reported inactive - bringup time for a managed node to reach active (configure + activate) before the expectation is enforced. Counted per NODE, not per matching entry: a node named by both a bare-name and a full-FQN entry still advances its streak once per tick, so mixing the two documented forms cannot halve the grace that was configured. Past the absence grace the streak keeps advancing while the node is ABSENT, so a node that leaves the graph while violating is still confirmed. Accepted range 0..300 - five minutes at the shipped 1 s cadence, and already an extravagant allowance for a managed node to reach active. The upper end is a real bound rather than a formality: grace also decides how long a node that LEFT the graph while not-active sits unsettled in the tracker, and GRAPH_NODE_INACTIVE’s clear is withheld for EVERY node while it does, so at the old maximum of INT_MAX - 1 the fault could neither raise nor heal for about 24 days. The check runs on the wide integer, so a value that only fits after truncation is rejected rather than silently turned into a hair-trigger. Anything outside the range warns and keeps the default. |
prune_grace |
int | 60 |
Consecutive ticks an IDLE tracked node - both clocks at zero and no matured ownership, so nothing to lose - may stay ABSENT from the graph before its bookkeeping is reclaimed. A node carrying evidence is never reclaimed by age at all, however long it stays gone, so this key cannot erase a fault’s own state; the map is bounded instead by tracked_node_cap (see “Bounded by evidence, not by age”). Injected for every detector at plugin scope and overridable per detector; a value outside 0..3600 warns and this detector keeps its own default of 60. The range check runs on the wide integer, so an out-of-int-range value is rejected rather than truncated. The value is used as written - there is no grace + 1 clamp, because there is no longer anything for one to protect. |
tracked_node_cap |
int | 512 |
The most nodes this detector keeps bookkeeping for at once. It is what bounds the map, since evidence is never reclaimed by age (see “Bounded by evidence, not by age”): at the cap, idle entries are reclaimed first, then entries for DEPARTED nodes are collapsed into a count, and only if every tracked node is PRESENT and carrying evidence is a newly matched node refused - which withholds GRAPH_NODE_INACTIVE’s clear and is reported both in the log and on GET /x-medkit-watchdog. 512 is comfortably above every node in a realistic graph (a full Nav2 stack plus perception is roughly a hundred nodes; a ten-robot fleet sharing one domain a few hundred), so raising it is only needed where require_active legitimately matches more PRESENT nodes than that. Accepted range 1..16384 - 16384 is about 8 MB of bookkeeping at roughly 500 bytes per tracked node, and a deployment needing more distinct required-node identities alive at once has identity churn rather than a large fleet. Anything outside the range, including 0 (a cap of nothing would mean checking nothing), warns and keeps the default; the check runs on the wide integer, so an out-of-int-range value is rejected rather than truncated. |
orphan keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | raise |
As above. |
grace |
int | 10 |
Consecutive sweeps a candidate pair must hold before it is reported. A staged bringup or a restart makes one side missing for a short time, which looks exactly like a typo until the other side appears. |
max_edit_distance |
int | 1 |
How many character edits the LEAF (the part after the last /) may differ by and still count as the same intended name. Values outside 1..8 keep the default: past a few edits a “near miss” starts matching unrelated topics. |
namespace_edit_distance |
int | 0 |
The same budget for the namespace, spent separately. 0 means the namespace must match to the character, which is what makes /robot1/scan and /robot2/scan two robots rather than a typo. Raise it to catch a misspelled namespace, and read the warning under the table first. Values outside 0..8 keep the default. |
allowlist |
string[] | [] |
Topic names never reported. Exact match only, never a prefix, so allowlisting /r1/scan does not also silence /r2/scan. |
What this detector will not catch, on purpose. A pair whose names match once every run of
digits is collapsed is treated as an enumeration of sensors, not a typo: /lidar_1 next to
/lidar_2, each one-sided, is an ordinary state on a multi-sensor robot and reporting it would
be a false alarm on every such machine. Collapsing runs rather than comparing character
positions is what also covers /lidar_9 next to /lidar_10. The cost is that a typo which
is a digit stops being reported. An appended digit is not that case: /scan1 collapses to
/scan#, which is not /scan, so it is still reported.
A misspelled namespace is opt-in, and it can be expensive. /robott/scan against
/robot/scan is one edit, but in the namespace, so by default it is not reported. Setting
namespace_edit_distance: 1 reports it. Check the naming scheme before doing that: on a fleet
named with letters, /amr_a and /amr_b are also one edit apart, and every robot in the
fleet will be reported as a typo. A numbered fleet is safe, since /robot1 against /robot2
differs only in its numeric field and the rule above already spares it.
What it cannot catch at all. A namespace added or dropped by mistake, /scan against
/robot/scan, is six edits apart, so no budget that is still specific will reach it. Do not
rely on this detector for that class.
param_drift keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. A bare off arrives as a YAML boolean; both forms disable the detector. |
baseline |
bool | true |
Self-capture the value of every parameter not listed in expect, then flag later changes to it. false checks only what expect pins, and skips reading the full parameter list. |
expect.<param> |
any | - | Absolute expected value for <param>, checked on every node that has it. This is the only rule that catches a value that was wrong from the first tick, which self-capture cannot see. A node that does not declare <param> answers NOT_FOUND, which is not a drift - so a misspelt name checks nothing at all. Once the sweep has covered every armed app - each one read, or given up on after repeated failed reads - and at least one of them answered, a pin that no app declares is warned about once, naming it. |
ignore |
string[] | [] |
Parameter-name globs never flagged. Applies to the self-captured set only, so it cannot silence an expect pin. A pattern of nothing but * matches every name and suppresses self-capture wholesale, which is indistinguishable at runtime from finding nothing; that combination warns at startup. |
max_reads_per_tick |
int | 8 |
Parameter-service ROUND TRIPS this detector may spend per tick, round-robin over nodes. Round trips, not calls: one call is several requests at the watched node, and the node pays for each of them. A baseline visit is charged 2 (a list, then one batched get) whatever the node’s parameter count; each expect pin is charged 3 (a list, a get and a descriptor read), or 2 on a node that does not declare it - see the table under “Coverage latency” for what each of those really costs the node. So a self-capture visit spends 2 of the budget and pinning N parameters spends up to 3N. It is an AMORTISED rate, not a per-tick cap: there is no pacing INSIDE a visit, so a node’s whole read set is issued back to back and the reader waits out the debt afterwards. Four pins against a budget of 8 fire 12 round trips inside one tick period and then idle for the rest of the next one; what the knob bounds is the sustained rate, not the burst. Accepted range 1..100000; anything else keeps the default and warns. |
prune_grace |
int | 60 |
Consecutive sweeps an app may be missing before its finding is dropped: the finding survives prune_grace absences and goes on the next one. Injected for every detector at plugin scope and overridable per detector; a value outside 0..3600 is rejected with a warning and this detector falls back to 2, not to the plugin default. The captured BASELINE has a shorter horizon of its own, and that one is NOT configurable - it is the compile-time kBaselineForgetGrace: the baseline is dropped on the third consecutive missed sweep, the two the reliability gate absorbs plus one, so an ordinary dropped discovery poll cannot re-capture a node on its already-drifted value. Setting prune_grace below 2 pulls that forward onto the sweep the finding itself is dropped on, so the two then go together. |
qos_mismatch keys
File truncated at 100 lines see the full file
Package Dependencies
System Dependencies
Dependant Packages
Launch files
Messages
Services
Plugins
Recent questions tagged ros2_medkit_graph_watchdog at Robotics Stack Exchange
Package Summary
| Version | 0.6.0 |
| License | Apache-2.0 |
| Build type | AMENT_CMAKE |
| Use | RECOMMENDED |
Repository Summary
| Checkout URI | https://github.com/selfpatch/ros2_medkit.git |
| VCS Type | git |
| VCS Version | main |
| Last Updated | 2026-08-17 |
| Dev Status | DEVELOPED |
| Released | RELEASED |
| Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
Maintainers
- bburda
Authors
ros2_medkit_graph_watchdog
Gateway plugin that detects silent faults in the ROS 2 graph: failures where every node is up, nothing logs an error, and the robot is still broken.
Detectors read the graph and raise faults through a ReportFault service client on the
gateway node; the faults surface via FaultManager on the gateway /faults API. That
client, like the lifecycle subscriptions below, lives in the plugin’s own callback group
and is driven by the plugin’s own executor from the tick thread, never by the gateway’s
ROS executor.
This package carries the plugin skeleton, the central reliability gate that holds raises
until the graph has quiesced, and four detectors, qos_mismatch, orphan,
param_drift and lifecycle_expectation. The remaining
silent-fault classes land in follow-up changes, each against its own issue; their fault
codes are already reserved in the frozen GRAPH_* namespace (see “Fault codes”).
Build
With the gateway built and sourced:
colcon build --packages-select ros2_medkit_graph_watchdog
colcon test --packages-select ros2_medkit_graph_watchdog && colcon test-result --verbose
Load into the gateway
ros2 run ros2_medkit_gateway gateway_node --ros-args \
-p plugins:="[graph_watchdog]" \
-p "plugins.graph_watchdog.path:=$(ros2 pkg prefix ros2_medkit_graph_watchdog)/lib/ros2_medkit_graph_watchdog/libros2_medkit_graph_watchdog.so"
Configuration (plugins.graph_watchdog.<key>)
| Key | Type | Default | Meaning |
|---|---|---|---|
tick_interval_ms |
int | 1000 |
Detector tick cadence. |
warmup_cycles |
int | 5 |
An entity must be continuously present for this many ticks before it arms. A mid-run restart re-warms it. Enforced centrally, see “Reliability (bringup-quiesce)” below. |
prune_grace |
int | 60 |
Default injected into every detector’s own config before configure(); a per-detector detectors.<id>.prune_grace overrides it. Used by detectors that keep per-key bookkeeping. |
detectors.<id>.mode |
string | bool | raise |
raise = push faults; advisory = observe, do not push; off = disabled. Bare off/no (YAML booleans) also disable; bare on/yes mean raise. |
detectors.<id>.<field> |
any | - | Per-detector thresholds, passed to that detector’s configure(). |
Nested delivery. Keys are written dotted in YAML or --ros-args, and the gateway
delivers them to the plugin as a nested object. A bare off is a YAML 1.1 boolean, so
mode: off arrives as false, not the string "off"; both forms disable the detector.
Unknown keys are reported. Anything under detectors.<id> that the detector does not
read produces one startup warning naming the key and listing the ones that exist, so a
typo like allow_list: for allowlist: cannot silently do nothing. A typo in the detector
id itself is warned about the same way, with the registered ids listed.
lifecycle_expectation keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. |
require_active |
string[] | [] |
Node names that must be in the active lifecycle state, each matched against a live node by its App::id, its full FQN (/ns/name), or the bare leaf of that FQN. A bare name matches that node in EVERY namespace (a fleet-wide “all controller_servers must be active”); use a full FQN to pin one robot’s. Both the bare and the FQN form match against the node’s stable FQN, so they survive the App::id renaming that a same-bare-name collision triggers; an entry written as an App::id also works, but can stop matching on a multi-robot graph for exactly that reason. Empty = the detector checks nothing and emits nothing. A require_active that is not a string array warns and is ignored; an empty-string entry warns and is skipped - never dropped silently, since the operator would go on believing the node is covered. |
grace |
int | 5 |
Consecutive not-active ticks a required node tolerates before being reported inactive - bringup time for a managed node to reach active (configure + activate) before the expectation is enforced. Counted per NODE, not per matching entry: a node named by both a bare-name and a full-FQN entry still advances its streak once per tick, so mixing the two documented forms cannot halve the grace that was configured. Past the absence grace the streak keeps advancing while the node is ABSENT, so a node that leaves the graph while violating is still confirmed. Accepted range 0..300 - five minutes at the shipped 1 s cadence, and already an extravagant allowance for a managed node to reach active. The upper end is a real bound rather than a formality: grace also decides how long a node that LEFT the graph while not-active sits unsettled in the tracker, and GRAPH_NODE_INACTIVE’s clear is withheld for EVERY node while it does, so at the old maximum of INT_MAX - 1 the fault could neither raise nor heal for about 24 days. The check runs on the wide integer, so a value that only fits after truncation is rejected rather than silently turned into a hair-trigger. Anything outside the range warns and keeps the default. |
prune_grace |
int | 60 |
Consecutive ticks an IDLE tracked node - both clocks at zero and no matured ownership, so nothing to lose - may stay ABSENT from the graph before its bookkeeping is reclaimed. A node carrying evidence is never reclaimed by age at all, however long it stays gone, so this key cannot erase a fault’s own state; the map is bounded instead by tracked_node_cap (see “Bounded by evidence, not by age”). Injected for every detector at plugin scope and overridable per detector; a value outside 0..3600 warns and this detector keeps its own default of 60. The range check runs on the wide integer, so an out-of-int-range value is rejected rather than truncated. The value is used as written - there is no grace + 1 clamp, because there is no longer anything for one to protect. |
tracked_node_cap |
int | 512 |
The most nodes this detector keeps bookkeeping for at once. It is what bounds the map, since evidence is never reclaimed by age (see “Bounded by evidence, not by age”): at the cap, idle entries are reclaimed first, then entries for DEPARTED nodes are collapsed into a count, and only if every tracked node is PRESENT and carrying evidence is a newly matched node refused - which withholds GRAPH_NODE_INACTIVE’s clear and is reported both in the log and on GET /x-medkit-watchdog. 512 is comfortably above every node in a realistic graph (a full Nav2 stack plus perception is roughly a hundred nodes; a ten-robot fleet sharing one domain a few hundred), so raising it is only needed where require_active legitimately matches more PRESENT nodes than that. Accepted range 1..16384 - 16384 is about 8 MB of bookkeeping at roughly 500 bytes per tracked node, and a deployment needing more distinct required-node identities alive at once has identity churn rather than a large fleet. Anything outside the range, including 0 (a cap of nothing would mean checking nothing), warns and keeps the default; the check runs on the wide integer, so an out-of-int-range value is rejected rather than truncated. |
orphan keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | raise |
As above. |
grace |
int | 10 |
Consecutive sweeps a candidate pair must hold before it is reported. A staged bringup or a restart makes one side missing for a short time, which looks exactly like a typo until the other side appears. |
max_edit_distance |
int | 1 |
How many character edits the LEAF (the part after the last /) may differ by and still count as the same intended name. Values outside 1..8 keep the default: past a few edits a “near miss” starts matching unrelated topics. |
namespace_edit_distance |
int | 0 |
The same budget for the namespace, spent separately. 0 means the namespace must match to the character, which is what makes /robot1/scan and /robot2/scan two robots rather than a typo. Raise it to catch a misspelled namespace, and read the warning under the table first. Values outside 0..8 keep the default. |
allowlist |
string[] | [] |
Topic names never reported. Exact match only, never a prefix, so allowlisting /r1/scan does not also silence /r2/scan. |
What this detector will not catch, on purpose. A pair whose names match once every run of
digits is collapsed is treated as an enumeration of sensors, not a typo: /lidar_1 next to
/lidar_2, each one-sided, is an ordinary state on a multi-sensor robot and reporting it would
be a false alarm on every such machine. Collapsing runs rather than comparing character
positions is what also covers /lidar_9 next to /lidar_10. The cost is that a typo which
is a digit stops being reported. An appended digit is not that case: /scan1 collapses to
/scan#, which is not /scan, so it is still reported.
A misspelled namespace is opt-in, and it can be expensive. /robott/scan against
/robot/scan is one edit, but in the namespace, so by default it is not reported. Setting
namespace_edit_distance: 1 reports it. Check the naming scheme before doing that: on a fleet
named with letters, /amr_a and /amr_b are also one edit apart, and every robot in the
fleet will be reported as a typo. A numbered fleet is safe, since /robot1 against /robot2
differs only in its numeric field and the rule above already spares it.
What it cannot catch at all. A namespace added or dropped by mistake, /scan against
/robot/scan, is six edits apart, so no budget that is still specific will reach it. Do not
rely on this detector for that class.
param_drift keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. A bare off arrives as a YAML boolean; both forms disable the detector. |
baseline |
bool | true |
Self-capture the value of every parameter not listed in expect, then flag later changes to it. false checks only what expect pins, and skips reading the full parameter list. |
expect.<param> |
any | - | Absolute expected value for <param>, checked on every node that has it. This is the only rule that catches a value that was wrong from the first tick, which self-capture cannot see. A node that does not declare <param> answers NOT_FOUND, which is not a drift - so a misspelt name checks nothing at all. Once the sweep has covered every armed app - each one read, or given up on after repeated failed reads - and at least one of them answered, a pin that no app declares is warned about once, naming it. |
ignore |
string[] | [] |
Parameter-name globs never flagged. Applies to the self-captured set only, so it cannot silence an expect pin. A pattern of nothing but * matches every name and suppresses self-capture wholesale, which is indistinguishable at runtime from finding nothing; that combination warns at startup. |
max_reads_per_tick |
int | 8 |
Parameter-service ROUND TRIPS this detector may spend per tick, round-robin over nodes. Round trips, not calls: one call is several requests at the watched node, and the node pays for each of them. A baseline visit is charged 2 (a list, then one batched get) whatever the node’s parameter count; each expect pin is charged 3 (a list, a get and a descriptor read), or 2 on a node that does not declare it - see the table under “Coverage latency” for what each of those really costs the node. So a self-capture visit spends 2 of the budget and pinning N parameters spends up to 3N. It is an AMORTISED rate, not a per-tick cap: there is no pacing INSIDE a visit, so a node’s whole read set is issued back to back and the reader waits out the debt afterwards. Four pins against a budget of 8 fire 12 round trips inside one tick period and then idle for the rest of the next one; what the knob bounds is the sustained rate, not the burst. Accepted range 1..100000; anything else keeps the default and warns. |
prune_grace |
int | 60 |
Consecutive sweeps an app may be missing before its finding is dropped: the finding survives prune_grace absences and goes on the next one. Injected for every detector at plugin scope and overridable per detector; a value outside 0..3600 is rejected with a warning and this detector falls back to 2, not to the plugin default. The captured BASELINE has a shorter horizon of its own, and that one is NOT configurable - it is the compile-time kBaselineForgetGrace: the baseline is dropped on the third consecutive missed sweep, the two the reliability gate absorbs plus one, so an ordinary dropped discovery poll cannot re-capture a node on its already-drifted value. Setting prune_grace below 2 pulls that forward onto the sweep the finding itself is dropped on, so the two then go together. |
qos_mismatch keys
File truncated at 100 lines see the full file
Package Dependencies
System Dependencies
Dependant Packages
Launch files
Messages
Services
Plugins
Recent questions tagged ros2_medkit_graph_watchdog at Robotics Stack Exchange
Package Summary
| Version | 0.6.0 |
| License | Apache-2.0 |
| Build type | AMENT_CMAKE |
| Use | RECOMMENDED |
Repository Summary
| Checkout URI | https://github.com/selfpatch/ros2_medkit.git |
| VCS Type | git |
| VCS Version | main |
| Last Updated | 2026-08-17 |
| Dev Status | DEVELOPED |
| Released | RELEASED |
| Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
Maintainers
- bburda
Authors
ros2_medkit_graph_watchdog
Gateway plugin that detects silent faults in the ROS 2 graph: failures where every node is up, nothing logs an error, and the robot is still broken.
Detectors read the graph and raise faults through a ReportFault service client on the
gateway node; the faults surface via FaultManager on the gateway /faults API. That
client, like the lifecycle subscriptions below, lives in the plugin’s own callback group
and is driven by the plugin’s own executor from the tick thread, never by the gateway’s
ROS executor.
This package carries the plugin skeleton, the central reliability gate that holds raises
until the graph has quiesced, and four detectors, qos_mismatch, orphan,
param_drift and lifecycle_expectation. The remaining
silent-fault classes land in follow-up changes, each against its own issue; their fault
codes are already reserved in the frozen GRAPH_* namespace (see “Fault codes”).
Build
With the gateway built and sourced:
colcon build --packages-select ros2_medkit_graph_watchdog
colcon test --packages-select ros2_medkit_graph_watchdog && colcon test-result --verbose
Load into the gateway
ros2 run ros2_medkit_gateway gateway_node --ros-args \
-p plugins:="[graph_watchdog]" \
-p "plugins.graph_watchdog.path:=$(ros2 pkg prefix ros2_medkit_graph_watchdog)/lib/ros2_medkit_graph_watchdog/libros2_medkit_graph_watchdog.so"
Configuration (plugins.graph_watchdog.<key>)
| Key | Type | Default | Meaning |
|---|---|---|---|
tick_interval_ms |
int | 1000 |
Detector tick cadence. |
warmup_cycles |
int | 5 |
An entity must be continuously present for this many ticks before it arms. A mid-run restart re-warms it. Enforced centrally, see “Reliability (bringup-quiesce)” below. |
prune_grace |
int | 60 |
Default injected into every detector’s own config before configure(); a per-detector detectors.<id>.prune_grace overrides it. Used by detectors that keep per-key bookkeeping. |
detectors.<id>.mode |
string | bool | raise |
raise = push faults; advisory = observe, do not push; off = disabled. Bare off/no (YAML booleans) also disable; bare on/yes mean raise. |
detectors.<id>.<field> |
any | - | Per-detector thresholds, passed to that detector’s configure(). |
Nested delivery. Keys are written dotted in YAML or --ros-args, and the gateway
delivers them to the plugin as a nested object. A bare off is a YAML 1.1 boolean, so
mode: off arrives as false, not the string "off"; both forms disable the detector.
Unknown keys are reported. Anything under detectors.<id> that the detector does not
read produces one startup warning naming the key and listing the ones that exist, so a
typo like allow_list: for allowlist: cannot silently do nothing. A typo in the detector
id itself is warned about the same way, with the registered ids listed.
lifecycle_expectation keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. |
require_active |
string[] | [] |
Node names that must be in the active lifecycle state, each matched against a live node by its App::id, its full FQN (/ns/name), or the bare leaf of that FQN. A bare name matches that node in EVERY namespace (a fleet-wide “all controller_servers must be active”); use a full FQN to pin one robot’s. Both the bare and the FQN form match against the node’s stable FQN, so they survive the App::id renaming that a same-bare-name collision triggers; an entry written as an App::id also works, but can stop matching on a multi-robot graph for exactly that reason. Empty = the detector checks nothing and emits nothing. A require_active that is not a string array warns and is ignored; an empty-string entry warns and is skipped - never dropped silently, since the operator would go on believing the node is covered. |
grace |
int | 5 |
Consecutive not-active ticks a required node tolerates before being reported inactive - bringup time for a managed node to reach active (configure + activate) before the expectation is enforced. Counted per NODE, not per matching entry: a node named by both a bare-name and a full-FQN entry still advances its streak once per tick, so mixing the two documented forms cannot halve the grace that was configured. Past the absence grace the streak keeps advancing while the node is ABSENT, so a node that leaves the graph while violating is still confirmed. Accepted range 0..300 - five minutes at the shipped 1 s cadence, and already an extravagant allowance for a managed node to reach active. The upper end is a real bound rather than a formality: grace also decides how long a node that LEFT the graph while not-active sits unsettled in the tracker, and GRAPH_NODE_INACTIVE’s clear is withheld for EVERY node while it does, so at the old maximum of INT_MAX - 1 the fault could neither raise nor heal for about 24 days. The check runs on the wide integer, so a value that only fits after truncation is rejected rather than silently turned into a hair-trigger. Anything outside the range warns and keeps the default. |
prune_grace |
int | 60 |
Consecutive ticks an IDLE tracked node - both clocks at zero and no matured ownership, so nothing to lose - may stay ABSENT from the graph before its bookkeeping is reclaimed. A node carrying evidence is never reclaimed by age at all, however long it stays gone, so this key cannot erase a fault’s own state; the map is bounded instead by tracked_node_cap (see “Bounded by evidence, not by age”). Injected for every detector at plugin scope and overridable per detector; a value outside 0..3600 warns and this detector keeps its own default of 60. The range check runs on the wide integer, so an out-of-int-range value is rejected rather than truncated. The value is used as written - there is no grace + 1 clamp, because there is no longer anything for one to protect. |
tracked_node_cap |
int | 512 |
The most nodes this detector keeps bookkeeping for at once. It is what bounds the map, since evidence is never reclaimed by age (see “Bounded by evidence, not by age”): at the cap, idle entries are reclaimed first, then entries for DEPARTED nodes are collapsed into a count, and only if every tracked node is PRESENT and carrying evidence is a newly matched node refused - which withholds GRAPH_NODE_INACTIVE’s clear and is reported both in the log and on GET /x-medkit-watchdog. 512 is comfortably above every node in a realistic graph (a full Nav2 stack plus perception is roughly a hundred nodes; a ten-robot fleet sharing one domain a few hundred), so raising it is only needed where require_active legitimately matches more PRESENT nodes than that. Accepted range 1..16384 - 16384 is about 8 MB of bookkeeping at roughly 500 bytes per tracked node, and a deployment needing more distinct required-node identities alive at once has identity churn rather than a large fleet. Anything outside the range, including 0 (a cap of nothing would mean checking nothing), warns and keeps the default; the check runs on the wide integer, so an out-of-int-range value is rejected rather than truncated. |
orphan keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | raise |
As above. |
grace |
int | 10 |
Consecutive sweeps a candidate pair must hold before it is reported. A staged bringup or a restart makes one side missing for a short time, which looks exactly like a typo until the other side appears. |
max_edit_distance |
int | 1 |
How many character edits the LEAF (the part after the last /) may differ by and still count as the same intended name. Values outside 1..8 keep the default: past a few edits a “near miss” starts matching unrelated topics. |
namespace_edit_distance |
int | 0 |
The same budget for the namespace, spent separately. 0 means the namespace must match to the character, which is what makes /robot1/scan and /robot2/scan two robots rather than a typo. Raise it to catch a misspelled namespace, and read the warning under the table first. Values outside 0..8 keep the default. |
allowlist |
string[] | [] |
Topic names never reported. Exact match only, never a prefix, so allowlisting /r1/scan does not also silence /r2/scan. |
What this detector will not catch, on purpose. A pair whose names match once every run of
digits is collapsed is treated as an enumeration of sensors, not a typo: /lidar_1 next to
/lidar_2, each one-sided, is an ordinary state on a multi-sensor robot and reporting it would
be a false alarm on every such machine. Collapsing runs rather than comparing character
positions is what also covers /lidar_9 next to /lidar_10. The cost is that a typo which
is a digit stops being reported. An appended digit is not that case: /scan1 collapses to
/scan#, which is not /scan, so it is still reported.
A misspelled namespace is opt-in, and it can be expensive. /robott/scan against
/robot/scan is one edit, but in the namespace, so by default it is not reported. Setting
namespace_edit_distance: 1 reports it. Check the naming scheme before doing that: on a fleet
named with letters, /amr_a and /amr_b are also one edit apart, and every robot in the
fleet will be reported as a typo. A numbered fleet is safe, since /robot1 against /robot2
differs only in its numeric field and the rule above already spares it.
What it cannot catch at all. A namespace added or dropped by mistake, /scan against
/robot/scan, is six edits apart, so no budget that is still specific will reach it. Do not
rely on this detector for that class.
param_drift keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. A bare off arrives as a YAML boolean; both forms disable the detector. |
baseline |
bool | true |
Self-capture the value of every parameter not listed in expect, then flag later changes to it. false checks only what expect pins, and skips reading the full parameter list. |
expect.<param> |
any | - | Absolute expected value for <param>, checked on every node that has it. This is the only rule that catches a value that was wrong from the first tick, which self-capture cannot see. A node that does not declare <param> answers NOT_FOUND, which is not a drift - so a misspelt name checks nothing at all. Once the sweep has covered every armed app - each one read, or given up on after repeated failed reads - and at least one of them answered, a pin that no app declares is warned about once, naming it. |
ignore |
string[] | [] |
Parameter-name globs never flagged. Applies to the self-captured set only, so it cannot silence an expect pin. A pattern of nothing but * matches every name and suppresses self-capture wholesale, which is indistinguishable at runtime from finding nothing; that combination warns at startup. |
max_reads_per_tick |
int | 8 |
Parameter-service ROUND TRIPS this detector may spend per tick, round-robin over nodes. Round trips, not calls: one call is several requests at the watched node, and the node pays for each of them. A baseline visit is charged 2 (a list, then one batched get) whatever the node’s parameter count; each expect pin is charged 3 (a list, a get and a descriptor read), or 2 on a node that does not declare it - see the table under “Coverage latency” for what each of those really costs the node. So a self-capture visit spends 2 of the budget and pinning N parameters spends up to 3N. It is an AMORTISED rate, not a per-tick cap: there is no pacing INSIDE a visit, so a node’s whole read set is issued back to back and the reader waits out the debt afterwards. Four pins against a budget of 8 fire 12 round trips inside one tick period and then idle for the rest of the next one; what the knob bounds is the sustained rate, not the burst. Accepted range 1..100000; anything else keeps the default and warns. |
prune_grace |
int | 60 |
Consecutive sweeps an app may be missing before its finding is dropped: the finding survives prune_grace absences and goes on the next one. Injected for every detector at plugin scope and overridable per detector; a value outside 0..3600 is rejected with a warning and this detector falls back to 2, not to the plugin default. The captured BASELINE has a shorter horizon of its own, and that one is NOT configurable - it is the compile-time kBaselineForgetGrace: the baseline is dropped on the third consecutive missed sweep, the two the reliability gate absorbs plus one, so an ordinary dropped discovery poll cannot re-capture a node on its already-drifted value. Setting prune_grace below 2 pulls that forward onto the sweep the finding itself is dropped on, so the two then go together. |
qos_mismatch keys
File truncated at 100 lines see the full file
Package Dependencies
System Dependencies
Dependant Packages
Launch files
Messages
Services
Plugins
Recent questions tagged ros2_medkit_graph_watchdog at Robotics Stack Exchange
Package Summary
| Version | 0.6.0 |
| License | Apache-2.0 |
| Build type | AMENT_CMAKE |
| Use | RECOMMENDED |
Repository Summary
| Checkout URI | https://github.com/selfpatch/ros2_medkit.git |
| VCS Type | git |
| VCS Version | main |
| Last Updated | 2026-08-17 |
| Dev Status | DEVELOPED |
| Released | RELEASED |
| Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
Maintainers
- bburda
Authors
ros2_medkit_graph_watchdog
Gateway plugin that detects silent faults in the ROS 2 graph: failures where every node is up, nothing logs an error, and the robot is still broken.
Detectors read the graph and raise faults through a ReportFault service client on the
gateway node; the faults surface via FaultManager on the gateway /faults API. That
client, like the lifecycle subscriptions below, lives in the plugin’s own callback group
and is driven by the plugin’s own executor from the tick thread, never by the gateway’s
ROS executor.
This package carries the plugin skeleton, the central reliability gate that holds raises
until the graph has quiesced, and four detectors, qos_mismatch, orphan,
param_drift and lifecycle_expectation. The remaining
silent-fault classes land in follow-up changes, each against its own issue; their fault
codes are already reserved in the frozen GRAPH_* namespace (see “Fault codes”).
Build
With the gateway built and sourced:
colcon build --packages-select ros2_medkit_graph_watchdog
colcon test --packages-select ros2_medkit_graph_watchdog && colcon test-result --verbose
Load into the gateway
ros2 run ros2_medkit_gateway gateway_node --ros-args \
-p plugins:="[graph_watchdog]" \
-p "plugins.graph_watchdog.path:=$(ros2 pkg prefix ros2_medkit_graph_watchdog)/lib/ros2_medkit_graph_watchdog/libros2_medkit_graph_watchdog.so"
Configuration (plugins.graph_watchdog.<key>)
| Key | Type | Default | Meaning |
|---|---|---|---|
tick_interval_ms |
int | 1000 |
Detector tick cadence. |
warmup_cycles |
int | 5 |
An entity must be continuously present for this many ticks before it arms. A mid-run restart re-warms it. Enforced centrally, see “Reliability (bringup-quiesce)” below. |
prune_grace |
int | 60 |
Default injected into every detector’s own config before configure(); a per-detector detectors.<id>.prune_grace overrides it. Used by detectors that keep per-key bookkeeping. |
detectors.<id>.mode |
string | bool | raise |
raise = push faults; advisory = observe, do not push; off = disabled. Bare off/no (YAML booleans) also disable; bare on/yes mean raise. |
detectors.<id>.<field> |
any | - | Per-detector thresholds, passed to that detector’s configure(). |
Nested delivery. Keys are written dotted in YAML or --ros-args, and the gateway
delivers them to the plugin as a nested object. A bare off is a YAML 1.1 boolean, so
mode: off arrives as false, not the string "off"; both forms disable the detector.
Unknown keys are reported. Anything under detectors.<id> that the detector does not
read produces one startup warning naming the key and listing the ones that exist, so a
typo like allow_list: for allowlist: cannot silently do nothing. A typo in the detector
id itself is warned about the same way, with the registered ids listed.
lifecycle_expectation keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. |
require_active |
string[] | [] |
Node names that must be in the active lifecycle state, each matched against a live node by its App::id, its full FQN (/ns/name), or the bare leaf of that FQN. A bare name matches that node in EVERY namespace (a fleet-wide “all controller_servers must be active”); use a full FQN to pin one robot’s. Both the bare and the FQN form match against the node’s stable FQN, so they survive the App::id renaming that a same-bare-name collision triggers; an entry written as an App::id also works, but can stop matching on a multi-robot graph for exactly that reason. Empty = the detector checks nothing and emits nothing. A require_active that is not a string array warns and is ignored; an empty-string entry warns and is skipped - never dropped silently, since the operator would go on believing the node is covered. |
grace |
int | 5 |
Consecutive not-active ticks a required node tolerates before being reported inactive - bringup time for a managed node to reach active (configure + activate) before the expectation is enforced. Counted per NODE, not per matching entry: a node named by both a bare-name and a full-FQN entry still advances its streak once per tick, so mixing the two documented forms cannot halve the grace that was configured. Past the absence grace the streak keeps advancing while the node is ABSENT, so a node that leaves the graph while violating is still confirmed. Accepted range 0..300 - five minutes at the shipped 1 s cadence, and already an extravagant allowance for a managed node to reach active. The upper end is a real bound rather than a formality: grace also decides how long a node that LEFT the graph while not-active sits unsettled in the tracker, and GRAPH_NODE_INACTIVE’s clear is withheld for EVERY node while it does, so at the old maximum of INT_MAX - 1 the fault could neither raise nor heal for about 24 days. The check runs on the wide integer, so a value that only fits after truncation is rejected rather than silently turned into a hair-trigger. Anything outside the range warns and keeps the default. |
prune_grace |
int | 60 |
Consecutive ticks an IDLE tracked node - both clocks at zero and no matured ownership, so nothing to lose - may stay ABSENT from the graph before its bookkeeping is reclaimed. A node carrying evidence is never reclaimed by age at all, however long it stays gone, so this key cannot erase a fault’s own state; the map is bounded instead by tracked_node_cap (see “Bounded by evidence, not by age”). Injected for every detector at plugin scope and overridable per detector; a value outside 0..3600 warns and this detector keeps its own default of 60. The range check runs on the wide integer, so an out-of-int-range value is rejected rather than truncated. The value is used as written - there is no grace + 1 clamp, because there is no longer anything for one to protect. |
tracked_node_cap |
int | 512 |
The most nodes this detector keeps bookkeeping for at once. It is what bounds the map, since evidence is never reclaimed by age (see “Bounded by evidence, not by age”): at the cap, idle entries are reclaimed first, then entries for DEPARTED nodes are collapsed into a count, and only if every tracked node is PRESENT and carrying evidence is a newly matched node refused - which withholds GRAPH_NODE_INACTIVE’s clear and is reported both in the log and on GET /x-medkit-watchdog. 512 is comfortably above every node in a realistic graph (a full Nav2 stack plus perception is roughly a hundred nodes; a ten-robot fleet sharing one domain a few hundred), so raising it is only needed where require_active legitimately matches more PRESENT nodes than that. Accepted range 1..16384 - 16384 is about 8 MB of bookkeeping at roughly 500 bytes per tracked node, and a deployment needing more distinct required-node identities alive at once has identity churn rather than a large fleet. Anything outside the range, including 0 (a cap of nothing would mean checking nothing), warns and keeps the default; the check runs on the wide integer, so an out-of-int-range value is rejected rather than truncated. |
orphan keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | raise |
As above. |
grace |
int | 10 |
Consecutive sweeps a candidate pair must hold before it is reported. A staged bringup or a restart makes one side missing for a short time, which looks exactly like a typo until the other side appears. |
max_edit_distance |
int | 1 |
How many character edits the LEAF (the part after the last /) may differ by and still count as the same intended name. Values outside 1..8 keep the default: past a few edits a “near miss” starts matching unrelated topics. |
namespace_edit_distance |
int | 0 |
The same budget for the namespace, spent separately. 0 means the namespace must match to the character, which is what makes /robot1/scan and /robot2/scan two robots rather than a typo. Raise it to catch a misspelled namespace, and read the warning under the table first. Values outside 0..8 keep the default. |
allowlist |
string[] | [] |
Topic names never reported. Exact match only, never a prefix, so allowlisting /r1/scan does not also silence /r2/scan. |
What this detector will not catch, on purpose. A pair whose names match once every run of
digits is collapsed is treated as an enumeration of sensors, not a typo: /lidar_1 next to
/lidar_2, each one-sided, is an ordinary state on a multi-sensor robot and reporting it would
be a false alarm on every such machine. Collapsing runs rather than comparing character
positions is what also covers /lidar_9 next to /lidar_10. The cost is that a typo which
is a digit stops being reported. An appended digit is not that case: /scan1 collapses to
/scan#, which is not /scan, so it is still reported.
A misspelled namespace is opt-in, and it can be expensive. /robott/scan against
/robot/scan is one edit, but in the namespace, so by default it is not reported. Setting
namespace_edit_distance: 1 reports it. Check the naming scheme before doing that: on a fleet
named with letters, /amr_a and /amr_b are also one edit apart, and every robot in the
fleet will be reported as a typo. A numbered fleet is safe, since /robot1 against /robot2
differs only in its numeric field and the rule above already spares it.
What it cannot catch at all. A namespace added or dropped by mistake, /scan against
/robot/scan, is six edits apart, so no budget that is still specific will reach it. Do not
rely on this detector for that class.
param_drift keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. A bare off arrives as a YAML boolean; both forms disable the detector. |
baseline |
bool | true |
Self-capture the value of every parameter not listed in expect, then flag later changes to it. false checks only what expect pins, and skips reading the full parameter list. |
expect.<param> |
any | - | Absolute expected value for <param>, checked on every node that has it. This is the only rule that catches a value that was wrong from the first tick, which self-capture cannot see. A node that does not declare <param> answers NOT_FOUND, which is not a drift - so a misspelt name checks nothing at all. Once the sweep has covered every armed app - each one read, or given up on after repeated failed reads - and at least one of them answered, a pin that no app declares is warned about once, naming it. |
ignore |
string[] | [] |
Parameter-name globs never flagged. Applies to the self-captured set only, so it cannot silence an expect pin. A pattern of nothing but * matches every name and suppresses self-capture wholesale, which is indistinguishable at runtime from finding nothing; that combination warns at startup. |
max_reads_per_tick |
int | 8 |
Parameter-service ROUND TRIPS this detector may spend per tick, round-robin over nodes. Round trips, not calls: one call is several requests at the watched node, and the node pays for each of them. A baseline visit is charged 2 (a list, then one batched get) whatever the node’s parameter count; each expect pin is charged 3 (a list, a get and a descriptor read), or 2 on a node that does not declare it - see the table under “Coverage latency” for what each of those really costs the node. So a self-capture visit spends 2 of the budget and pinning N parameters spends up to 3N. It is an AMORTISED rate, not a per-tick cap: there is no pacing INSIDE a visit, so a node’s whole read set is issued back to back and the reader waits out the debt afterwards. Four pins against a budget of 8 fire 12 round trips inside one tick period and then idle for the rest of the next one; what the knob bounds is the sustained rate, not the burst. Accepted range 1..100000; anything else keeps the default and warns. |
prune_grace |
int | 60 |
Consecutive sweeps an app may be missing before its finding is dropped: the finding survives prune_grace absences and goes on the next one. Injected for every detector at plugin scope and overridable per detector; a value outside 0..3600 is rejected with a warning and this detector falls back to 2, not to the plugin default. The captured BASELINE has a shorter horizon of its own, and that one is NOT configurable - it is the compile-time kBaselineForgetGrace: the baseline is dropped on the third consecutive missed sweep, the two the reliability gate absorbs plus one, so an ordinary dropped discovery poll cannot re-capture a node on its already-drifted value. Setting prune_grace below 2 pulls that forward onto the sweep the finding itself is dropped on, so the two then go together. |
qos_mismatch keys
File truncated at 100 lines see the full file
Package Dependencies
System Dependencies
Dependant Packages
Launch files
Messages
Services
Plugins
Recent questions tagged ros2_medkit_graph_watchdog at Robotics Stack Exchange
Package Summary
| Version | 0.6.0 |
| License | Apache-2.0 |
| Build type | AMENT_CMAKE |
| Use | RECOMMENDED |
Repository Summary
| Checkout URI | https://github.com/selfpatch/ros2_medkit.git |
| VCS Type | git |
| VCS Version | main |
| Last Updated | 2026-08-17 |
| Dev Status | DEVELOPED |
| Released | RELEASED |
| Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
Maintainers
- bburda
Authors
ros2_medkit_graph_watchdog
Gateway plugin that detects silent faults in the ROS 2 graph: failures where every node is up, nothing logs an error, and the robot is still broken.
Detectors read the graph and raise faults through a ReportFault service client on the
gateway node; the faults surface via FaultManager on the gateway /faults API. That
client, like the lifecycle subscriptions below, lives in the plugin’s own callback group
and is driven by the plugin’s own executor from the tick thread, never by the gateway’s
ROS executor.
This package carries the plugin skeleton, the central reliability gate that holds raises
until the graph has quiesced, and four detectors, qos_mismatch, orphan,
param_drift and lifecycle_expectation. The remaining
silent-fault classes land in follow-up changes, each against its own issue; their fault
codes are already reserved in the frozen GRAPH_* namespace (see “Fault codes”).
Build
With the gateway built and sourced:
colcon build --packages-select ros2_medkit_graph_watchdog
colcon test --packages-select ros2_medkit_graph_watchdog && colcon test-result --verbose
Load into the gateway
ros2 run ros2_medkit_gateway gateway_node --ros-args \
-p plugins:="[graph_watchdog]" \
-p "plugins.graph_watchdog.path:=$(ros2 pkg prefix ros2_medkit_graph_watchdog)/lib/ros2_medkit_graph_watchdog/libros2_medkit_graph_watchdog.so"
Configuration (plugins.graph_watchdog.<key>)
| Key | Type | Default | Meaning |
|---|---|---|---|
tick_interval_ms |
int | 1000 |
Detector tick cadence. |
warmup_cycles |
int | 5 |
An entity must be continuously present for this many ticks before it arms. A mid-run restart re-warms it. Enforced centrally, see “Reliability (bringup-quiesce)” below. |
prune_grace |
int | 60 |
Default injected into every detector’s own config before configure(); a per-detector detectors.<id>.prune_grace overrides it. Used by detectors that keep per-key bookkeeping. |
detectors.<id>.mode |
string | bool | raise |
raise = push faults; advisory = observe, do not push; off = disabled. Bare off/no (YAML booleans) also disable; bare on/yes mean raise. |
detectors.<id>.<field> |
any | - | Per-detector thresholds, passed to that detector’s configure(). |
Nested delivery. Keys are written dotted in YAML or --ros-args, and the gateway
delivers them to the plugin as a nested object. A bare off is a YAML 1.1 boolean, so
mode: off arrives as false, not the string "off"; both forms disable the detector.
Unknown keys are reported. Anything under detectors.<id> that the detector does not
read produces one startup warning naming the key and listing the ones that exist, so a
typo like allow_list: for allowlist: cannot silently do nothing. A typo in the detector
id itself is warned about the same way, with the registered ids listed.
lifecycle_expectation keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. |
require_active |
string[] | [] |
Node names that must be in the active lifecycle state, each matched against a live node by its App::id, its full FQN (/ns/name), or the bare leaf of that FQN. A bare name matches that node in EVERY namespace (a fleet-wide “all controller_servers must be active”); use a full FQN to pin one robot’s. Both the bare and the FQN form match against the node’s stable FQN, so they survive the App::id renaming that a same-bare-name collision triggers; an entry written as an App::id also works, but can stop matching on a multi-robot graph for exactly that reason. Empty = the detector checks nothing and emits nothing. A require_active that is not a string array warns and is ignored; an empty-string entry warns and is skipped - never dropped silently, since the operator would go on believing the node is covered. |
grace |
int | 5 |
Consecutive not-active ticks a required node tolerates before being reported inactive - bringup time for a managed node to reach active (configure + activate) before the expectation is enforced. Counted per NODE, not per matching entry: a node named by both a bare-name and a full-FQN entry still advances its streak once per tick, so mixing the two documented forms cannot halve the grace that was configured. Past the absence grace the streak keeps advancing while the node is ABSENT, so a node that leaves the graph while violating is still confirmed. Accepted range 0..300 - five minutes at the shipped 1 s cadence, and already an extravagant allowance for a managed node to reach active. The upper end is a real bound rather than a formality: grace also decides how long a node that LEFT the graph while not-active sits unsettled in the tracker, and GRAPH_NODE_INACTIVE’s clear is withheld for EVERY node while it does, so at the old maximum of INT_MAX - 1 the fault could neither raise nor heal for about 24 days. The check runs on the wide integer, so a value that only fits after truncation is rejected rather than silently turned into a hair-trigger. Anything outside the range warns and keeps the default. |
prune_grace |
int | 60 |
Consecutive ticks an IDLE tracked node - both clocks at zero and no matured ownership, so nothing to lose - may stay ABSENT from the graph before its bookkeeping is reclaimed. A node carrying evidence is never reclaimed by age at all, however long it stays gone, so this key cannot erase a fault’s own state; the map is bounded instead by tracked_node_cap (see “Bounded by evidence, not by age”). Injected for every detector at plugin scope and overridable per detector; a value outside 0..3600 warns and this detector keeps its own default of 60. The range check runs on the wide integer, so an out-of-int-range value is rejected rather than truncated. The value is used as written - there is no grace + 1 clamp, because there is no longer anything for one to protect. |
tracked_node_cap |
int | 512 |
The most nodes this detector keeps bookkeeping for at once. It is what bounds the map, since evidence is never reclaimed by age (see “Bounded by evidence, not by age”): at the cap, idle entries are reclaimed first, then entries for DEPARTED nodes are collapsed into a count, and only if every tracked node is PRESENT and carrying evidence is a newly matched node refused - which withholds GRAPH_NODE_INACTIVE’s clear and is reported both in the log and on GET /x-medkit-watchdog. 512 is comfortably above every node in a realistic graph (a full Nav2 stack plus perception is roughly a hundred nodes; a ten-robot fleet sharing one domain a few hundred), so raising it is only needed where require_active legitimately matches more PRESENT nodes than that. Accepted range 1..16384 - 16384 is about 8 MB of bookkeeping at roughly 500 bytes per tracked node, and a deployment needing more distinct required-node identities alive at once has identity churn rather than a large fleet. Anything outside the range, including 0 (a cap of nothing would mean checking nothing), warns and keeps the default; the check runs on the wide integer, so an out-of-int-range value is rejected rather than truncated. |
orphan keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | raise |
As above. |
grace |
int | 10 |
Consecutive sweeps a candidate pair must hold before it is reported. A staged bringup or a restart makes one side missing for a short time, which looks exactly like a typo until the other side appears. |
max_edit_distance |
int | 1 |
How many character edits the LEAF (the part after the last /) may differ by and still count as the same intended name. Values outside 1..8 keep the default: past a few edits a “near miss” starts matching unrelated topics. |
namespace_edit_distance |
int | 0 |
The same budget for the namespace, spent separately. 0 means the namespace must match to the character, which is what makes /robot1/scan and /robot2/scan two robots rather than a typo. Raise it to catch a misspelled namespace, and read the warning under the table first. Values outside 0..8 keep the default. |
allowlist |
string[] | [] |
Topic names never reported. Exact match only, never a prefix, so allowlisting /r1/scan does not also silence /r2/scan. |
What this detector will not catch, on purpose. A pair whose names match once every run of
digits is collapsed is treated as an enumeration of sensors, not a typo: /lidar_1 next to
/lidar_2, each one-sided, is an ordinary state on a multi-sensor robot and reporting it would
be a false alarm on every such machine. Collapsing runs rather than comparing character
positions is what also covers /lidar_9 next to /lidar_10. The cost is that a typo which
is a digit stops being reported. An appended digit is not that case: /scan1 collapses to
/scan#, which is not /scan, so it is still reported.
A misspelled namespace is opt-in, and it can be expensive. /robott/scan against
/robot/scan is one edit, but in the namespace, so by default it is not reported. Setting
namespace_edit_distance: 1 reports it. Check the naming scheme before doing that: on a fleet
named with letters, /amr_a and /amr_b are also one edit apart, and every robot in the
fleet will be reported as a typo. A numbered fleet is safe, since /robot1 against /robot2
differs only in its numeric field and the rule above already spares it.
What it cannot catch at all. A namespace added or dropped by mistake, /scan against
/robot/scan, is six edits apart, so no budget that is still specific will reach it. Do not
rely on this detector for that class.
param_drift keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. A bare off arrives as a YAML boolean; both forms disable the detector. |
baseline |
bool | true |
Self-capture the value of every parameter not listed in expect, then flag later changes to it. false checks only what expect pins, and skips reading the full parameter list. |
expect.<param> |
any | - | Absolute expected value for <param>, checked on every node that has it. This is the only rule that catches a value that was wrong from the first tick, which self-capture cannot see. A node that does not declare <param> answers NOT_FOUND, which is not a drift - so a misspelt name checks nothing at all. Once the sweep has covered every armed app - each one read, or given up on after repeated failed reads - and at least one of them answered, a pin that no app declares is warned about once, naming it. |
ignore |
string[] | [] |
Parameter-name globs never flagged. Applies to the self-captured set only, so it cannot silence an expect pin. A pattern of nothing but * matches every name and suppresses self-capture wholesale, which is indistinguishable at runtime from finding nothing; that combination warns at startup. |
max_reads_per_tick |
int | 8 |
Parameter-service ROUND TRIPS this detector may spend per tick, round-robin over nodes. Round trips, not calls: one call is several requests at the watched node, and the node pays for each of them. A baseline visit is charged 2 (a list, then one batched get) whatever the node’s parameter count; each expect pin is charged 3 (a list, a get and a descriptor read), or 2 on a node that does not declare it - see the table under “Coverage latency” for what each of those really costs the node. So a self-capture visit spends 2 of the budget and pinning N parameters spends up to 3N. It is an AMORTISED rate, not a per-tick cap: there is no pacing INSIDE a visit, so a node’s whole read set is issued back to back and the reader waits out the debt afterwards. Four pins against a budget of 8 fire 12 round trips inside one tick period and then idle for the rest of the next one; what the knob bounds is the sustained rate, not the burst. Accepted range 1..100000; anything else keeps the default and warns. |
prune_grace |
int | 60 |
Consecutive sweeps an app may be missing before its finding is dropped: the finding survives prune_grace absences and goes on the next one. Injected for every detector at plugin scope and overridable per detector; a value outside 0..3600 is rejected with a warning and this detector falls back to 2, not to the plugin default. The captured BASELINE has a shorter horizon of its own, and that one is NOT configurable - it is the compile-time kBaselineForgetGrace: the baseline is dropped on the third consecutive missed sweep, the two the reliability gate absorbs plus one, so an ordinary dropped discovery poll cannot re-capture a node on its already-drifted value. Setting prune_grace below 2 pulls that forward onto the sweep the finding itself is dropped on, so the two then go together. |
qos_mismatch keys
File truncated at 100 lines see the full file
Package Dependencies
System Dependencies
Dependant Packages
Launch files
Messages
Services
Plugins
Recent questions tagged ros2_medkit_graph_watchdog at Robotics Stack Exchange
Package Summary
| Version | 0.6.0 |
| License | Apache-2.0 |
| Build type | AMENT_CMAKE |
| Use | RECOMMENDED |
Repository Summary
| Checkout URI | https://github.com/selfpatch/ros2_medkit.git |
| VCS Type | git |
| VCS Version | main |
| Last Updated | 2026-08-17 |
| Dev Status | DEVELOPED |
| Released | RELEASED |
| Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
Maintainers
- bburda
Authors
ros2_medkit_graph_watchdog
Gateway plugin that detects silent faults in the ROS 2 graph: failures where every node is up, nothing logs an error, and the robot is still broken.
Detectors read the graph and raise faults through a ReportFault service client on the
gateway node; the faults surface via FaultManager on the gateway /faults API. That
client, like the lifecycle subscriptions below, lives in the plugin’s own callback group
and is driven by the plugin’s own executor from the tick thread, never by the gateway’s
ROS executor.
This package carries the plugin skeleton, the central reliability gate that holds raises
until the graph has quiesced, and four detectors, qos_mismatch, orphan,
param_drift and lifecycle_expectation. The remaining
silent-fault classes land in follow-up changes, each against its own issue; their fault
codes are already reserved in the frozen GRAPH_* namespace (see “Fault codes”).
Build
With the gateway built and sourced:
colcon build --packages-select ros2_medkit_graph_watchdog
colcon test --packages-select ros2_medkit_graph_watchdog && colcon test-result --verbose
Load into the gateway
ros2 run ros2_medkit_gateway gateway_node --ros-args \
-p plugins:="[graph_watchdog]" \
-p "plugins.graph_watchdog.path:=$(ros2 pkg prefix ros2_medkit_graph_watchdog)/lib/ros2_medkit_graph_watchdog/libros2_medkit_graph_watchdog.so"
Configuration (plugins.graph_watchdog.<key>)
| Key | Type | Default | Meaning |
|---|---|---|---|
tick_interval_ms |
int | 1000 |
Detector tick cadence. |
warmup_cycles |
int | 5 |
An entity must be continuously present for this many ticks before it arms. A mid-run restart re-warms it. Enforced centrally, see “Reliability (bringup-quiesce)” below. |
prune_grace |
int | 60 |
Default injected into every detector’s own config before configure(); a per-detector detectors.<id>.prune_grace overrides it. Used by detectors that keep per-key bookkeeping. |
detectors.<id>.mode |
string | bool | raise |
raise = push faults; advisory = observe, do not push; off = disabled. Bare off/no (YAML booleans) also disable; bare on/yes mean raise. |
detectors.<id>.<field> |
any | - | Per-detector thresholds, passed to that detector’s configure(). |
Nested delivery. Keys are written dotted in YAML or --ros-args, and the gateway
delivers them to the plugin as a nested object. A bare off is a YAML 1.1 boolean, so
mode: off arrives as false, not the string "off"; both forms disable the detector.
Unknown keys are reported. Anything under detectors.<id> that the detector does not
read produces one startup warning naming the key and listing the ones that exist, so a
typo like allow_list: for allowlist: cannot silently do nothing. A typo in the detector
id itself is warned about the same way, with the registered ids listed.
lifecycle_expectation keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. |
require_active |
string[] | [] |
Node names that must be in the active lifecycle state, each matched against a live node by its App::id, its full FQN (/ns/name), or the bare leaf of that FQN. A bare name matches that node in EVERY namespace (a fleet-wide “all controller_servers must be active”); use a full FQN to pin one robot’s. Both the bare and the FQN form match against the node’s stable FQN, so they survive the App::id renaming that a same-bare-name collision triggers; an entry written as an App::id also works, but can stop matching on a multi-robot graph for exactly that reason. Empty = the detector checks nothing and emits nothing. A require_active that is not a string array warns and is ignored; an empty-string entry warns and is skipped - never dropped silently, since the operator would go on believing the node is covered. |
grace |
int | 5 |
Consecutive not-active ticks a required node tolerates before being reported inactive - bringup time for a managed node to reach active (configure + activate) before the expectation is enforced. Counted per NODE, not per matching entry: a node named by both a bare-name and a full-FQN entry still advances its streak once per tick, so mixing the two documented forms cannot halve the grace that was configured. Past the absence grace the streak keeps advancing while the node is ABSENT, so a node that leaves the graph while violating is still confirmed. Accepted range 0..300 - five minutes at the shipped 1 s cadence, and already an extravagant allowance for a managed node to reach active. The upper end is a real bound rather than a formality: grace also decides how long a node that LEFT the graph while not-active sits unsettled in the tracker, and GRAPH_NODE_INACTIVE’s clear is withheld for EVERY node while it does, so at the old maximum of INT_MAX - 1 the fault could neither raise nor heal for about 24 days. The check runs on the wide integer, so a value that only fits after truncation is rejected rather than silently turned into a hair-trigger. Anything outside the range warns and keeps the default. |
prune_grace |
int | 60 |
Consecutive ticks an IDLE tracked node - both clocks at zero and no matured ownership, so nothing to lose - may stay ABSENT from the graph before its bookkeeping is reclaimed. A node carrying evidence is never reclaimed by age at all, however long it stays gone, so this key cannot erase a fault’s own state; the map is bounded instead by tracked_node_cap (see “Bounded by evidence, not by age”). Injected for every detector at plugin scope and overridable per detector; a value outside 0..3600 warns and this detector keeps its own default of 60. The range check runs on the wide integer, so an out-of-int-range value is rejected rather than truncated. The value is used as written - there is no grace + 1 clamp, because there is no longer anything for one to protect. |
tracked_node_cap |
int | 512 |
The most nodes this detector keeps bookkeeping for at once. It is what bounds the map, since evidence is never reclaimed by age (see “Bounded by evidence, not by age”): at the cap, idle entries are reclaimed first, then entries for DEPARTED nodes are collapsed into a count, and only if every tracked node is PRESENT and carrying evidence is a newly matched node refused - which withholds GRAPH_NODE_INACTIVE’s clear and is reported both in the log and on GET /x-medkit-watchdog. 512 is comfortably above every node in a realistic graph (a full Nav2 stack plus perception is roughly a hundred nodes; a ten-robot fleet sharing one domain a few hundred), so raising it is only needed where require_active legitimately matches more PRESENT nodes than that. Accepted range 1..16384 - 16384 is about 8 MB of bookkeeping at roughly 500 bytes per tracked node, and a deployment needing more distinct required-node identities alive at once has identity churn rather than a large fleet. Anything outside the range, including 0 (a cap of nothing would mean checking nothing), warns and keeps the default; the check runs on the wide integer, so an out-of-int-range value is rejected rather than truncated. |
orphan keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | raise |
As above. |
grace |
int | 10 |
Consecutive sweeps a candidate pair must hold before it is reported. A staged bringup or a restart makes one side missing for a short time, which looks exactly like a typo until the other side appears. |
max_edit_distance |
int | 1 |
How many character edits the LEAF (the part after the last /) may differ by and still count as the same intended name. Values outside 1..8 keep the default: past a few edits a “near miss” starts matching unrelated topics. |
namespace_edit_distance |
int | 0 |
The same budget for the namespace, spent separately. 0 means the namespace must match to the character, which is what makes /robot1/scan and /robot2/scan two robots rather than a typo. Raise it to catch a misspelled namespace, and read the warning under the table first. Values outside 0..8 keep the default. |
allowlist |
string[] | [] |
Topic names never reported. Exact match only, never a prefix, so allowlisting /r1/scan does not also silence /r2/scan. |
What this detector will not catch, on purpose. A pair whose names match once every run of
digits is collapsed is treated as an enumeration of sensors, not a typo: /lidar_1 next to
/lidar_2, each one-sided, is an ordinary state on a multi-sensor robot and reporting it would
be a false alarm on every such machine. Collapsing runs rather than comparing character
positions is what also covers /lidar_9 next to /lidar_10. The cost is that a typo which
is a digit stops being reported. An appended digit is not that case: /scan1 collapses to
/scan#, which is not /scan, so it is still reported.
A misspelled namespace is opt-in, and it can be expensive. /robott/scan against
/robot/scan is one edit, but in the namespace, so by default it is not reported. Setting
namespace_edit_distance: 1 reports it. Check the naming scheme before doing that: on a fleet
named with letters, /amr_a and /amr_b are also one edit apart, and every robot in the
fleet will be reported as a typo. A numbered fleet is safe, since /robot1 against /robot2
differs only in its numeric field and the rule above already spares it.
What it cannot catch at all. A namespace added or dropped by mistake, /scan against
/robot/scan, is six edits apart, so no budget that is still specific will reach it. Do not
rely on this detector for that class.
param_drift keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. A bare off arrives as a YAML boolean; both forms disable the detector. |
baseline |
bool | true |
Self-capture the value of every parameter not listed in expect, then flag later changes to it. false checks only what expect pins, and skips reading the full parameter list. |
expect.<param> |
any | - | Absolute expected value for <param>, checked on every node that has it. This is the only rule that catches a value that was wrong from the first tick, which self-capture cannot see. A node that does not declare <param> answers NOT_FOUND, which is not a drift - so a misspelt name checks nothing at all. Once the sweep has covered every armed app - each one read, or given up on after repeated failed reads - and at least one of them answered, a pin that no app declares is warned about once, naming it. |
ignore |
string[] | [] |
Parameter-name globs never flagged. Applies to the self-captured set only, so it cannot silence an expect pin. A pattern of nothing but * matches every name and suppresses self-capture wholesale, which is indistinguishable at runtime from finding nothing; that combination warns at startup. |
max_reads_per_tick |
int | 8 |
Parameter-service ROUND TRIPS this detector may spend per tick, round-robin over nodes. Round trips, not calls: one call is several requests at the watched node, and the node pays for each of them. A baseline visit is charged 2 (a list, then one batched get) whatever the node’s parameter count; each expect pin is charged 3 (a list, a get and a descriptor read), or 2 on a node that does not declare it - see the table under “Coverage latency” for what each of those really costs the node. So a self-capture visit spends 2 of the budget and pinning N parameters spends up to 3N. It is an AMORTISED rate, not a per-tick cap: there is no pacing INSIDE a visit, so a node’s whole read set is issued back to back and the reader waits out the debt afterwards. Four pins against a budget of 8 fire 12 round trips inside one tick period and then idle for the rest of the next one; what the knob bounds is the sustained rate, not the burst. Accepted range 1..100000; anything else keeps the default and warns. |
prune_grace |
int | 60 |
Consecutive sweeps an app may be missing before its finding is dropped: the finding survives prune_grace absences and goes on the next one. Injected for every detector at plugin scope and overridable per detector; a value outside 0..3600 is rejected with a warning and this detector falls back to 2, not to the plugin default. The captured BASELINE has a shorter horizon of its own, and that one is NOT configurable - it is the compile-time kBaselineForgetGrace: the baseline is dropped on the third consecutive missed sweep, the two the reliability gate absorbs plus one, so an ordinary dropped discovery poll cannot re-capture a node on its already-drifted value. Setting prune_grace below 2 pulls that forward onto the sweep the finding itself is dropped on, so the two then go together. |
qos_mismatch keys
File truncated at 100 lines see the full file
Package Dependencies
System Dependencies
Dependant Packages
Launch files
Messages
Services
Plugins
Recent questions tagged ros2_medkit_graph_watchdog at Robotics Stack Exchange
Package Summary
| Version | 0.6.0 |
| License | Apache-2.0 |
| Build type | AMENT_CMAKE |
| Use | RECOMMENDED |
Repository Summary
| Checkout URI | https://github.com/selfpatch/ros2_medkit.git |
| VCS Type | git |
| VCS Version | main |
| Last Updated | 2026-08-17 |
| Dev Status | DEVELOPED |
| Released | RELEASED |
| Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
Maintainers
- bburda
Authors
ros2_medkit_graph_watchdog
Gateway plugin that detects silent faults in the ROS 2 graph: failures where every node is up, nothing logs an error, and the robot is still broken.
Detectors read the graph and raise faults through a ReportFault service client on the
gateway node; the faults surface via FaultManager on the gateway /faults API. That
client, like the lifecycle subscriptions below, lives in the plugin’s own callback group
and is driven by the plugin’s own executor from the tick thread, never by the gateway’s
ROS executor.
This package carries the plugin skeleton, the central reliability gate that holds raises
until the graph has quiesced, and four detectors, qos_mismatch, orphan,
param_drift and lifecycle_expectation. The remaining
silent-fault classes land in follow-up changes, each against its own issue; their fault
codes are already reserved in the frozen GRAPH_* namespace (see “Fault codes”).
Build
With the gateway built and sourced:
colcon build --packages-select ros2_medkit_graph_watchdog
colcon test --packages-select ros2_medkit_graph_watchdog && colcon test-result --verbose
Load into the gateway
ros2 run ros2_medkit_gateway gateway_node --ros-args \
-p plugins:="[graph_watchdog]" \
-p "plugins.graph_watchdog.path:=$(ros2 pkg prefix ros2_medkit_graph_watchdog)/lib/ros2_medkit_graph_watchdog/libros2_medkit_graph_watchdog.so"
Configuration (plugins.graph_watchdog.<key>)
| Key | Type | Default | Meaning |
|---|---|---|---|
tick_interval_ms |
int | 1000 |
Detector tick cadence. |
warmup_cycles |
int | 5 |
An entity must be continuously present for this many ticks before it arms. A mid-run restart re-warms it. Enforced centrally, see “Reliability (bringup-quiesce)” below. |
prune_grace |
int | 60 |
Default injected into every detector’s own config before configure(); a per-detector detectors.<id>.prune_grace overrides it. Used by detectors that keep per-key bookkeeping. |
detectors.<id>.mode |
string | bool | raise |
raise = push faults; advisory = observe, do not push; off = disabled. Bare off/no (YAML booleans) also disable; bare on/yes mean raise. |
detectors.<id>.<field> |
any | - | Per-detector thresholds, passed to that detector’s configure(). |
Nested delivery. Keys are written dotted in YAML or --ros-args, and the gateway
delivers them to the plugin as a nested object. A bare off is a YAML 1.1 boolean, so
mode: off arrives as false, not the string "off"; both forms disable the detector.
Unknown keys are reported. Anything under detectors.<id> that the detector does not
read produces one startup warning naming the key and listing the ones that exist, so a
typo like allow_list: for allowlist: cannot silently do nothing. A typo in the detector
id itself is warned about the same way, with the registered ids listed.
lifecycle_expectation keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. |
require_active |
string[] | [] |
Node names that must be in the active lifecycle state, each matched against a live node by its App::id, its full FQN (/ns/name), or the bare leaf of that FQN. A bare name matches that node in EVERY namespace (a fleet-wide “all controller_servers must be active”); use a full FQN to pin one robot’s. Both the bare and the FQN form match against the node’s stable FQN, so they survive the App::id renaming that a same-bare-name collision triggers; an entry written as an App::id also works, but can stop matching on a multi-robot graph for exactly that reason. Empty = the detector checks nothing and emits nothing. A require_active that is not a string array warns and is ignored; an empty-string entry warns and is skipped - never dropped silently, since the operator would go on believing the node is covered. |
grace |
int | 5 |
Consecutive not-active ticks a required node tolerates before being reported inactive - bringup time for a managed node to reach active (configure + activate) before the expectation is enforced. Counted per NODE, not per matching entry: a node named by both a bare-name and a full-FQN entry still advances its streak once per tick, so mixing the two documented forms cannot halve the grace that was configured. Past the absence grace the streak keeps advancing while the node is ABSENT, so a node that leaves the graph while violating is still confirmed. Accepted range 0..300 - five minutes at the shipped 1 s cadence, and already an extravagant allowance for a managed node to reach active. The upper end is a real bound rather than a formality: grace also decides how long a node that LEFT the graph while not-active sits unsettled in the tracker, and GRAPH_NODE_INACTIVE’s clear is withheld for EVERY node while it does, so at the old maximum of INT_MAX - 1 the fault could neither raise nor heal for about 24 days. The check runs on the wide integer, so a value that only fits after truncation is rejected rather than silently turned into a hair-trigger. Anything outside the range warns and keeps the default. |
prune_grace |
int | 60 |
Consecutive ticks an IDLE tracked node - both clocks at zero and no matured ownership, so nothing to lose - may stay ABSENT from the graph before its bookkeeping is reclaimed. A node carrying evidence is never reclaimed by age at all, however long it stays gone, so this key cannot erase a fault’s own state; the map is bounded instead by tracked_node_cap (see “Bounded by evidence, not by age”). Injected for every detector at plugin scope and overridable per detector; a value outside 0..3600 warns and this detector keeps its own default of 60. The range check runs on the wide integer, so an out-of-int-range value is rejected rather than truncated. The value is used as written - there is no grace + 1 clamp, because there is no longer anything for one to protect. |
tracked_node_cap |
int | 512 |
The most nodes this detector keeps bookkeeping for at once. It is what bounds the map, since evidence is never reclaimed by age (see “Bounded by evidence, not by age”): at the cap, idle entries are reclaimed first, then entries for DEPARTED nodes are collapsed into a count, and only if every tracked node is PRESENT and carrying evidence is a newly matched node refused - which withholds GRAPH_NODE_INACTIVE’s clear and is reported both in the log and on GET /x-medkit-watchdog. 512 is comfortably above every node in a realistic graph (a full Nav2 stack plus perception is roughly a hundred nodes; a ten-robot fleet sharing one domain a few hundred), so raising it is only needed where require_active legitimately matches more PRESENT nodes than that. Accepted range 1..16384 - 16384 is about 8 MB of bookkeeping at roughly 500 bytes per tracked node, and a deployment needing more distinct required-node identities alive at once has identity churn rather than a large fleet. Anything outside the range, including 0 (a cap of nothing would mean checking nothing), warns and keeps the default; the check runs on the wide integer, so an out-of-int-range value is rejected rather than truncated. |
orphan keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | raise |
As above. |
grace |
int | 10 |
Consecutive sweeps a candidate pair must hold before it is reported. A staged bringup or a restart makes one side missing for a short time, which looks exactly like a typo until the other side appears. |
max_edit_distance |
int | 1 |
How many character edits the LEAF (the part after the last /) may differ by and still count as the same intended name. Values outside 1..8 keep the default: past a few edits a “near miss” starts matching unrelated topics. |
namespace_edit_distance |
int | 0 |
The same budget for the namespace, spent separately. 0 means the namespace must match to the character, which is what makes /robot1/scan and /robot2/scan two robots rather than a typo. Raise it to catch a misspelled namespace, and read the warning under the table first. Values outside 0..8 keep the default. |
allowlist |
string[] | [] |
Topic names never reported. Exact match only, never a prefix, so allowlisting /r1/scan does not also silence /r2/scan. |
What this detector will not catch, on purpose. A pair whose names match once every run of
digits is collapsed is treated as an enumeration of sensors, not a typo: /lidar_1 next to
/lidar_2, each one-sided, is an ordinary state on a multi-sensor robot and reporting it would
be a false alarm on every such machine. Collapsing runs rather than comparing character
positions is what also covers /lidar_9 next to /lidar_10. The cost is that a typo which
is a digit stops being reported. An appended digit is not that case: /scan1 collapses to
/scan#, which is not /scan, so it is still reported.
A misspelled namespace is opt-in, and it can be expensive. /robott/scan against
/robot/scan is one edit, but in the namespace, so by default it is not reported. Setting
namespace_edit_distance: 1 reports it. Check the naming scheme before doing that: on a fleet
named with letters, /amr_a and /amr_b are also one edit apart, and every robot in the
fleet will be reported as a typo. A numbered fleet is safe, since /robot1 against /robot2
differs only in its numeric field and the rule above already spares it.
What it cannot catch at all. A namespace added or dropped by mistake, /scan against
/robot/scan, is six edits apart, so no budget that is still specific will reach it. Do not
rely on this detector for that class.
param_drift keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. A bare off arrives as a YAML boolean; both forms disable the detector. |
baseline |
bool | true |
Self-capture the value of every parameter not listed in expect, then flag later changes to it. false checks only what expect pins, and skips reading the full parameter list. |
expect.<param> |
any | - | Absolute expected value for <param>, checked on every node that has it. This is the only rule that catches a value that was wrong from the first tick, which self-capture cannot see. A node that does not declare <param> answers NOT_FOUND, which is not a drift - so a misspelt name checks nothing at all. Once the sweep has covered every armed app - each one read, or given up on after repeated failed reads - and at least one of them answered, a pin that no app declares is warned about once, naming it. |
ignore |
string[] | [] |
Parameter-name globs never flagged. Applies to the self-captured set only, so it cannot silence an expect pin. A pattern of nothing but * matches every name and suppresses self-capture wholesale, which is indistinguishable at runtime from finding nothing; that combination warns at startup. |
max_reads_per_tick |
int | 8 |
Parameter-service ROUND TRIPS this detector may spend per tick, round-robin over nodes. Round trips, not calls: one call is several requests at the watched node, and the node pays for each of them. A baseline visit is charged 2 (a list, then one batched get) whatever the node’s parameter count; each expect pin is charged 3 (a list, a get and a descriptor read), or 2 on a node that does not declare it - see the table under “Coverage latency” for what each of those really costs the node. So a self-capture visit spends 2 of the budget and pinning N parameters spends up to 3N. It is an AMORTISED rate, not a per-tick cap: there is no pacing INSIDE a visit, so a node’s whole read set is issued back to back and the reader waits out the debt afterwards. Four pins against a budget of 8 fire 12 round trips inside one tick period and then idle for the rest of the next one; what the knob bounds is the sustained rate, not the burst. Accepted range 1..100000; anything else keeps the default and warns. |
prune_grace |
int | 60 |
Consecutive sweeps an app may be missing before its finding is dropped: the finding survives prune_grace absences and goes on the next one. Injected for every detector at plugin scope and overridable per detector; a value outside 0..3600 is rejected with a warning and this detector falls back to 2, not to the plugin default. The captured BASELINE has a shorter horizon of its own, and that one is NOT configurable - it is the compile-time kBaselineForgetGrace: the baseline is dropped on the third consecutive missed sweep, the two the reliability gate absorbs plus one, so an ordinary dropped discovery poll cannot re-capture a node on its already-drifted value. Setting prune_grace below 2 pulls that forward onto the sweep the finding itself is dropped on, so the two then go together. |
qos_mismatch keys
File truncated at 100 lines see the full file
Package Dependencies
System Dependencies
Dependant Packages
Launch files
Messages
Services
Plugins
Recent questions tagged ros2_medkit_graph_watchdog at Robotics Stack Exchange
Package Summary
| Version | 0.6.0 |
| License | Apache-2.0 |
| Build type | AMENT_CMAKE |
| Use | RECOMMENDED |
Repository Summary
| Checkout URI | https://github.com/selfpatch/ros2_medkit.git |
| VCS Type | git |
| VCS Version | main |
| Last Updated | 2026-08-17 |
| Dev Status | DEVELOPED |
| Released | RELEASED |
| Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
Maintainers
- bburda
Authors
ros2_medkit_graph_watchdog
Gateway plugin that detects silent faults in the ROS 2 graph: failures where every node is up, nothing logs an error, and the robot is still broken.
Detectors read the graph and raise faults through a ReportFault service client on the
gateway node; the faults surface via FaultManager on the gateway /faults API. That
client, like the lifecycle subscriptions below, lives in the plugin’s own callback group
and is driven by the plugin’s own executor from the tick thread, never by the gateway’s
ROS executor.
This package carries the plugin skeleton, the central reliability gate that holds raises
until the graph has quiesced, and four detectors, qos_mismatch, orphan,
param_drift and lifecycle_expectation. The remaining
silent-fault classes land in follow-up changes, each against its own issue; their fault
codes are already reserved in the frozen GRAPH_* namespace (see “Fault codes”).
Build
With the gateway built and sourced:
colcon build --packages-select ros2_medkit_graph_watchdog
colcon test --packages-select ros2_medkit_graph_watchdog && colcon test-result --verbose
Load into the gateway
ros2 run ros2_medkit_gateway gateway_node --ros-args \
-p plugins:="[graph_watchdog]" \
-p "plugins.graph_watchdog.path:=$(ros2 pkg prefix ros2_medkit_graph_watchdog)/lib/ros2_medkit_graph_watchdog/libros2_medkit_graph_watchdog.so"
Configuration (plugins.graph_watchdog.<key>)
| Key | Type | Default | Meaning |
|---|---|---|---|
tick_interval_ms |
int | 1000 |
Detector tick cadence. |
warmup_cycles |
int | 5 |
An entity must be continuously present for this many ticks before it arms. A mid-run restart re-warms it. Enforced centrally, see “Reliability (bringup-quiesce)” below. |
prune_grace |
int | 60 |
Default injected into every detector’s own config before configure(); a per-detector detectors.<id>.prune_grace overrides it. Used by detectors that keep per-key bookkeeping. |
detectors.<id>.mode |
string | bool | raise |
raise = push faults; advisory = observe, do not push; off = disabled. Bare off/no (YAML booleans) also disable; bare on/yes mean raise. |
detectors.<id>.<field> |
any | - | Per-detector thresholds, passed to that detector’s configure(). |
Nested delivery. Keys are written dotted in YAML or --ros-args, and the gateway
delivers them to the plugin as a nested object. A bare off is a YAML 1.1 boolean, so
mode: off arrives as false, not the string "off"; both forms disable the detector.
Unknown keys are reported. Anything under detectors.<id> that the detector does not
read produces one startup warning naming the key and listing the ones that exist, so a
typo like allow_list: for allowlist: cannot silently do nothing. A typo in the detector
id itself is warned about the same way, with the registered ids listed.
lifecycle_expectation keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. |
require_active |
string[] | [] |
Node names that must be in the active lifecycle state, each matched against a live node by its App::id, its full FQN (/ns/name), or the bare leaf of that FQN. A bare name matches that node in EVERY namespace (a fleet-wide “all controller_servers must be active”); use a full FQN to pin one robot’s. Both the bare and the FQN form match against the node’s stable FQN, so they survive the App::id renaming that a same-bare-name collision triggers; an entry written as an App::id also works, but can stop matching on a multi-robot graph for exactly that reason. Empty = the detector checks nothing and emits nothing. A require_active that is not a string array warns and is ignored; an empty-string entry warns and is skipped - never dropped silently, since the operator would go on believing the node is covered. |
grace |
int | 5 |
Consecutive not-active ticks a required node tolerates before being reported inactive - bringup time for a managed node to reach active (configure + activate) before the expectation is enforced. Counted per NODE, not per matching entry: a node named by both a bare-name and a full-FQN entry still advances its streak once per tick, so mixing the two documented forms cannot halve the grace that was configured. Past the absence grace the streak keeps advancing while the node is ABSENT, so a node that leaves the graph while violating is still confirmed. Accepted range 0..300 - five minutes at the shipped 1 s cadence, and already an extravagant allowance for a managed node to reach active. The upper end is a real bound rather than a formality: grace also decides how long a node that LEFT the graph while not-active sits unsettled in the tracker, and GRAPH_NODE_INACTIVE’s clear is withheld for EVERY node while it does, so at the old maximum of INT_MAX - 1 the fault could neither raise nor heal for about 24 days. The check runs on the wide integer, so a value that only fits after truncation is rejected rather than silently turned into a hair-trigger. Anything outside the range warns and keeps the default. |
prune_grace |
int | 60 |
Consecutive ticks an IDLE tracked node - both clocks at zero and no matured ownership, so nothing to lose - may stay ABSENT from the graph before its bookkeeping is reclaimed. A node carrying evidence is never reclaimed by age at all, however long it stays gone, so this key cannot erase a fault’s own state; the map is bounded instead by tracked_node_cap (see “Bounded by evidence, not by age”). Injected for every detector at plugin scope and overridable per detector; a value outside 0..3600 warns and this detector keeps its own default of 60. The range check runs on the wide integer, so an out-of-int-range value is rejected rather than truncated. The value is used as written - there is no grace + 1 clamp, because there is no longer anything for one to protect. |
tracked_node_cap |
int | 512 |
The most nodes this detector keeps bookkeeping for at once. It is what bounds the map, since evidence is never reclaimed by age (see “Bounded by evidence, not by age”): at the cap, idle entries are reclaimed first, then entries for DEPARTED nodes are collapsed into a count, and only if every tracked node is PRESENT and carrying evidence is a newly matched node refused - which withholds GRAPH_NODE_INACTIVE’s clear and is reported both in the log and on GET /x-medkit-watchdog. 512 is comfortably above every node in a realistic graph (a full Nav2 stack plus perception is roughly a hundred nodes; a ten-robot fleet sharing one domain a few hundred), so raising it is only needed where require_active legitimately matches more PRESENT nodes than that. Accepted range 1..16384 - 16384 is about 8 MB of bookkeeping at roughly 500 bytes per tracked node, and a deployment needing more distinct required-node identities alive at once has identity churn rather than a large fleet. Anything outside the range, including 0 (a cap of nothing would mean checking nothing), warns and keeps the default; the check runs on the wide integer, so an out-of-int-range value is rejected rather than truncated. |
orphan keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | raise |
As above. |
grace |
int | 10 |
Consecutive sweeps a candidate pair must hold before it is reported. A staged bringup or a restart makes one side missing for a short time, which looks exactly like a typo until the other side appears. |
max_edit_distance |
int | 1 |
How many character edits the LEAF (the part after the last /) may differ by and still count as the same intended name. Values outside 1..8 keep the default: past a few edits a “near miss” starts matching unrelated topics. |
namespace_edit_distance |
int | 0 |
The same budget for the namespace, spent separately. 0 means the namespace must match to the character, which is what makes /robot1/scan and /robot2/scan two robots rather than a typo. Raise it to catch a misspelled namespace, and read the warning under the table first. Values outside 0..8 keep the default. |
allowlist |
string[] | [] |
Topic names never reported. Exact match only, never a prefix, so allowlisting /r1/scan does not also silence /r2/scan. |
What this detector will not catch, on purpose. A pair whose names match once every run of
digits is collapsed is treated as an enumeration of sensors, not a typo: /lidar_1 next to
/lidar_2, each one-sided, is an ordinary state on a multi-sensor robot and reporting it would
be a false alarm on every such machine. Collapsing runs rather than comparing character
positions is what also covers /lidar_9 next to /lidar_10. The cost is that a typo which
is a digit stops being reported. An appended digit is not that case: /scan1 collapses to
/scan#, which is not /scan, so it is still reported.
A misspelled namespace is opt-in, and it can be expensive. /robott/scan against
/robot/scan is one edit, but in the namespace, so by default it is not reported. Setting
namespace_edit_distance: 1 reports it. Check the naming scheme before doing that: on a fleet
named with letters, /amr_a and /amr_b are also one edit apart, and every robot in the
fleet will be reported as a typo. A numbered fleet is safe, since /robot1 against /robot2
differs only in its numeric field and the rule above already spares it.
What it cannot catch at all. A namespace added or dropped by mistake, /scan against
/robot/scan, is six edits apart, so no budget that is still specific will reach it. Do not
rely on this detector for that class.
param_drift keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. A bare off arrives as a YAML boolean; both forms disable the detector. |
baseline |
bool | true |
Self-capture the value of every parameter not listed in expect, then flag later changes to it. false checks only what expect pins, and skips reading the full parameter list. |
expect.<param> |
any | - | Absolute expected value for <param>, checked on every node that has it. This is the only rule that catches a value that was wrong from the first tick, which self-capture cannot see. A node that does not declare <param> answers NOT_FOUND, which is not a drift - so a misspelt name checks nothing at all. Once the sweep has covered every armed app - each one read, or given up on after repeated failed reads - and at least one of them answered, a pin that no app declares is warned about once, naming it. |
ignore |
string[] | [] |
Parameter-name globs never flagged. Applies to the self-captured set only, so it cannot silence an expect pin. A pattern of nothing but * matches every name and suppresses self-capture wholesale, which is indistinguishable at runtime from finding nothing; that combination warns at startup. |
max_reads_per_tick |
int | 8 |
Parameter-service ROUND TRIPS this detector may spend per tick, round-robin over nodes. Round trips, not calls: one call is several requests at the watched node, and the node pays for each of them. A baseline visit is charged 2 (a list, then one batched get) whatever the node’s parameter count; each expect pin is charged 3 (a list, a get and a descriptor read), or 2 on a node that does not declare it - see the table under “Coverage latency” for what each of those really costs the node. So a self-capture visit spends 2 of the budget and pinning N parameters spends up to 3N. It is an AMORTISED rate, not a per-tick cap: there is no pacing INSIDE a visit, so a node’s whole read set is issued back to back and the reader waits out the debt afterwards. Four pins against a budget of 8 fire 12 round trips inside one tick period and then idle for the rest of the next one; what the knob bounds is the sustained rate, not the burst. Accepted range 1..100000; anything else keeps the default and warns. |
prune_grace |
int | 60 |
Consecutive sweeps an app may be missing before its finding is dropped: the finding survives prune_grace absences and goes on the next one. Injected for every detector at plugin scope and overridable per detector; a value outside 0..3600 is rejected with a warning and this detector falls back to 2, not to the plugin default. The captured BASELINE has a shorter horizon of its own, and that one is NOT configurable - it is the compile-time kBaselineForgetGrace: the baseline is dropped on the third consecutive missed sweep, the two the reliability gate absorbs plus one, so an ordinary dropped discovery poll cannot re-capture a node on its already-drifted value. Setting prune_grace below 2 pulls that forward onto the sweep the finding itself is dropped on, so the two then go together. |
qos_mismatch keys
File truncated at 100 lines see the full file
Package Dependencies
System Dependencies
Dependant Packages
Launch files
Messages
Services
Plugins
Recent questions tagged ros2_medkit_graph_watchdog at Robotics Stack Exchange
Package Summary
| Version | 0.6.0 |
| License | Apache-2.0 |
| Build type | AMENT_CMAKE |
| Use | RECOMMENDED |
Repository Summary
| Checkout URI | https://github.com/selfpatch/ros2_medkit.git |
| VCS Type | git |
| VCS Version | main |
| Last Updated | 2026-08-17 |
| Dev Status | DEVELOPED |
| Released | RELEASED |
| Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
Maintainers
- bburda
Authors
ros2_medkit_graph_watchdog
Gateway plugin that detects silent faults in the ROS 2 graph: failures where every node is up, nothing logs an error, and the robot is still broken.
Detectors read the graph and raise faults through a ReportFault service client on the
gateway node; the faults surface via FaultManager on the gateway /faults API. That
client, like the lifecycle subscriptions below, lives in the plugin’s own callback group
and is driven by the plugin’s own executor from the tick thread, never by the gateway’s
ROS executor.
This package carries the plugin skeleton, the central reliability gate that holds raises
until the graph has quiesced, and four detectors, qos_mismatch, orphan,
param_drift and lifecycle_expectation. The remaining
silent-fault classes land in follow-up changes, each against its own issue; their fault
codes are already reserved in the frozen GRAPH_* namespace (see “Fault codes”).
Build
With the gateway built and sourced:
colcon build --packages-select ros2_medkit_graph_watchdog
colcon test --packages-select ros2_medkit_graph_watchdog && colcon test-result --verbose
Load into the gateway
ros2 run ros2_medkit_gateway gateway_node --ros-args \
-p plugins:="[graph_watchdog]" \
-p "plugins.graph_watchdog.path:=$(ros2 pkg prefix ros2_medkit_graph_watchdog)/lib/ros2_medkit_graph_watchdog/libros2_medkit_graph_watchdog.so"
Configuration (plugins.graph_watchdog.<key>)
| Key | Type | Default | Meaning |
|---|---|---|---|
tick_interval_ms |
int | 1000 |
Detector tick cadence. |
warmup_cycles |
int | 5 |
An entity must be continuously present for this many ticks before it arms. A mid-run restart re-warms it. Enforced centrally, see “Reliability (bringup-quiesce)” below. |
prune_grace |
int | 60 |
Default injected into every detector’s own config before configure(); a per-detector detectors.<id>.prune_grace overrides it. Used by detectors that keep per-key bookkeeping. |
detectors.<id>.mode |
string | bool | raise |
raise = push faults; advisory = observe, do not push; off = disabled. Bare off/no (YAML booleans) also disable; bare on/yes mean raise. |
detectors.<id>.<field> |
any | - | Per-detector thresholds, passed to that detector’s configure(). |
Nested delivery. Keys are written dotted in YAML or --ros-args, and the gateway
delivers them to the plugin as a nested object. A bare off is a YAML 1.1 boolean, so
mode: off arrives as false, not the string "off"; both forms disable the detector.
Unknown keys are reported. Anything under detectors.<id> that the detector does not
read produces one startup warning naming the key and listing the ones that exist, so a
typo like allow_list: for allowlist: cannot silently do nothing. A typo in the detector
id itself is warned about the same way, with the registered ids listed.
lifecycle_expectation keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. |
require_active |
string[] | [] |
Node names that must be in the active lifecycle state, each matched against a live node by its App::id, its full FQN (/ns/name), or the bare leaf of that FQN. A bare name matches that node in EVERY namespace (a fleet-wide “all controller_servers must be active”); use a full FQN to pin one robot’s. Both the bare and the FQN form match against the node’s stable FQN, so they survive the App::id renaming that a same-bare-name collision triggers; an entry written as an App::id also works, but can stop matching on a multi-robot graph for exactly that reason. Empty = the detector checks nothing and emits nothing. A require_active that is not a string array warns and is ignored; an empty-string entry warns and is skipped - never dropped silently, since the operator would go on believing the node is covered. |
grace |
int | 5 |
Consecutive not-active ticks a required node tolerates before being reported inactive - bringup time for a managed node to reach active (configure + activate) before the expectation is enforced. Counted per NODE, not per matching entry: a node named by both a bare-name and a full-FQN entry still advances its streak once per tick, so mixing the two documented forms cannot halve the grace that was configured. Past the absence grace the streak keeps advancing while the node is ABSENT, so a node that leaves the graph while violating is still confirmed. Accepted range 0..300 - five minutes at the shipped 1 s cadence, and already an extravagant allowance for a managed node to reach active. The upper end is a real bound rather than a formality: grace also decides how long a node that LEFT the graph while not-active sits unsettled in the tracker, and GRAPH_NODE_INACTIVE’s clear is withheld for EVERY node while it does, so at the old maximum of INT_MAX - 1 the fault could neither raise nor heal for about 24 days. The check runs on the wide integer, so a value that only fits after truncation is rejected rather than silently turned into a hair-trigger. Anything outside the range warns and keeps the default. |
prune_grace |
int | 60 |
Consecutive ticks an IDLE tracked node - both clocks at zero and no matured ownership, so nothing to lose - may stay ABSENT from the graph before its bookkeeping is reclaimed. A node carrying evidence is never reclaimed by age at all, however long it stays gone, so this key cannot erase a fault’s own state; the map is bounded instead by tracked_node_cap (see “Bounded by evidence, not by age”). Injected for every detector at plugin scope and overridable per detector; a value outside 0..3600 warns and this detector keeps its own default of 60. The range check runs on the wide integer, so an out-of-int-range value is rejected rather than truncated. The value is used as written - there is no grace + 1 clamp, because there is no longer anything for one to protect. |
tracked_node_cap |
int | 512 |
The most nodes this detector keeps bookkeeping for at once. It is what bounds the map, since evidence is never reclaimed by age (see “Bounded by evidence, not by age”): at the cap, idle entries are reclaimed first, then entries for DEPARTED nodes are collapsed into a count, and only if every tracked node is PRESENT and carrying evidence is a newly matched node refused - which withholds GRAPH_NODE_INACTIVE’s clear and is reported both in the log and on GET /x-medkit-watchdog. 512 is comfortably above every node in a realistic graph (a full Nav2 stack plus perception is roughly a hundred nodes; a ten-robot fleet sharing one domain a few hundred), so raising it is only needed where require_active legitimately matches more PRESENT nodes than that. Accepted range 1..16384 - 16384 is about 8 MB of bookkeeping at roughly 500 bytes per tracked node, and a deployment needing more distinct required-node identities alive at once has identity churn rather than a large fleet. Anything outside the range, including 0 (a cap of nothing would mean checking nothing), warns and keeps the default; the check runs on the wide integer, so an out-of-int-range value is rejected rather than truncated. |
orphan keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | raise |
As above. |
grace |
int | 10 |
Consecutive sweeps a candidate pair must hold before it is reported. A staged bringup or a restart makes one side missing for a short time, which looks exactly like a typo until the other side appears. |
max_edit_distance |
int | 1 |
How many character edits the LEAF (the part after the last /) may differ by and still count as the same intended name. Values outside 1..8 keep the default: past a few edits a “near miss” starts matching unrelated topics. |
namespace_edit_distance |
int | 0 |
The same budget for the namespace, spent separately. 0 means the namespace must match to the character, which is what makes /robot1/scan and /robot2/scan two robots rather than a typo. Raise it to catch a misspelled namespace, and read the warning under the table first. Values outside 0..8 keep the default. |
allowlist |
string[] | [] |
Topic names never reported. Exact match only, never a prefix, so allowlisting /r1/scan does not also silence /r2/scan. |
What this detector will not catch, on purpose. A pair whose names match once every run of
digits is collapsed is treated as an enumeration of sensors, not a typo: /lidar_1 next to
/lidar_2, each one-sided, is an ordinary state on a multi-sensor robot and reporting it would
be a false alarm on every such machine. Collapsing runs rather than comparing character
positions is what also covers /lidar_9 next to /lidar_10. The cost is that a typo which
is a digit stops being reported. An appended digit is not that case: /scan1 collapses to
/scan#, which is not /scan, so it is still reported.
A misspelled namespace is opt-in, and it can be expensive. /robott/scan against
/robot/scan is one edit, but in the namespace, so by default it is not reported. Setting
namespace_edit_distance: 1 reports it. Check the naming scheme before doing that: on a fleet
named with letters, /amr_a and /amr_b are also one edit apart, and every robot in the
fleet will be reported as a typo. A numbered fleet is safe, since /robot1 against /robot2
differs only in its numeric field and the rule above already spares it.
What it cannot catch at all. A namespace added or dropped by mistake, /scan against
/robot/scan, is six edits apart, so no budget that is still specific will reach it. Do not
rely on this detector for that class.
param_drift keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. A bare off arrives as a YAML boolean; both forms disable the detector. |
baseline |
bool | true |
Self-capture the value of every parameter not listed in expect, then flag later changes to it. false checks only what expect pins, and skips reading the full parameter list. |
expect.<param> |
any | - | Absolute expected value for <param>, checked on every node that has it. This is the only rule that catches a value that was wrong from the first tick, which self-capture cannot see. A node that does not declare <param> answers NOT_FOUND, which is not a drift - so a misspelt name checks nothing at all. Once the sweep has covered every armed app - each one read, or given up on after repeated failed reads - and at least one of them answered, a pin that no app declares is warned about once, naming it. |
ignore |
string[] | [] |
Parameter-name globs never flagged. Applies to the self-captured set only, so it cannot silence an expect pin. A pattern of nothing but * matches every name and suppresses self-capture wholesale, which is indistinguishable at runtime from finding nothing; that combination warns at startup. |
max_reads_per_tick |
int | 8 |
Parameter-service ROUND TRIPS this detector may spend per tick, round-robin over nodes. Round trips, not calls: one call is several requests at the watched node, and the node pays for each of them. A baseline visit is charged 2 (a list, then one batched get) whatever the node’s parameter count; each expect pin is charged 3 (a list, a get and a descriptor read), or 2 on a node that does not declare it - see the table under “Coverage latency” for what each of those really costs the node. So a self-capture visit spends 2 of the budget and pinning N parameters spends up to 3N. It is an AMORTISED rate, not a per-tick cap: there is no pacing INSIDE a visit, so a node’s whole read set is issued back to back and the reader waits out the debt afterwards. Four pins against a budget of 8 fire 12 round trips inside one tick period and then idle for the rest of the next one; what the knob bounds is the sustained rate, not the burst. Accepted range 1..100000; anything else keeps the default and warns. |
prune_grace |
int | 60 |
Consecutive sweeps an app may be missing before its finding is dropped: the finding survives prune_grace absences and goes on the next one. Injected for every detector at plugin scope and overridable per detector; a value outside 0..3600 is rejected with a warning and this detector falls back to 2, not to the plugin default. The captured BASELINE has a shorter horizon of its own, and that one is NOT configurable - it is the compile-time kBaselineForgetGrace: the baseline is dropped on the third consecutive missed sweep, the two the reliability gate absorbs plus one, so an ordinary dropped discovery poll cannot re-capture a node on its already-drifted value. Setting prune_grace below 2 pulls that forward onto the sweep the finding itself is dropped on, so the two then go together. |
qos_mismatch keys
File truncated at 100 lines see the full file
Package Dependencies
System Dependencies
Dependant Packages
Launch files
Messages
Services
Plugins
Recent questions tagged ros2_medkit_graph_watchdog at Robotics Stack Exchange
Package Summary
| Version | 0.6.0 |
| License | Apache-2.0 |
| Build type | AMENT_CMAKE |
| Use | RECOMMENDED |
Repository Summary
| Checkout URI | https://github.com/selfpatch/ros2_medkit.git |
| VCS Type | git |
| VCS Version | main |
| Last Updated | 2026-08-17 |
| Dev Status | DEVELOPED |
| Released | RELEASED |
| Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
Maintainers
- bburda
Authors
ros2_medkit_graph_watchdog
Gateway plugin that detects silent faults in the ROS 2 graph: failures where every node is up, nothing logs an error, and the robot is still broken.
Detectors read the graph and raise faults through a ReportFault service client on the
gateway node; the faults surface via FaultManager on the gateway /faults API. That
client, like the lifecycle subscriptions below, lives in the plugin’s own callback group
and is driven by the plugin’s own executor from the tick thread, never by the gateway’s
ROS executor.
This package carries the plugin skeleton, the central reliability gate that holds raises
until the graph has quiesced, and four detectors, qos_mismatch, orphan,
param_drift and lifecycle_expectation. The remaining
silent-fault classes land in follow-up changes, each against its own issue; their fault
codes are already reserved in the frozen GRAPH_* namespace (see “Fault codes”).
Build
With the gateway built and sourced:
colcon build --packages-select ros2_medkit_graph_watchdog
colcon test --packages-select ros2_medkit_graph_watchdog && colcon test-result --verbose
Load into the gateway
ros2 run ros2_medkit_gateway gateway_node --ros-args \
-p plugins:="[graph_watchdog]" \
-p "plugins.graph_watchdog.path:=$(ros2 pkg prefix ros2_medkit_graph_watchdog)/lib/ros2_medkit_graph_watchdog/libros2_medkit_graph_watchdog.so"
Configuration (plugins.graph_watchdog.<key>)
| Key | Type | Default | Meaning |
|---|---|---|---|
tick_interval_ms |
int | 1000 |
Detector tick cadence. |
warmup_cycles |
int | 5 |
An entity must be continuously present for this many ticks before it arms. A mid-run restart re-warms it. Enforced centrally, see “Reliability (bringup-quiesce)” below. |
prune_grace |
int | 60 |
Default injected into every detector’s own config before configure(); a per-detector detectors.<id>.prune_grace overrides it. Used by detectors that keep per-key bookkeeping. |
detectors.<id>.mode |
string | bool | raise |
raise = push faults; advisory = observe, do not push; off = disabled. Bare off/no (YAML booleans) also disable; bare on/yes mean raise. |
detectors.<id>.<field> |
any | - | Per-detector thresholds, passed to that detector’s configure(). |
Nested delivery. Keys are written dotted in YAML or --ros-args, and the gateway
delivers them to the plugin as a nested object. A bare off is a YAML 1.1 boolean, so
mode: off arrives as false, not the string "off"; both forms disable the detector.
Unknown keys are reported. Anything under detectors.<id> that the detector does not
read produces one startup warning naming the key and listing the ones that exist, so a
typo like allow_list: for allowlist: cannot silently do nothing. A typo in the detector
id itself is warned about the same way, with the registered ids listed.
lifecycle_expectation keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. |
require_active |
string[] | [] |
Node names that must be in the active lifecycle state, each matched against a live node by its App::id, its full FQN (/ns/name), or the bare leaf of that FQN. A bare name matches that node in EVERY namespace (a fleet-wide “all controller_servers must be active”); use a full FQN to pin one robot’s. Both the bare and the FQN form match against the node’s stable FQN, so they survive the App::id renaming that a same-bare-name collision triggers; an entry written as an App::id also works, but can stop matching on a multi-robot graph for exactly that reason. Empty = the detector checks nothing and emits nothing. A require_active that is not a string array warns and is ignored; an empty-string entry warns and is skipped - never dropped silently, since the operator would go on believing the node is covered. |
grace |
int | 5 |
Consecutive not-active ticks a required node tolerates before being reported inactive - bringup time for a managed node to reach active (configure + activate) before the expectation is enforced. Counted per NODE, not per matching entry: a node named by both a bare-name and a full-FQN entry still advances its streak once per tick, so mixing the two documented forms cannot halve the grace that was configured. Past the absence grace the streak keeps advancing while the node is ABSENT, so a node that leaves the graph while violating is still confirmed. Accepted range 0..300 - five minutes at the shipped 1 s cadence, and already an extravagant allowance for a managed node to reach active. The upper end is a real bound rather than a formality: grace also decides how long a node that LEFT the graph while not-active sits unsettled in the tracker, and GRAPH_NODE_INACTIVE’s clear is withheld for EVERY node while it does, so at the old maximum of INT_MAX - 1 the fault could neither raise nor heal for about 24 days. The check runs on the wide integer, so a value that only fits after truncation is rejected rather than silently turned into a hair-trigger. Anything outside the range warns and keeps the default. |
prune_grace |
int | 60 |
Consecutive ticks an IDLE tracked node - both clocks at zero and no matured ownership, so nothing to lose - may stay ABSENT from the graph before its bookkeeping is reclaimed. A node carrying evidence is never reclaimed by age at all, however long it stays gone, so this key cannot erase a fault’s own state; the map is bounded instead by tracked_node_cap (see “Bounded by evidence, not by age”). Injected for every detector at plugin scope and overridable per detector; a value outside 0..3600 warns and this detector keeps its own default of 60. The range check runs on the wide integer, so an out-of-int-range value is rejected rather than truncated. The value is used as written - there is no grace + 1 clamp, because there is no longer anything for one to protect. |
tracked_node_cap |
int | 512 |
The most nodes this detector keeps bookkeeping for at once. It is what bounds the map, since evidence is never reclaimed by age (see “Bounded by evidence, not by age”): at the cap, idle entries are reclaimed first, then entries for DEPARTED nodes are collapsed into a count, and only if every tracked node is PRESENT and carrying evidence is a newly matched node refused - which withholds GRAPH_NODE_INACTIVE’s clear and is reported both in the log and on GET /x-medkit-watchdog. 512 is comfortably above every node in a realistic graph (a full Nav2 stack plus perception is roughly a hundred nodes; a ten-robot fleet sharing one domain a few hundred), so raising it is only needed where require_active legitimately matches more PRESENT nodes than that. Accepted range 1..16384 - 16384 is about 8 MB of bookkeeping at roughly 500 bytes per tracked node, and a deployment needing more distinct required-node identities alive at once has identity churn rather than a large fleet. Anything outside the range, including 0 (a cap of nothing would mean checking nothing), warns and keeps the default; the check runs on the wide integer, so an out-of-int-range value is rejected rather than truncated. |
orphan keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | raise |
As above. |
grace |
int | 10 |
Consecutive sweeps a candidate pair must hold before it is reported. A staged bringup or a restart makes one side missing for a short time, which looks exactly like a typo until the other side appears. |
max_edit_distance |
int | 1 |
How many character edits the LEAF (the part after the last /) may differ by and still count as the same intended name. Values outside 1..8 keep the default: past a few edits a “near miss” starts matching unrelated topics. |
namespace_edit_distance |
int | 0 |
The same budget for the namespace, spent separately. 0 means the namespace must match to the character, which is what makes /robot1/scan and /robot2/scan two robots rather than a typo. Raise it to catch a misspelled namespace, and read the warning under the table first. Values outside 0..8 keep the default. |
allowlist |
string[] | [] |
Topic names never reported. Exact match only, never a prefix, so allowlisting /r1/scan does not also silence /r2/scan. |
What this detector will not catch, on purpose. A pair whose names match once every run of
digits is collapsed is treated as an enumeration of sensors, not a typo: /lidar_1 next to
/lidar_2, each one-sided, is an ordinary state on a multi-sensor robot and reporting it would
be a false alarm on every such machine. Collapsing runs rather than comparing character
positions is what also covers /lidar_9 next to /lidar_10. The cost is that a typo which
is a digit stops being reported. An appended digit is not that case: /scan1 collapses to
/scan#, which is not /scan, so it is still reported.
A misspelled namespace is opt-in, and it can be expensive. /robott/scan against
/robot/scan is one edit, but in the namespace, so by default it is not reported. Setting
namespace_edit_distance: 1 reports it. Check the naming scheme before doing that: on a fleet
named with letters, /amr_a and /amr_b are also one edit apart, and every robot in the
fleet will be reported as a typo. A numbered fleet is safe, since /robot1 against /robot2
differs only in its numeric field and the rule above already spares it.
What it cannot catch at all. A namespace added or dropped by mistake, /scan against
/robot/scan, is six edits apart, so no budget that is still specific will reach it. Do not
rely on this detector for that class.
param_drift keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. A bare off arrives as a YAML boolean; both forms disable the detector. |
baseline |
bool | true |
Self-capture the value of every parameter not listed in expect, then flag later changes to it. false checks only what expect pins, and skips reading the full parameter list. |
expect.<param> |
any | - | Absolute expected value for <param>, checked on every node that has it. This is the only rule that catches a value that was wrong from the first tick, which self-capture cannot see. A node that does not declare <param> answers NOT_FOUND, which is not a drift - so a misspelt name checks nothing at all. Once the sweep has covered every armed app - each one read, or given up on after repeated failed reads - and at least one of them answered, a pin that no app declares is warned about once, naming it. |
ignore |
string[] | [] |
Parameter-name globs never flagged. Applies to the self-captured set only, so it cannot silence an expect pin. A pattern of nothing but * matches every name and suppresses self-capture wholesale, which is indistinguishable at runtime from finding nothing; that combination warns at startup. |
max_reads_per_tick |
int | 8 |
Parameter-service ROUND TRIPS this detector may spend per tick, round-robin over nodes. Round trips, not calls: one call is several requests at the watched node, and the node pays for each of them. A baseline visit is charged 2 (a list, then one batched get) whatever the node’s parameter count; each expect pin is charged 3 (a list, a get and a descriptor read), or 2 on a node that does not declare it - see the table under “Coverage latency” for what each of those really costs the node. So a self-capture visit spends 2 of the budget and pinning N parameters spends up to 3N. It is an AMORTISED rate, not a per-tick cap: there is no pacing INSIDE a visit, so a node’s whole read set is issued back to back and the reader waits out the debt afterwards. Four pins against a budget of 8 fire 12 round trips inside one tick period and then idle for the rest of the next one; what the knob bounds is the sustained rate, not the burst. Accepted range 1..100000; anything else keeps the default and warns. |
prune_grace |
int | 60 |
Consecutive sweeps an app may be missing before its finding is dropped: the finding survives prune_grace absences and goes on the next one. Injected for every detector at plugin scope and overridable per detector; a value outside 0..3600 is rejected with a warning and this detector falls back to 2, not to the plugin default. The captured BASELINE has a shorter horizon of its own, and that one is NOT configurable - it is the compile-time kBaselineForgetGrace: the baseline is dropped on the third consecutive missed sweep, the two the reliability gate absorbs plus one, so an ordinary dropped discovery poll cannot re-capture a node on its already-drifted value. Setting prune_grace below 2 pulls that forward onto the sweep the finding itself is dropped on, so the two then go together. |
qos_mismatch keys
File truncated at 100 lines see the full file
Package Dependencies
System Dependencies
Dependant Packages
Launch files
Messages
Services
Plugins
Recent questions tagged ros2_medkit_graph_watchdog at Robotics Stack Exchange
Package Summary
| Version | 0.6.0 |
| License | Apache-2.0 |
| Build type | AMENT_CMAKE |
| Use | RECOMMENDED |
Repository Summary
| Checkout URI | https://github.com/selfpatch/ros2_medkit.git |
| VCS Type | git |
| VCS Version | main |
| Last Updated | 2026-08-17 |
| Dev Status | DEVELOPED |
| Released | RELEASED |
| Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
Maintainers
- bburda
Authors
ros2_medkit_graph_watchdog
Gateway plugin that detects silent faults in the ROS 2 graph: failures where every node is up, nothing logs an error, and the robot is still broken.
Detectors read the graph and raise faults through a ReportFault service client on the
gateway node; the faults surface via FaultManager on the gateway /faults API. That
client, like the lifecycle subscriptions below, lives in the plugin’s own callback group
and is driven by the plugin’s own executor from the tick thread, never by the gateway’s
ROS executor.
This package carries the plugin skeleton, the central reliability gate that holds raises
until the graph has quiesced, and four detectors, qos_mismatch, orphan,
param_drift and lifecycle_expectation. The remaining
silent-fault classes land in follow-up changes, each against its own issue; their fault
codes are already reserved in the frozen GRAPH_* namespace (see “Fault codes”).
Build
With the gateway built and sourced:
colcon build --packages-select ros2_medkit_graph_watchdog
colcon test --packages-select ros2_medkit_graph_watchdog && colcon test-result --verbose
Load into the gateway
ros2 run ros2_medkit_gateway gateway_node --ros-args \
-p plugins:="[graph_watchdog]" \
-p "plugins.graph_watchdog.path:=$(ros2 pkg prefix ros2_medkit_graph_watchdog)/lib/ros2_medkit_graph_watchdog/libros2_medkit_graph_watchdog.so"
Configuration (plugins.graph_watchdog.<key>)
| Key | Type | Default | Meaning |
|---|---|---|---|
tick_interval_ms |
int | 1000 |
Detector tick cadence. |
warmup_cycles |
int | 5 |
An entity must be continuously present for this many ticks before it arms. A mid-run restart re-warms it. Enforced centrally, see “Reliability (bringup-quiesce)” below. |
prune_grace |
int | 60 |
Default injected into every detector’s own config before configure(); a per-detector detectors.<id>.prune_grace overrides it. Used by detectors that keep per-key bookkeeping. |
detectors.<id>.mode |
string | bool | raise |
raise = push faults; advisory = observe, do not push; off = disabled. Bare off/no (YAML booleans) also disable; bare on/yes mean raise. |
detectors.<id>.<field> |
any | - | Per-detector thresholds, passed to that detector’s configure(). |
Nested delivery. Keys are written dotted in YAML or --ros-args, and the gateway
delivers them to the plugin as a nested object. A bare off is a YAML 1.1 boolean, so
mode: off arrives as false, not the string "off"; both forms disable the detector.
Unknown keys are reported. Anything under detectors.<id> that the detector does not
read produces one startup warning naming the key and listing the ones that exist, so a
typo like allow_list: for allowlist: cannot silently do nothing. A typo in the detector
id itself is warned about the same way, with the registered ids listed.
lifecycle_expectation keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. |
require_active |
string[] | [] |
Node names that must be in the active lifecycle state, each matched against a live node by its App::id, its full FQN (/ns/name), or the bare leaf of that FQN. A bare name matches that node in EVERY namespace (a fleet-wide “all controller_servers must be active”); use a full FQN to pin one robot’s. Both the bare and the FQN form match against the node’s stable FQN, so they survive the App::id renaming that a same-bare-name collision triggers; an entry written as an App::id also works, but can stop matching on a multi-robot graph for exactly that reason. Empty = the detector checks nothing and emits nothing. A require_active that is not a string array warns and is ignored; an empty-string entry warns and is skipped - never dropped silently, since the operator would go on believing the node is covered. |
grace |
int | 5 |
Consecutive not-active ticks a required node tolerates before being reported inactive - bringup time for a managed node to reach active (configure + activate) before the expectation is enforced. Counted per NODE, not per matching entry: a node named by both a bare-name and a full-FQN entry still advances its streak once per tick, so mixing the two documented forms cannot halve the grace that was configured. Past the absence grace the streak keeps advancing while the node is ABSENT, so a node that leaves the graph while violating is still confirmed. Accepted range 0..300 - five minutes at the shipped 1 s cadence, and already an extravagant allowance for a managed node to reach active. The upper end is a real bound rather than a formality: grace also decides how long a node that LEFT the graph while not-active sits unsettled in the tracker, and GRAPH_NODE_INACTIVE’s clear is withheld for EVERY node while it does, so at the old maximum of INT_MAX - 1 the fault could neither raise nor heal for about 24 days. The check runs on the wide integer, so a value that only fits after truncation is rejected rather than silently turned into a hair-trigger. Anything outside the range warns and keeps the default. |
prune_grace |
int | 60 |
Consecutive ticks an IDLE tracked node - both clocks at zero and no matured ownership, so nothing to lose - may stay ABSENT from the graph before its bookkeeping is reclaimed. A node carrying evidence is never reclaimed by age at all, however long it stays gone, so this key cannot erase a fault’s own state; the map is bounded instead by tracked_node_cap (see “Bounded by evidence, not by age”). Injected for every detector at plugin scope and overridable per detector; a value outside 0..3600 warns and this detector keeps its own default of 60. The range check runs on the wide integer, so an out-of-int-range value is rejected rather than truncated. The value is used as written - there is no grace + 1 clamp, because there is no longer anything for one to protect. |
tracked_node_cap |
int | 512 |
The most nodes this detector keeps bookkeeping for at once. It is what bounds the map, since evidence is never reclaimed by age (see “Bounded by evidence, not by age”): at the cap, idle entries are reclaimed first, then entries for DEPARTED nodes are collapsed into a count, and only if every tracked node is PRESENT and carrying evidence is a newly matched node refused - which withholds GRAPH_NODE_INACTIVE’s clear and is reported both in the log and on GET /x-medkit-watchdog. 512 is comfortably above every node in a realistic graph (a full Nav2 stack plus perception is roughly a hundred nodes; a ten-robot fleet sharing one domain a few hundred), so raising it is only needed where require_active legitimately matches more PRESENT nodes than that. Accepted range 1..16384 - 16384 is about 8 MB of bookkeeping at roughly 500 bytes per tracked node, and a deployment needing more distinct required-node identities alive at once has identity churn rather than a large fleet. Anything outside the range, including 0 (a cap of nothing would mean checking nothing), warns and keeps the default; the check runs on the wide integer, so an out-of-int-range value is rejected rather than truncated. |
orphan keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | raise |
As above. |
grace |
int | 10 |
Consecutive sweeps a candidate pair must hold before it is reported. A staged bringup or a restart makes one side missing for a short time, which looks exactly like a typo until the other side appears. |
max_edit_distance |
int | 1 |
How many character edits the LEAF (the part after the last /) may differ by and still count as the same intended name. Values outside 1..8 keep the default: past a few edits a “near miss” starts matching unrelated topics. |
namespace_edit_distance |
int | 0 |
The same budget for the namespace, spent separately. 0 means the namespace must match to the character, which is what makes /robot1/scan and /robot2/scan two robots rather than a typo. Raise it to catch a misspelled namespace, and read the warning under the table first. Values outside 0..8 keep the default. |
allowlist |
string[] | [] |
Topic names never reported. Exact match only, never a prefix, so allowlisting /r1/scan does not also silence /r2/scan. |
What this detector will not catch, on purpose. A pair whose names match once every run of
digits is collapsed is treated as an enumeration of sensors, not a typo: /lidar_1 next to
/lidar_2, each one-sided, is an ordinary state on a multi-sensor robot and reporting it would
be a false alarm on every such machine. Collapsing runs rather than comparing character
positions is what also covers /lidar_9 next to /lidar_10. The cost is that a typo which
is a digit stops being reported. An appended digit is not that case: /scan1 collapses to
/scan#, which is not /scan, so it is still reported.
A misspelled namespace is opt-in, and it can be expensive. /robott/scan against
/robot/scan is one edit, but in the namespace, so by default it is not reported. Setting
namespace_edit_distance: 1 reports it. Check the naming scheme before doing that: on a fleet
named with letters, /amr_a and /amr_b are also one edit apart, and every robot in the
fleet will be reported as a typo. A numbered fleet is safe, since /robot1 against /robot2
differs only in its numeric field and the rule above already spares it.
What it cannot catch at all. A namespace added or dropped by mistake, /scan against
/robot/scan, is six edits apart, so no budget that is still specific will reach it. Do not
rely on this detector for that class.
param_drift keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. A bare off arrives as a YAML boolean; both forms disable the detector. |
baseline |
bool | true |
Self-capture the value of every parameter not listed in expect, then flag later changes to it. false checks only what expect pins, and skips reading the full parameter list. |
expect.<param> |
any | - | Absolute expected value for <param>, checked on every node that has it. This is the only rule that catches a value that was wrong from the first tick, which self-capture cannot see. A node that does not declare <param> answers NOT_FOUND, which is not a drift - so a misspelt name checks nothing at all. Once the sweep has covered every armed app - each one read, or given up on after repeated failed reads - and at least one of them answered, a pin that no app declares is warned about once, naming it. |
ignore |
string[] | [] |
Parameter-name globs never flagged. Applies to the self-captured set only, so it cannot silence an expect pin. A pattern of nothing but * matches every name and suppresses self-capture wholesale, which is indistinguishable at runtime from finding nothing; that combination warns at startup. |
max_reads_per_tick |
int | 8 |
Parameter-service ROUND TRIPS this detector may spend per tick, round-robin over nodes. Round trips, not calls: one call is several requests at the watched node, and the node pays for each of them. A baseline visit is charged 2 (a list, then one batched get) whatever the node’s parameter count; each expect pin is charged 3 (a list, a get and a descriptor read), or 2 on a node that does not declare it - see the table under “Coverage latency” for what each of those really costs the node. So a self-capture visit spends 2 of the budget and pinning N parameters spends up to 3N. It is an AMORTISED rate, not a per-tick cap: there is no pacing INSIDE a visit, so a node’s whole read set is issued back to back and the reader waits out the debt afterwards. Four pins against a budget of 8 fire 12 round trips inside one tick period and then idle for the rest of the next one; what the knob bounds is the sustained rate, not the burst. Accepted range 1..100000; anything else keeps the default and warns. |
prune_grace |
int | 60 |
Consecutive sweeps an app may be missing before its finding is dropped: the finding survives prune_grace absences and goes on the next one. Injected for every detector at plugin scope and overridable per detector; a value outside 0..3600 is rejected with a warning and this detector falls back to 2, not to the plugin default. The captured BASELINE has a shorter horizon of its own, and that one is NOT configurable - it is the compile-time kBaselineForgetGrace: the baseline is dropped on the third consecutive missed sweep, the two the reliability gate absorbs plus one, so an ordinary dropped discovery poll cannot re-capture a node on its already-drifted value. Setting prune_grace below 2 pulls that forward onto the sweep the finding itself is dropped on, so the two then go together. |
qos_mismatch keys
File truncated at 100 lines see the full file
Package Dependencies
System Dependencies
Dependant Packages
Launch files
Messages
Services
Plugins
Recent questions tagged ros2_medkit_graph_watchdog at Robotics Stack Exchange
Package Summary
| Version | 0.6.0 |
| License | Apache-2.0 |
| Build type | AMENT_CMAKE |
| Use | RECOMMENDED |
Repository Summary
| Checkout URI | https://github.com/selfpatch/ros2_medkit.git |
| VCS Type | git |
| VCS Version | main |
| Last Updated | 2026-08-17 |
| Dev Status | DEVELOPED |
| Released | RELEASED |
| Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
Maintainers
- bburda
Authors
ros2_medkit_graph_watchdog
Gateway plugin that detects silent faults in the ROS 2 graph: failures where every node is up, nothing logs an error, and the robot is still broken.
Detectors read the graph and raise faults through a ReportFault service client on the
gateway node; the faults surface via FaultManager on the gateway /faults API. That
client, like the lifecycle subscriptions below, lives in the plugin’s own callback group
and is driven by the plugin’s own executor from the tick thread, never by the gateway’s
ROS executor.
This package carries the plugin skeleton, the central reliability gate that holds raises
until the graph has quiesced, and four detectors, qos_mismatch, orphan,
param_drift and lifecycle_expectation. The remaining
silent-fault classes land in follow-up changes, each against its own issue; their fault
codes are already reserved in the frozen GRAPH_* namespace (see “Fault codes”).
Build
With the gateway built and sourced:
colcon build --packages-select ros2_medkit_graph_watchdog
colcon test --packages-select ros2_medkit_graph_watchdog && colcon test-result --verbose
Load into the gateway
ros2 run ros2_medkit_gateway gateway_node --ros-args \
-p plugins:="[graph_watchdog]" \
-p "plugins.graph_watchdog.path:=$(ros2 pkg prefix ros2_medkit_graph_watchdog)/lib/ros2_medkit_graph_watchdog/libros2_medkit_graph_watchdog.so"
Configuration (plugins.graph_watchdog.<key>)
| Key | Type | Default | Meaning |
|---|---|---|---|
tick_interval_ms |
int | 1000 |
Detector tick cadence. |
warmup_cycles |
int | 5 |
An entity must be continuously present for this many ticks before it arms. A mid-run restart re-warms it. Enforced centrally, see “Reliability (bringup-quiesce)” below. |
prune_grace |
int | 60 |
Default injected into every detector’s own config before configure(); a per-detector detectors.<id>.prune_grace overrides it. Used by detectors that keep per-key bookkeeping. |
detectors.<id>.mode |
string | bool | raise |
raise = push faults; advisory = observe, do not push; off = disabled. Bare off/no (YAML booleans) also disable; bare on/yes mean raise. |
detectors.<id>.<field> |
any | - | Per-detector thresholds, passed to that detector’s configure(). |
Nested delivery. Keys are written dotted in YAML or --ros-args, and the gateway
delivers them to the plugin as a nested object. A bare off is a YAML 1.1 boolean, so
mode: off arrives as false, not the string "off"; both forms disable the detector.
Unknown keys are reported. Anything under detectors.<id> that the detector does not
read produces one startup warning naming the key and listing the ones that exist, so a
typo like allow_list: for allowlist: cannot silently do nothing. A typo in the detector
id itself is warned about the same way, with the registered ids listed.
lifecycle_expectation keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. |
require_active |
string[] | [] |
Node names that must be in the active lifecycle state, each matched against a live node by its App::id, its full FQN (/ns/name), or the bare leaf of that FQN. A bare name matches that node in EVERY namespace (a fleet-wide “all controller_servers must be active”); use a full FQN to pin one robot’s. Both the bare and the FQN form match against the node’s stable FQN, so they survive the App::id renaming that a same-bare-name collision triggers; an entry written as an App::id also works, but can stop matching on a multi-robot graph for exactly that reason. Empty = the detector checks nothing and emits nothing. A require_active that is not a string array warns and is ignored; an empty-string entry warns and is skipped - never dropped silently, since the operator would go on believing the node is covered. |
grace |
int | 5 |
Consecutive not-active ticks a required node tolerates before being reported inactive - bringup time for a managed node to reach active (configure + activate) before the expectation is enforced. Counted per NODE, not per matching entry: a node named by both a bare-name and a full-FQN entry still advances its streak once per tick, so mixing the two documented forms cannot halve the grace that was configured. Past the absence grace the streak keeps advancing while the node is ABSENT, so a node that leaves the graph while violating is still confirmed. Accepted range 0..300 - five minutes at the shipped 1 s cadence, and already an extravagant allowance for a managed node to reach active. The upper end is a real bound rather than a formality: grace also decides how long a node that LEFT the graph while not-active sits unsettled in the tracker, and GRAPH_NODE_INACTIVE’s clear is withheld for EVERY node while it does, so at the old maximum of INT_MAX - 1 the fault could neither raise nor heal for about 24 days. The check runs on the wide integer, so a value that only fits after truncation is rejected rather than silently turned into a hair-trigger. Anything outside the range warns and keeps the default. |
prune_grace |
int | 60 |
Consecutive ticks an IDLE tracked node - both clocks at zero and no matured ownership, so nothing to lose - may stay ABSENT from the graph before its bookkeeping is reclaimed. A node carrying evidence is never reclaimed by age at all, however long it stays gone, so this key cannot erase a fault’s own state; the map is bounded instead by tracked_node_cap (see “Bounded by evidence, not by age”). Injected for every detector at plugin scope and overridable per detector; a value outside 0..3600 warns and this detector keeps its own default of 60. The range check runs on the wide integer, so an out-of-int-range value is rejected rather than truncated. The value is used as written - there is no grace + 1 clamp, because there is no longer anything for one to protect. |
tracked_node_cap |
int | 512 |
The most nodes this detector keeps bookkeeping for at once. It is what bounds the map, since evidence is never reclaimed by age (see “Bounded by evidence, not by age”): at the cap, idle entries are reclaimed first, then entries for DEPARTED nodes are collapsed into a count, and only if every tracked node is PRESENT and carrying evidence is a newly matched node refused - which withholds GRAPH_NODE_INACTIVE’s clear and is reported both in the log and on GET /x-medkit-watchdog. 512 is comfortably above every node in a realistic graph (a full Nav2 stack plus perception is roughly a hundred nodes; a ten-robot fleet sharing one domain a few hundred), so raising it is only needed where require_active legitimately matches more PRESENT nodes than that. Accepted range 1..16384 - 16384 is about 8 MB of bookkeeping at roughly 500 bytes per tracked node, and a deployment needing more distinct required-node identities alive at once has identity churn rather than a large fleet. Anything outside the range, including 0 (a cap of nothing would mean checking nothing), warns and keeps the default; the check runs on the wide integer, so an out-of-int-range value is rejected rather than truncated. |
orphan keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | raise |
As above. |
grace |
int | 10 |
Consecutive sweeps a candidate pair must hold before it is reported. A staged bringup or a restart makes one side missing for a short time, which looks exactly like a typo until the other side appears. |
max_edit_distance |
int | 1 |
How many character edits the LEAF (the part after the last /) may differ by and still count as the same intended name. Values outside 1..8 keep the default: past a few edits a “near miss” starts matching unrelated topics. |
namespace_edit_distance |
int | 0 |
The same budget for the namespace, spent separately. 0 means the namespace must match to the character, which is what makes /robot1/scan and /robot2/scan two robots rather than a typo. Raise it to catch a misspelled namespace, and read the warning under the table first. Values outside 0..8 keep the default. |
allowlist |
string[] | [] |
Topic names never reported. Exact match only, never a prefix, so allowlisting /r1/scan does not also silence /r2/scan. |
What this detector will not catch, on purpose. A pair whose names match once every run of
digits is collapsed is treated as an enumeration of sensors, not a typo: /lidar_1 next to
/lidar_2, each one-sided, is an ordinary state on a multi-sensor robot and reporting it would
be a false alarm on every such machine. Collapsing runs rather than comparing character
positions is what also covers /lidar_9 next to /lidar_10. The cost is that a typo which
is a digit stops being reported. An appended digit is not that case: /scan1 collapses to
/scan#, which is not /scan, so it is still reported.
A misspelled namespace is opt-in, and it can be expensive. /robott/scan against
/robot/scan is one edit, but in the namespace, so by default it is not reported. Setting
namespace_edit_distance: 1 reports it. Check the naming scheme before doing that: on a fleet
named with letters, /amr_a and /amr_b are also one edit apart, and every robot in the
fleet will be reported as a typo. A numbered fleet is safe, since /robot1 against /robot2
differs only in its numeric field and the rule above already spares it.
What it cannot catch at all. A namespace added or dropped by mistake, /scan against
/robot/scan, is six edits apart, so no budget that is still specific will reach it. Do not
rely on this detector for that class.
param_drift keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. A bare off arrives as a YAML boolean; both forms disable the detector. |
baseline |
bool | true |
Self-capture the value of every parameter not listed in expect, then flag later changes to it. false checks only what expect pins, and skips reading the full parameter list. |
expect.<param> |
any | - | Absolute expected value for <param>, checked on every node that has it. This is the only rule that catches a value that was wrong from the first tick, which self-capture cannot see. A node that does not declare <param> answers NOT_FOUND, which is not a drift - so a misspelt name checks nothing at all. Once the sweep has covered every armed app - each one read, or given up on after repeated failed reads - and at least one of them answered, a pin that no app declares is warned about once, naming it. |
ignore |
string[] | [] |
Parameter-name globs never flagged. Applies to the self-captured set only, so it cannot silence an expect pin. A pattern of nothing but * matches every name and suppresses self-capture wholesale, which is indistinguishable at runtime from finding nothing; that combination warns at startup. |
max_reads_per_tick |
int | 8 |
Parameter-service ROUND TRIPS this detector may spend per tick, round-robin over nodes. Round trips, not calls: one call is several requests at the watched node, and the node pays for each of them. A baseline visit is charged 2 (a list, then one batched get) whatever the node’s parameter count; each expect pin is charged 3 (a list, a get and a descriptor read), or 2 on a node that does not declare it - see the table under “Coverage latency” for what each of those really costs the node. So a self-capture visit spends 2 of the budget and pinning N parameters spends up to 3N. It is an AMORTISED rate, not a per-tick cap: there is no pacing INSIDE a visit, so a node’s whole read set is issued back to back and the reader waits out the debt afterwards. Four pins against a budget of 8 fire 12 round trips inside one tick period and then idle for the rest of the next one; what the knob bounds is the sustained rate, not the burst. Accepted range 1..100000; anything else keeps the default and warns. |
prune_grace |
int | 60 |
Consecutive sweeps an app may be missing before its finding is dropped: the finding survives prune_grace absences and goes on the next one. Injected for every detector at plugin scope and overridable per detector; a value outside 0..3600 is rejected with a warning and this detector falls back to 2, not to the plugin default. The captured BASELINE has a shorter horizon of its own, and that one is NOT configurable - it is the compile-time kBaselineForgetGrace: the baseline is dropped on the third consecutive missed sweep, the two the reliability gate absorbs plus one, so an ordinary dropped discovery poll cannot re-capture a node on its already-drifted value. Setting prune_grace below 2 pulls that forward onto the sweep the finding itself is dropped on, so the two then go together. |
qos_mismatch keys
File truncated at 100 lines see the full file
Package Dependencies
System Dependencies
Dependant Packages
Launch files
Messages
Services
Plugins
Recent questions tagged ros2_medkit_graph_watchdog at Robotics Stack Exchange
Package Summary
| Version | 0.6.0 |
| License | Apache-2.0 |
| Build type | AMENT_CMAKE |
| Use | RECOMMENDED |
Repository Summary
| Checkout URI | https://github.com/selfpatch/ros2_medkit.git |
| VCS Type | git |
| VCS Version | main |
| Last Updated | 2026-08-17 |
| Dev Status | DEVELOPED |
| Released | RELEASED |
| Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
Maintainers
- bburda
Authors
ros2_medkit_graph_watchdog
Gateway plugin that detects silent faults in the ROS 2 graph: failures where every node is up, nothing logs an error, and the robot is still broken.
Detectors read the graph and raise faults through a ReportFault service client on the
gateway node; the faults surface via FaultManager on the gateway /faults API. That
client, like the lifecycle subscriptions below, lives in the plugin’s own callback group
and is driven by the plugin’s own executor from the tick thread, never by the gateway’s
ROS executor.
This package carries the plugin skeleton, the central reliability gate that holds raises
until the graph has quiesced, and four detectors, qos_mismatch, orphan,
param_drift and lifecycle_expectation. The remaining
silent-fault classes land in follow-up changes, each against its own issue; their fault
codes are already reserved in the frozen GRAPH_* namespace (see “Fault codes”).
Build
With the gateway built and sourced:
colcon build --packages-select ros2_medkit_graph_watchdog
colcon test --packages-select ros2_medkit_graph_watchdog && colcon test-result --verbose
Load into the gateway
ros2 run ros2_medkit_gateway gateway_node --ros-args \
-p plugins:="[graph_watchdog]" \
-p "plugins.graph_watchdog.path:=$(ros2 pkg prefix ros2_medkit_graph_watchdog)/lib/ros2_medkit_graph_watchdog/libros2_medkit_graph_watchdog.so"
Configuration (plugins.graph_watchdog.<key>)
| Key | Type | Default | Meaning |
|---|---|---|---|
tick_interval_ms |
int | 1000 |
Detector tick cadence. |
warmup_cycles |
int | 5 |
An entity must be continuously present for this many ticks before it arms. A mid-run restart re-warms it. Enforced centrally, see “Reliability (bringup-quiesce)” below. |
prune_grace |
int | 60 |
Default injected into every detector’s own config before configure(); a per-detector detectors.<id>.prune_grace overrides it. Used by detectors that keep per-key bookkeeping. |
detectors.<id>.mode |
string | bool | raise |
raise = push faults; advisory = observe, do not push; off = disabled. Bare off/no (YAML booleans) also disable; bare on/yes mean raise. |
detectors.<id>.<field> |
any | - | Per-detector thresholds, passed to that detector’s configure(). |
Nested delivery. Keys are written dotted in YAML or --ros-args, and the gateway
delivers them to the plugin as a nested object. A bare off is a YAML 1.1 boolean, so
mode: off arrives as false, not the string "off"; both forms disable the detector.
Unknown keys are reported. Anything under detectors.<id> that the detector does not
read produces one startup warning naming the key and listing the ones that exist, so a
typo like allow_list: for allowlist: cannot silently do nothing. A typo in the detector
id itself is warned about the same way, with the registered ids listed.
lifecycle_expectation keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. |
require_active |
string[] | [] |
Node names that must be in the active lifecycle state, each matched against a live node by its App::id, its full FQN (/ns/name), or the bare leaf of that FQN. A bare name matches that node in EVERY namespace (a fleet-wide “all controller_servers must be active”); use a full FQN to pin one robot’s. Both the bare and the FQN form match against the node’s stable FQN, so they survive the App::id renaming that a same-bare-name collision triggers; an entry written as an App::id also works, but can stop matching on a multi-robot graph for exactly that reason. Empty = the detector checks nothing and emits nothing. A require_active that is not a string array warns and is ignored; an empty-string entry warns and is skipped - never dropped silently, since the operator would go on believing the node is covered. |
grace |
int | 5 |
Consecutive not-active ticks a required node tolerates before being reported inactive - bringup time for a managed node to reach active (configure + activate) before the expectation is enforced. Counted per NODE, not per matching entry: a node named by both a bare-name and a full-FQN entry still advances its streak once per tick, so mixing the two documented forms cannot halve the grace that was configured. Past the absence grace the streak keeps advancing while the node is ABSENT, so a node that leaves the graph while violating is still confirmed. Accepted range 0..300 - five minutes at the shipped 1 s cadence, and already an extravagant allowance for a managed node to reach active. The upper end is a real bound rather than a formality: grace also decides how long a node that LEFT the graph while not-active sits unsettled in the tracker, and GRAPH_NODE_INACTIVE’s clear is withheld for EVERY node while it does, so at the old maximum of INT_MAX - 1 the fault could neither raise nor heal for about 24 days. The check runs on the wide integer, so a value that only fits after truncation is rejected rather than silently turned into a hair-trigger. Anything outside the range warns and keeps the default. |
prune_grace |
int | 60 |
Consecutive ticks an IDLE tracked node - both clocks at zero and no matured ownership, so nothing to lose - may stay ABSENT from the graph before its bookkeeping is reclaimed. A node carrying evidence is never reclaimed by age at all, however long it stays gone, so this key cannot erase a fault’s own state; the map is bounded instead by tracked_node_cap (see “Bounded by evidence, not by age”). Injected for every detector at plugin scope and overridable per detector; a value outside 0..3600 warns and this detector keeps its own default of 60. The range check runs on the wide integer, so an out-of-int-range value is rejected rather than truncated. The value is used as written - there is no grace + 1 clamp, because there is no longer anything for one to protect. |
tracked_node_cap |
int | 512 |
The most nodes this detector keeps bookkeeping for at once. It is what bounds the map, since evidence is never reclaimed by age (see “Bounded by evidence, not by age”): at the cap, idle entries are reclaimed first, then entries for DEPARTED nodes are collapsed into a count, and only if every tracked node is PRESENT and carrying evidence is a newly matched node refused - which withholds GRAPH_NODE_INACTIVE’s clear and is reported both in the log and on GET /x-medkit-watchdog. 512 is comfortably above every node in a realistic graph (a full Nav2 stack plus perception is roughly a hundred nodes; a ten-robot fleet sharing one domain a few hundred), so raising it is only needed where require_active legitimately matches more PRESENT nodes than that. Accepted range 1..16384 - 16384 is about 8 MB of bookkeeping at roughly 500 bytes per tracked node, and a deployment needing more distinct required-node identities alive at once has identity churn rather than a large fleet. Anything outside the range, including 0 (a cap of nothing would mean checking nothing), warns and keeps the default; the check runs on the wide integer, so an out-of-int-range value is rejected rather than truncated. |
orphan keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | raise |
As above. |
grace |
int | 10 |
Consecutive sweeps a candidate pair must hold before it is reported. A staged bringup or a restart makes one side missing for a short time, which looks exactly like a typo until the other side appears. |
max_edit_distance |
int | 1 |
How many character edits the LEAF (the part after the last /) may differ by and still count as the same intended name. Values outside 1..8 keep the default: past a few edits a “near miss” starts matching unrelated topics. |
namespace_edit_distance |
int | 0 |
The same budget for the namespace, spent separately. 0 means the namespace must match to the character, which is what makes /robot1/scan and /robot2/scan two robots rather than a typo. Raise it to catch a misspelled namespace, and read the warning under the table first. Values outside 0..8 keep the default. |
allowlist |
string[] | [] |
Topic names never reported. Exact match only, never a prefix, so allowlisting /r1/scan does not also silence /r2/scan. |
What this detector will not catch, on purpose. A pair whose names match once every run of
digits is collapsed is treated as an enumeration of sensors, not a typo: /lidar_1 next to
/lidar_2, each one-sided, is an ordinary state on a multi-sensor robot and reporting it would
be a false alarm on every such machine. Collapsing runs rather than comparing character
positions is what also covers /lidar_9 next to /lidar_10. The cost is that a typo which
is a digit stops being reported. An appended digit is not that case: /scan1 collapses to
/scan#, which is not /scan, so it is still reported.
A misspelled namespace is opt-in, and it can be expensive. /robott/scan against
/robot/scan is one edit, but in the namespace, so by default it is not reported. Setting
namespace_edit_distance: 1 reports it. Check the naming scheme before doing that: on a fleet
named with letters, /amr_a and /amr_b are also one edit apart, and every robot in the
fleet will be reported as a typo. A numbered fleet is safe, since /robot1 against /robot2
differs only in its numeric field and the rule above already spares it.
What it cannot catch at all. A namespace added or dropped by mistake, /scan against
/robot/scan, is six edits apart, so no budget that is still specific will reach it. Do not
rely on this detector for that class.
param_drift keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. A bare off arrives as a YAML boolean; both forms disable the detector. |
baseline |
bool | true |
Self-capture the value of every parameter not listed in expect, then flag later changes to it. false checks only what expect pins, and skips reading the full parameter list. |
expect.<param> |
any | - | Absolute expected value for <param>, checked on every node that has it. This is the only rule that catches a value that was wrong from the first tick, which self-capture cannot see. A node that does not declare <param> answers NOT_FOUND, which is not a drift - so a misspelt name checks nothing at all. Once the sweep has covered every armed app - each one read, or given up on after repeated failed reads - and at least one of them answered, a pin that no app declares is warned about once, naming it. |
ignore |
string[] | [] |
Parameter-name globs never flagged. Applies to the self-captured set only, so it cannot silence an expect pin. A pattern of nothing but * matches every name and suppresses self-capture wholesale, which is indistinguishable at runtime from finding nothing; that combination warns at startup. |
max_reads_per_tick |
int | 8 |
Parameter-service ROUND TRIPS this detector may spend per tick, round-robin over nodes. Round trips, not calls: one call is several requests at the watched node, and the node pays for each of them. A baseline visit is charged 2 (a list, then one batched get) whatever the node’s parameter count; each expect pin is charged 3 (a list, a get and a descriptor read), or 2 on a node that does not declare it - see the table under “Coverage latency” for what each of those really costs the node. So a self-capture visit spends 2 of the budget and pinning N parameters spends up to 3N. It is an AMORTISED rate, not a per-tick cap: there is no pacing INSIDE a visit, so a node’s whole read set is issued back to back and the reader waits out the debt afterwards. Four pins against a budget of 8 fire 12 round trips inside one tick period and then idle for the rest of the next one; what the knob bounds is the sustained rate, not the burst. Accepted range 1..100000; anything else keeps the default and warns. |
prune_grace |
int | 60 |
Consecutive sweeps an app may be missing before its finding is dropped: the finding survives prune_grace absences and goes on the next one. Injected for every detector at plugin scope and overridable per detector; a value outside 0..3600 is rejected with a warning and this detector falls back to 2, not to the plugin default. The captured BASELINE has a shorter horizon of its own, and that one is NOT configurable - it is the compile-time kBaselineForgetGrace: the baseline is dropped on the third consecutive missed sweep, the two the reliability gate absorbs plus one, so an ordinary dropped discovery poll cannot re-capture a node on its already-drifted value. Setting prune_grace below 2 pulls that forward onto the sweep the finding itself is dropped on, so the two then go together. |
qos_mismatch keys
File truncated at 100 lines see the full file
Package Dependencies
System Dependencies
Dependant Packages
Launch files
Messages
Services
Plugins
Recent questions tagged ros2_medkit_graph_watchdog at Robotics Stack Exchange
Package Summary
| Version | 0.6.0 |
| License | Apache-2.0 |
| Build type | AMENT_CMAKE |
| Use | RECOMMENDED |
Repository Summary
| Checkout URI | https://github.com/selfpatch/ros2_medkit.git |
| VCS Type | git |
| VCS Version | main |
| Last Updated | 2026-08-17 |
| Dev Status | DEVELOPED |
| Released | RELEASED |
| Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
Maintainers
- bburda
Authors
ros2_medkit_graph_watchdog
Gateway plugin that detects silent faults in the ROS 2 graph: failures where every node is up, nothing logs an error, and the robot is still broken.
Detectors read the graph and raise faults through a ReportFault service client on the
gateway node; the faults surface via FaultManager on the gateway /faults API. That
client, like the lifecycle subscriptions below, lives in the plugin’s own callback group
and is driven by the plugin’s own executor from the tick thread, never by the gateway’s
ROS executor.
This package carries the plugin skeleton, the central reliability gate that holds raises
until the graph has quiesced, and four detectors, qos_mismatch, orphan,
param_drift and lifecycle_expectation. The remaining
silent-fault classes land in follow-up changes, each against its own issue; their fault
codes are already reserved in the frozen GRAPH_* namespace (see “Fault codes”).
Build
With the gateway built and sourced:
colcon build --packages-select ros2_medkit_graph_watchdog
colcon test --packages-select ros2_medkit_graph_watchdog && colcon test-result --verbose
Load into the gateway
ros2 run ros2_medkit_gateway gateway_node --ros-args \
-p plugins:="[graph_watchdog]" \
-p "plugins.graph_watchdog.path:=$(ros2 pkg prefix ros2_medkit_graph_watchdog)/lib/ros2_medkit_graph_watchdog/libros2_medkit_graph_watchdog.so"
Configuration (plugins.graph_watchdog.<key>)
| Key | Type | Default | Meaning |
|---|---|---|---|
tick_interval_ms |
int | 1000 |
Detector tick cadence. |
warmup_cycles |
int | 5 |
An entity must be continuously present for this many ticks before it arms. A mid-run restart re-warms it. Enforced centrally, see “Reliability (bringup-quiesce)” below. |
prune_grace |
int | 60 |
Default injected into every detector’s own config before configure(); a per-detector detectors.<id>.prune_grace overrides it. Used by detectors that keep per-key bookkeeping. |
detectors.<id>.mode |
string | bool | raise |
raise = push faults; advisory = observe, do not push; off = disabled. Bare off/no (YAML booleans) also disable; bare on/yes mean raise. |
detectors.<id>.<field> |
any | - | Per-detector thresholds, passed to that detector’s configure(). |
Nested delivery. Keys are written dotted in YAML or --ros-args, and the gateway
delivers them to the plugin as a nested object. A bare off is a YAML 1.1 boolean, so
mode: off arrives as false, not the string "off"; both forms disable the detector.
Unknown keys are reported. Anything under detectors.<id> that the detector does not
read produces one startup warning naming the key and listing the ones that exist, so a
typo like allow_list: for allowlist: cannot silently do nothing. A typo in the detector
id itself is warned about the same way, with the registered ids listed.
lifecycle_expectation keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. |
require_active |
string[] | [] |
Node names that must be in the active lifecycle state, each matched against a live node by its App::id, its full FQN (/ns/name), or the bare leaf of that FQN. A bare name matches that node in EVERY namespace (a fleet-wide “all controller_servers must be active”); use a full FQN to pin one robot’s. Both the bare and the FQN form match against the node’s stable FQN, so they survive the App::id renaming that a same-bare-name collision triggers; an entry written as an App::id also works, but can stop matching on a multi-robot graph for exactly that reason. Empty = the detector checks nothing and emits nothing. A require_active that is not a string array warns and is ignored; an empty-string entry warns and is skipped - never dropped silently, since the operator would go on believing the node is covered. |
grace |
int | 5 |
Consecutive not-active ticks a required node tolerates before being reported inactive - bringup time for a managed node to reach active (configure + activate) before the expectation is enforced. Counted per NODE, not per matching entry: a node named by both a bare-name and a full-FQN entry still advances its streak once per tick, so mixing the two documented forms cannot halve the grace that was configured. Past the absence grace the streak keeps advancing while the node is ABSENT, so a node that leaves the graph while violating is still confirmed. Accepted range 0..300 - five minutes at the shipped 1 s cadence, and already an extravagant allowance for a managed node to reach active. The upper end is a real bound rather than a formality: grace also decides how long a node that LEFT the graph while not-active sits unsettled in the tracker, and GRAPH_NODE_INACTIVE’s clear is withheld for EVERY node while it does, so at the old maximum of INT_MAX - 1 the fault could neither raise nor heal for about 24 days. The check runs on the wide integer, so a value that only fits after truncation is rejected rather than silently turned into a hair-trigger. Anything outside the range warns and keeps the default. |
prune_grace |
int | 60 |
Consecutive ticks an IDLE tracked node - both clocks at zero and no matured ownership, so nothing to lose - may stay ABSENT from the graph before its bookkeeping is reclaimed. A node carrying evidence is never reclaimed by age at all, however long it stays gone, so this key cannot erase a fault’s own state; the map is bounded instead by tracked_node_cap (see “Bounded by evidence, not by age”). Injected for every detector at plugin scope and overridable per detector; a value outside 0..3600 warns and this detector keeps its own default of 60. The range check runs on the wide integer, so an out-of-int-range value is rejected rather than truncated. The value is used as written - there is no grace + 1 clamp, because there is no longer anything for one to protect. |
tracked_node_cap |
int | 512 |
The most nodes this detector keeps bookkeeping for at once. It is what bounds the map, since evidence is never reclaimed by age (see “Bounded by evidence, not by age”): at the cap, idle entries are reclaimed first, then entries for DEPARTED nodes are collapsed into a count, and only if every tracked node is PRESENT and carrying evidence is a newly matched node refused - which withholds GRAPH_NODE_INACTIVE’s clear and is reported both in the log and on GET /x-medkit-watchdog. 512 is comfortably above every node in a realistic graph (a full Nav2 stack plus perception is roughly a hundred nodes; a ten-robot fleet sharing one domain a few hundred), so raising it is only needed where require_active legitimately matches more PRESENT nodes than that. Accepted range 1..16384 - 16384 is about 8 MB of bookkeeping at roughly 500 bytes per tracked node, and a deployment needing more distinct required-node identities alive at once has identity churn rather than a large fleet. Anything outside the range, including 0 (a cap of nothing would mean checking nothing), warns and keeps the default; the check runs on the wide integer, so an out-of-int-range value is rejected rather than truncated. |
orphan keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | raise |
As above. |
grace |
int | 10 |
Consecutive sweeps a candidate pair must hold before it is reported. A staged bringup or a restart makes one side missing for a short time, which looks exactly like a typo until the other side appears. |
max_edit_distance |
int | 1 |
How many character edits the LEAF (the part after the last /) may differ by and still count as the same intended name. Values outside 1..8 keep the default: past a few edits a “near miss” starts matching unrelated topics. |
namespace_edit_distance |
int | 0 |
The same budget for the namespace, spent separately. 0 means the namespace must match to the character, which is what makes /robot1/scan and /robot2/scan two robots rather than a typo. Raise it to catch a misspelled namespace, and read the warning under the table first. Values outside 0..8 keep the default. |
allowlist |
string[] | [] |
Topic names never reported. Exact match only, never a prefix, so allowlisting /r1/scan does not also silence /r2/scan. |
What this detector will not catch, on purpose. A pair whose names match once every run of
digits is collapsed is treated as an enumeration of sensors, not a typo: /lidar_1 next to
/lidar_2, each one-sided, is an ordinary state on a multi-sensor robot and reporting it would
be a false alarm on every such machine. Collapsing runs rather than comparing character
positions is what also covers /lidar_9 next to /lidar_10. The cost is that a typo which
is a digit stops being reported. An appended digit is not that case: /scan1 collapses to
/scan#, which is not /scan, so it is still reported.
A misspelled namespace is opt-in, and it can be expensive. /robott/scan against
/robot/scan is one edit, but in the namespace, so by default it is not reported. Setting
namespace_edit_distance: 1 reports it. Check the naming scheme before doing that: on a fleet
named with letters, /amr_a and /amr_b are also one edit apart, and every robot in the
fleet will be reported as a typo. A numbered fleet is safe, since /robot1 against /robot2
differs only in its numeric field and the rule above already spares it.
What it cannot catch at all. A namespace added or dropped by mistake, /scan against
/robot/scan, is six edits apart, so no budget that is still specific will reach it. Do not
rely on this detector for that class.
param_drift keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. A bare off arrives as a YAML boolean; both forms disable the detector. |
baseline |
bool | true |
Self-capture the value of every parameter not listed in expect, then flag later changes to it. false checks only what expect pins, and skips reading the full parameter list. |
expect.<param> |
any | - | Absolute expected value for <param>, checked on every node that has it. This is the only rule that catches a value that was wrong from the first tick, which self-capture cannot see. A node that does not declare <param> answers NOT_FOUND, which is not a drift - so a misspelt name checks nothing at all. Once the sweep has covered every armed app - each one read, or given up on after repeated failed reads - and at least one of them answered, a pin that no app declares is warned about once, naming it. |
ignore |
string[] | [] |
Parameter-name globs never flagged. Applies to the self-captured set only, so it cannot silence an expect pin. A pattern of nothing but * matches every name and suppresses self-capture wholesale, which is indistinguishable at runtime from finding nothing; that combination warns at startup. |
max_reads_per_tick |
int | 8 |
Parameter-service ROUND TRIPS this detector may spend per tick, round-robin over nodes. Round trips, not calls: one call is several requests at the watched node, and the node pays for each of them. A baseline visit is charged 2 (a list, then one batched get) whatever the node’s parameter count; each expect pin is charged 3 (a list, a get and a descriptor read), or 2 on a node that does not declare it - see the table under “Coverage latency” for what each of those really costs the node. So a self-capture visit spends 2 of the budget and pinning N parameters spends up to 3N. It is an AMORTISED rate, not a per-tick cap: there is no pacing INSIDE a visit, so a node’s whole read set is issued back to back and the reader waits out the debt afterwards. Four pins against a budget of 8 fire 12 round trips inside one tick period and then idle for the rest of the next one; what the knob bounds is the sustained rate, not the burst. Accepted range 1..100000; anything else keeps the default and warns. |
prune_grace |
int | 60 |
Consecutive sweeps an app may be missing before its finding is dropped: the finding survives prune_grace absences and goes on the next one. Injected for every detector at plugin scope and overridable per detector; a value outside 0..3600 is rejected with a warning and this detector falls back to 2, not to the plugin default. The captured BASELINE has a shorter horizon of its own, and that one is NOT configurable - it is the compile-time kBaselineForgetGrace: the baseline is dropped on the third consecutive missed sweep, the two the reliability gate absorbs plus one, so an ordinary dropped discovery poll cannot re-capture a node on its already-drifted value. Setting prune_grace below 2 pulls that forward onto the sweep the finding itself is dropped on, so the two then go together. |
qos_mismatch keys
File truncated at 100 lines see the full file