ROS Index
Package Summary
| Version | 0.6.0 |
| License | Apache-2.0 |
| Build type | AMENT_CMAKE |
| Use | RECOMMENDED |
Repository Summary
| Checkout URI | https://github.com/selfpatch/ros2_medkit.git |
| VCS Type | git |
| VCS Version | main |
| Last Updated | 2026-07-31 |
| Dev Status | DEVELOPED |
| Released | RELEASED |
| Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
Maintainers
- bburda
Authors
ros2_medkit_graph_watchdog
Gateway plugin that detects silent faults in the ROS 2 graph: failures where every node is up, nothing logs an error, and the robot is still broken.
Detectors read the graph and raise faults through a ReportFault service client on the
gateway node; the faults surface via FaultManager on the gateway /faults API.
This package carries the plugin skeleton, the central reliability gate that holds raises
until the graph has quiesced, and three detectors, qos_mismatch, orphan and
param_drift. The remaining
silent-fault classes land in follow-up changes, each against its own issue; their fault
codes are already reserved in the frozen GRAPH_* namespace (see “Fault codes”).
Build
With the gateway built and sourced:
colcon build --packages-select ros2_medkit_graph_watchdog
colcon test --packages-select ros2_medkit_graph_watchdog && colcon test-result --verbose
Load into the gateway
ros2 run ros2_medkit_gateway gateway_node --ros-args \
-p plugins:="[graph_watchdog]" \
-p "plugins.graph_watchdog.path:=$(ros2 pkg prefix ros2_medkit_graph_watchdog)/lib/ros2_medkit_graph_watchdog/libros2_medkit_graph_watchdog.so"
Configuration (plugins.graph_watchdog.<key>)
| Key | Type | Default | Meaning |
|---|---|---|---|
tick_interval_ms |
int | 1000 |
Detector tick cadence. |
warmup_cycles |
int | 5 |
An entity must be continuously present for this many ticks before it arms. A mid-run restart re-warms it. Enforced centrally, see “Reliability (bringup-quiesce)” below. |
prune_grace |
int | 60 |
Default injected into every detector’s own config before configure(); a per-detector detectors.<id>.prune_grace overrides it. Used by detectors that keep per-key bookkeeping. |
detectors.<id>.mode |
string | bool | raise |
raise = push faults; advisory = observe, do not push; off = disabled. Bare off/no (YAML booleans) also disable; bare on/yes mean raise. |
detectors.<id>.<field> |
any | - | Per-detector thresholds, passed to that detector’s configure(). |
Nested delivery. Keys are written dotted in YAML or --ros-args, and the gateway
delivers them to the plugin as a nested object. A bare off is a YAML 1.1 boolean, so
mode: off arrives as false, not the string "off"; both forms disable the detector.
Unknown keys are reported. Anything under detectors.<id> that the detector does not
read produces one startup warning naming the key and listing the ones that exist, so a
typo like allow_list: for allowlist: cannot silently do nothing. A typo in the detector
id itself is warned about the same way, with the registered ids listed.
orphan keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | raise |
As above. |
grace |
int | 10 |
Consecutive sweeps a candidate pair must hold before it is reported. A staged bringup or a restart makes one side missing for a short time, which looks exactly like a typo until the other side appears. |
max_edit_distance |
int | 1 |
How many character edits the LEAF (the part after the last /) may differ by and still count as the same intended name. Values outside 1..8 keep the default: past a few edits a “near miss” starts matching unrelated topics. |
namespace_edit_distance |
int | 0 |
The same budget for the namespace, spent separately. 0 means the namespace must match to the character, which is what makes /robot1/scan and /robot2/scan two robots rather than a typo. Raise it to catch a misspelled namespace, and read the warning under the table first. Values outside 0..8 keep the default. |
allowlist |
string[] | [] |
Topic names never reported. Exact match only, never a prefix, so allowlisting /r1/scan does not also silence /r2/scan. |
What this detector will not catch, on purpose. A pair whose names match once every run of
digits is collapsed is treated as an enumeration of sensors, not a typo: /lidar_1 next to
/lidar_2, each one-sided, is an ordinary state on a multi-sensor robot and reporting it would
be a false alarm on every such machine. Collapsing runs rather than comparing character
positions is what also covers /lidar_9 next to /lidar_10. The cost is that a typo which
is a digit stops being reported. An appended digit is not that case: /scan1 collapses to
/scan#, which is not /scan, so it is still reported.
A misspelled namespace is opt-in, and it can be expensive. /robott/scan against
/robot/scan is one edit, but in the namespace, so by default it is not reported. Setting
namespace_edit_distance: 1 reports it. Check the naming scheme before doing that: on a fleet
named with letters, /amr_a and /amr_b are also one edit apart, and every robot in the
fleet will be reported as a typo. A numbered fleet is safe, since /robot1 against /robot2
differs only in its numeric field and the rule above already spares it.
What it cannot catch at all. A namespace added or dropped by mistake, /scan against
/robot/scan, is six edits apart, so no budget that is still specific will reach it. Do not
rely on this detector for that class.
param_drift keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. A bare off arrives as a YAML boolean; both forms disable the detector. |
baseline |
bool | true |
Self-capture the value of every parameter not listed in expect, then flag later changes to it. false checks only what expect pins, and skips reading the full parameter list. |
expect.<param> |
any | - | Absolute expected value for <param>, checked on every node that has it. This is the only rule that catches a value that was wrong from the first tick, which self-capture cannot see. A node that does not declare <param> answers NOT_FOUND, which is not a drift - so a misspelt name checks nothing at all. Once the sweep has covered every armed app - each one read, or given up on after repeated failed reads - and at least one of them answered, a pin that no app declares is warned about once, naming it. |
ignore |
string[] | [] |
Parameter-name globs never flagged. Applies to the self-captured set only, so it cannot silence an expect pin. A pattern of nothing but * matches every name and suppresses self-capture wholesale, which is indistinguishable at runtime from finding nothing; that combination warns at startup. |
max_reads_per_tick |
int | 8 |
Parameter-service ROUND TRIPS this detector may spend per tick, round-robin over nodes. Round trips, not calls: one call is several requests at the watched node, and the node pays for each of them. A baseline visit is charged 2 (a list, then one batched get) whatever the node’s parameter count; each expect pin is charged 3 (a list, a get and a descriptor read), or 2 on a node that does not declare it - see the table under “Coverage latency” for what each of those really costs the node. So a self-capture visit spends 2 of the budget and pinning N parameters spends up to 3N. It is an AMORTISED rate, not a per-tick cap: there is no pacing INSIDE a visit, so a node’s whole read set is issued back to back and the reader waits out the debt afterwards. Four pins against a budget of 8 fire 12 round trips inside one tick period and then idle for the rest of the next one; what the knob bounds is the sustained rate, not the burst. Accepted range 1..100000; anything else keeps the default and warns. |
prune_grace |
int | 60 |
Consecutive sweeps an app may be missing before its finding is dropped: the finding survives prune_grace absences and goes on the next one. Injected for every detector at plugin scope and overridable per detector; a value outside 0..3600 is rejected with a warning and this detector falls back to 2, not to the plugin default. The captured BASELINE has a shorter horizon of its own, and that one is NOT configurable - it is the compile-time kBaselineForgetGrace: the baseline is dropped on the third consecutive missed sweep, the two the reliability gate absorbs plus one, so an ordinary dropped discovery poll cannot re-capture a node on its already-drifted value. Setting prune_grace below 2 pulls that forward onto the sweep the finding itself is dropped on, so the two then go together. |
qos_mismatch keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | raise |
As above. |
grace |
int | 3 |
Consecutive ticks a topic must stay affected before it is reported. Endpoint discovery is not atomic, so a subscriber can be visible before a publisher’s QoS is, which reads as starvation for a tick or two. |
allowlist |
string[] | [] |
Subscriber FQNs never reported. Exact match only. |
Closing the loop: healing config is required
Every detector here reports level-triggered: it re-raises FAILED while the condition holds and emits PASSED on every clean sweep. That is what lets the fault_manager’s debounce counter walk from CONFIRMED back to HEALED. The fault_manager only counts PASSED events
File truncated at 100 lines see the full file
Package Dependencies
System Dependencies
Dependant Packages
Launch files
Messages
Services
Plugins
Recent questions tagged ros2_medkit_graph_watchdog at Robotics Stack Exchange
Package Summary
| Version | 0.6.0 |
| License | Apache-2.0 |
| Build type | AMENT_CMAKE |
| Use | RECOMMENDED |
Repository Summary
| Checkout URI | https://github.com/selfpatch/ros2_medkit.git |
| VCS Type | git |
| VCS Version | main |
| Last Updated | 2026-07-31 |
| Dev Status | DEVELOPED |
| Released | RELEASED |
| Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
Maintainers
- bburda
Authors
ros2_medkit_graph_watchdog
Gateway plugin that detects silent faults in the ROS 2 graph: failures where every node is up, nothing logs an error, and the robot is still broken.
Detectors read the graph and raise faults through a ReportFault service client on the
gateway node; the faults surface via FaultManager on the gateway /faults API.
This package carries the plugin skeleton, the central reliability gate that holds raises
until the graph has quiesced, and three detectors, qos_mismatch, orphan and
param_drift. The remaining
silent-fault classes land in follow-up changes, each against its own issue; their fault
codes are already reserved in the frozen GRAPH_* namespace (see “Fault codes”).
Build
With the gateway built and sourced:
colcon build --packages-select ros2_medkit_graph_watchdog
colcon test --packages-select ros2_medkit_graph_watchdog && colcon test-result --verbose
Load into the gateway
ros2 run ros2_medkit_gateway gateway_node --ros-args \
-p plugins:="[graph_watchdog]" \
-p "plugins.graph_watchdog.path:=$(ros2 pkg prefix ros2_medkit_graph_watchdog)/lib/ros2_medkit_graph_watchdog/libros2_medkit_graph_watchdog.so"
Configuration (plugins.graph_watchdog.<key>)
| Key | Type | Default | Meaning |
|---|---|---|---|
tick_interval_ms |
int | 1000 |
Detector tick cadence. |
warmup_cycles |
int | 5 |
An entity must be continuously present for this many ticks before it arms. A mid-run restart re-warms it. Enforced centrally, see “Reliability (bringup-quiesce)” below. |
prune_grace |
int | 60 |
Default injected into every detector’s own config before configure(); a per-detector detectors.<id>.prune_grace overrides it. Used by detectors that keep per-key bookkeeping. |
detectors.<id>.mode |
string | bool | raise |
raise = push faults; advisory = observe, do not push; off = disabled. Bare off/no (YAML booleans) also disable; bare on/yes mean raise. |
detectors.<id>.<field> |
any | - | Per-detector thresholds, passed to that detector’s configure(). |
Nested delivery. Keys are written dotted in YAML or --ros-args, and the gateway
delivers them to the plugin as a nested object. A bare off is a YAML 1.1 boolean, so
mode: off arrives as false, not the string "off"; both forms disable the detector.
Unknown keys are reported. Anything under detectors.<id> that the detector does not
read produces one startup warning naming the key and listing the ones that exist, so a
typo like allow_list: for allowlist: cannot silently do nothing. A typo in the detector
id itself is warned about the same way, with the registered ids listed.
orphan keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | raise |
As above. |
grace |
int | 10 |
Consecutive sweeps a candidate pair must hold before it is reported. A staged bringup or a restart makes one side missing for a short time, which looks exactly like a typo until the other side appears. |
max_edit_distance |
int | 1 |
How many character edits the LEAF (the part after the last /) may differ by and still count as the same intended name. Values outside 1..8 keep the default: past a few edits a “near miss” starts matching unrelated topics. |
namespace_edit_distance |
int | 0 |
The same budget for the namespace, spent separately. 0 means the namespace must match to the character, which is what makes /robot1/scan and /robot2/scan two robots rather than a typo. Raise it to catch a misspelled namespace, and read the warning under the table first. Values outside 0..8 keep the default. |
allowlist |
string[] | [] |
Topic names never reported. Exact match only, never a prefix, so allowlisting /r1/scan does not also silence /r2/scan. |
What this detector will not catch, on purpose. A pair whose names match once every run of
digits is collapsed is treated as an enumeration of sensors, not a typo: /lidar_1 next to
/lidar_2, each one-sided, is an ordinary state on a multi-sensor robot and reporting it would
be a false alarm on every such machine. Collapsing runs rather than comparing character
positions is what also covers /lidar_9 next to /lidar_10. The cost is that a typo which
is a digit stops being reported. An appended digit is not that case: /scan1 collapses to
/scan#, which is not /scan, so it is still reported.
A misspelled namespace is opt-in, and it can be expensive. /robott/scan against
/robot/scan is one edit, but in the namespace, so by default it is not reported. Setting
namespace_edit_distance: 1 reports it. Check the naming scheme before doing that: on a fleet
named with letters, /amr_a and /amr_b are also one edit apart, and every robot in the
fleet will be reported as a typo. A numbered fleet is safe, since /robot1 against /robot2
differs only in its numeric field and the rule above already spares it.
What it cannot catch at all. A namespace added or dropped by mistake, /scan against
/robot/scan, is six edits apart, so no budget that is still specific will reach it. Do not
rely on this detector for that class.
param_drift keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. A bare off arrives as a YAML boolean; both forms disable the detector. |
baseline |
bool | true |
Self-capture the value of every parameter not listed in expect, then flag later changes to it. false checks only what expect pins, and skips reading the full parameter list. |
expect.<param> |
any | - | Absolute expected value for <param>, checked on every node that has it. This is the only rule that catches a value that was wrong from the first tick, which self-capture cannot see. A node that does not declare <param> answers NOT_FOUND, which is not a drift - so a misspelt name checks nothing at all. Once the sweep has covered every armed app - each one read, or given up on after repeated failed reads - and at least one of them answered, a pin that no app declares is warned about once, naming it. |
ignore |
string[] | [] |
Parameter-name globs never flagged. Applies to the self-captured set only, so it cannot silence an expect pin. A pattern of nothing but * matches every name and suppresses self-capture wholesale, which is indistinguishable at runtime from finding nothing; that combination warns at startup. |
max_reads_per_tick |
int | 8 |
Parameter-service ROUND TRIPS this detector may spend per tick, round-robin over nodes. Round trips, not calls: one call is several requests at the watched node, and the node pays for each of them. A baseline visit is charged 2 (a list, then one batched get) whatever the node’s parameter count; each expect pin is charged 3 (a list, a get and a descriptor read), or 2 on a node that does not declare it - see the table under “Coverage latency” for what each of those really costs the node. So a self-capture visit spends 2 of the budget and pinning N parameters spends up to 3N. It is an AMORTISED rate, not a per-tick cap: there is no pacing INSIDE a visit, so a node’s whole read set is issued back to back and the reader waits out the debt afterwards. Four pins against a budget of 8 fire 12 round trips inside one tick period and then idle for the rest of the next one; what the knob bounds is the sustained rate, not the burst. Accepted range 1..100000; anything else keeps the default and warns. |
prune_grace |
int | 60 |
Consecutive sweeps an app may be missing before its finding is dropped: the finding survives prune_grace absences and goes on the next one. Injected for every detector at plugin scope and overridable per detector; a value outside 0..3600 is rejected with a warning and this detector falls back to 2, not to the plugin default. The captured BASELINE has a shorter horizon of its own, and that one is NOT configurable - it is the compile-time kBaselineForgetGrace: the baseline is dropped on the third consecutive missed sweep, the two the reliability gate absorbs plus one, so an ordinary dropped discovery poll cannot re-capture a node on its already-drifted value. Setting prune_grace below 2 pulls that forward onto the sweep the finding itself is dropped on, so the two then go together. |
qos_mismatch keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | raise |
As above. |
grace |
int | 3 |
Consecutive ticks a topic must stay affected before it is reported. Endpoint discovery is not atomic, so a subscriber can be visible before a publisher’s QoS is, which reads as starvation for a tick or two. |
allowlist |
string[] | [] |
Subscriber FQNs never reported. Exact match only. |
Closing the loop: healing config is required
Every detector here reports level-triggered: it re-raises FAILED while the condition holds and emits PASSED on every clean sweep. That is what lets the fault_manager’s debounce counter walk from CONFIRMED back to HEALED. The fault_manager only counts PASSED events
File truncated at 100 lines see the full file
Package Dependencies
System Dependencies
Dependant Packages
Launch files
Messages
Services
Plugins
Recent questions tagged ros2_medkit_graph_watchdog at Robotics Stack Exchange
Package Summary
| Version | 0.6.0 |
| License | Apache-2.0 |
| Build type | AMENT_CMAKE |
| Use | RECOMMENDED |
Repository Summary
| Checkout URI | https://github.com/selfpatch/ros2_medkit.git |
| VCS Type | git |
| VCS Version | main |
| Last Updated | 2026-07-31 |
| Dev Status | DEVELOPED |
| Released | RELEASED |
| Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
Maintainers
- bburda
Authors
ros2_medkit_graph_watchdog
Gateway plugin that detects silent faults in the ROS 2 graph: failures where every node is up, nothing logs an error, and the robot is still broken.
Detectors read the graph and raise faults through a ReportFault service client on the
gateway node; the faults surface via FaultManager on the gateway /faults API.
This package carries the plugin skeleton, the central reliability gate that holds raises
until the graph has quiesced, and three detectors, qos_mismatch, orphan and
param_drift. The remaining
silent-fault classes land in follow-up changes, each against its own issue; their fault
codes are already reserved in the frozen GRAPH_* namespace (see “Fault codes”).
Build
With the gateway built and sourced:
colcon build --packages-select ros2_medkit_graph_watchdog
colcon test --packages-select ros2_medkit_graph_watchdog && colcon test-result --verbose
Load into the gateway
ros2 run ros2_medkit_gateway gateway_node --ros-args \
-p plugins:="[graph_watchdog]" \
-p "plugins.graph_watchdog.path:=$(ros2 pkg prefix ros2_medkit_graph_watchdog)/lib/ros2_medkit_graph_watchdog/libros2_medkit_graph_watchdog.so"
Configuration (plugins.graph_watchdog.<key>)
| Key | Type | Default | Meaning |
|---|---|---|---|
tick_interval_ms |
int | 1000 |
Detector tick cadence. |
warmup_cycles |
int | 5 |
An entity must be continuously present for this many ticks before it arms. A mid-run restart re-warms it. Enforced centrally, see “Reliability (bringup-quiesce)” below. |
prune_grace |
int | 60 |
Default injected into every detector’s own config before configure(); a per-detector detectors.<id>.prune_grace overrides it. Used by detectors that keep per-key bookkeeping. |
detectors.<id>.mode |
string | bool | raise |
raise = push faults; advisory = observe, do not push; off = disabled. Bare off/no (YAML booleans) also disable; bare on/yes mean raise. |
detectors.<id>.<field> |
any | - | Per-detector thresholds, passed to that detector’s configure(). |
Nested delivery. Keys are written dotted in YAML or --ros-args, and the gateway
delivers them to the plugin as a nested object. A bare off is a YAML 1.1 boolean, so
mode: off arrives as false, not the string "off"; both forms disable the detector.
Unknown keys are reported. Anything under detectors.<id> that the detector does not
read produces one startup warning naming the key and listing the ones that exist, so a
typo like allow_list: for allowlist: cannot silently do nothing. A typo in the detector
id itself is warned about the same way, with the registered ids listed.
orphan keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | raise |
As above. |
grace |
int | 10 |
Consecutive sweeps a candidate pair must hold before it is reported. A staged bringup or a restart makes one side missing for a short time, which looks exactly like a typo until the other side appears. |
max_edit_distance |
int | 1 |
How many character edits the LEAF (the part after the last /) may differ by and still count as the same intended name. Values outside 1..8 keep the default: past a few edits a “near miss” starts matching unrelated topics. |
namespace_edit_distance |
int | 0 |
The same budget for the namespace, spent separately. 0 means the namespace must match to the character, which is what makes /robot1/scan and /robot2/scan two robots rather than a typo. Raise it to catch a misspelled namespace, and read the warning under the table first. Values outside 0..8 keep the default. |
allowlist |
string[] | [] |
Topic names never reported. Exact match only, never a prefix, so allowlisting /r1/scan does not also silence /r2/scan. |
What this detector will not catch, on purpose. A pair whose names match once every run of
digits is collapsed is treated as an enumeration of sensors, not a typo: /lidar_1 next to
/lidar_2, each one-sided, is an ordinary state on a multi-sensor robot and reporting it would
be a false alarm on every such machine. Collapsing runs rather than comparing character
positions is what also covers /lidar_9 next to /lidar_10. The cost is that a typo which
is a digit stops being reported. An appended digit is not that case: /scan1 collapses to
/scan#, which is not /scan, so it is still reported.
A misspelled namespace is opt-in, and it can be expensive. /robott/scan against
/robot/scan is one edit, but in the namespace, so by default it is not reported. Setting
namespace_edit_distance: 1 reports it. Check the naming scheme before doing that: on a fleet
named with letters, /amr_a and /amr_b are also one edit apart, and every robot in the
fleet will be reported as a typo. A numbered fleet is safe, since /robot1 against /robot2
differs only in its numeric field and the rule above already spares it.
What it cannot catch at all. A namespace added or dropped by mistake, /scan against
/robot/scan, is six edits apart, so no budget that is still specific will reach it. Do not
rely on this detector for that class.
param_drift keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. A bare off arrives as a YAML boolean; both forms disable the detector. |
baseline |
bool | true |
Self-capture the value of every parameter not listed in expect, then flag later changes to it. false checks only what expect pins, and skips reading the full parameter list. |
expect.<param> |
any | - | Absolute expected value for <param>, checked on every node that has it. This is the only rule that catches a value that was wrong from the first tick, which self-capture cannot see. A node that does not declare <param> answers NOT_FOUND, which is not a drift - so a misspelt name checks nothing at all. Once the sweep has covered every armed app - each one read, or given up on after repeated failed reads - and at least one of them answered, a pin that no app declares is warned about once, naming it. |
ignore |
string[] | [] |
Parameter-name globs never flagged. Applies to the self-captured set only, so it cannot silence an expect pin. A pattern of nothing but * matches every name and suppresses self-capture wholesale, which is indistinguishable at runtime from finding nothing; that combination warns at startup. |
max_reads_per_tick |
int | 8 |
Parameter-service ROUND TRIPS this detector may spend per tick, round-robin over nodes. Round trips, not calls: one call is several requests at the watched node, and the node pays for each of them. A baseline visit is charged 2 (a list, then one batched get) whatever the node’s parameter count; each expect pin is charged 3 (a list, a get and a descriptor read), or 2 on a node that does not declare it - see the table under “Coverage latency” for what each of those really costs the node. So a self-capture visit spends 2 of the budget and pinning N parameters spends up to 3N. It is an AMORTISED rate, not a per-tick cap: there is no pacing INSIDE a visit, so a node’s whole read set is issued back to back and the reader waits out the debt afterwards. Four pins against a budget of 8 fire 12 round trips inside one tick period and then idle for the rest of the next one; what the knob bounds is the sustained rate, not the burst. Accepted range 1..100000; anything else keeps the default and warns. |
prune_grace |
int | 60 |
Consecutive sweeps an app may be missing before its finding is dropped: the finding survives prune_grace absences and goes on the next one. Injected for every detector at plugin scope and overridable per detector; a value outside 0..3600 is rejected with a warning and this detector falls back to 2, not to the plugin default. The captured BASELINE has a shorter horizon of its own, and that one is NOT configurable - it is the compile-time kBaselineForgetGrace: the baseline is dropped on the third consecutive missed sweep, the two the reliability gate absorbs plus one, so an ordinary dropped discovery poll cannot re-capture a node on its already-drifted value. Setting prune_grace below 2 pulls that forward onto the sweep the finding itself is dropped on, so the two then go together. |
qos_mismatch keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | raise |
As above. |
grace |
int | 3 |
Consecutive ticks a topic must stay affected before it is reported. Endpoint discovery is not atomic, so a subscriber can be visible before a publisher’s QoS is, which reads as starvation for a tick or two. |
allowlist |
string[] | [] |
Subscriber FQNs never reported. Exact match only. |
Closing the loop: healing config is required
Every detector here reports level-triggered: it re-raises FAILED while the condition holds and emits PASSED on every clean sweep. That is what lets the fault_manager’s debounce counter walk from CONFIRMED back to HEALED. The fault_manager only counts PASSED events
File truncated at 100 lines see the full file
Package Dependencies
System Dependencies
Dependant Packages
Launch files
Messages
Services
Plugins
Recent questions tagged ros2_medkit_graph_watchdog at Robotics Stack Exchange
Package Summary
| Version | 0.6.0 |
| License | Apache-2.0 |
| Build type | AMENT_CMAKE |
| Use | RECOMMENDED |
Repository Summary
| Checkout URI | https://github.com/selfpatch/ros2_medkit.git |
| VCS Type | git |
| VCS Version | main |
| Last Updated | 2026-07-31 |
| Dev Status | DEVELOPED |
| Released | RELEASED |
| Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
Maintainers
- bburda
Authors
ros2_medkit_graph_watchdog
Gateway plugin that detects silent faults in the ROS 2 graph: failures where every node is up, nothing logs an error, and the robot is still broken.
Detectors read the graph and raise faults through a ReportFault service client on the
gateway node; the faults surface via FaultManager on the gateway /faults API.
This package carries the plugin skeleton, the central reliability gate that holds raises
until the graph has quiesced, and three detectors, qos_mismatch, orphan and
param_drift. The remaining
silent-fault classes land in follow-up changes, each against its own issue; their fault
codes are already reserved in the frozen GRAPH_* namespace (see “Fault codes”).
Build
With the gateway built and sourced:
colcon build --packages-select ros2_medkit_graph_watchdog
colcon test --packages-select ros2_medkit_graph_watchdog && colcon test-result --verbose
Load into the gateway
ros2 run ros2_medkit_gateway gateway_node --ros-args \
-p plugins:="[graph_watchdog]" \
-p "plugins.graph_watchdog.path:=$(ros2 pkg prefix ros2_medkit_graph_watchdog)/lib/ros2_medkit_graph_watchdog/libros2_medkit_graph_watchdog.so"
Configuration (plugins.graph_watchdog.<key>)
| Key | Type | Default | Meaning |
|---|---|---|---|
tick_interval_ms |
int | 1000 |
Detector tick cadence. |
warmup_cycles |
int | 5 |
An entity must be continuously present for this many ticks before it arms. A mid-run restart re-warms it. Enforced centrally, see “Reliability (bringup-quiesce)” below. |
prune_grace |
int | 60 |
Default injected into every detector’s own config before configure(); a per-detector detectors.<id>.prune_grace overrides it. Used by detectors that keep per-key bookkeeping. |
detectors.<id>.mode |
string | bool | raise |
raise = push faults; advisory = observe, do not push; off = disabled. Bare off/no (YAML booleans) also disable; bare on/yes mean raise. |
detectors.<id>.<field> |
any | - | Per-detector thresholds, passed to that detector’s configure(). |
Nested delivery. Keys are written dotted in YAML or --ros-args, and the gateway
delivers them to the plugin as a nested object. A bare off is a YAML 1.1 boolean, so
mode: off arrives as false, not the string "off"; both forms disable the detector.
Unknown keys are reported. Anything under detectors.<id> that the detector does not
read produces one startup warning naming the key and listing the ones that exist, so a
typo like allow_list: for allowlist: cannot silently do nothing. A typo in the detector
id itself is warned about the same way, with the registered ids listed.
orphan keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | raise |
As above. |
grace |
int | 10 |
Consecutive sweeps a candidate pair must hold before it is reported. A staged bringup or a restart makes one side missing for a short time, which looks exactly like a typo until the other side appears. |
max_edit_distance |
int | 1 |
How many character edits the LEAF (the part after the last /) may differ by and still count as the same intended name. Values outside 1..8 keep the default: past a few edits a “near miss” starts matching unrelated topics. |
namespace_edit_distance |
int | 0 |
The same budget for the namespace, spent separately. 0 means the namespace must match to the character, which is what makes /robot1/scan and /robot2/scan two robots rather than a typo. Raise it to catch a misspelled namespace, and read the warning under the table first. Values outside 0..8 keep the default. |
allowlist |
string[] | [] |
Topic names never reported. Exact match only, never a prefix, so allowlisting /r1/scan does not also silence /r2/scan. |
What this detector will not catch, on purpose. A pair whose names match once every run of
digits is collapsed is treated as an enumeration of sensors, not a typo: /lidar_1 next to
/lidar_2, each one-sided, is an ordinary state on a multi-sensor robot and reporting it would
be a false alarm on every such machine. Collapsing runs rather than comparing character
positions is what also covers /lidar_9 next to /lidar_10. The cost is that a typo which
is a digit stops being reported. An appended digit is not that case: /scan1 collapses to
/scan#, which is not /scan, so it is still reported.
A misspelled namespace is opt-in, and it can be expensive. /robott/scan against
/robot/scan is one edit, but in the namespace, so by default it is not reported. Setting
namespace_edit_distance: 1 reports it. Check the naming scheme before doing that: on a fleet
named with letters, /amr_a and /amr_b are also one edit apart, and every robot in the
fleet will be reported as a typo. A numbered fleet is safe, since /robot1 against /robot2
differs only in its numeric field and the rule above already spares it.
What it cannot catch at all. A namespace added or dropped by mistake, /scan against
/robot/scan, is six edits apart, so no budget that is still specific will reach it. Do not
rely on this detector for that class.
param_drift keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. A bare off arrives as a YAML boolean; both forms disable the detector. |
baseline |
bool | true |
Self-capture the value of every parameter not listed in expect, then flag later changes to it. false checks only what expect pins, and skips reading the full parameter list. |
expect.<param> |
any | - | Absolute expected value for <param>, checked on every node that has it. This is the only rule that catches a value that was wrong from the first tick, which self-capture cannot see. A node that does not declare <param> answers NOT_FOUND, which is not a drift - so a misspelt name checks nothing at all. Once the sweep has covered every armed app - each one read, or given up on after repeated failed reads - and at least one of them answered, a pin that no app declares is warned about once, naming it. |
ignore |
string[] | [] |
Parameter-name globs never flagged. Applies to the self-captured set only, so it cannot silence an expect pin. A pattern of nothing but * matches every name and suppresses self-capture wholesale, which is indistinguishable at runtime from finding nothing; that combination warns at startup. |
max_reads_per_tick |
int | 8 |
Parameter-service ROUND TRIPS this detector may spend per tick, round-robin over nodes. Round trips, not calls: one call is several requests at the watched node, and the node pays for each of them. A baseline visit is charged 2 (a list, then one batched get) whatever the node’s parameter count; each expect pin is charged 3 (a list, a get and a descriptor read), or 2 on a node that does not declare it - see the table under “Coverage latency” for what each of those really costs the node. So a self-capture visit spends 2 of the budget and pinning N parameters spends up to 3N. It is an AMORTISED rate, not a per-tick cap: there is no pacing INSIDE a visit, so a node’s whole read set is issued back to back and the reader waits out the debt afterwards. Four pins against a budget of 8 fire 12 round trips inside one tick period and then idle for the rest of the next one; what the knob bounds is the sustained rate, not the burst. Accepted range 1..100000; anything else keeps the default and warns. |
prune_grace |
int | 60 |
Consecutive sweeps an app may be missing before its finding is dropped: the finding survives prune_grace absences and goes on the next one. Injected for every detector at plugin scope and overridable per detector; a value outside 0..3600 is rejected with a warning and this detector falls back to 2, not to the plugin default. The captured BASELINE has a shorter horizon of its own, and that one is NOT configurable - it is the compile-time kBaselineForgetGrace: the baseline is dropped on the third consecutive missed sweep, the two the reliability gate absorbs plus one, so an ordinary dropped discovery poll cannot re-capture a node on its already-drifted value. Setting prune_grace below 2 pulls that forward onto the sweep the finding itself is dropped on, so the two then go together. |
qos_mismatch keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | raise |
As above. |
grace |
int | 3 |
Consecutive ticks a topic must stay affected before it is reported. Endpoint discovery is not atomic, so a subscriber can be visible before a publisher’s QoS is, which reads as starvation for a tick or two. |
allowlist |
string[] | [] |
Subscriber FQNs never reported. Exact match only. |
Closing the loop: healing config is required
Every detector here reports level-triggered: it re-raises FAILED while the condition holds and emits PASSED on every clean sweep. That is what lets the fault_manager’s debounce counter walk from CONFIRMED back to HEALED. The fault_manager only counts PASSED events
File truncated at 100 lines see the full file
Package Dependencies
System Dependencies
Dependant Packages
Launch files
Messages
Services
Plugins
Recent questions tagged ros2_medkit_graph_watchdog at Robotics Stack Exchange
Package Summary
| Version | 0.6.0 |
| License | Apache-2.0 |
| Build type | AMENT_CMAKE |
| Use | RECOMMENDED |
Repository Summary
| Checkout URI | https://github.com/selfpatch/ros2_medkit.git |
| VCS Type | git |
| VCS Version | main |
| Last Updated | 2026-07-31 |
| Dev Status | DEVELOPED |
| Released | RELEASED |
| Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
Maintainers
- bburda
Authors
ros2_medkit_graph_watchdog
Gateway plugin that detects silent faults in the ROS 2 graph: failures where every node is up, nothing logs an error, and the robot is still broken.
Detectors read the graph and raise faults through a ReportFault service client on the
gateway node; the faults surface via FaultManager on the gateway /faults API.
This package carries the plugin skeleton, the central reliability gate that holds raises
until the graph has quiesced, and three detectors, qos_mismatch, orphan and
param_drift. The remaining
silent-fault classes land in follow-up changes, each against its own issue; their fault
codes are already reserved in the frozen GRAPH_* namespace (see “Fault codes”).
Build
With the gateway built and sourced:
colcon build --packages-select ros2_medkit_graph_watchdog
colcon test --packages-select ros2_medkit_graph_watchdog && colcon test-result --verbose
Load into the gateway
ros2 run ros2_medkit_gateway gateway_node --ros-args \
-p plugins:="[graph_watchdog]" \
-p "plugins.graph_watchdog.path:=$(ros2 pkg prefix ros2_medkit_graph_watchdog)/lib/ros2_medkit_graph_watchdog/libros2_medkit_graph_watchdog.so"
Configuration (plugins.graph_watchdog.<key>)
| Key | Type | Default | Meaning |
|---|---|---|---|
tick_interval_ms |
int | 1000 |
Detector tick cadence. |
warmup_cycles |
int | 5 |
An entity must be continuously present for this many ticks before it arms. A mid-run restart re-warms it. Enforced centrally, see “Reliability (bringup-quiesce)” below. |
prune_grace |
int | 60 |
Default injected into every detector’s own config before configure(); a per-detector detectors.<id>.prune_grace overrides it. Used by detectors that keep per-key bookkeeping. |
detectors.<id>.mode |
string | bool | raise |
raise = push faults; advisory = observe, do not push; off = disabled. Bare off/no (YAML booleans) also disable; bare on/yes mean raise. |
detectors.<id>.<field> |
any | - | Per-detector thresholds, passed to that detector’s configure(). |
Nested delivery. Keys are written dotted in YAML or --ros-args, and the gateway
delivers them to the plugin as a nested object. A bare off is a YAML 1.1 boolean, so
mode: off arrives as false, not the string "off"; both forms disable the detector.
Unknown keys are reported. Anything under detectors.<id> that the detector does not
read produces one startup warning naming the key and listing the ones that exist, so a
typo like allow_list: for allowlist: cannot silently do nothing. A typo in the detector
id itself is warned about the same way, with the registered ids listed.
orphan keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | raise |
As above. |
grace |
int | 10 |
Consecutive sweeps a candidate pair must hold before it is reported. A staged bringup or a restart makes one side missing for a short time, which looks exactly like a typo until the other side appears. |
max_edit_distance |
int | 1 |
How many character edits the LEAF (the part after the last /) may differ by and still count as the same intended name. Values outside 1..8 keep the default: past a few edits a “near miss” starts matching unrelated topics. |
namespace_edit_distance |
int | 0 |
The same budget for the namespace, spent separately. 0 means the namespace must match to the character, which is what makes /robot1/scan and /robot2/scan two robots rather than a typo. Raise it to catch a misspelled namespace, and read the warning under the table first. Values outside 0..8 keep the default. |
allowlist |
string[] | [] |
Topic names never reported. Exact match only, never a prefix, so allowlisting /r1/scan does not also silence /r2/scan. |
What this detector will not catch, on purpose. A pair whose names match once every run of
digits is collapsed is treated as an enumeration of sensors, not a typo: /lidar_1 next to
/lidar_2, each one-sided, is an ordinary state on a multi-sensor robot and reporting it would
be a false alarm on every such machine. Collapsing runs rather than comparing character
positions is what also covers /lidar_9 next to /lidar_10. The cost is that a typo which
is a digit stops being reported. An appended digit is not that case: /scan1 collapses to
/scan#, which is not /scan, so it is still reported.
A misspelled namespace is opt-in, and it can be expensive. /robott/scan against
/robot/scan is one edit, but in the namespace, so by default it is not reported. Setting
namespace_edit_distance: 1 reports it. Check the naming scheme before doing that: on a fleet
named with letters, /amr_a and /amr_b are also one edit apart, and every robot in the
fleet will be reported as a typo. A numbered fleet is safe, since /robot1 against /robot2
differs only in its numeric field and the rule above already spares it.
What it cannot catch at all. A namespace added or dropped by mistake, /scan against
/robot/scan, is six edits apart, so no budget that is still specific will reach it. Do not
rely on this detector for that class.
param_drift keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. A bare off arrives as a YAML boolean; both forms disable the detector. |
baseline |
bool | true |
Self-capture the value of every parameter not listed in expect, then flag later changes to it. false checks only what expect pins, and skips reading the full parameter list. |
expect.<param> |
any | - | Absolute expected value for <param>, checked on every node that has it. This is the only rule that catches a value that was wrong from the first tick, which self-capture cannot see. A node that does not declare <param> answers NOT_FOUND, which is not a drift - so a misspelt name checks nothing at all. Once the sweep has covered every armed app - each one read, or given up on after repeated failed reads - and at least one of them answered, a pin that no app declares is warned about once, naming it. |
ignore |
string[] | [] |
Parameter-name globs never flagged. Applies to the self-captured set only, so it cannot silence an expect pin. A pattern of nothing but * matches every name and suppresses self-capture wholesale, which is indistinguishable at runtime from finding nothing; that combination warns at startup. |
max_reads_per_tick |
int | 8 |
Parameter-service ROUND TRIPS this detector may spend per tick, round-robin over nodes. Round trips, not calls: one call is several requests at the watched node, and the node pays for each of them. A baseline visit is charged 2 (a list, then one batched get) whatever the node’s parameter count; each expect pin is charged 3 (a list, a get and a descriptor read), or 2 on a node that does not declare it - see the table under “Coverage latency” for what each of those really costs the node. So a self-capture visit spends 2 of the budget and pinning N parameters spends up to 3N. It is an AMORTISED rate, not a per-tick cap: there is no pacing INSIDE a visit, so a node’s whole read set is issued back to back and the reader waits out the debt afterwards. Four pins against a budget of 8 fire 12 round trips inside one tick period and then idle for the rest of the next one; what the knob bounds is the sustained rate, not the burst. Accepted range 1..100000; anything else keeps the default and warns. |
prune_grace |
int | 60 |
Consecutive sweeps an app may be missing before its finding is dropped: the finding survives prune_grace absences and goes on the next one. Injected for every detector at plugin scope and overridable per detector; a value outside 0..3600 is rejected with a warning and this detector falls back to 2, not to the plugin default. The captured BASELINE has a shorter horizon of its own, and that one is NOT configurable - it is the compile-time kBaselineForgetGrace: the baseline is dropped on the third consecutive missed sweep, the two the reliability gate absorbs plus one, so an ordinary dropped discovery poll cannot re-capture a node on its already-drifted value. Setting prune_grace below 2 pulls that forward onto the sweep the finding itself is dropped on, so the two then go together. |
qos_mismatch keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | raise |
As above. |
grace |
int | 3 |
Consecutive ticks a topic must stay affected before it is reported. Endpoint discovery is not atomic, so a subscriber can be visible before a publisher’s QoS is, which reads as starvation for a tick or two. |
allowlist |
string[] | [] |
Subscriber FQNs never reported. Exact match only. |
Closing the loop: healing config is required
Every detector here reports level-triggered: it re-raises FAILED while the condition holds and emits PASSED on every clean sweep. That is what lets the fault_manager’s debounce counter walk from CONFIRMED back to HEALED. The fault_manager only counts PASSED events
File truncated at 100 lines see the full file
Package Dependencies
System Dependencies
Dependant Packages
Launch files
Messages
Services
Plugins
Recent questions tagged ros2_medkit_graph_watchdog at Robotics Stack Exchange
Package Summary
| Version | 0.6.0 |
| License | Apache-2.0 |
| Build type | AMENT_CMAKE |
| Use | RECOMMENDED |
Repository Summary
| Checkout URI | https://github.com/selfpatch/ros2_medkit.git |
| VCS Type | git |
| VCS Version | main |
| Last Updated | 2026-07-31 |
| Dev Status | DEVELOPED |
| Released | RELEASED |
| Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
Maintainers
- bburda
Authors
ros2_medkit_graph_watchdog
Gateway plugin that detects silent faults in the ROS 2 graph: failures where every node is up, nothing logs an error, and the robot is still broken.
Detectors read the graph and raise faults through a ReportFault service client on the
gateway node; the faults surface via FaultManager on the gateway /faults API.
This package carries the plugin skeleton, the central reliability gate that holds raises
until the graph has quiesced, and three detectors, qos_mismatch, orphan and
param_drift. The remaining
silent-fault classes land in follow-up changes, each against its own issue; their fault
codes are already reserved in the frozen GRAPH_* namespace (see “Fault codes”).
Build
With the gateway built and sourced:
colcon build --packages-select ros2_medkit_graph_watchdog
colcon test --packages-select ros2_medkit_graph_watchdog && colcon test-result --verbose
Load into the gateway
ros2 run ros2_medkit_gateway gateway_node --ros-args \
-p plugins:="[graph_watchdog]" \
-p "plugins.graph_watchdog.path:=$(ros2 pkg prefix ros2_medkit_graph_watchdog)/lib/ros2_medkit_graph_watchdog/libros2_medkit_graph_watchdog.so"
Configuration (plugins.graph_watchdog.<key>)
| Key | Type | Default | Meaning |
|---|---|---|---|
tick_interval_ms |
int | 1000 |
Detector tick cadence. |
warmup_cycles |
int | 5 |
An entity must be continuously present for this many ticks before it arms. A mid-run restart re-warms it. Enforced centrally, see “Reliability (bringup-quiesce)” below. |
prune_grace |
int | 60 |
Default injected into every detector’s own config before configure(); a per-detector detectors.<id>.prune_grace overrides it. Used by detectors that keep per-key bookkeeping. |
detectors.<id>.mode |
string | bool | raise |
raise = push faults; advisory = observe, do not push; off = disabled. Bare off/no (YAML booleans) also disable; bare on/yes mean raise. |
detectors.<id>.<field> |
any | - | Per-detector thresholds, passed to that detector’s configure(). |
Nested delivery. Keys are written dotted in YAML or --ros-args, and the gateway
delivers them to the plugin as a nested object. A bare off is a YAML 1.1 boolean, so
mode: off arrives as false, not the string "off"; both forms disable the detector.
Unknown keys are reported. Anything under detectors.<id> that the detector does not
read produces one startup warning naming the key and listing the ones that exist, so a
typo like allow_list: for allowlist: cannot silently do nothing. A typo in the detector
id itself is warned about the same way, with the registered ids listed.
orphan keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | raise |
As above. |
grace |
int | 10 |
Consecutive sweeps a candidate pair must hold before it is reported. A staged bringup or a restart makes one side missing for a short time, which looks exactly like a typo until the other side appears. |
max_edit_distance |
int | 1 |
How many character edits the LEAF (the part after the last /) may differ by and still count as the same intended name. Values outside 1..8 keep the default: past a few edits a “near miss” starts matching unrelated topics. |
namespace_edit_distance |
int | 0 |
The same budget for the namespace, spent separately. 0 means the namespace must match to the character, which is what makes /robot1/scan and /robot2/scan two robots rather than a typo. Raise it to catch a misspelled namespace, and read the warning under the table first. Values outside 0..8 keep the default. |
allowlist |
string[] | [] |
Topic names never reported. Exact match only, never a prefix, so allowlisting /r1/scan does not also silence /r2/scan. |
What this detector will not catch, on purpose. A pair whose names match once every run of
digits is collapsed is treated as an enumeration of sensors, not a typo: /lidar_1 next to
/lidar_2, each one-sided, is an ordinary state on a multi-sensor robot and reporting it would
be a false alarm on every such machine. Collapsing runs rather than comparing character
positions is what also covers /lidar_9 next to /lidar_10. The cost is that a typo which
is a digit stops being reported. An appended digit is not that case: /scan1 collapses to
/scan#, which is not /scan, so it is still reported.
A misspelled namespace is opt-in, and it can be expensive. /robott/scan against
/robot/scan is one edit, but in the namespace, so by default it is not reported. Setting
namespace_edit_distance: 1 reports it. Check the naming scheme before doing that: on a fleet
named with letters, /amr_a and /amr_b are also one edit apart, and every robot in the
fleet will be reported as a typo. A numbered fleet is safe, since /robot1 against /robot2
differs only in its numeric field and the rule above already spares it.
What it cannot catch at all. A namespace added or dropped by mistake, /scan against
/robot/scan, is six edits apart, so no budget that is still specific will reach it. Do not
rely on this detector for that class.
param_drift keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. A bare off arrives as a YAML boolean; both forms disable the detector. |
baseline |
bool | true |
Self-capture the value of every parameter not listed in expect, then flag later changes to it. false checks only what expect pins, and skips reading the full parameter list. |
expect.<param> |
any | - | Absolute expected value for <param>, checked on every node that has it. This is the only rule that catches a value that was wrong from the first tick, which self-capture cannot see. A node that does not declare <param> answers NOT_FOUND, which is not a drift - so a misspelt name checks nothing at all. Once the sweep has covered every armed app - each one read, or given up on after repeated failed reads - and at least one of them answered, a pin that no app declares is warned about once, naming it. |
ignore |
string[] | [] |
Parameter-name globs never flagged. Applies to the self-captured set only, so it cannot silence an expect pin. A pattern of nothing but * matches every name and suppresses self-capture wholesale, which is indistinguishable at runtime from finding nothing; that combination warns at startup. |
max_reads_per_tick |
int | 8 |
Parameter-service ROUND TRIPS this detector may spend per tick, round-robin over nodes. Round trips, not calls: one call is several requests at the watched node, and the node pays for each of them. A baseline visit is charged 2 (a list, then one batched get) whatever the node’s parameter count; each expect pin is charged 3 (a list, a get and a descriptor read), or 2 on a node that does not declare it - see the table under “Coverage latency” for what each of those really costs the node. So a self-capture visit spends 2 of the budget and pinning N parameters spends up to 3N. It is an AMORTISED rate, not a per-tick cap: there is no pacing INSIDE a visit, so a node’s whole read set is issued back to back and the reader waits out the debt afterwards. Four pins against a budget of 8 fire 12 round trips inside one tick period and then idle for the rest of the next one; what the knob bounds is the sustained rate, not the burst. Accepted range 1..100000; anything else keeps the default and warns. |
prune_grace |
int | 60 |
Consecutive sweeps an app may be missing before its finding is dropped: the finding survives prune_grace absences and goes on the next one. Injected for every detector at plugin scope and overridable per detector; a value outside 0..3600 is rejected with a warning and this detector falls back to 2, not to the plugin default. The captured BASELINE has a shorter horizon of its own, and that one is NOT configurable - it is the compile-time kBaselineForgetGrace: the baseline is dropped on the third consecutive missed sweep, the two the reliability gate absorbs plus one, so an ordinary dropped discovery poll cannot re-capture a node on its already-drifted value. Setting prune_grace below 2 pulls that forward onto the sweep the finding itself is dropped on, so the two then go together. |
qos_mismatch keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | raise |
As above. |
grace |
int | 3 |
Consecutive ticks a topic must stay affected before it is reported. Endpoint discovery is not atomic, so a subscriber can be visible before a publisher’s QoS is, which reads as starvation for a tick or two. |
allowlist |
string[] | [] |
Subscriber FQNs never reported. Exact match only. |
Closing the loop: healing config is required
Every detector here reports level-triggered: it re-raises FAILED while the condition holds and emits PASSED on every clean sweep. That is what lets the fault_manager’s debounce counter walk from CONFIRMED back to HEALED. The fault_manager only counts PASSED events
File truncated at 100 lines see the full file
Package Dependencies
System Dependencies
Dependant Packages
Launch files
Messages
Services
Plugins
Recent questions tagged ros2_medkit_graph_watchdog at Robotics Stack Exchange
Package Summary
| Version | 0.6.0 |
| License | Apache-2.0 |
| Build type | AMENT_CMAKE |
| Use | RECOMMENDED |
Repository Summary
| Checkout URI | https://github.com/selfpatch/ros2_medkit.git |
| VCS Type | git |
| VCS Version | main |
| Last Updated | 2026-07-31 |
| Dev Status | DEVELOPED |
| Released | RELEASED |
| Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
Maintainers
- bburda
Authors
ros2_medkit_graph_watchdog
Gateway plugin that detects silent faults in the ROS 2 graph: failures where every node is up, nothing logs an error, and the robot is still broken.
Detectors read the graph and raise faults through a ReportFault service client on the
gateway node; the faults surface via FaultManager on the gateway /faults API.
This package carries the plugin skeleton, the central reliability gate that holds raises
until the graph has quiesced, and three detectors, qos_mismatch, orphan and
param_drift. The remaining
silent-fault classes land in follow-up changes, each against its own issue; their fault
codes are already reserved in the frozen GRAPH_* namespace (see “Fault codes”).
Build
With the gateway built and sourced:
colcon build --packages-select ros2_medkit_graph_watchdog
colcon test --packages-select ros2_medkit_graph_watchdog && colcon test-result --verbose
Load into the gateway
ros2 run ros2_medkit_gateway gateway_node --ros-args \
-p plugins:="[graph_watchdog]" \
-p "plugins.graph_watchdog.path:=$(ros2 pkg prefix ros2_medkit_graph_watchdog)/lib/ros2_medkit_graph_watchdog/libros2_medkit_graph_watchdog.so"
Configuration (plugins.graph_watchdog.<key>)
| Key | Type | Default | Meaning |
|---|---|---|---|
tick_interval_ms |
int | 1000 |
Detector tick cadence. |
warmup_cycles |
int | 5 |
An entity must be continuously present for this many ticks before it arms. A mid-run restart re-warms it. Enforced centrally, see “Reliability (bringup-quiesce)” below. |
prune_grace |
int | 60 |
Default injected into every detector’s own config before configure(); a per-detector detectors.<id>.prune_grace overrides it. Used by detectors that keep per-key bookkeeping. |
detectors.<id>.mode |
string | bool | raise |
raise = push faults; advisory = observe, do not push; off = disabled. Bare off/no (YAML booleans) also disable; bare on/yes mean raise. |
detectors.<id>.<field> |
any | - | Per-detector thresholds, passed to that detector’s configure(). |
Nested delivery. Keys are written dotted in YAML or --ros-args, and the gateway
delivers them to the plugin as a nested object. A bare off is a YAML 1.1 boolean, so
mode: off arrives as false, not the string "off"; both forms disable the detector.
Unknown keys are reported. Anything under detectors.<id> that the detector does not
read produces one startup warning naming the key and listing the ones that exist, so a
typo like allow_list: for allowlist: cannot silently do nothing. A typo in the detector
id itself is warned about the same way, with the registered ids listed.
orphan keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | raise |
As above. |
grace |
int | 10 |
Consecutive sweeps a candidate pair must hold before it is reported. A staged bringup or a restart makes one side missing for a short time, which looks exactly like a typo until the other side appears. |
max_edit_distance |
int | 1 |
How many character edits the LEAF (the part after the last /) may differ by and still count as the same intended name. Values outside 1..8 keep the default: past a few edits a “near miss” starts matching unrelated topics. |
namespace_edit_distance |
int | 0 |
The same budget for the namespace, spent separately. 0 means the namespace must match to the character, which is what makes /robot1/scan and /robot2/scan two robots rather than a typo. Raise it to catch a misspelled namespace, and read the warning under the table first. Values outside 0..8 keep the default. |
allowlist |
string[] | [] |
Topic names never reported. Exact match only, never a prefix, so allowlisting /r1/scan does not also silence /r2/scan. |
What this detector will not catch, on purpose. A pair whose names match once every run of
digits is collapsed is treated as an enumeration of sensors, not a typo: /lidar_1 next to
/lidar_2, each one-sided, is an ordinary state on a multi-sensor robot and reporting it would
be a false alarm on every such machine. Collapsing runs rather than comparing character
positions is what also covers /lidar_9 next to /lidar_10. The cost is that a typo which
is a digit stops being reported. An appended digit is not that case: /scan1 collapses to
/scan#, which is not /scan, so it is still reported.
A misspelled namespace is opt-in, and it can be expensive. /robott/scan against
/robot/scan is one edit, but in the namespace, so by default it is not reported. Setting
namespace_edit_distance: 1 reports it. Check the naming scheme before doing that: on a fleet
named with letters, /amr_a and /amr_b are also one edit apart, and every robot in the
fleet will be reported as a typo. A numbered fleet is safe, since /robot1 against /robot2
differs only in its numeric field and the rule above already spares it.
What it cannot catch at all. A namespace added or dropped by mistake, /scan against
/robot/scan, is six edits apart, so no budget that is still specific will reach it. Do not
rely on this detector for that class.
param_drift keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. A bare off arrives as a YAML boolean; both forms disable the detector. |
baseline |
bool | true |
Self-capture the value of every parameter not listed in expect, then flag later changes to it. false checks only what expect pins, and skips reading the full parameter list. |
expect.<param> |
any | - | Absolute expected value for <param>, checked on every node that has it. This is the only rule that catches a value that was wrong from the first tick, which self-capture cannot see. A node that does not declare <param> answers NOT_FOUND, which is not a drift - so a misspelt name checks nothing at all. Once the sweep has covered every armed app - each one read, or given up on after repeated failed reads - and at least one of them answered, a pin that no app declares is warned about once, naming it. |
ignore |
string[] | [] |
Parameter-name globs never flagged. Applies to the self-captured set only, so it cannot silence an expect pin. A pattern of nothing but * matches every name and suppresses self-capture wholesale, which is indistinguishable at runtime from finding nothing; that combination warns at startup. |
max_reads_per_tick |
int | 8 |
Parameter-service ROUND TRIPS this detector may spend per tick, round-robin over nodes. Round trips, not calls: one call is several requests at the watched node, and the node pays for each of them. A baseline visit is charged 2 (a list, then one batched get) whatever the node’s parameter count; each expect pin is charged 3 (a list, a get and a descriptor read), or 2 on a node that does not declare it - see the table under “Coverage latency” for what each of those really costs the node. So a self-capture visit spends 2 of the budget and pinning N parameters spends up to 3N. It is an AMORTISED rate, not a per-tick cap: there is no pacing INSIDE a visit, so a node’s whole read set is issued back to back and the reader waits out the debt afterwards. Four pins against a budget of 8 fire 12 round trips inside one tick period and then idle for the rest of the next one; what the knob bounds is the sustained rate, not the burst. Accepted range 1..100000; anything else keeps the default and warns. |
prune_grace |
int | 60 |
Consecutive sweeps an app may be missing before its finding is dropped: the finding survives prune_grace absences and goes on the next one. Injected for every detector at plugin scope and overridable per detector; a value outside 0..3600 is rejected with a warning and this detector falls back to 2, not to the plugin default. The captured BASELINE has a shorter horizon of its own, and that one is NOT configurable - it is the compile-time kBaselineForgetGrace: the baseline is dropped on the third consecutive missed sweep, the two the reliability gate absorbs plus one, so an ordinary dropped discovery poll cannot re-capture a node on its already-drifted value. Setting prune_grace below 2 pulls that forward onto the sweep the finding itself is dropped on, so the two then go together. |
qos_mismatch keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | raise |
As above. |
grace |
int | 3 |
Consecutive ticks a topic must stay affected before it is reported. Endpoint discovery is not atomic, so a subscriber can be visible before a publisher’s QoS is, which reads as starvation for a tick or two. |
allowlist |
string[] | [] |
Subscriber FQNs never reported. Exact match only. |
Closing the loop: healing config is required
Every detector here reports level-triggered: it re-raises FAILED while the condition holds and emits PASSED on every clean sweep. That is what lets the fault_manager’s debounce counter walk from CONFIRMED back to HEALED. The fault_manager only counts PASSED events
File truncated at 100 lines see the full file
Package Dependencies
System Dependencies
Dependant Packages
Launch files
Messages
Services
Plugins
Recent questions tagged ros2_medkit_graph_watchdog at Robotics Stack Exchange
Package Summary
| Version | 0.6.0 |
| License | Apache-2.0 |
| Build type | AMENT_CMAKE |
| Use | RECOMMENDED |
Repository Summary
| Checkout URI | https://github.com/selfpatch/ros2_medkit.git |
| VCS Type | git |
| VCS Version | main |
| Last Updated | 2026-07-31 |
| Dev Status | DEVELOPED |
| Released | RELEASED |
| Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
Maintainers
- bburda
Authors
ros2_medkit_graph_watchdog
Gateway plugin that detects silent faults in the ROS 2 graph: failures where every node is up, nothing logs an error, and the robot is still broken.
Detectors read the graph and raise faults through a ReportFault service client on the
gateway node; the faults surface via FaultManager on the gateway /faults API.
This package carries the plugin skeleton, the central reliability gate that holds raises
until the graph has quiesced, and three detectors, qos_mismatch, orphan and
param_drift. The remaining
silent-fault classes land in follow-up changes, each against its own issue; their fault
codes are already reserved in the frozen GRAPH_* namespace (see “Fault codes”).
Build
With the gateway built and sourced:
colcon build --packages-select ros2_medkit_graph_watchdog
colcon test --packages-select ros2_medkit_graph_watchdog && colcon test-result --verbose
Load into the gateway
ros2 run ros2_medkit_gateway gateway_node --ros-args \
-p plugins:="[graph_watchdog]" \
-p "plugins.graph_watchdog.path:=$(ros2 pkg prefix ros2_medkit_graph_watchdog)/lib/ros2_medkit_graph_watchdog/libros2_medkit_graph_watchdog.so"
Configuration (plugins.graph_watchdog.<key>)
| Key | Type | Default | Meaning |
|---|---|---|---|
tick_interval_ms |
int | 1000 |
Detector tick cadence. |
warmup_cycles |
int | 5 |
An entity must be continuously present for this many ticks before it arms. A mid-run restart re-warms it. Enforced centrally, see “Reliability (bringup-quiesce)” below. |
prune_grace |
int | 60 |
Default injected into every detector’s own config before configure(); a per-detector detectors.<id>.prune_grace overrides it. Used by detectors that keep per-key bookkeeping. |
detectors.<id>.mode |
string | bool | raise |
raise = push faults; advisory = observe, do not push; off = disabled. Bare off/no (YAML booleans) also disable; bare on/yes mean raise. |
detectors.<id>.<field> |
any | - | Per-detector thresholds, passed to that detector’s configure(). |
Nested delivery. Keys are written dotted in YAML or --ros-args, and the gateway
delivers them to the plugin as a nested object. A bare off is a YAML 1.1 boolean, so
mode: off arrives as false, not the string "off"; both forms disable the detector.
Unknown keys are reported. Anything under detectors.<id> that the detector does not
read produces one startup warning naming the key and listing the ones that exist, so a
typo like allow_list: for allowlist: cannot silently do nothing. A typo in the detector
id itself is warned about the same way, with the registered ids listed.
orphan keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | raise |
As above. |
grace |
int | 10 |
Consecutive sweeps a candidate pair must hold before it is reported. A staged bringup or a restart makes one side missing for a short time, which looks exactly like a typo until the other side appears. |
max_edit_distance |
int | 1 |
How many character edits the LEAF (the part after the last /) may differ by and still count as the same intended name. Values outside 1..8 keep the default: past a few edits a “near miss” starts matching unrelated topics. |
namespace_edit_distance |
int | 0 |
The same budget for the namespace, spent separately. 0 means the namespace must match to the character, which is what makes /robot1/scan and /robot2/scan two robots rather than a typo. Raise it to catch a misspelled namespace, and read the warning under the table first. Values outside 0..8 keep the default. |
allowlist |
string[] | [] |
Topic names never reported. Exact match only, never a prefix, so allowlisting /r1/scan does not also silence /r2/scan. |
What this detector will not catch, on purpose. A pair whose names match once every run of
digits is collapsed is treated as an enumeration of sensors, not a typo: /lidar_1 next to
/lidar_2, each one-sided, is an ordinary state on a multi-sensor robot and reporting it would
be a false alarm on every such machine. Collapsing runs rather than comparing character
positions is what also covers /lidar_9 next to /lidar_10. The cost is that a typo which
is a digit stops being reported. An appended digit is not that case: /scan1 collapses to
/scan#, which is not /scan, so it is still reported.
A misspelled namespace is opt-in, and it can be expensive. /robott/scan against
/robot/scan is one edit, but in the namespace, so by default it is not reported. Setting
namespace_edit_distance: 1 reports it. Check the naming scheme before doing that: on a fleet
named with letters, /amr_a and /amr_b are also one edit apart, and every robot in the
fleet will be reported as a typo. A numbered fleet is safe, since /robot1 against /robot2
differs only in its numeric field and the rule above already spares it.
What it cannot catch at all. A namespace added or dropped by mistake, /scan against
/robot/scan, is six edits apart, so no budget that is still specific will reach it. Do not
rely on this detector for that class.
param_drift keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. A bare off arrives as a YAML boolean; both forms disable the detector. |
baseline |
bool | true |
Self-capture the value of every parameter not listed in expect, then flag later changes to it. false checks only what expect pins, and skips reading the full parameter list. |
expect.<param> |
any | - | Absolute expected value for <param>, checked on every node that has it. This is the only rule that catches a value that was wrong from the first tick, which self-capture cannot see. A node that does not declare <param> answers NOT_FOUND, which is not a drift - so a misspelt name checks nothing at all. Once the sweep has covered every armed app - each one read, or given up on after repeated failed reads - and at least one of them answered, a pin that no app declares is warned about once, naming it. |
ignore |
string[] | [] |
Parameter-name globs never flagged. Applies to the self-captured set only, so it cannot silence an expect pin. A pattern of nothing but * matches every name and suppresses self-capture wholesale, which is indistinguishable at runtime from finding nothing; that combination warns at startup. |
max_reads_per_tick |
int | 8 |
Parameter-service ROUND TRIPS this detector may spend per tick, round-robin over nodes. Round trips, not calls: one call is several requests at the watched node, and the node pays for each of them. A baseline visit is charged 2 (a list, then one batched get) whatever the node’s parameter count; each expect pin is charged 3 (a list, a get and a descriptor read), or 2 on a node that does not declare it - see the table under “Coverage latency” for what each of those really costs the node. So a self-capture visit spends 2 of the budget and pinning N parameters spends up to 3N. It is an AMORTISED rate, not a per-tick cap: there is no pacing INSIDE a visit, so a node’s whole read set is issued back to back and the reader waits out the debt afterwards. Four pins against a budget of 8 fire 12 round trips inside one tick period and then idle for the rest of the next one; what the knob bounds is the sustained rate, not the burst. Accepted range 1..100000; anything else keeps the default and warns. |
prune_grace |
int | 60 |
Consecutive sweeps an app may be missing before its finding is dropped: the finding survives prune_grace absences and goes on the next one. Injected for every detector at plugin scope and overridable per detector; a value outside 0..3600 is rejected with a warning and this detector falls back to 2, not to the plugin default. The captured BASELINE has a shorter horizon of its own, and that one is NOT configurable - it is the compile-time kBaselineForgetGrace: the baseline is dropped on the third consecutive missed sweep, the two the reliability gate absorbs plus one, so an ordinary dropped discovery poll cannot re-capture a node on its already-drifted value. Setting prune_grace below 2 pulls that forward onto the sweep the finding itself is dropped on, so the two then go together. |
qos_mismatch keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | raise |
As above. |
grace |
int | 3 |
Consecutive ticks a topic must stay affected before it is reported. Endpoint discovery is not atomic, so a subscriber can be visible before a publisher’s QoS is, which reads as starvation for a tick or two. |
allowlist |
string[] | [] |
Subscriber FQNs never reported. Exact match only. |
Closing the loop: healing config is required
Every detector here reports level-triggered: it re-raises FAILED while the condition holds and emits PASSED on every clean sweep. That is what lets the fault_manager’s debounce counter walk from CONFIRMED back to HEALED. The fault_manager only counts PASSED events
File truncated at 100 lines see the full file
Package Dependencies
System Dependencies
Dependant Packages
Launch files
Messages
Services
Plugins
Recent questions tagged ros2_medkit_graph_watchdog at Robotics Stack Exchange
Package Summary
| Version | 0.6.0 |
| License | Apache-2.0 |
| Build type | AMENT_CMAKE |
| Use | RECOMMENDED |
Repository Summary
| Checkout URI | https://github.com/selfpatch/ros2_medkit.git |
| VCS Type | git |
| VCS Version | main |
| Last Updated | 2026-07-31 |
| Dev Status | DEVELOPED |
| Released | RELEASED |
| Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
Maintainers
- bburda
Authors
ros2_medkit_graph_watchdog
Gateway plugin that detects silent faults in the ROS 2 graph: failures where every node is up, nothing logs an error, and the robot is still broken.
Detectors read the graph and raise faults through a ReportFault service client on the
gateway node; the faults surface via FaultManager on the gateway /faults API.
This package carries the plugin skeleton, the central reliability gate that holds raises
until the graph has quiesced, and three detectors, qos_mismatch, orphan and
param_drift. The remaining
silent-fault classes land in follow-up changes, each against its own issue; their fault
codes are already reserved in the frozen GRAPH_* namespace (see “Fault codes”).
Build
With the gateway built and sourced:
colcon build --packages-select ros2_medkit_graph_watchdog
colcon test --packages-select ros2_medkit_graph_watchdog && colcon test-result --verbose
Load into the gateway
ros2 run ros2_medkit_gateway gateway_node --ros-args \
-p plugins:="[graph_watchdog]" \
-p "plugins.graph_watchdog.path:=$(ros2 pkg prefix ros2_medkit_graph_watchdog)/lib/ros2_medkit_graph_watchdog/libros2_medkit_graph_watchdog.so"
Configuration (plugins.graph_watchdog.<key>)
| Key | Type | Default | Meaning |
|---|---|---|---|
tick_interval_ms |
int | 1000 |
Detector tick cadence. |
warmup_cycles |
int | 5 |
An entity must be continuously present for this many ticks before it arms. A mid-run restart re-warms it. Enforced centrally, see “Reliability (bringup-quiesce)” below. |
prune_grace |
int | 60 |
Default injected into every detector’s own config before configure(); a per-detector detectors.<id>.prune_grace overrides it. Used by detectors that keep per-key bookkeeping. |
detectors.<id>.mode |
string | bool | raise |
raise = push faults; advisory = observe, do not push; off = disabled. Bare off/no (YAML booleans) also disable; bare on/yes mean raise. |
detectors.<id>.<field> |
any | - | Per-detector thresholds, passed to that detector’s configure(). |
Nested delivery. Keys are written dotted in YAML or --ros-args, and the gateway
delivers them to the plugin as a nested object. A bare off is a YAML 1.1 boolean, so
mode: off arrives as false, not the string "off"; both forms disable the detector.
Unknown keys are reported. Anything under detectors.<id> that the detector does not
read produces one startup warning naming the key and listing the ones that exist, so a
typo like allow_list: for allowlist: cannot silently do nothing. A typo in the detector
id itself is warned about the same way, with the registered ids listed.
orphan keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | raise |
As above. |
grace |
int | 10 |
Consecutive sweeps a candidate pair must hold before it is reported. A staged bringup or a restart makes one side missing for a short time, which looks exactly like a typo until the other side appears. |
max_edit_distance |
int | 1 |
How many character edits the LEAF (the part after the last /) may differ by and still count as the same intended name. Values outside 1..8 keep the default: past a few edits a “near miss” starts matching unrelated topics. |
namespace_edit_distance |
int | 0 |
The same budget for the namespace, spent separately. 0 means the namespace must match to the character, which is what makes /robot1/scan and /robot2/scan two robots rather than a typo. Raise it to catch a misspelled namespace, and read the warning under the table first. Values outside 0..8 keep the default. |
allowlist |
string[] | [] |
Topic names never reported. Exact match only, never a prefix, so allowlisting /r1/scan does not also silence /r2/scan. |
What this detector will not catch, on purpose. A pair whose names match once every run of
digits is collapsed is treated as an enumeration of sensors, not a typo: /lidar_1 next to
/lidar_2, each one-sided, is an ordinary state on a multi-sensor robot and reporting it would
be a false alarm on every such machine. Collapsing runs rather than comparing character
positions is what also covers /lidar_9 next to /lidar_10. The cost is that a typo which
is a digit stops being reported. An appended digit is not that case: /scan1 collapses to
/scan#, which is not /scan, so it is still reported.
A misspelled namespace is opt-in, and it can be expensive. /robott/scan against
/robot/scan is one edit, but in the namespace, so by default it is not reported. Setting
namespace_edit_distance: 1 reports it. Check the naming scheme before doing that: on a fleet
named with letters, /amr_a and /amr_b are also one edit apart, and every robot in the
fleet will be reported as a typo. A numbered fleet is safe, since /robot1 against /robot2
differs only in its numeric field and the rule above already spares it.
What it cannot catch at all. A namespace added or dropped by mistake, /scan against
/robot/scan, is six edits apart, so no budget that is still specific will reach it. Do not
rely on this detector for that class.
param_drift keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. A bare off arrives as a YAML boolean; both forms disable the detector. |
baseline |
bool | true |
Self-capture the value of every parameter not listed in expect, then flag later changes to it. false checks only what expect pins, and skips reading the full parameter list. |
expect.<param> |
any | - | Absolute expected value for <param>, checked on every node that has it. This is the only rule that catches a value that was wrong from the first tick, which self-capture cannot see. A node that does not declare <param> answers NOT_FOUND, which is not a drift - so a misspelt name checks nothing at all. Once the sweep has covered every armed app - each one read, or given up on after repeated failed reads - and at least one of them answered, a pin that no app declares is warned about once, naming it. |
ignore |
string[] | [] |
Parameter-name globs never flagged. Applies to the self-captured set only, so it cannot silence an expect pin. A pattern of nothing but * matches every name and suppresses self-capture wholesale, which is indistinguishable at runtime from finding nothing; that combination warns at startup. |
max_reads_per_tick |
int | 8 |
Parameter-service ROUND TRIPS this detector may spend per tick, round-robin over nodes. Round trips, not calls: one call is several requests at the watched node, and the node pays for each of them. A baseline visit is charged 2 (a list, then one batched get) whatever the node’s parameter count; each expect pin is charged 3 (a list, a get and a descriptor read), or 2 on a node that does not declare it - see the table under “Coverage latency” for what each of those really costs the node. So a self-capture visit spends 2 of the budget and pinning N parameters spends up to 3N. It is an AMORTISED rate, not a per-tick cap: there is no pacing INSIDE a visit, so a node’s whole read set is issued back to back and the reader waits out the debt afterwards. Four pins against a budget of 8 fire 12 round trips inside one tick period and then idle for the rest of the next one; what the knob bounds is the sustained rate, not the burst. Accepted range 1..100000; anything else keeps the default and warns. |
prune_grace |
int | 60 |
Consecutive sweeps an app may be missing before its finding is dropped: the finding survives prune_grace absences and goes on the next one. Injected for every detector at plugin scope and overridable per detector; a value outside 0..3600 is rejected with a warning and this detector falls back to 2, not to the plugin default. The captured BASELINE has a shorter horizon of its own, and that one is NOT configurable - it is the compile-time kBaselineForgetGrace: the baseline is dropped on the third consecutive missed sweep, the two the reliability gate absorbs plus one, so an ordinary dropped discovery poll cannot re-capture a node on its already-drifted value. Setting prune_grace below 2 pulls that forward onto the sweep the finding itself is dropped on, so the two then go together. |
qos_mismatch keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | raise |
As above. |
grace |
int | 3 |
Consecutive ticks a topic must stay affected before it is reported. Endpoint discovery is not atomic, so a subscriber can be visible before a publisher’s QoS is, which reads as starvation for a tick or two. |
allowlist |
string[] | [] |
Subscriber FQNs never reported. Exact match only. |
Closing the loop: healing config is required
Every detector here reports level-triggered: it re-raises FAILED while the condition holds and emits PASSED on every clean sweep. That is what lets the fault_manager’s debounce counter walk from CONFIRMED back to HEALED. The fault_manager only counts PASSED events
File truncated at 100 lines see the full file
Package Dependencies
System Dependencies
Dependant Packages
Launch files
Messages
Services
Plugins
Recent questions tagged ros2_medkit_graph_watchdog at Robotics Stack Exchange
Package Summary
| Version | 0.6.0 |
| License | Apache-2.0 |
| Build type | AMENT_CMAKE |
| Use | RECOMMENDED |
Repository Summary
| Checkout URI | https://github.com/selfpatch/ros2_medkit.git |
| VCS Type | git |
| VCS Version | main |
| Last Updated | 2026-07-31 |
| Dev Status | DEVELOPED |
| Released | RELEASED |
| Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
Maintainers
- bburda
Authors
ros2_medkit_graph_watchdog
Gateway plugin that detects silent faults in the ROS 2 graph: failures where every node is up, nothing logs an error, and the robot is still broken.
Detectors read the graph and raise faults through a ReportFault service client on the
gateway node; the faults surface via FaultManager on the gateway /faults API.
This package carries the plugin skeleton, the central reliability gate that holds raises
until the graph has quiesced, and three detectors, qos_mismatch, orphan and
param_drift. The remaining
silent-fault classes land in follow-up changes, each against its own issue; their fault
codes are already reserved in the frozen GRAPH_* namespace (see “Fault codes”).
Build
With the gateway built and sourced:
colcon build --packages-select ros2_medkit_graph_watchdog
colcon test --packages-select ros2_medkit_graph_watchdog && colcon test-result --verbose
Load into the gateway
ros2 run ros2_medkit_gateway gateway_node --ros-args \
-p plugins:="[graph_watchdog]" \
-p "plugins.graph_watchdog.path:=$(ros2 pkg prefix ros2_medkit_graph_watchdog)/lib/ros2_medkit_graph_watchdog/libros2_medkit_graph_watchdog.so"
Configuration (plugins.graph_watchdog.<key>)
| Key | Type | Default | Meaning |
|---|---|---|---|
tick_interval_ms |
int | 1000 |
Detector tick cadence. |
warmup_cycles |
int | 5 |
An entity must be continuously present for this many ticks before it arms. A mid-run restart re-warms it. Enforced centrally, see “Reliability (bringup-quiesce)” below. |
prune_grace |
int | 60 |
Default injected into every detector’s own config before configure(); a per-detector detectors.<id>.prune_grace overrides it. Used by detectors that keep per-key bookkeeping. |
detectors.<id>.mode |
string | bool | raise |
raise = push faults; advisory = observe, do not push; off = disabled. Bare off/no (YAML booleans) also disable; bare on/yes mean raise. |
detectors.<id>.<field> |
any | - | Per-detector thresholds, passed to that detector’s configure(). |
Nested delivery. Keys are written dotted in YAML or --ros-args, and the gateway
delivers them to the plugin as a nested object. A bare off is a YAML 1.1 boolean, so
mode: off arrives as false, not the string "off"; both forms disable the detector.
Unknown keys are reported. Anything under detectors.<id> that the detector does not
read produces one startup warning naming the key and listing the ones that exist, so a
typo like allow_list: for allowlist: cannot silently do nothing. A typo in the detector
id itself is warned about the same way, with the registered ids listed.
orphan keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | raise |
As above. |
grace |
int | 10 |
Consecutive sweeps a candidate pair must hold before it is reported. A staged bringup or a restart makes one side missing for a short time, which looks exactly like a typo until the other side appears. |
max_edit_distance |
int | 1 |
How many character edits the LEAF (the part after the last /) may differ by and still count as the same intended name. Values outside 1..8 keep the default: past a few edits a “near miss” starts matching unrelated topics. |
namespace_edit_distance |
int | 0 |
The same budget for the namespace, spent separately. 0 means the namespace must match to the character, which is what makes /robot1/scan and /robot2/scan two robots rather than a typo. Raise it to catch a misspelled namespace, and read the warning under the table first. Values outside 0..8 keep the default. |
allowlist |
string[] | [] |
Topic names never reported. Exact match only, never a prefix, so allowlisting /r1/scan does not also silence /r2/scan. |
What this detector will not catch, on purpose. A pair whose names match once every run of
digits is collapsed is treated as an enumeration of sensors, not a typo: /lidar_1 next to
/lidar_2, each one-sided, is an ordinary state on a multi-sensor robot and reporting it would
be a false alarm on every such machine. Collapsing runs rather than comparing character
positions is what also covers /lidar_9 next to /lidar_10. The cost is that a typo which
is a digit stops being reported. An appended digit is not that case: /scan1 collapses to
/scan#, which is not /scan, so it is still reported.
A misspelled namespace is opt-in, and it can be expensive. /robott/scan against
/robot/scan is one edit, but in the namespace, so by default it is not reported. Setting
namespace_edit_distance: 1 reports it. Check the naming scheme before doing that: on a fleet
named with letters, /amr_a and /amr_b are also one edit apart, and every robot in the
fleet will be reported as a typo. A numbered fleet is safe, since /robot1 against /robot2
differs only in its numeric field and the rule above already spares it.
What it cannot catch at all. A namespace added or dropped by mistake, /scan against
/robot/scan, is six edits apart, so no budget that is still specific will reach it. Do not
rely on this detector for that class.
param_drift keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. A bare off arrives as a YAML boolean; both forms disable the detector. |
baseline |
bool | true |
Self-capture the value of every parameter not listed in expect, then flag later changes to it. false checks only what expect pins, and skips reading the full parameter list. |
expect.<param> |
any | - | Absolute expected value for <param>, checked on every node that has it. This is the only rule that catches a value that was wrong from the first tick, which self-capture cannot see. A node that does not declare <param> answers NOT_FOUND, which is not a drift - so a misspelt name checks nothing at all. Once the sweep has covered every armed app - each one read, or given up on after repeated failed reads - and at least one of them answered, a pin that no app declares is warned about once, naming it. |
ignore |
string[] | [] |
Parameter-name globs never flagged. Applies to the self-captured set only, so it cannot silence an expect pin. A pattern of nothing but * matches every name and suppresses self-capture wholesale, which is indistinguishable at runtime from finding nothing; that combination warns at startup. |
max_reads_per_tick |
int | 8 |
Parameter-service ROUND TRIPS this detector may spend per tick, round-robin over nodes. Round trips, not calls: one call is several requests at the watched node, and the node pays for each of them. A baseline visit is charged 2 (a list, then one batched get) whatever the node’s parameter count; each expect pin is charged 3 (a list, a get and a descriptor read), or 2 on a node that does not declare it - see the table under “Coverage latency” for what each of those really costs the node. So a self-capture visit spends 2 of the budget and pinning N parameters spends up to 3N. It is an AMORTISED rate, not a per-tick cap: there is no pacing INSIDE a visit, so a node’s whole read set is issued back to back and the reader waits out the debt afterwards. Four pins against a budget of 8 fire 12 round trips inside one tick period and then idle for the rest of the next one; what the knob bounds is the sustained rate, not the burst. Accepted range 1..100000; anything else keeps the default and warns. |
prune_grace |
int | 60 |
Consecutive sweeps an app may be missing before its finding is dropped: the finding survives prune_grace absences and goes on the next one. Injected for every detector at plugin scope and overridable per detector; a value outside 0..3600 is rejected with a warning and this detector falls back to 2, not to the plugin default. The captured BASELINE has a shorter horizon of its own, and that one is NOT configurable - it is the compile-time kBaselineForgetGrace: the baseline is dropped on the third consecutive missed sweep, the two the reliability gate absorbs plus one, so an ordinary dropped discovery poll cannot re-capture a node on its already-drifted value. Setting prune_grace below 2 pulls that forward onto the sweep the finding itself is dropped on, so the two then go together. |
qos_mismatch keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | raise |
As above. |
grace |
int | 3 |
Consecutive ticks a topic must stay affected before it is reported. Endpoint discovery is not atomic, so a subscriber can be visible before a publisher’s QoS is, which reads as starvation for a tick or two. |
allowlist |
string[] | [] |
Subscriber FQNs never reported. Exact match only. |
Closing the loop: healing config is required
Every detector here reports level-triggered: it re-raises FAILED while the condition holds and emits PASSED on every clean sweep. That is what lets the fault_manager’s debounce counter walk from CONFIRMED back to HEALED. The fault_manager only counts PASSED events
File truncated at 100 lines see the full file
Package Dependencies
System Dependencies
Dependant Packages
Launch files
Messages
Services
Plugins
Recent questions tagged ros2_medkit_graph_watchdog at Robotics Stack Exchange
Package Summary
| Version | 0.6.0 |
| License | Apache-2.0 |
| Build type | AMENT_CMAKE |
| Use | RECOMMENDED |
Repository Summary
| Checkout URI | https://github.com/selfpatch/ros2_medkit.git |
| VCS Type | git |
| VCS Version | main |
| Last Updated | 2026-07-31 |
| Dev Status | DEVELOPED |
| Released | RELEASED |
| Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
Maintainers
- bburda
Authors
ros2_medkit_graph_watchdog
Gateway plugin that detects silent faults in the ROS 2 graph: failures where every node is up, nothing logs an error, and the robot is still broken.
Detectors read the graph and raise faults through a ReportFault service client on the
gateway node; the faults surface via FaultManager on the gateway /faults API.
This package carries the plugin skeleton, the central reliability gate that holds raises
until the graph has quiesced, and three detectors, qos_mismatch, orphan and
param_drift. The remaining
silent-fault classes land in follow-up changes, each against its own issue; their fault
codes are already reserved in the frozen GRAPH_* namespace (see “Fault codes”).
Build
With the gateway built and sourced:
colcon build --packages-select ros2_medkit_graph_watchdog
colcon test --packages-select ros2_medkit_graph_watchdog && colcon test-result --verbose
Load into the gateway
ros2 run ros2_medkit_gateway gateway_node --ros-args \
-p plugins:="[graph_watchdog]" \
-p "plugins.graph_watchdog.path:=$(ros2 pkg prefix ros2_medkit_graph_watchdog)/lib/ros2_medkit_graph_watchdog/libros2_medkit_graph_watchdog.so"
Configuration (plugins.graph_watchdog.<key>)
| Key | Type | Default | Meaning |
|---|---|---|---|
tick_interval_ms |
int | 1000 |
Detector tick cadence. |
warmup_cycles |
int | 5 |
An entity must be continuously present for this many ticks before it arms. A mid-run restart re-warms it. Enforced centrally, see “Reliability (bringup-quiesce)” below. |
prune_grace |
int | 60 |
Default injected into every detector’s own config before configure(); a per-detector detectors.<id>.prune_grace overrides it. Used by detectors that keep per-key bookkeeping. |
detectors.<id>.mode |
string | bool | raise |
raise = push faults; advisory = observe, do not push; off = disabled. Bare off/no (YAML booleans) also disable; bare on/yes mean raise. |
detectors.<id>.<field> |
any | - | Per-detector thresholds, passed to that detector’s configure(). |
Nested delivery. Keys are written dotted in YAML or --ros-args, and the gateway
delivers them to the plugin as a nested object. A bare off is a YAML 1.1 boolean, so
mode: off arrives as false, not the string "off"; both forms disable the detector.
Unknown keys are reported. Anything under detectors.<id> that the detector does not
read produces one startup warning naming the key and listing the ones that exist, so a
typo like allow_list: for allowlist: cannot silently do nothing. A typo in the detector
id itself is warned about the same way, with the registered ids listed.
orphan keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | raise |
As above. |
grace |
int | 10 |
Consecutive sweeps a candidate pair must hold before it is reported. A staged bringup or a restart makes one side missing for a short time, which looks exactly like a typo until the other side appears. |
max_edit_distance |
int | 1 |
How many character edits the LEAF (the part after the last /) may differ by and still count as the same intended name. Values outside 1..8 keep the default: past a few edits a “near miss” starts matching unrelated topics. |
namespace_edit_distance |
int | 0 |
The same budget for the namespace, spent separately. 0 means the namespace must match to the character, which is what makes /robot1/scan and /robot2/scan two robots rather than a typo. Raise it to catch a misspelled namespace, and read the warning under the table first. Values outside 0..8 keep the default. |
allowlist |
string[] | [] |
Topic names never reported. Exact match only, never a prefix, so allowlisting /r1/scan does not also silence /r2/scan. |
What this detector will not catch, on purpose. A pair whose names match once every run of
digits is collapsed is treated as an enumeration of sensors, not a typo: /lidar_1 next to
/lidar_2, each one-sided, is an ordinary state on a multi-sensor robot and reporting it would
be a false alarm on every such machine. Collapsing runs rather than comparing character
positions is what also covers /lidar_9 next to /lidar_10. The cost is that a typo which
is a digit stops being reported. An appended digit is not that case: /scan1 collapses to
/scan#, which is not /scan, so it is still reported.
A misspelled namespace is opt-in, and it can be expensive. /robott/scan against
/robot/scan is one edit, but in the namespace, so by default it is not reported. Setting
namespace_edit_distance: 1 reports it. Check the naming scheme before doing that: on a fleet
named with letters, /amr_a and /amr_b are also one edit apart, and every robot in the
fleet will be reported as a typo. A numbered fleet is safe, since /robot1 against /robot2
differs only in its numeric field and the rule above already spares it.
What it cannot catch at all. A namespace added or dropped by mistake, /scan against
/robot/scan, is six edits apart, so no budget that is still specific will reach it. Do not
rely on this detector for that class.
param_drift keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. A bare off arrives as a YAML boolean; both forms disable the detector. |
baseline |
bool | true |
Self-capture the value of every parameter not listed in expect, then flag later changes to it. false checks only what expect pins, and skips reading the full parameter list. |
expect.<param> |
any | - | Absolute expected value for <param>, checked on every node that has it. This is the only rule that catches a value that was wrong from the first tick, which self-capture cannot see. A node that does not declare <param> answers NOT_FOUND, which is not a drift - so a misspelt name checks nothing at all. Once the sweep has covered every armed app - each one read, or given up on after repeated failed reads - and at least one of them answered, a pin that no app declares is warned about once, naming it. |
ignore |
string[] | [] |
Parameter-name globs never flagged. Applies to the self-captured set only, so it cannot silence an expect pin. A pattern of nothing but * matches every name and suppresses self-capture wholesale, which is indistinguishable at runtime from finding nothing; that combination warns at startup. |
max_reads_per_tick |
int | 8 |
Parameter-service ROUND TRIPS this detector may spend per tick, round-robin over nodes. Round trips, not calls: one call is several requests at the watched node, and the node pays for each of them. A baseline visit is charged 2 (a list, then one batched get) whatever the node’s parameter count; each expect pin is charged 3 (a list, a get and a descriptor read), or 2 on a node that does not declare it - see the table under “Coverage latency” for what each of those really costs the node. So a self-capture visit spends 2 of the budget and pinning N parameters spends up to 3N. It is an AMORTISED rate, not a per-tick cap: there is no pacing INSIDE a visit, so a node’s whole read set is issued back to back and the reader waits out the debt afterwards. Four pins against a budget of 8 fire 12 round trips inside one tick period and then idle for the rest of the next one; what the knob bounds is the sustained rate, not the burst. Accepted range 1..100000; anything else keeps the default and warns. |
prune_grace |
int | 60 |
Consecutive sweeps an app may be missing before its finding is dropped: the finding survives prune_grace absences and goes on the next one. Injected for every detector at plugin scope and overridable per detector; a value outside 0..3600 is rejected with a warning and this detector falls back to 2, not to the plugin default. The captured BASELINE has a shorter horizon of its own, and that one is NOT configurable - it is the compile-time kBaselineForgetGrace: the baseline is dropped on the third consecutive missed sweep, the two the reliability gate absorbs plus one, so an ordinary dropped discovery poll cannot re-capture a node on its already-drifted value. Setting prune_grace below 2 pulls that forward onto the sweep the finding itself is dropped on, so the two then go together. |
qos_mismatch keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | raise |
As above. |
grace |
int | 3 |
Consecutive ticks a topic must stay affected before it is reported. Endpoint discovery is not atomic, so a subscriber can be visible before a publisher’s QoS is, which reads as starvation for a tick or two. |
allowlist |
string[] | [] |
Subscriber FQNs never reported. Exact match only. |
Closing the loop: healing config is required
Every detector here reports level-triggered: it re-raises FAILED while the condition holds and emits PASSED on every clean sweep. That is what lets the fault_manager’s debounce counter walk from CONFIRMED back to HEALED. The fault_manager only counts PASSED events
File truncated at 100 lines see the full file
Package Dependencies
System Dependencies
Dependant Packages
Launch files
Messages
Services
Plugins
Recent questions tagged ros2_medkit_graph_watchdog at Robotics Stack Exchange
Package Summary
| Version | 0.6.0 |
| License | Apache-2.0 |
| Build type | AMENT_CMAKE |
| Use | RECOMMENDED |
Repository Summary
| Checkout URI | https://github.com/selfpatch/ros2_medkit.git |
| VCS Type | git |
| VCS Version | main |
| Last Updated | 2026-07-31 |
| Dev Status | DEVELOPED |
| Released | RELEASED |
| Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
Maintainers
- bburda
Authors
ros2_medkit_graph_watchdog
Gateway plugin that detects silent faults in the ROS 2 graph: failures where every node is up, nothing logs an error, and the robot is still broken.
Detectors read the graph and raise faults through a ReportFault service client on the
gateway node; the faults surface via FaultManager on the gateway /faults API.
This package carries the plugin skeleton, the central reliability gate that holds raises
until the graph has quiesced, and three detectors, qos_mismatch, orphan and
param_drift. The remaining
silent-fault classes land in follow-up changes, each against its own issue; their fault
codes are already reserved in the frozen GRAPH_* namespace (see “Fault codes”).
Build
With the gateway built and sourced:
colcon build --packages-select ros2_medkit_graph_watchdog
colcon test --packages-select ros2_medkit_graph_watchdog && colcon test-result --verbose
Load into the gateway
ros2 run ros2_medkit_gateway gateway_node --ros-args \
-p plugins:="[graph_watchdog]" \
-p "plugins.graph_watchdog.path:=$(ros2 pkg prefix ros2_medkit_graph_watchdog)/lib/ros2_medkit_graph_watchdog/libros2_medkit_graph_watchdog.so"
Configuration (plugins.graph_watchdog.<key>)
| Key | Type | Default | Meaning |
|---|---|---|---|
tick_interval_ms |
int | 1000 |
Detector tick cadence. |
warmup_cycles |
int | 5 |
An entity must be continuously present for this many ticks before it arms. A mid-run restart re-warms it. Enforced centrally, see “Reliability (bringup-quiesce)” below. |
prune_grace |
int | 60 |
Default injected into every detector’s own config before configure(); a per-detector detectors.<id>.prune_grace overrides it. Used by detectors that keep per-key bookkeeping. |
detectors.<id>.mode |
string | bool | raise |
raise = push faults; advisory = observe, do not push; off = disabled. Bare off/no (YAML booleans) also disable; bare on/yes mean raise. |
detectors.<id>.<field> |
any | - | Per-detector thresholds, passed to that detector’s configure(). |
Nested delivery. Keys are written dotted in YAML or --ros-args, and the gateway
delivers them to the plugin as a nested object. A bare off is a YAML 1.1 boolean, so
mode: off arrives as false, not the string "off"; both forms disable the detector.
Unknown keys are reported. Anything under detectors.<id> that the detector does not
read produces one startup warning naming the key and listing the ones that exist, so a
typo like allow_list: for allowlist: cannot silently do nothing. A typo in the detector
id itself is warned about the same way, with the registered ids listed.
orphan keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | raise |
As above. |
grace |
int | 10 |
Consecutive sweeps a candidate pair must hold before it is reported. A staged bringup or a restart makes one side missing for a short time, which looks exactly like a typo until the other side appears. |
max_edit_distance |
int | 1 |
How many character edits the LEAF (the part after the last /) may differ by and still count as the same intended name. Values outside 1..8 keep the default: past a few edits a “near miss” starts matching unrelated topics. |
namespace_edit_distance |
int | 0 |
The same budget for the namespace, spent separately. 0 means the namespace must match to the character, which is what makes /robot1/scan and /robot2/scan two robots rather than a typo. Raise it to catch a misspelled namespace, and read the warning under the table first. Values outside 0..8 keep the default. |
allowlist |
string[] | [] |
Topic names never reported. Exact match only, never a prefix, so allowlisting /r1/scan does not also silence /r2/scan. |
What this detector will not catch, on purpose. A pair whose names match once every run of
digits is collapsed is treated as an enumeration of sensors, not a typo: /lidar_1 next to
/lidar_2, each one-sided, is an ordinary state on a multi-sensor robot and reporting it would
be a false alarm on every such machine. Collapsing runs rather than comparing character
positions is what also covers /lidar_9 next to /lidar_10. The cost is that a typo which
is a digit stops being reported. An appended digit is not that case: /scan1 collapses to
/scan#, which is not /scan, so it is still reported.
A misspelled namespace is opt-in, and it can be expensive. /robott/scan against
/robot/scan is one edit, but in the namespace, so by default it is not reported. Setting
namespace_edit_distance: 1 reports it. Check the naming scheme before doing that: on a fleet
named with letters, /amr_a and /amr_b are also one edit apart, and every robot in the
fleet will be reported as a typo. A numbered fleet is safe, since /robot1 against /robot2
differs only in its numeric field and the rule above already spares it.
What it cannot catch at all. A namespace added or dropped by mistake, /scan against
/robot/scan, is six edits apart, so no budget that is still specific will reach it. Do not
rely on this detector for that class.
param_drift keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. A bare off arrives as a YAML boolean; both forms disable the detector. |
baseline |
bool | true |
Self-capture the value of every parameter not listed in expect, then flag later changes to it. false checks only what expect pins, and skips reading the full parameter list. |
expect.<param> |
any | - | Absolute expected value for <param>, checked on every node that has it. This is the only rule that catches a value that was wrong from the first tick, which self-capture cannot see. A node that does not declare <param> answers NOT_FOUND, which is not a drift - so a misspelt name checks nothing at all. Once the sweep has covered every armed app - each one read, or given up on after repeated failed reads - and at least one of them answered, a pin that no app declares is warned about once, naming it. |
ignore |
string[] | [] |
Parameter-name globs never flagged. Applies to the self-captured set only, so it cannot silence an expect pin. A pattern of nothing but * matches every name and suppresses self-capture wholesale, which is indistinguishable at runtime from finding nothing; that combination warns at startup. |
max_reads_per_tick |
int | 8 |
Parameter-service ROUND TRIPS this detector may spend per tick, round-robin over nodes. Round trips, not calls: one call is several requests at the watched node, and the node pays for each of them. A baseline visit is charged 2 (a list, then one batched get) whatever the node’s parameter count; each expect pin is charged 3 (a list, a get and a descriptor read), or 2 on a node that does not declare it - see the table under “Coverage latency” for what each of those really costs the node. So a self-capture visit spends 2 of the budget and pinning N parameters spends up to 3N. It is an AMORTISED rate, not a per-tick cap: there is no pacing INSIDE a visit, so a node’s whole read set is issued back to back and the reader waits out the debt afterwards. Four pins against a budget of 8 fire 12 round trips inside one tick period and then idle for the rest of the next one; what the knob bounds is the sustained rate, not the burst. Accepted range 1..100000; anything else keeps the default and warns. |
prune_grace |
int | 60 |
Consecutive sweeps an app may be missing before its finding is dropped: the finding survives prune_grace absences and goes on the next one. Injected for every detector at plugin scope and overridable per detector; a value outside 0..3600 is rejected with a warning and this detector falls back to 2, not to the plugin default. The captured BASELINE has a shorter horizon of its own, and that one is NOT configurable - it is the compile-time kBaselineForgetGrace: the baseline is dropped on the third consecutive missed sweep, the two the reliability gate absorbs plus one, so an ordinary dropped discovery poll cannot re-capture a node on its already-drifted value. Setting prune_grace below 2 pulls that forward onto the sweep the finding itself is dropped on, so the two then go together. |
qos_mismatch keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | raise |
As above. |
grace |
int | 3 |
Consecutive ticks a topic must stay affected before it is reported. Endpoint discovery is not atomic, so a subscriber can be visible before a publisher’s QoS is, which reads as starvation for a tick or two. |
allowlist |
string[] | [] |
Subscriber FQNs never reported. Exact match only. |
Closing the loop: healing config is required
Every detector here reports level-triggered: it re-raises FAILED while the condition holds and emits PASSED on every clean sweep. That is what lets the fault_manager’s debounce counter walk from CONFIRMED back to HEALED. The fault_manager only counts PASSED events
File truncated at 100 lines see the full file
Package Dependencies
System Dependencies
Dependant Packages
Launch files
Messages
Services
Plugins
Recent questions tagged ros2_medkit_graph_watchdog at Robotics Stack Exchange
Package Summary
| Version | 0.6.0 |
| License | Apache-2.0 |
| Build type | AMENT_CMAKE |
| Use | RECOMMENDED |
Repository Summary
| Checkout URI | https://github.com/selfpatch/ros2_medkit.git |
| VCS Type | git |
| VCS Version | main |
| Last Updated | 2026-07-31 |
| Dev Status | DEVELOPED |
| Released | RELEASED |
| Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
Maintainers
- bburda
Authors
ros2_medkit_graph_watchdog
Gateway plugin that detects silent faults in the ROS 2 graph: failures where every node is up, nothing logs an error, and the robot is still broken.
Detectors read the graph and raise faults through a ReportFault service client on the
gateway node; the faults surface via FaultManager on the gateway /faults API.
This package carries the plugin skeleton, the central reliability gate that holds raises
until the graph has quiesced, and three detectors, qos_mismatch, orphan and
param_drift. The remaining
silent-fault classes land in follow-up changes, each against its own issue; their fault
codes are already reserved in the frozen GRAPH_* namespace (see “Fault codes”).
Build
With the gateway built and sourced:
colcon build --packages-select ros2_medkit_graph_watchdog
colcon test --packages-select ros2_medkit_graph_watchdog && colcon test-result --verbose
Load into the gateway
ros2 run ros2_medkit_gateway gateway_node --ros-args \
-p plugins:="[graph_watchdog]" \
-p "plugins.graph_watchdog.path:=$(ros2 pkg prefix ros2_medkit_graph_watchdog)/lib/ros2_medkit_graph_watchdog/libros2_medkit_graph_watchdog.so"
Configuration (plugins.graph_watchdog.<key>)
| Key | Type | Default | Meaning |
|---|---|---|---|
tick_interval_ms |
int | 1000 |
Detector tick cadence. |
warmup_cycles |
int | 5 |
An entity must be continuously present for this many ticks before it arms. A mid-run restart re-warms it. Enforced centrally, see “Reliability (bringup-quiesce)” below. |
prune_grace |
int | 60 |
Default injected into every detector’s own config before configure(); a per-detector detectors.<id>.prune_grace overrides it. Used by detectors that keep per-key bookkeeping. |
detectors.<id>.mode |
string | bool | raise |
raise = push faults; advisory = observe, do not push; off = disabled. Bare off/no (YAML booleans) also disable; bare on/yes mean raise. |
detectors.<id>.<field> |
any | - | Per-detector thresholds, passed to that detector’s configure(). |
Nested delivery. Keys are written dotted in YAML or --ros-args, and the gateway
delivers them to the plugin as a nested object. A bare off is a YAML 1.1 boolean, so
mode: off arrives as false, not the string "off"; both forms disable the detector.
Unknown keys are reported. Anything under detectors.<id> that the detector does not
read produces one startup warning naming the key and listing the ones that exist, so a
typo like allow_list: for allowlist: cannot silently do nothing. A typo in the detector
id itself is warned about the same way, with the registered ids listed.
orphan keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | raise |
As above. |
grace |
int | 10 |
Consecutive sweeps a candidate pair must hold before it is reported. A staged bringup or a restart makes one side missing for a short time, which looks exactly like a typo until the other side appears. |
max_edit_distance |
int | 1 |
How many character edits the LEAF (the part after the last /) may differ by and still count as the same intended name. Values outside 1..8 keep the default: past a few edits a “near miss” starts matching unrelated topics. |
namespace_edit_distance |
int | 0 |
The same budget for the namespace, spent separately. 0 means the namespace must match to the character, which is what makes /robot1/scan and /robot2/scan two robots rather than a typo. Raise it to catch a misspelled namespace, and read the warning under the table first. Values outside 0..8 keep the default. |
allowlist |
string[] | [] |
Topic names never reported. Exact match only, never a prefix, so allowlisting /r1/scan does not also silence /r2/scan. |
What this detector will not catch, on purpose. A pair whose names match once every run of
digits is collapsed is treated as an enumeration of sensors, not a typo: /lidar_1 next to
/lidar_2, each one-sided, is an ordinary state on a multi-sensor robot and reporting it would
be a false alarm on every such machine. Collapsing runs rather than comparing character
positions is what also covers /lidar_9 next to /lidar_10. The cost is that a typo which
is a digit stops being reported. An appended digit is not that case: /scan1 collapses to
/scan#, which is not /scan, so it is still reported.
A misspelled namespace is opt-in, and it can be expensive. /robott/scan against
/robot/scan is one edit, but in the namespace, so by default it is not reported. Setting
namespace_edit_distance: 1 reports it. Check the naming scheme before doing that: on a fleet
named with letters, /amr_a and /amr_b are also one edit apart, and every robot in the
fleet will be reported as a typo. A numbered fleet is safe, since /robot1 against /robot2
differs only in its numeric field and the rule above already spares it.
What it cannot catch at all. A namespace added or dropped by mistake, /scan against
/robot/scan, is six edits apart, so no budget that is still specific will reach it. Do not
rely on this detector for that class.
param_drift keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. A bare off arrives as a YAML boolean; both forms disable the detector. |
baseline |
bool | true |
Self-capture the value of every parameter not listed in expect, then flag later changes to it. false checks only what expect pins, and skips reading the full parameter list. |
expect.<param> |
any | - | Absolute expected value for <param>, checked on every node that has it. This is the only rule that catches a value that was wrong from the first tick, which self-capture cannot see. A node that does not declare <param> answers NOT_FOUND, which is not a drift - so a misspelt name checks nothing at all. Once the sweep has covered every armed app - each one read, or given up on after repeated failed reads - and at least one of them answered, a pin that no app declares is warned about once, naming it. |
ignore |
string[] | [] |
Parameter-name globs never flagged. Applies to the self-captured set only, so it cannot silence an expect pin. A pattern of nothing but * matches every name and suppresses self-capture wholesale, which is indistinguishable at runtime from finding nothing; that combination warns at startup. |
max_reads_per_tick |
int | 8 |
Parameter-service ROUND TRIPS this detector may spend per tick, round-robin over nodes. Round trips, not calls: one call is several requests at the watched node, and the node pays for each of them. A baseline visit is charged 2 (a list, then one batched get) whatever the node’s parameter count; each expect pin is charged 3 (a list, a get and a descriptor read), or 2 on a node that does not declare it - see the table under “Coverage latency” for what each of those really costs the node. So a self-capture visit spends 2 of the budget and pinning N parameters spends up to 3N. It is an AMORTISED rate, not a per-tick cap: there is no pacing INSIDE a visit, so a node’s whole read set is issued back to back and the reader waits out the debt afterwards. Four pins against a budget of 8 fire 12 round trips inside one tick period and then idle for the rest of the next one; what the knob bounds is the sustained rate, not the burst. Accepted range 1..100000; anything else keeps the default and warns. |
prune_grace |
int | 60 |
Consecutive sweeps an app may be missing before its finding is dropped: the finding survives prune_grace absences and goes on the next one. Injected for every detector at plugin scope and overridable per detector; a value outside 0..3600 is rejected with a warning and this detector falls back to 2, not to the plugin default. The captured BASELINE has a shorter horizon of its own, and that one is NOT configurable - it is the compile-time kBaselineForgetGrace: the baseline is dropped on the third consecutive missed sweep, the two the reliability gate absorbs plus one, so an ordinary dropped discovery poll cannot re-capture a node on its already-drifted value. Setting prune_grace below 2 pulls that forward onto the sweep the finding itself is dropped on, so the two then go together. |
qos_mismatch keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | raise |
As above. |
grace |
int | 3 |
Consecutive ticks a topic must stay affected before it is reported. Endpoint discovery is not atomic, so a subscriber can be visible before a publisher’s QoS is, which reads as starvation for a tick or two. |
allowlist |
string[] | [] |
Subscriber FQNs never reported. Exact match only. |
Closing the loop: healing config is required
Every detector here reports level-triggered: it re-raises FAILED while the condition holds and emits PASSED on every clean sweep. That is what lets the fault_manager’s debounce counter walk from CONFIRMED back to HEALED. The fault_manager only counts PASSED events
File truncated at 100 lines see the full file
Package Dependencies
System Dependencies
Dependant Packages
Launch files
Messages
Services
Plugins
Recent questions tagged ros2_medkit_graph_watchdog at Robotics Stack Exchange
Package Summary
| Version | 0.6.0 |
| License | Apache-2.0 |
| Build type | AMENT_CMAKE |
| Use | RECOMMENDED |
Repository Summary
| Checkout URI | https://github.com/selfpatch/ros2_medkit.git |
| VCS Type | git |
| VCS Version | main |
| Last Updated | 2026-07-31 |
| Dev Status | DEVELOPED |
| Released | RELEASED |
| Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
Maintainers
- bburda
Authors
ros2_medkit_graph_watchdog
Gateway plugin that detects silent faults in the ROS 2 graph: failures where every node is up, nothing logs an error, and the robot is still broken.
Detectors read the graph and raise faults through a ReportFault service client on the
gateway node; the faults surface via FaultManager on the gateway /faults API.
This package carries the plugin skeleton, the central reliability gate that holds raises
until the graph has quiesced, and three detectors, qos_mismatch, orphan and
param_drift. The remaining
silent-fault classes land in follow-up changes, each against its own issue; their fault
codes are already reserved in the frozen GRAPH_* namespace (see “Fault codes”).
Build
With the gateway built and sourced:
colcon build --packages-select ros2_medkit_graph_watchdog
colcon test --packages-select ros2_medkit_graph_watchdog && colcon test-result --verbose
Load into the gateway
ros2 run ros2_medkit_gateway gateway_node --ros-args \
-p plugins:="[graph_watchdog]" \
-p "plugins.graph_watchdog.path:=$(ros2 pkg prefix ros2_medkit_graph_watchdog)/lib/ros2_medkit_graph_watchdog/libros2_medkit_graph_watchdog.so"
Configuration (plugins.graph_watchdog.<key>)
| Key | Type | Default | Meaning |
|---|---|---|---|
tick_interval_ms |
int | 1000 |
Detector tick cadence. |
warmup_cycles |
int | 5 |
An entity must be continuously present for this many ticks before it arms. A mid-run restart re-warms it. Enforced centrally, see “Reliability (bringup-quiesce)” below. |
prune_grace |
int | 60 |
Default injected into every detector’s own config before configure(); a per-detector detectors.<id>.prune_grace overrides it. Used by detectors that keep per-key bookkeeping. |
detectors.<id>.mode |
string | bool | raise |
raise = push faults; advisory = observe, do not push; off = disabled. Bare off/no (YAML booleans) also disable; bare on/yes mean raise. |
detectors.<id>.<field> |
any | - | Per-detector thresholds, passed to that detector’s configure(). |
Nested delivery. Keys are written dotted in YAML or --ros-args, and the gateway
delivers them to the plugin as a nested object. A bare off is a YAML 1.1 boolean, so
mode: off arrives as false, not the string "off"; both forms disable the detector.
Unknown keys are reported. Anything under detectors.<id> that the detector does not
read produces one startup warning naming the key and listing the ones that exist, so a
typo like allow_list: for allowlist: cannot silently do nothing. A typo in the detector
id itself is warned about the same way, with the registered ids listed.
orphan keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | raise |
As above. |
grace |
int | 10 |
Consecutive sweeps a candidate pair must hold before it is reported. A staged bringup or a restart makes one side missing for a short time, which looks exactly like a typo until the other side appears. |
max_edit_distance |
int | 1 |
How many character edits the LEAF (the part after the last /) may differ by and still count as the same intended name. Values outside 1..8 keep the default: past a few edits a “near miss” starts matching unrelated topics. |
namespace_edit_distance |
int | 0 |
The same budget for the namespace, spent separately. 0 means the namespace must match to the character, which is what makes /robot1/scan and /robot2/scan two robots rather than a typo. Raise it to catch a misspelled namespace, and read the warning under the table first. Values outside 0..8 keep the default. |
allowlist |
string[] | [] |
Topic names never reported. Exact match only, never a prefix, so allowlisting /r1/scan does not also silence /r2/scan. |
What this detector will not catch, on purpose. A pair whose names match once every run of
digits is collapsed is treated as an enumeration of sensors, not a typo: /lidar_1 next to
/lidar_2, each one-sided, is an ordinary state on a multi-sensor robot and reporting it would
be a false alarm on every such machine. Collapsing runs rather than comparing character
positions is what also covers /lidar_9 next to /lidar_10. The cost is that a typo which
is a digit stops being reported. An appended digit is not that case: /scan1 collapses to
/scan#, which is not /scan, so it is still reported.
A misspelled namespace is opt-in, and it can be expensive. /robott/scan against
/robot/scan is one edit, but in the namespace, so by default it is not reported. Setting
namespace_edit_distance: 1 reports it. Check the naming scheme before doing that: on a fleet
named with letters, /amr_a and /amr_b are also one edit apart, and every robot in the
fleet will be reported as a typo. A numbered fleet is safe, since /robot1 against /robot2
differs only in its numeric field and the rule above already spares it.
What it cannot catch at all. A namespace added or dropped by mistake, /scan against
/robot/scan, is six edits apart, so no budget that is still specific will reach it. Do not
rely on this detector for that class.
param_drift keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. A bare off arrives as a YAML boolean; both forms disable the detector. |
baseline |
bool | true |
Self-capture the value of every parameter not listed in expect, then flag later changes to it. false checks only what expect pins, and skips reading the full parameter list. |
expect.<param> |
any | - | Absolute expected value for <param>, checked on every node that has it. This is the only rule that catches a value that was wrong from the first tick, which self-capture cannot see. A node that does not declare <param> answers NOT_FOUND, which is not a drift - so a misspelt name checks nothing at all. Once the sweep has covered every armed app - each one read, or given up on after repeated failed reads - and at least one of them answered, a pin that no app declares is warned about once, naming it. |
ignore |
string[] | [] |
Parameter-name globs never flagged. Applies to the self-captured set only, so it cannot silence an expect pin. A pattern of nothing but * matches every name and suppresses self-capture wholesale, which is indistinguishable at runtime from finding nothing; that combination warns at startup. |
max_reads_per_tick |
int | 8 |
Parameter-service ROUND TRIPS this detector may spend per tick, round-robin over nodes. Round trips, not calls: one call is several requests at the watched node, and the node pays for each of them. A baseline visit is charged 2 (a list, then one batched get) whatever the node’s parameter count; each expect pin is charged 3 (a list, a get and a descriptor read), or 2 on a node that does not declare it - see the table under “Coverage latency” for what each of those really costs the node. So a self-capture visit spends 2 of the budget and pinning N parameters spends up to 3N. It is an AMORTISED rate, not a per-tick cap: there is no pacing INSIDE a visit, so a node’s whole read set is issued back to back and the reader waits out the debt afterwards. Four pins against a budget of 8 fire 12 round trips inside one tick period and then idle for the rest of the next one; what the knob bounds is the sustained rate, not the burst. Accepted range 1..100000; anything else keeps the default and warns. |
prune_grace |
int | 60 |
Consecutive sweeps an app may be missing before its finding is dropped: the finding survives prune_grace absences and goes on the next one. Injected for every detector at plugin scope and overridable per detector; a value outside 0..3600 is rejected with a warning and this detector falls back to 2, not to the plugin default. The captured BASELINE has a shorter horizon of its own, and that one is NOT configurable - it is the compile-time kBaselineForgetGrace: the baseline is dropped on the third consecutive missed sweep, the two the reliability gate absorbs plus one, so an ordinary dropped discovery poll cannot re-capture a node on its already-drifted value. Setting prune_grace below 2 pulls that forward onto the sweep the finding itself is dropped on, so the two then go together. |
qos_mismatch keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | raise |
As above. |
grace |
int | 3 |
Consecutive ticks a topic must stay affected before it is reported. Endpoint discovery is not atomic, so a subscriber can be visible before a publisher’s QoS is, which reads as starvation for a tick or two. |
allowlist |
string[] | [] |
Subscriber FQNs never reported. Exact match only. |
Closing the loop: healing config is required
Every detector here reports level-triggered: it re-raises FAILED while the condition holds and emits PASSED on every clean sweep. That is what lets the fault_manager’s debounce counter walk from CONFIRMED back to HEALED. The fault_manager only counts PASSED events
File truncated at 100 lines see the full file
Package Dependencies
System Dependencies
Dependant Packages
Launch files
Messages
Services
Plugins
Recent questions tagged ros2_medkit_graph_watchdog at Robotics Stack Exchange
Package Summary
| Version | 0.6.0 |
| License | Apache-2.0 |
| Build type | AMENT_CMAKE |
| Use | RECOMMENDED |
Repository Summary
| Checkout URI | https://github.com/selfpatch/ros2_medkit.git |
| VCS Type | git |
| VCS Version | main |
| Last Updated | 2026-07-31 |
| Dev Status | DEVELOPED |
| Released | RELEASED |
| Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
Maintainers
- bburda
Authors
ros2_medkit_graph_watchdog
Gateway plugin that detects silent faults in the ROS 2 graph: failures where every node is up, nothing logs an error, and the robot is still broken.
Detectors read the graph and raise faults through a ReportFault service client on the
gateway node; the faults surface via FaultManager on the gateway /faults API.
This package carries the plugin skeleton, the central reliability gate that holds raises
until the graph has quiesced, and three detectors, qos_mismatch, orphan and
param_drift. The remaining
silent-fault classes land in follow-up changes, each against its own issue; their fault
codes are already reserved in the frozen GRAPH_* namespace (see “Fault codes”).
Build
With the gateway built and sourced:
colcon build --packages-select ros2_medkit_graph_watchdog
colcon test --packages-select ros2_medkit_graph_watchdog && colcon test-result --verbose
Load into the gateway
ros2 run ros2_medkit_gateway gateway_node --ros-args \
-p plugins:="[graph_watchdog]" \
-p "plugins.graph_watchdog.path:=$(ros2 pkg prefix ros2_medkit_graph_watchdog)/lib/ros2_medkit_graph_watchdog/libros2_medkit_graph_watchdog.so"
Configuration (plugins.graph_watchdog.<key>)
| Key | Type | Default | Meaning |
|---|---|---|---|
tick_interval_ms |
int | 1000 |
Detector tick cadence. |
warmup_cycles |
int | 5 |
An entity must be continuously present for this many ticks before it arms. A mid-run restart re-warms it. Enforced centrally, see “Reliability (bringup-quiesce)” below. |
prune_grace |
int | 60 |
Default injected into every detector’s own config before configure(); a per-detector detectors.<id>.prune_grace overrides it. Used by detectors that keep per-key bookkeeping. |
detectors.<id>.mode |
string | bool | raise |
raise = push faults; advisory = observe, do not push; off = disabled. Bare off/no (YAML booleans) also disable; bare on/yes mean raise. |
detectors.<id>.<field> |
any | - | Per-detector thresholds, passed to that detector’s configure(). |
Nested delivery. Keys are written dotted in YAML or --ros-args, and the gateway
delivers them to the plugin as a nested object. A bare off is a YAML 1.1 boolean, so
mode: off arrives as false, not the string "off"; both forms disable the detector.
Unknown keys are reported. Anything under detectors.<id> that the detector does not
read produces one startup warning naming the key and listing the ones that exist, so a
typo like allow_list: for allowlist: cannot silently do nothing. A typo in the detector
id itself is warned about the same way, with the registered ids listed.
orphan keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | raise |
As above. |
grace |
int | 10 |
Consecutive sweeps a candidate pair must hold before it is reported. A staged bringup or a restart makes one side missing for a short time, which looks exactly like a typo until the other side appears. |
max_edit_distance |
int | 1 |
How many character edits the LEAF (the part after the last /) may differ by and still count as the same intended name. Values outside 1..8 keep the default: past a few edits a “near miss” starts matching unrelated topics. |
namespace_edit_distance |
int | 0 |
The same budget for the namespace, spent separately. 0 means the namespace must match to the character, which is what makes /robot1/scan and /robot2/scan two robots rather than a typo. Raise it to catch a misspelled namespace, and read the warning under the table first. Values outside 0..8 keep the default. |
allowlist |
string[] | [] |
Topic names never reported. Exact match only, never a prefix, so allowlisting /r1/scan does not also silence /r2/scan. |
What this detector will not catch, on purpose. A pair whose names match once every run of
digits is collapsed is treated as an enumeration of sensors, not a typo: /lidar_1 next to
/lidar_2, each one-sided, is an ordinary state on a multi-sensor robot and reporting it would
be a false alarm on every such machine. Collapsing runs rather than comparing character
positions is what also covers /lidar_9 next to /lidar_10. The cost is that a typo which
is a digit stops being reported. An appended digit is not that case: /scan1 collapses to
/scan#, which is not /scan, so it is still reported.
A misspelled namespace is opt-in, and it can be expensive. /robott/scan against
/robot/scan is one edit, but in the namespace, so by default it is not reported. Setting
namespace_edit_distance: 1 reports it. Check the naming scheme before doing that: on a fleet
named with letters, /amr_a and /amr_b are also one edit apart, and every robot in the
fleet will be reported as a typo. A numbered fleet is safe, since /robot1 against /robot2
differs only in its numeric field and the rule above already spares it.
What it cannot catch at all. A namespace added or dropped by mistake, /scan against
/robot/scan, is six edits apart, so no budget that is still specific will reach it. Do not
rely on this detector for that class.
param_drift keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. A bare off arrives as a YAML boolean; both forms disable the detector. |
baseline |
bool | true |
Self-capture the value of every parameter not listed in expect, then flag later changes to it. false checks only what expect pins, and skips reading the full parameter list. |
expect.<param> |
any | - | Absolute expected value for <param>, checked on every node that has it. This is the only rule that catches a value that was wrong from the first tick, which self-capture cannot see. A node that does not declare <param> answers NOT_FOUND, which is not a drift - so a misspelt name checks nothing at all. Once the sweep has covered every armed app - each one read, or given up on after repeated failed reads - and at least one of them answered, a pin that no app declares is warned about once, naming it. |
ignore |
string[] | [] |
Parameter-name globs never flagged. Applies to the self-captured set only, so it cannot silence an expect pin. A pattern of nothing but * matches every name and suppresses self-capture wholesale, which is indistinguishable at runtime from finding nothing; that combination warns at startup. |
max_reads_per_tick |
int | 8 |
Parameter-service ROUND TRIPS this detector may spend per tick, round-robin over nodes. Round trips, not calls: one call is several requests at the watched node, and the node pays for each of them. A baseline visit is charged 2 (a list, then one batched get) whatever the node’s parameter count; each expect pin is charged 3 (a list, a get and a descriptor read), or 2 on a node that does not declare it - see the table under “Coverage latency” for what each of those really costs the node. So a self-capture visit spends 2 of the budget and pinning N parameters spends up to 3N. It is an AMORTISED rate, not a per-tick cap: there is no pacing INSIDE a visit, so a node’s whole read set is issued back to back and the reader waits out the debt afterwards. Four pins against a budget of 8 fire 12 round trips inside one tick period and then idle for the rest of the next one; what the knob bounds is the sustained rate, not the burst. Accepted range 1..100000; anything else keeps the default and warns. |
prune_grace |
int | 60 |
Consecutive sweeps an app may be missing before its finding is dropped: the finding survives prune_grace absences and goes on the next one. Injected for every detector at plugin scope and overridable per detector; a value outside 0..3600 is rejected with a warning and this detector falls back to 2, not to the plugin default. The captured BASELINE has a shorter horizon of its own, and that one is NOT configurable - it is the compile-time kBaselineForgetGrace: the baseline is dropped on the third consecutive missed sweep, the two the reliability gate absorbs plus one, so an ordinary dropped discovery poll cannot re-capture a node on its already-drifted value. Setting prune_grace below 2 pulls that forward onto the sweep the finding itself is dropped on, so the two then go together. |
qos_mismatch keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | raise |
As above. |
grace |
int | 3 |
Consecutive ticks a topic must stay affected before it is reported. Endpoint discovery is not atomic, so a subscriber can be visible before a publisher’s QoS is, which reads as starvation for a tick or two. |
allowlist |
string[] | [] |
Subscriber FQNs never reported. Exact match only. |
Closing the loop: healing config is required
Every detector here reports level-triggered: it re-raises FAILED while the condition holds and emits PASSED on every clean sweep. That is what lets the fault_manager’s debounce counter walk from CONFIRMED back to HEALED. The fault_manager only counts PASSED events
File truncated at 100 lines see the full file
Package Dependencies
System Dependencies
Dependant Packages
Launch files
Messages
Services
Plugins
Recent questions tagged ros2_medkit_graph_watchdog at Robotics Stack Exchange
Package Summary
| Version | 0.6.0 |
| License | Apache-2.0 |
| Build type | AMENT_CMAKE |
| Use | RECOMMENDED |
Repository Summary
| Checkout URI | https://github.com/selfpatch/ros2_medkit.git |
| VCS Type | git |
| VCS Version | main |
| Last Updated | 2026-07-31 |
| Dev Status | DEVELOPED |
| Released | RELEASED |
| Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
Maintainers
- bburda
Authors
ros2_medkit_graph_watchdog
Gateway plugin that detects silent faults in the ROS 2 graph: failures where every node is up, nothing logs an error, and the robot is still broken.
Detectors read the graph and raise faults through a ReportFault service client on the
gateway node; the faults surface via FaultManager on the gateway /faults API.
This package carries the plugin skeleton, the central reliability gate that holds raises
until the graph has quiesced, and three detectors, qos_mismatch, orphan and
param_drift. The remaining
silent-fault classes land in follow-up changes, each against its own issue; their fault
codes are already reserved in the frozen GRAPH_* namespace (see “Fault codes”).
Build
With the gateway built and sourced:
colcon build --packages-select ros2_medkit_graph_watchdog
colcon test --packages-select ros2_medkit_graph_watchdog && colcon test-result --verbose
Load into the gateway
ros2 run ros2_medkit_gateway gateway_node --ros-args \
-p plugins:="[graph_watchdog]" \
-p "plugins.graph_watchdog.path:=$(ros2 pkg prefix ros2_medkit_graph_watchdog)/lib/ros2_medkit_graph_watchdog/libros2_medkit_graph_watchdog.so"
Configuration (plugins.graph_watchdog.<key>)
| Key | Type | Default | Meaning |
|---|---|---|---|
tick_interval_ms |
int | 1000 |
Detector tick cadence. |
warmup_cycles |
int | 5 |
An entity must be continuously present for this many ticks before it arms. A mid-run restart re-warms it. Enforced centrally, see “Reliability (bringup-quiesce)” below. |
prune_grace |
int | 60 |
Default injected into every detector’s own config before configure(); a per-detector detectors.<id>.prune_grace overrides it. Used by detectors that keep per-key bookkeeping. |
detectors.<id>.mode |
string | bool | raise |
raise = push faults; advisory = observe, do not push; off = disabled. Bare off/no (YAML booleans) also disable; bare on/yes mean raise. |
detectors.<id>.<field> |
any | - | Per-detector thresholds, passed to that detector’s configure(). |
Nested delivery. Keys are written dotted in YAML or --ros-args, and the gateway
delivers them to the plugin as a nested object. A bare off is a YAML 1.1 boolean, so
mode: off arrives as false, not the string "off"; both forms disable the detector.
Unknown keys are reported. Anything under detectors.<id> that the detector does not
read produces one startup warning naming the key and listing the ones that exist, so a
typo like allow_list: for allowlist: cannot silently do nothing. A typo in the detector
id itself is warned about the same way, with the registered ids listed.
orphan keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | raise |
As above. |
grace |
int | 10 |
Consecutive sweeps a candidate pair must hold before it is reported. A staged bringup or a restart makes one side missing for a short time, which looks exactly like a typo until the other side appears. |
max_edit_distance |
int | 1 |
How many character edits the LEAF (the part after the last /) may differ by and still count as the same intended name. Values outside 1..8 keep the default: past a few edits a “near miss” starts matching unrelated topics. |
namespace_edit_distance |
int | 0 |
The same budget for the namespace, spent separately. 0 means the namespace must match to the character, which is what makes /robot1/scan and /robot2/scan two robots rather than a typo. Raise it to catch a misspelled namespace, and read the warning under the table first. Values outside 0..8 keep the default. |
allowlist |
string[] | [] |
Topic names never reported. Exact match only, never a prefix, so allowlisting /r1/scan does not also silence /r2/scan. |
What this detector will not catch, on purpose. A pair whose names match once every run of
digits is collapsed is treated as an enumeration of sensors, not a typo: /lidar_1 next to
/lidar_2, each one-sided, is an ordinary state on a multi-sensor robot and reporting it would
be a false alarm on every such machine. Collapsing runs rather than comparing character
positions is what also covers /lidar_9 next to /lidar_10. The cost is that a typo which
is a digit stops being reported. An appended digit is not that case: /scan1 collapses to
/scan#, which is not /scan, so it is still reported.
A misspelled namespace is opt-in, and it can be expensive. /robott/scan against
/robot/scan is one edit, but in the namespace, so by default it is not reported. Setting
namespace_edit_distance: 1 reports it. Check the naming scheme before doing that: on a fleet
named with letters, /amr_a and /amr_b are also one edit apart, and every robot in the
fleet will be reported as a typo. A numbered fleet is safe, since /robot1 against /robot2
differs only in its numeric field and the rule above already spares it.
What it cannot catch at all. A namespace added or dropped by mistake, /scan against
/robot/scan, is six edits apart, so no budget that is still specific will reach it. Do not
rely on this detector for that class.
param_drift keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. A bare off arrives as a YAML boolean; both forms disable the detector. |
baseline |
bool | true |
Self-capture the value of every parameter not listed in expect, then flag later changes to it. false checks only what expect pins, and skips reading the full parameter list. |
expect.<param> |
any | - | Absolute expected value for <param>, checked on every node that has it. This is the only rule that catches a value that was wrong from the first tick, which self-capture cannot see. A node that does not declare <param> answers NOT_FOUND, which is not a drift - so a misspelt name checks nothing at all. Once the sweep has covered every armed app - each one read, or given up on after repeated failed reads - and at least one of them answered, a pin that no app declares is warned about once, naming it. |
ignore |
string[] | [] |
Parameter-name globs never flagged. Applies to the self-captured set only, so it cannot silence an expect pin. A pattern of nothing but * matches every name and suppresses self-capture wholesale, which is indistinguishable at runtime from finding nothing; that combination warns at startup. |
max_reads_per_tick |
int | 8 |
Parameter-service ROUND TRIPS this detector may spend per tick, round-robin over nodes. Round trips, not calls: one call is several requests at the watched node, and the node pays for each of them. A baseline visit is charged 2 (a list, then one batched get) whatever the node’s parameter count; each expect pin is charged 3 (a list, a get and a descriptor read), or 2 on a node that does not declare it - see the table under “Coverage latency” for what each of those really costs the node. So a self-capture visit spends 2 of the budget and pinning N parameters spends up to 3N. It is an AMORTISED rate, not a per-tick cap: there is no pacing INSIDE a visit, so a node’s whole read set is issued back to back and the reader waits out the debt afterwards. Four pins against a budget of 8 fire 12 round trips inside one tick period and then idle for the rest of the next one; what the knob bounds is the sustained rate, not the burst. Accepted range 1..100000; anything else keeps the default and warns. |
prune_grace |
int | 60 |
Consecutive sweeps an app may be missing before its finding is dropped: the finding survives prune_grace absences and goes on the next one. Injected for every detector at plugin scope and overridable per detector; a value outside 0..3600 is rejected with a warning and this detector falls back to 2, not to the plugin default. The captured BASELINE has a shorter horizon of its own, and that one is NOT configurable - it is the compile-time kBaselineForgetGrace: the baseline is dropped on the third consecutive missed sweep, the two the reliability gate absorbs plus one, so an ordinary dropped discovery poll cannot re-capture a node on its already-drifted value. Setting prune_grace below 2 pulls that forward onto the sweep the finding itself is dropped on, so the two then go together. |
qos_mismatch keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | raise |
As above. |
grace |
int | 3 |
Consecutive ticks a topic must stay affected before it is reported. Endpoint discovery is not atomic, so a subscriber can be visible before a publisher’s QoS is, which reads as starvation for a tick or two. |
allowlist |
string[] | [] |
Subscriber FQNs never reported. Exact match only. |
Closing the loop: healing config is required
Every detector here reports level-triggered: it re-raises FAILED while the condition holds and emits PASSED on every clean sweep. That is what lets the fault_manager’s debounce counter walk from CONFIRMED back to HEALED. The fault_manager only counts PASSED events
File truncated at 100 lines see the full file
Package Dependencies
System Dependencies
Dependant Packages
Launch files
Messages
Services
Plugins
Recent questions tagged ros2_medkit_graph_watchdog at Robotics Stack Exchange
Package Summary
| Version | 0.6.0 |
| License | Apache-2.0 |
| Build type | AMENT_CMAKE |
| Use | RECOMMENDED |
Repository Summary
| Checkout URI | https://github.com/selfpatch/ros2_medkit.git |
| VCS Type | git |
| VCS Version | main |
| Last Updated | 2026-07-31 |
| Dev Status | DEVELOPED |
| Released | RELEASED |
| Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
Maintainers
- bburda
Authors
ros2_medkit_graph_watchdog
Gateway plugin that detects silent faults in the ROS 2 graph: failures where every node is up, nothing logs an error, and the robot is still broken.
Detectors read the graph and raise faults through a ReportFault service client on the
gateway node; the faults surface via FaultManager on the gateway /faults API.
This package carries the plugin skeleton, the central reliability gate that holds raises
until the graph has quiesced, and three detectors, qos_mismatch, orphan and
param_drift. The remaining
silent-fault classes land in follow-up changes, each against its own issue; their fault
codes are already reserved in the frozen GRAPH_* namespace (see “Fault codes”).
Build
With the gateway built and sourced:
colcon build --packages-select ros2_medkit_graph_watchdog
colcon test --packages-select ros2_medkit_graph_watchdog && colcon test-result --verbose
Load into the gateway
ros2 run ros2_medkit_gateway gateway_node --ros-args \
-p plugins:="[graph_watchdog]" \
-p "plugins.graph_watchdog.path:=$(ros2 pkg prefix ros2_medkit_graph_watchdog)/lib/ros2_medkit_graph_watchdog/libros2_medkit_graph_watchdog.so"
Configuration (plugins.graph_watchdog.<key>)
| Key | Type | Default | Meaning |
|---|---|---|---|
tick_interval_ms |
int | 1000 |
Detector tick cadence. |
warmup_cycles |
int | 5 |
An entity must be continuously present for this many ticks before it arms. A mid-run restart re-warms it. Enforced centrally, see “Reliability (bringup-quiesce)” below. |
prune_grace |
int | 60 |
Default injected into every detector’s own config before configure(); a per-detector detectors.<id>.prune_grace overrides it. Used by detectors that keep per-key bookkeeping. |
detectors.<id>.mode |
string | bool | raise |
raise = push faults; advisory = observe, do not push; off = disabled. Bare off/no (YAML booleans) also disable; bare on/yes mean raise. |
detectors.<id>.<field> |
any | - | Per-detector thresholds, passed to that detector’s configure(). |
Nested delivery. Keys are written dotted in YAML or --ros-args, and the gateway
delivers them to the plugin as a nested object. A bare off is a YAML 1.1 boolean, so
mode: off arrives as false, not the string "off"; both forms disable the detector.
Unknown keys are reported. Anything under detectors.<id> that the detector does not
read produces one startup warning naming the key and listing the ones that exist, so a
typo like allow_list: for allowlist: cannot silently do nothing. A typo in the detector
id itself is warned about the same way, with the registered ids listed.
orphan keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | raise |
As above. |
grace |
int | 10 |
Consecutive sweeps a candidate pair must hold before it is reported. A staged bringup or a restart makes one side missing for a short time, which looks exactly like a typo until the other side appears. |
max_edit_distance |
int | 1 |
How many character edits the LEAF (the part after the last /) may differ by and still count as the same intended name. Values outside 1..8 keep the default: past a few edits a “near miss” starts matching unrelated topics. |
namespace_edit_distance |
int | 0 |
The same budget for the namespace, spent separately. 0 means the namespace must match to the character, which is what makes /robot1/scan and /robot2/scan two robots rather than a typo. Raise it to catch a misspelled namespace, and read the warning under the table first. Values outside 0..8 keep the default. |
allowlist |
string[] | [] |
Topic names never reported. Exact match only, never a prefix, so allowlisting /r1/scan does not also silence /r2/scan. |
What this detector will not catch, on purpose. A pair whose names match once every run of
digits is collapsed is treated as an enumeration of sensors, not a typo: /lidar_1 next to
/lidar_2, each one-sided, is an ordinary state on a multi-sensor robot and reporting it would
be a false alarm on every such machine. Collapsing runs rather than comparing character
positions is what also covers /lidar_9 next to /lidar_10. The cost is that a typo which
is a digit stops being reported. An appended digit is not that case: /scan1 collapses to
/scan#, which is not /scan, so it is still reported.
A misspelled namespace is opt-in, and it can be expensive. /robott/scan against
/robot/scan is one edit, but in the namespace, so by default it is not reported. Setting
namespace_edit_distance: 1 reports it. Check the naming scheme before doing that: on a fleet
named with letters, /amr_a and /amr_b are also one edit apart, and every robot in the
fleet will be reported as a typo. A numbered fleet is safe, since /robot1 against /robot2
differs only in its numeric field and the rule above already spares it.
What it cannot catch at all. A namespace added or dropped by mistake, /scan against
/robot/scan, is six edits apart, so no budget that is still specific will reach it. Do not
rely on this detector for that class.
param_drift keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. A bare off arrives as a YAML boolean; both forms disable the detector. |
baseline |
bool | true |
Self-capture the value of every parameter not listed in expect, then flag later changes to it. false checks only what expect pins, and skips reading the full parameter list. |
expect.<param> |
any | - | Absolute expected value for <param>, checked on every node that has it. This is the only rule that catches a value that was wrong from the first tick, which self-capture cannot see. A node that does not declare <param> answers NOT_FOUND, which is not a drift - so a misspelt name checks nothing at all. Once the sweep has covered every armed app - each one read, or given up on after repeated failed reads - and at least one of them answered, a pin that no app declares is warned about once, naming it. |
ignore |
string[] | [] |
Parameter-name globs never flagged. Applies to the self-captured set only, so it cannot silence an expect pin. A pattern of nothing but * matches every name and suppresses self-capture wholesale, which is indistinguishable at runtime from finding nothing; that combination warns at startup. |
max_reads_per_tick |
int | 8 |
Parameter-service ROUND TRIPS this detector may spend per tick, round-robin over nodes. Round trips, not calls: one call is several requests at the watched node, and the node pays for each of them. A baseline visit is charged 2 (a list, then one batched get) whatever the node’s parameter count; each expect pin is charged 3 (a list, a get and a descriptor read), or 2 on a node that does not declare it - see the table under “Coverage latency” for what each of those really costs the node. So a self-capture visit spends 2 of the budget and pinning N parameters spends up to 3N. It is an AMORTISED rate, not a per-tick cap: there is no pacing INSIDE a visit, so a node’s whole read set is issued back to back and the reader waits out the debt afterwards. Four pins against a budget of 8 fire 12 round trips inside one tick period and then idle for the rest of the next one; what the knob bounds is the sustained rate, not the burst. Accepted range 1..100000; anything else keeps the default and warns. |
prune_grace |
int | 60 |
Consecutive sweeps an app may be missing before its finding is dropped: the finding survives prune_grace absences and goes on the next one. Injected for every detector at plugin scope and overridable per detector; a value outside 0..3600 is rejected with a warning and this detector falls back to 2, not to the plugin default. The captured BASELINE has a shorter horizon of its own, and that one is NOT configurable - it is the compile-time kBaselineForgetGrace: the baseline is dropped on the third consecutive missed sweep, the two the reliability gate absorbs plus one, so an ordinary dropped discovery poll cannot re-capture a node on its already-drifted value. Setting prune_grace below 2 pulls that forward onto the sweep the finding itself is dropped on, so the two then go together. |
qos_mismatch keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | raise |
As above. |
grace |
int | 3 |
Consecutive ticks a topic must stay affected before it is reported. Endpoint discovery is not atomic, so a subscriber can be visible before a publisher’s QoS is, which reads as starvation for a tick or two. |
allowlist |
string[] | [] |
Subscriber FQNs never reported. Exact match only. |
Closing the loop: healing config is required
Every detector here reports level-triggered: it re-raises FAILED while the condition holds and emits PASSED on every clean sweep. That is what lets the fault_manager’s debounce counter walk from CONFIRMED back to HEALED. The fault_manager only counts PASSED events
File truncated at 100 lines see the full file
Package Dependencies
System Dependencies
Dependant Packages
Launch files
Messages
Services
Plugins
Recent questions tagged ros2_medkit_graph_watchdog at Robotics Stack Exchange
Package Summary
| Version | 0.6.0 |
| License | Apache-2.0 |
| Build type | AMENT_CMAKE |
| Use | RECOMMENDED |
Repository Summary
| Checkout URI | https://github.com/selfpatch/ros2_medkit.git |
| VCS Type | git |
| VCS Version | main |
| Last Updated | 2026-07-31 |
| Dev Status | DEVELOPED |
| Released | RELEASED |
| Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
Maintainers
- bburda
Authors
ros2_medkit_graph_watchdog
Gateway plugin that detects silent faults in the ROS 2 graph: failures where every node is up, nothing logs an error, and the robot is still broken.
Detectors read the graph and raise faults through a ReportFault service client on the
gateway node; the faults surface via FaultManager on the gateway /faults API.
This package carries the plugin skeleton, the central reliability gate that holds raises
until the graph has quiesced, and three detectors, qos_mismatch, orphan and
param_drift. The remaining
silent-fault classes land in follow-up changes, each against its own issue; their fault
codes are already reserved in the frozen GRAPH_* namespace (see “Fault codes”).
Build
With the gateway built and sourced:
colcon build --packages-select ros2_medkit_graph_watchdog
colcon test --packages-select ros2_medkit_graph_watchdog && colcon test-result --verbose
Load into the gateway
ros2 run ros2_medkit_gateway gateway_node --ros-args \
-p plugins:="[graph_watchdog]" \
-p "plugins.graph_watchdog.path:=$(ros2 pkg prefix ros2_medkit_graph_watchdog)/lib/ros2_medkit_graph_watchdog/libros2_medkit_graph_watchdog.so"
Configuration (plugins.graph_watchdog.<key>)
| Key | Type | Default | Meaning |
|---|---|---|---|
tick_interval_ms |
int | 1000 |
Detector tick cadence. |
warmup_cycles |
int | 5 |
An entity must be continuously present for this many ticks before it arms. A mid-run restart re-warms it. Enforced centrally, see “Reliability (bringup-quiesce)” below. |
prune_grace |
int | 60 |
Default injected into every detector’s own config before configure(); a per-detector detectors.<id>.prune_grace overrides it. Used by detectors that keep per-key bookkeeping. |
detectors.<id>.mode |
string | bool | raise |
raise = push faults; advisory = observe, do not push; off = disabled. Bare off/no (YAML booleans) also disable; bare on/yes mean raise. |
detectors.<id>.<field> |
any | - | Per-detector thresholds, passed to that detector’s configure(). |
Nested delivery. Keys are written dotted in YAML or --ros-args, and the gateway
delivers them to the plugin as a nested object. A bare off is a YAML 1.1 boolean, so
mode: off arrives as false, not the string "off"; both forms disable the detector.
Unknown keys are reported. Anything under detectors.<id> that the detector does not
read produces one startup warning naming the key and listing the ones that exist, so a
typo like allow_list: for allowlist: cannot silently do nothing. A typo in the detector
id itself is warned about the same way, with the registered ids listed.
orphan keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | raise |
As above. |
grace |
int | 10 |
Consecutive sweeps a candidate pair must hold before it is reported. A staged bringup or a restart makes one side missing for a short time, which looks exactly like a typo until the other side appears. |
max_edit_distance |
int | 1 |
How many character edits the LEAF (the part after the last /) may differ by and still count as the same intended name. Values outside 1..8 keep the default: past a few edits a “near miss” starts matching unrelated topics. |
namespace_edit_distance |
int | 0 |
The same budget for the namespace, spent separately. 0 means the namespace must match to the character, which is what makes /robot1/scan and /robot2/scan two robots rather than a typo. Raise it to catch a misspelled namespace, and read the warning under the table first. Values outside 0..8 keep the default. |
allowlist |
string[] | [] |
Topic names never reported. Exact match only, never a prefix, so allowlisting /r1/scan does not also silence /r2/scan. |
What this detector will not catch, on purpose. A pair whose names match once every run of
digits is collapsed is treated as an enumeration of sensors, not a typo: /lidar_1 next to
/lidar_2, each one-sided, is an ordinary state on a multi-sensor robot and reporting it would
be a false alarm on every such machine. Collapsing runs rather than comparing character
positions is what also covers /lidar_9 next to /lidar_10. The cost is that a typo which
is a digit stops being reported. An appended digit is not that case: /scan1 collapses to
/scan#, which is not /scan, so it is still reported.
A misspelled namespace is opt-in, and it can be expensive. /robott/scan against
/robot/scan is one edit, but in the namespace, so by default it is not reported. Setting
namespace_edit_distance: 1 reports it. Check the naming scheme before doing that: on a fleet
named with letters, /amr_a and /amr_b are also one edit apart, and every robot in the
fleet will be reported as a typo. A numbered fleet is safe, since /robot1 against /robot2
differs only in its numeric field and the rule above already spares it.
What it cannot catch at all. A namespace added or dropped by mistake, /scan against
/robot/scan, is six edits apart, so no budget that is still specific will reach it. Do not
rely on this detector for that class.
param_drift keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. A bare off arrives as a YAML boolean; both forms disable the detector. |
baseline |
bool | true |
Self-capture the value of every parameter not listed in expect, then flag later changes to it. false checks only what expect pins, and skips reading the full parameter list. |
expect.<param> |
any | - | Absolute expected value for <param>, checked on every node that has it. This is the only rule that catches a value that was wrong from the first tick, which self-capture cannot see. A node that does not declare <param> answers NOT_FOUND, which is not a drift - so a misspelt name checks nothing at all. Once the sweep has covered every armed app - each one read, or given up on after repeated failed reads - and at least one of them answered, a pin that no app declares is warned about once, naming it. |
ignore |
string[] | [] |
Parameter-name globs never flagged. Applies to the self-captured set only, so it cannot silence an expect pin. A pattern of nothing but * matches every name and suppresses self-capture wholesale, which is indistinguishable at runtime from finding nothing; that combination warns at startup. |
max_reads_per_tick |
int | 8 |
Parameter-service ROUND TRIPS this detector may spend per tick, round-robin over nodes. Round trips, not calls: one call is several requests at the watched node, and the node pays for each of them. A baseline visit is charged 2 (a list, then one batched get) whatever the node’s parameter count; each expect pin is charged 3 (a list, a get and a descriptor read), or 2 on a node that does not declare it - see the table under “Coverage latency” for what each of those really costs the node. So a self-capture visit spends 2 of the budget and pinning N parameters spends up to 3N. It is an AMORTISED rate, not a per-tick cap: there is no pacing INSIDE a visit, so a node’s whole read set is issued back to back and the reader waits out the debt afterwards. Four pins against a budget of 8 fire 12 round trips inside one tick period and then idle for the rest of the next one; what the knob bounds is the sustained rate, not the burst. Accepted range 1..100000; anything else keeps the default and warns. |
prune_grace |
int | 60 |
Consecutive sweeps an app may be missing before its finding is dropped: the finding survives prune_grace absences and goes on the next one. Injected for every detector at plugin scope and overridable per detector; a value outside 0..3600 is rejected with a warning and this detector falls back to 2, not to the plugin default. The captured BASELINE has a shorter horizon of its own, and that one is NOT configurable - it is the compile-time kBaselineForgetGrace: the baseline is dropped on the third consecutive missed sweep, the two the reliability gate absorbs plus one, so an ordinary dropped discovery poll cannot re-capture a node on its already-drifted value. Setting prune_grace below 2 pulls that forward onto the sweep the finding itself is dropped on, so the two then go together. |
qos_mismatch keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | raise |
As above. |
grace |
int | 3 |
Consecutive ticks a topic must stay affected before it is reported. Endpoint discovery is not atomic, so a subscriber can be visible before a publisher’s QoS is, which reads as starvation for a tick or two. |
allowlist |
string[] | [] |
Subscriber FQNs never reported. Exact match only. |
Closing the loop: healing config is required
Every detector here reports level-triggered: it re-raises FAILED while the condition holds and emits PASSED on every clean sweep. That is what lets the fault_manager’s debounce counter walk from CONFIRMED back to HEALED. The fault_manager only counts PASSED events
File truncated at 100 lines see the full file
Package Dependencies
System Dependencies
Dependant Packages
Launch files
Messages
Services
Plugins
Recent questions tagged ros2_medkit_graph_watchdog at Robotics Stack Exchange
Package Summary
| Version | 0.6.0 |
| License | Apache-2.0 |
| Build type | AMENT_CMAKE |
| Use | RECOMMENDED |
Repository Summary
| Checkout URI | https://github.com/selfpatch/ros2_medkit.git |
| VCS Type | git |
| VCS Version | main |
| Last Updated | 2026-07-31 |
| Dev Status | DEVELOPED |
| Released | RELEASED |
| Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
Maintainers
- bburda
Authors
ros2_medkit_graph_watchdog
Gateway plugin that detects silent faults in the ROS 2 graph: failures where every node is up, nothing logs an error, and the robot is still broken.
Detectors read the graph and raise faults through a ReportFault service client on the
gateway node; the faults surface via FaultManager on the gateway /faults API.
This package carries the plugin skeleton, the central reliability gate that holds raises
until the graph has quiesced, and three detectors, qos_mismatch, orphan and
param_drift. The remaining
silent-fault classes land in follow-up changes, each against its own issue; their fault
codes are already reserved in the frozen GRAPH_* namespace (see “Fault codes”).
Build
With the gateway built and sourced:
colcon build --packages-select ros2_medkit_graph_watchdog
colcon test --packages-select ros2_medkit_graph_watchdog && colcon test-result --verbose
Load into the gateway
ros2 run ros2_medkit_gateway gateway_node --ros-args \
-p plugins:="[graph_watchdog]" \
-p "plugins.graph_watchdog.path:=$(ros2 pkg prefix ros2_medkit_graph_watchdog)/lib/ros2_medkit_graph_watchdog/libros2_medkit_graph_watchdog.so"
Configuration (plugins.graph_watchdog.<key>)
| Key | Type | Default | Meaning |
|---|---|---|---|
tick_interval_ms |
int | 1000 |
Detector tick cadence. |
warmup_cycles |
int | 5 |
An entity must be continuously present for this many ticks before it arms. A mid-run restart re-warms it. Enforced centrally, see “Reliability (bringup-quiesce)” below. |
prune_grace |
int | 60 |
Default injected into every detector’s own config before configure(); a per-detector detectors.<id>.prune_grace overrides it. Used by detectors that keep per-key bookkeeping. |
detectors.<id>.mode |
string | bool | raise |
raise = push faults; advisory = observe, do not push; off = disabled. Bare off/no (YAML booleans) also disable; bare on/yes mean raise. |
detectors.<id>.<field> |
any | - | Per-detector thresholds, passed to that detector’s configure(). |
Nested delivery. Keys are written dotted in YAML or --ros-args, and the gateway
delivers them to the plugin as a nested object. A bare off is a YAML 1.1 boolean, so
mode: off arrives as false, not the string "off"; both forms disable the detector.
Unknown keys are reported. Anything under detectors.<id> that the detector does not
read produces one startup warning naming the key and listing the ones that exist, so a
typo like allow_list: for allowlist: cannot silently do nothing. A typo in the detector
id itself is warned about the same way, with the registered ids listed.
orphan keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | raise |
As above. |
grace |
int | 10 |
Consecutive sweeps a candidate pair must hold before it is reported. A staged bringup or a restart makes one side missing for a short time, which looks exactly like a typo until the other side appears. |
max_edit_distance |
int | 1 |
How many character edits the LEAF (the part after the last /) may differ by and still count as the same intended name. Values outside 1..8 keep the default: past a few edits a “near miss” starts matching unrelated topics. |
namespace_edit_distance |
int | 0 |
The same budget for the namespace, spent separately. 0 means the namespace must match to the character, which is what makes /robot1/scan and /robot2/scan two robots rather than a typo. Raise it to catch a misspelled namespace, and read the warning under the table first. Values outside 0..8 keep the default. |
allowlist |
string[] | [] |
Topic names never reported. Exact match only, never a prefix, so allowlisting /r1/scan does not also silence /r2/scan. |
What this detector will not catch, on purpose. A pair whose names match once every run of
digits is collapsed is treated as an enumeration of sensors, not a typo: /lidar_1 next to
/lidar_2, each one-sided, is an ordinary state on a multi-sensor robot and reporting it would
be a false alarm on every such machine. Collapsing runs rather than comparing character
positions is what also covers /lidar_9 next to /lidar_10. The cost is that a typo which
is a digit stops being reported. An appended digit is not that case: /scan1 collapses to
/scan#, which is not /scan, so it is still reported.
A misspelled namespace is opt-in, and it can be expensive. /robott/scan against
/robot/scan is one edit, but in the namespace, so by default it is not reported. Setting
namespace_edit_distance: 1 reports it. Check the naming scheme before doing that: on a fleet
named with letters, /amr_a and /amr_b are also one edit apart, and every robot in the
fleet will be reported as a typo. A numbered fleet is safe, since /robot1 against /robot2
differs only in its numeric field and the rule above already spares it.
What it cannot catch at all. A namespace added or dropped by mistake, /scan against
/robot/scan, is six edits apart, so no budget that is still specific will reach it. Do not
rely on this detector for that class.
param_drift keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. A bare off arrives as a YAML boolean; both forms disable the detector. |
baseline |
bool | true |
Self-capture the value of every parameter not listed in expect, then flag later changes to it. false checks only what expect pins, and skips reading the full parameter list. |
expect.<param> |
any | - | Absolute expected value for <param>, checked on every node that has it. This is the only rule that catches a value that was wrong from the first tick, which self-capture cannot see. A node that does not declare <param> answers NOT_FOUND, which is not a drift - so a misspelt name checks nothing at all. Once the sweep has covered every armed app - each one read, or given up on after repeated failed reads - and at least one of them answered, a pin that no app declares is warned about once, naming it. |
ignore |
string[] | [] |
Parameter-name globs never flagged. Applies to the self-captured set only, so it cannot silence an expect pin. A pattern of nothing but * matches every name and suppresses self-capture wholesale, which is indistinguishable at runtime from finding nothing; that combination warns at startup. |
max_reads_per_tick |
int | 8 |
Parameter-service ROUND TRIPS this detector may spend per tick, round-robin over nodes. Round trips, not calls: one call is several requests at the watched node, and the node pays for each of them. A baseline visit is charged 2 (a list, then one batched get) whatever the node’s parameter count; each expect pin is charged 3 (a list, a get and a descriptor read), or 2 on a node that does not declare it - see the table under “Coverage latency” for what each of those really costs the node. So a self-capture visit spends 2 of the budget and pinning N parameters spends up to 3N. It is an AMORTISED rate, not a per-tick cap: there is no pacing INSIDE a visit, so a node’s whole read set is issued back to back and the reader waits out the debt afterwards. Four pins against a budget of 8 fire 12 round trips inside one tick period and then idle for the rest of the next one; what the knob bounds is the sustained rate, not the burst. Accepted range 1..100000; anything else keeps the default and warns. |
prune_grace |
int | 60 |
Consecutive sweeps an app may be missing before its finding is dropped: the finding survives prune_grace absences and goes on the next one. Injected for every detector at plugin scope and overridable per detector; a value outside 0..3600 is rejected with a warning and this detector falls back to 2, not to the plugin default. The captured BASELINE has a shorter horizon of its own, and that one is NOT configurable - it is the compile-time kBaselineForgetGrace: the baseline is dropped on the third consecutive missed sweep, the two the reliability gate absorbs plus one, so an ordinary dropped discovery poll cannot re-capture a node on its already-drifted value. Setting prune_grace below 2 pulls that forward onto the sweep the finding itself is dropped on, so the two then go together. |
qos_mismatch keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | raise |
As above. |
grace |
int | 3 |
Consecutive ticks a topic must stay affected before it is reported. Endpoint discovery is not atomic, so a subscriber can be visible before a publisher’s QoS is, which reads as starvation for a tick or two. |
allowlist |
string[] | [] |
Subscriber FQNs never reported. Exact match only. |
Closing the loop: healing config is required
Every detector here reports level-triggered: it re-raises FAILED while the condition holds and emits PASSED on every clean sweep. That is what lets the fault_manager’s debounce counter walk from CONFIRMED back to HEALED. The fault_manager only counts PASSED events
File truncated at 100 lines see the full file
Package Dependencies
System Dependencies
Dependant Packages
Launch files
Messages
Services
Plugins
Recent questions tagged ros2_medkit_graph_watchdog at Robotics Stack Exchange
Package Summary
| Version | 0.6.0 |
| License | Apache-2.0 |
| Build type | AMENT_CMAKE |
| Use | RECOMMENDED |
Repository Summary
| Checkout URI | https://github.com/selfpatch/ros2_medkit.git |
| VCS Type | git |
| VCS Version | main |
| Last Updated | 2026-07-31 |
| Dev Status | DEVELOPED |
| Released | RELEASED |
| Contributing |
Help Wanted (-)
Good First Issues (-) Pull Requests to Review (-) |
Package Description
Maintainers
- bburda
Authors
ros2_medkit_graph_watchdog
Gateway plugin that detects silent faults in the ROS 2 graph: failures where every node is up, nothing logs an error, and the robot is still broken.
Detectors read the graph and raise faults through a ReportFault service client on the
gateway node; the faults surface via FaultManager on the gateway /faults API.
This package carries the plugin skeleton, the central reliability gate that holds raises
until the graph has quiesced, and three detectors, qos_mismatch, orphan and
param_drift. The remaining
silent-fault classes land in follow-up changes, each against its own issue; their fault
codes are already reserved in the frozen GRAPH_* namespace (see “Fault codes”).
Build
With the gateway built and sourced:
colcon build --packages-select ros2_medkit_graph_watchdog
colcon test --packages-select ros2_medkit_graph_watchdog && colcon test-result --verbose
Load into the gateway
ros2 run ros2_medkit_gateway gateway_node --ros-args \
-p plugins:="[graph_watchdog]" \
-p "plugins.graph_watchdog.path:=$(ros2 pkg prefix ros2_medkit_graph_watchdog)/lib/ros2_medkit_graph_watchdog/libros2_medkit_graph_watchdog.so"
Configuration (plugins.graph_watchdog.<key>)
| Key | Type | Default | Meaning |
|---|---|---|---|
tick_interval_ms |
int | 1000 |
Detector tick cadence. |
warmup_cycles |
int | 5 |
An entity must be continuously present for this many ticks before it arms. A mid-run restart re-warms it. Enforced centrally, see “Reliability (bringup-quiesce)” below. |
prune_grace |
int | 60 |
Default injected into every detector’s own config before configure(); a per-detector detectors.<id>.prune_grace overrides it. Used by detectors that keep per-key bookkeeping. |
detectors.<id>.mode |
string | bool | raise |
raise = push faults; advisory = observe, do not push; off = disabled. Bare off/no (YAML booleans) also disable; bare on/yes mean raise. |
detectors.<id>.<field> |
any | - | Per-detector thresholds, passed to that detector’s configure(). |
Nested delivery. Keys are written dotted in YAML or --ros-args, and the gateway
delivers them to the plugin as a nested object. A bare off is a YAML 1.1 boolean, so
mode: off arrives as false, not the string "off"; both forms disable the detector.
Unknown keys are reported. Anything under detectors.<id> that the detector does not
read produces one startup warning naming the key and listing the ones that exist, so a
typo like allow_list: for allowlist: cannot silently do nothing. A typo in the detector
id itself is warned about the same way, with the registered ids listed.
orphan keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | raise |
As above. |
grace |
int | 10 |
Consecutive sweeps a candidate pair must hold before it is reported. A staged bringup or a restart makes one side missing for a short time, which looks exactly like a typo until the other side appears. |
max_edit_distance |
int | 1 |
How many character edits the LEAF (the part after the last /) may differ by and still count as the same intended name. Values outside 1..8 keep the default: past a few edits a “near miss” starts matching unrelated topics. |
namespace_edit_distance |
int | 0 |
The same budget for the namespace, spent separately. 0 means the namespace must match to the character, which is what makes /robot1/scan and /robot2/scan two robots rather than a typo. Raise it to catch a misspelled namespace, and read the warning under the table first. Values outside 0..8 keep the default. |
allowlist |
string[] | [] |
Topic names never reported. Exact match only, never a prefix, so allowlisting /r1/scan does not also silence /r2/scan. |
What this detector will not catch, on purpose. A pair whose names match once every run of
digits is collapsed is treated as an enumeration of sensors, not a typo: /lidar_1 next to
/lidar_2, each one-sided, is an ordinary state on a multi-sensor robot and reporting it would
be a false alarm on every such machine. Collapsing runs rather than comparing character
positions is what also covers /lidar_9 next to /lidar_10. The cost is that a typo which
is a digit stops being reported. An appended digit is not that case: /scan1 collapses to
/scan#, which is not /scan, so it is still reported.
A misspelled namespace is opt-in, and it can be expensive. /robott/scan against
/robot/scan is one edit, but in the namespace, so by default it is not reported. Setting
namespace_edit_distance: 1 reports it. Check the naming scheme before doing that: on a fleet
named with letters, /amr_a and /amr_b are also one edit apart, and every robot in the
fleet will be reported as a typo. A numbered fleet is safe, since /robot1 against /robot2
differs only in its numeric field and the rule above already spares it.
What it cannot catch at all. A namespace added or dropped by mistake, /scan against
/robot/scan, is six edits apart, so no budget that is still specific will reach it. Do not
rely on this detector for that class.
param_drift keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | bool | raise |
As above. A bare off arrives as a YAML boolean; both forms disable the detector. |
baseline |
bool | true |
Self-capture the value of every parameter not listed in expect, then flag later changes to it. false checks only what expect pins, and skips reading the full parameter list. |
expect.<param> |
any | - | Absolute expected value for <param>, checked on every node that has it. This is the only rule that catches a value that was wrong from the first tick, which self-capture cannot see. A node that does not declare <param> answers NOT_FOUND, which is not a drift - so a misspelt name checks nothing at all. Once the sweep has covered every armed app - each one read, or given up on after repeated failed reads - and at least one of them answered, a pin that no app declares is warned about once, naming it. |
ignore |
string[] | [] |
Parameter-name globs never flagged. Applies to the self-captured set only, so it cannot silence an expect pin. A pattern of nothing but * matches every name and suppresses self-capture wholesale, which is indistinguishable at runtime from finding nothing; that combination warns at startup. |
max_reads_per_tick |
int | 8 |
Parameter-service ROUND TRIPS this detector may spend per tick, round-robin over nodes. Round trips, not calls: one call is several requests at the watched node, and the node pays for each of them. A baseline visit is charged 2 (a list, then one batched get) whatever the node’s parameter count; each expect pin is charged 3 (a list, a get and a descriptor read), or 2 on a node that does not declare it - see the table under “Coverage latency” for what each of those really costs the node. So a self-capture visit spends 2 of the budget and pinning N parameters spends up to 3N. It is an AMORTISED rate, not a per-tick cap: there is no pacing INSIDE a visit, so a node’s whole read set is issued back to back and the reader waits out the debt afterwards. Four pins against a budget of 8 fire 12 round trips inside one tick period and then idle for the rest of the next one; what the knob bounds is the sustained rate, not the burst. Accepted range 1..100000; anything else keeps the default and warns. |
prune_grace |
int | 60 |
Consecutive sweeps an app may be missing before its finding is dropped: the finding survives prune_grace absences and goes on the next one. Injected for every detector at plugin scope and overridable per detector; a value outside 0..3600 is rejected with a warning and this detector falls back to 2, not to the plugin default. The captured BASELINE has a shorter horizon of its own, and that one is NOT configurable - it is the compile-time kBaselineForgetGrace: the baseline is dropped on the third consecutive missed sweep, the two the reliability gate absorbs plus one, so an ordinary dropped discovery poll cannot re-capture a node on its already-drifted value. Setting prune_grace below 2 pulls that forward onto the sweep the finding itself is dropped on, so the two then go together. |
qos_mismatch keys
| Key | Type | Default | Meaning |
|---|---|---|---|
mode |
string | raise |
As above. |
grace |
int | 3 |
Consecutive ticks a topic must stay affected before it is reported. Endpoint discovery is not atomic, so a subscriber can be visible before a publisher’s QoS is, which reads as starvation for a tick or two. |
allowlist |
string[] | [] |
Subscriber FQNs never reported. Exact match only. |
Closing the loop: healing config is required
Every detector here reports level-triggered: it re-raises FAILED while the condition holds and emits PASSED on every clean sweep. That is what lets the fault_manager’s debounce counter walk from CONFIRMED back to HEALED. The fault_manager only counts PASSED events
File truncated at 100 lines see the full file