ros2_medkit_graph_watchdog

Gateway plugin that detects silent faults in the ROS 2 graph: failures where every node is up, nothing logs an error, and the robot is still broken.

Detectors read the graph and raise faults through a ReportFault service client on the gateway node; the faults surface via FaultManager on the gateway /faults API.

This package carries the plugin skeleton, the central reliability gate that holds raises until the graph has quiesced, and two detectors, qos_mismatch and orphan. The remaining silent-fault classes land in follow-up changes, each against its own issue; their fault codes are already reserved in the frozen GRAPH_* namespace (see “Fault codes”).

Build

With the gateway built and sourced:

colcon build --packages-select ros2_medkit_graph_watchdog
colcon test --packages-select ros2_medkit_graph_watchdog && colcon test-result --verbose

Load into the gateway

ros2 run ros2_medkit_gateway gateway_node --ros-args \
  -p plugins:="[graph_watchdog]" \
  -p "plugins.graph_watchdog.path:=$(ros2 pkg prefix ros2_medkit_graph_watchdog)/lib/ros2_medkit_graph_watchdog/libros2_medkit_graph_watchdog.so"

Configuration (plugins.graph_watchdog.<key>)

Nested delivery. Keys are written dotted in YAML or --ros-args, and the gateway delivers them to the plugin as a nested object. A bare off is a YAML 1.1 boolean, so mode: off arrives as false, not the string "off"; both forms disable the detector.

Unknown keys are reported. Anything under detectors.<id> that the detector does not read produces one startup warning naming the key and listing the ones that exist, so a typo like allow_list: for allowlist: cannot silently do nothing. A typo in the detector id itself is warned about the same way, with the registered ids listed.

orphan keys

What this detector will not catch, on purpose. A pair whose names match once every run of digits is collapsed is treated as an enumeration of sensors, not a typo: /lidar_1 next to /lidar_2, each one-sided, is an ordinary state on a multi-sensor robot and reporting it would be a false alarm on every such machine. Collapsing runs rather than comparing character positions is what also covers /lidar_9 next to /lidar_10. The cost is that a typo which is a digit stops being reported. An appended digit is not that case: /scan1 collapses to /scan#, which is not /scan, so it is still reported.

A misspelled namespace is opt-in, and it can be expensive. /robott/scan against /robot/scan is one edit, but in the namespace, so by default it is not reported. Setting namespace_edit_distance: 1 reports it. Check the naming scheme before doing that: on a fleet named with letters, /amr_a and /amr_b are also one edit apart, and every robot in the fleet will be reported as a typo. A numbered fleet is safe, since /robot1 against /robot2 differs only in its numeric field and the rule above already spares it.

What it cannot catch at all. A namespace added or dropped by mistake, /scan against /robot/scan, is six edits apart, so no budget that is still specific will reach it. Do not rely on this detector for that class.

qos_mismatch keys

Where GRAPH_* faults hang

Every GRAPH_* fault is raised with source_id: graph_watchdog. That is an App the plugin publishes itself, from its own IntrospectionProvider, marked external: true. So:

• GET /api/v1/apps/graph_watchdog/faults lists them.
• GET /api/v1/components/<host>/faults lists them too, because the App is attached to the host Component and a Component resolves its fault scope through its child apps.

It has to be an entity the plugin owns. Scoping these faults to the host Component directly - the obvious choice - makes them reachable from no endpoint at all: collect_component_app_fqns only puts a Component’s own id in the scope set when external is true, and a runtime host Component built by HostInfoProvider never sets it. There is no server-level /faults/{code} route either, so the flat /faults list was the only place these faults existed.

File truncated at 100 lines see the full file