ros2_medkit_fault_manager

Central fault manager node for the ros2_medkit fault management system.

Overview

The FaultManager node provides a central point for fault aggregation and lifecycle management. It receives fault reports from multiple sources, aggregates them by fault_code, and provides query and clearing interfaces.

Quick Start

By default, faults are confirmed immediately when reported - no additional configuration needed.

# Start the fault manager
ros2 launch ros2_medkit_fault_manager fault_manager.launch.py

# Report a fault - it's immediately CONFIRMED
ros2 service call /fault_manager/report_fault ros2_medkit_msgs/srv/ReportFault \
  "{fault_code: 'MOTOR_OVERHEAT', event_type: 0, severity: 2, description: 'Motor temp exceeded', source_id: '/motor_node'}"

# Query faults
ros2 service call /fault_manager/list_faults ros2_medkit_msgs/srv/ListFaults \
  "{statuses: ['CONFIRMED']}"

# Clear a fault (cascade-clears correlated symptoms by default)
ros2 service call /fault_manager/clear_fault ros2_medkit_msgs/srv/ClearFault \
  "{fault_code: 'MOTOR_OVERHEAT', skip_correlation_auto_clear: false}"

# Clear without touching correlated symptoms
ros2 service call /fault_manager/clear_fault ros2_medkit_msgs/srv/ClearFault \
  "{fault_code: 'MOTOR_OVERHEAT', skip_correlation_auto_clear: true}"

Note: The skip_correlation_auto_clear request field was added post-0.4.0. Adding a request field changes the service type hash, so callers built against ros2_medkit_msgs 0.4.0 or earlier must rebuild to keep talking to fault_manager.

Services

Features

• Multi-source aggregation: Same fault_code from different sources creates a single fault
• Occurrence tracking: Counts total reports and tracks all reporting sources
• Severity escalation: Fault severity is updated if a higher severity is reported
• Persistent storage: SQLite backend ensures faults survive node restarts
• Debounce filtering (optional): AUTOSAR DEM-style counter-based fault confirmation with per-entity threshold overrides
• Snapshot capture: Captures topic data when faults are confirmed for debugging (snapshots are deleted when fault is cleared)
• Near-miss series: Appends one entry per FAILED report that moved the debounce counter without confirming, bounded per fault code and retained when the fault is cleared
• Freeze-frame retention: One compact JSON freeze-frame per fault code, retained across clear_fault (see below)
• Fault correlation (optional): Root cause analysis with symptom muting and auto-clear
• Tamper-evident audit log (optional): Append-only, hash-chained record of fault state transitions for verifiable history

Parameters

Snapshot Parameters

Snapshots capture topic data when faults are confirmed for post-mortem debugging.

Each confirm also writes a freeze-frame: a single compact JSON object mapping every captured topic to its value at confirmation time, keyed by fault code. It differs from per-topic snapshots in two ways: snapshots are deleted when the fault is cleared, while the freeze-frame is retained across clear_fault (once the snapshots are gone, ~/get_fault serves the retained frame so the confirmed-state record stays available after acknowledgement); and a re-confirm that captures nothing (e.g. source publishers down) never overwrites an existing non-empty frame. A fault code with no configured capture set gets no freeze-frame row; a configured capture that samples nothing on its first run records an empty {} frame. Freeze-frame storage is bounded by the number of distinct fault codes (one row per code, replaced in place) and rows are never evicted.

Under a fault storm, captures are bounded by a worker pool (capture_pool_size) draining a bounded queue (capture_queue_depth); excess captures are dropped per capture_queue_full_policy and logged (throttled). The pool is shared and is created when snapshots or rosbag is enabled, so these parameters bound both. capture_pool_size parallelizes freeze-frame snapshot capture only - rosbag stays single-writer regardless of pool size, and correlated faults confirming inside one post-roll window share a single recording.

That single-writer property also shapes what each fault of a burst gets. Nothing is buffered while a post-fault window is open (messages go straight into the open bag), and the flush that opened that bag already emptied the ring buffer, so a fault confirming right after the window closes has no pre-fault history available. What a confirmation gets is decided by the buffer, so a fault arriving before any captured topic has published lands the same way. It gets a post-fault-only bag: its own recording holding just its duration_after_sec window, entered through the same post-roll state machine, so later faults of the burst attach to it normally. With duration_after_sec: 0 there is no window to record into and such a fault gets no bag; if the bag cannot be written at all, no recording is opened and no metadata row is stored. The duration_sec on a stored bag is the span the recording was open rather than the configured windows, so a post-fault-only bag usually reports roughly duration_after_sec where a full one reports its buffered history too. It is a recording span, not a content span: a window during which nothing was published still reports the seconds it covered. It can also exceed duration_sec + duration_after_sec, because the ring buffer is pruned only when a message arrives - a topic that stops publishing keeps its last window buffered until the next confirmation flushes it, which is deliberate for a black box. See docs/config/fault-manager.rst for the full lifecycle.

Topic Resolution Priority:

1. fault_specific - Exact match for fault code (configured via YAML config file)
2. patterns - Regex pattern match (configured via YAML config file)
3. default_topics - Fallback for all faults

File truncated at 100 lines see the full file

Changelog for package ros2_medkit_fault_manager

0.7.0 (2026-08-27)

• Rosbag black-box recordings are no longer limited to one per fault code. A fault that re-confirms keeps a bounded history of recordings instead of overwriting the previous one, controlled by the new snapshots.rosbag.max_bags_per_fault (default 1, which reproduces the previous behaviour exactly; 0 = unlimited). Retention is keep-newest and the bag is unlinked only when no fault still references it, so a burst that shares one recording behaves as before. Internally the rosbag_files grain changed from "one row per fault" to "one row per (fault, recording) link": recording_id is now a stored, indexed column, and the legacy column-level UNIQUE(fault_code) is replaced by a UNIQUE INDEX on (fault_code, file_path) through an automatic, idempotent table rebuild on first open. Four latent defects are fixed on the way: quota eviction deleted by fault code rather than by recording, get_rosbag_file had no ORDER BY and would have served an arbitrary recording, the stale-row self-heals deleted a fault's entire history because one bag had vanished from disk, and both delete_rosbag_file / delete_rosbag_files read only the first file_path of a fault, so deleting a fault with several recordings removed every row but left all but one bag on disk - unreachable and still charged against the quota (#623, #620)
• Optional append-only, hash-chained audit log of fault state transitions: each transition appends one immutable row (record_hash = sha256(prev_hash + canonical(event)) via OpenSSL EVP SHA-256) with a persisted chain head, a verify routine, a read API, and retention that seals a segment anchor before pruning. Time-based (PREFAILED->CONFIRMED) auto-confirmations are also audited. verify reads the chain head directly from the database, so deleting the newest row together with the head row is reported as tampering instead of silently recovering. BEFORE UPDATE / BEFORE DELETE triggers reject out-of-band edits as defense-in-depth. The chain is unkeyed and stored in a single writable file, so verify detects edits/deletions that did not recompute the chain (casual or accidental tampering); it is not a defence against an attacker who can rewrite the whole file. Off by default (#487, #483)
• Breaking: the default rosbag storage format is mcap again. snapshots.rosbag.format now defaults to "mcap", so black-box recordings land as .mcap files instead of .db3, and the filename a bulk-data download serves changes with them. The storage plugins are declared explicitly and the plugin loader is serialised, which is what makes the format selectable reliably rather than dependent on load order. Set snapshots.rosbag.format: sqlite3 to keep the previous on-disk format (#610)
• Freeze-frame: a compact JSON snapshot of the entity's data is persisted when a fault is confirmed, so the state at the moment of confirmation survives the fault being cleared (#491)
• A fault that confirms inside an active post-roll window keeps its black-box recording instead of finding the buffer already finalised (#561), and a fault landing on a recording-window boundary gets its own bag rather than none (#594)
• The debounce counter is clamped and the confirmed / healed status is latched, so a counter cannot run past its threshold and a status cannot silently regress (#484)
• A PASSED event no longer re-dates a fault - first_occurred keeps marking the start of the current occurrence - and genuine SSE loss is counted rather than absorbed (#573)
• The near-miss series survives a fault being cleared, so acknowledging a fault no longer discards the evidence gathered around it (#629)
• The fault manager's YAML parameter file supports launch substitutions, so a path can be composed at launch time instead of being fixed in the file (#634)
• Build and test only: the package is instrumented for coverage (#582), every launch test lives under test/integration and takes a DDS domain at run time (#628, #551, #597)
• A fault_code up to the advertised 256 characters is accepted, where the manager previously stopped at 128, and a long code no longer costs the recording: the rosbag filename is truncated with a digest instead of exceeding the filesystem's component limit (#591)
• Contributors: \@bburda, \@mfaferek93, \@nnarain

0.6.0 (2026-06-22)

• Bounded concurrent snapshot capture under fault storms with a CaptureThreadPool and configurable capture pool / queue / overflow-policy parameters. The rosbag leg is serialized and the cooldown map is bounded, so a burst of simultaneous faults can no longer exhaust capture threads or grow memory without limit (#456)
• Entity-scoped rosbag capture by default

File truncated at 100 lines see the full file