grepros

grep for ROS bag files and live topics: read, filter, export.

Searches through ROS messages and matches any message field value by regular expression patterns or plain text, regardless of field type. Can also look for specific values in specific message fields only.

By default, matches are printed to console. Additionally, matches can be written to a bagfile or HTML/CSV/MCAP/Parquet/Postgres/SQL/SQLite, or published to live topics.

Supports both ROS1 and ROS2. ROS environment variables need to be set, at least ROS_VERSION.

Supported bag formats: .bag (ROS1), .db3 (ROS2), .mcap (ROS1, ROS2).

In ROS1, messages can be grepped even if Python packages for message types are not installed. Using ROS1 live topics requires ROS master to be running.

Using ROS2 requires Python packages for message types to be available in path.

Supports loading custom plugins, mainly for additional output formats.

Usable as a Python library, see LIBRARY.md. Full API documentation available at https://suurjaak.github.io/grepros.

• Example usage
• Installation
• Inputs
• Outputs
• Matching and filtering
• Plugins
• Notes on ROS1 vs ROS2
• All command-line arguments
• Dependencies
• Attribution
• License

Example usage

Search for “my text” in all bags under current directory and subdirectories:

grepros -r "my text"

Print 30 lines of the first message from each live ROS topic:

grepros --max-per-topic 1 --lines-per-message 30 --live

Find first message containing “future” (case-sensitive) in my.bag:

grepros future -I --max-count 1 --name my.bag

Find 10 messages, from geometry_msgs package, in “map” frame, from bags in current directory, reindexing any unindexed bags:

grepros frame_id=map --type geometry_msgs/* --max-count 10 --reindex-if-unindexed

Pipe all diagnostics messages with “CPU usage” from live ROS topics to my.bag:

grepros "CPU usage" --type *DiagnosticArray --no-console-output --write my.bag

Find messages with field “key” containing “0xA002”, in topics ending with “diagnostics”, in bags under “/tmp”:

grepros key=0xA002 --topic *diagnostics --path /tmp

Find diagnostics_msgs messages in bags in current directory, containing “navigation” in fields “name” or “message”, print only header stamp and values:

grepros --type diagnostic_msgs/* --select-field name message \
        --emit-field header.stamp status.values -- navigation

Print first message from each lidar topic on ROS1 host 1.2.3.4, without highlight:

ROS_MASTER_URI=http://1.2.3.4::11311 \
grepros --live --topic *lidar* --max-per-topic 1 --no-highlight

Export all bag messages to SQLite and Postgres, print only export progress:

grepros -n my.bag --write my.bag.sqlite --no-console-output --no-verbose --progress

grepros -n my.bag --write postgresql://user@host/dbname \
        --no-console-output --no-verbose --progress

Patterns use Python regular expression syntax, message matches if all match. ‘*’ wildcards use simple globbing as zero or more characters, target matches if any value matches.

Note that some expressions may need to be quoted to avoid shell auto-unescaping or auto-expanding them, e.g. linear.x=2.?5 should be given as "linear.x=2\.?5".

Care must also be taken with unquoted wildcards, as they will auto-expanded by shell if they happen to match paths on disk.

File truncated at 100 lines see the full file

Changelog for package grepros

1.2.2 (2024-05-06)

• fix recursing into subdirectories when looking for bagfiles
• fix ROS2Bag creating blank file when opening a non-existent file for reading
• handle deserialization error when grepping live topics in ROS1
• tweak word wrap in HTML output

1.2.1 (2024-04-29)

• fix match highlighting regression in HTML output
• fix stripping trailing zeroes from timestamps and highlighting message definition comments in HTML output

1.2.0 (2024-04-22)

• add --expression option to give pattern as a logical expression like [this AND (this2 OR NOT "skip this")]{.title-ref}
• add --time-scale option to emit messages in bag timeline
• count --every-nth-message from --start-index
• support explicit integer indexes in filtering nested message fields
• add separate match counts to progress bar where relevant
• print error stacktraces when --verbose
• make api.dict_to_message() support nested ROS messages in dictionary
• make api.get_message_value() accept a default
• support fractional values in --every-nth-interval option
• support [--lines-per-message 0]{.title-ref}
• stop bag reading when reaching --end-index in each topic
• stop bag and live reading when reaching --end-time in each topic
• add result message count in HTML output
• add total topic count to output metadata when grepping live topics
• rename TopicSource and TopicSink to LiveSource and LiveSink
• fix parsing and hashing message type definitions with leading whitespace
• fix negative --start-index and --end-index ignoring different counts in bag topics
• fix --every-nth-interval not working when messages continuously close
• fix emitting invalid YAML for nested message arrays
• fix using --filename with --recursive
• fix preparing patterns for early message skip
• fix program not exiting properly on exception in background thread, like ROS master shutting down when grepping live topics
• fix intermittent concurrency issues in HTML output
• fix api.make_bag_time() and api.make_live_time() for Duration parameter
• fix handling already deserialized clock messages when grepping ROS1 live topics in sim time
• fix invalid error message when trying to write to unwritable path
• improve auto-detecting MCAP and ROS2 .db3 bags
• validate all arguments, check signedness
• print detailed output statistics only in verbose mode
• default to non-verbose mode when using progress bar
• add Source.configure() and Sink.configure()
• avoid UserWarning from PyArrow in Parquet output
• avoid pointless stacktrace upon multiple Ctrl-C when grepping live topics

1.1.0 (2023-12-30)

• add support for splitting files in bag/HTML/MCAP/SQLite output
• add api.to_duration()
• do not use partial match for topic and typename filters without wildcards
• support patterns yielding zero-length matches like "(?!not_this)"
• support search over message fields in conditions as [--condition "value in msg"]{.title-ref}
• support giving dashed names like "rollover-size" in format-specific write options as underscored "rollover_size" instead
• disallow unknown command-line flags and partial abbreviations
• fix matching nested message values for patterns using start or end flags
• fix ISO datetime support in earliest and latest timestamp arguments
• fix --end-line not being used
• fix api.dict_to_message() erroring on temporal types in dict
• fix api.get_ros_time_category() erroring on time/duration instances in ROS2
• fix including topics in match only used for conditions not matching
• fix using len() in conditions on topics not reached yet
• ensure api.get_message_value() returning collections as lists not tuples

1.0.4 (2023-09-28)

• fix Parquet sink validation resulting in silent failure if additional columns specified

1.0.3 (2023-08-31)

• fix Unicode characters in HTML output template

1.0.2 (2023-08-30)

• fix raising error in ROS1 live topics if message class not locally available
• fix generating ROS1 message classes dynamically in multi-threaded environment

File truncated at 100 lines see the full file