Repository Summary
| Checkout URI | https://github.com/suurjaak/grepros.git |
| VCS Type | git |
| VCS Version | master |
| Last Updated | 2023-03-30 |
| Dev Status | DEVELOPED |
| CI status | No Continuous Integration |
| Released | RELEASED |
| Tags | No category tags. |
| Contributing |
Help Wanted (0)
Good First Issues (0) Pull Requests to Review (0) |
Packages
| Name | Version |
|---|---|
| grepros | 0.6.0 |
README
grepros
grep for ROS bag files and live topics.
Searches through ROS messages and matches any message field value by regular expression patterns or plain text, regardless of field type. Can also look for specific values in specific message fields only.
By default, matches are printed to console. Additionally, matches can be written to a bagfile or HTML/CSV/MCAP/Parquet/Postgres/SQL/SQLite, or published to live topics.
Supports both ROS1 and ROS2. ROS environment variables need to be set, at least ROS_VERSION.
In ROS1, messages can be grepped even if Python packages for message types are not installed. Using ROS1 live topics requires ROS master to be running.
Using ROS2 requires Python packages for message types to be available in path.
Supports loading custom plugins, mainly for additional output formats.
API documentation available at https://suurjaak.github.io/grepros.
- Example usage
- Installation
- Inputs
- Outputs
- Matching and filtering
- Plugins
- SQL dialects
- Notes on ROS1 vs ROS2
- All command-line arguments
- Dependencies
- Attribution
- License
Example usage
Search for "my text" in all bags under current directory and subdirectories:
grepros -r "my text"
Print 30 lines of the first message from each live ROS topic:
grepros --max-per-topic 1 --lines-per-message 30 --live
Find first message containing "future" (case-sensitive) in my.bag:
grepros future -I --max-count 1 --name my.bag
Find 10 messages, from geometry_msgs package, in "map" frame, from bags in current directory, reindexing any unindexed bags:
grepros frame_id=map --type geometry_msgs/* --max-count 10 --reindex-if-unindexed
Pipe all diagnostics messages with "CPU usage" from live ROS topics to my.bag:
grepros "CPU usage" --type *DiagnosticArray --no-console-output --write my.bag
Find messages with field "key" containing "0xA002", in topics ending with "diagnostics", in bags under "/tmp":
grepros key=0xA002 --topic *diagnostics --path /tmp
Find diagnostics_msgs messages in bags in current directory, containing "navigation" in fields "name" or "message", print only header stamp and values:
grepros --type diagnostic_msgs/* --select-field name message \
--print-field header.stamp status.values -- navigation
Print first message from each lidar topic on host 1.2.3.4:
ROS_MASTER_URI=http://1.2.3.4::11311 \
grepros --live --topic *lidar* --max-per-topic 1
Export all bag messages to SQLite and Postgres, print only export progress:
grepros -n my.bag --write my.bag.sqlite --no-console-output --no-verbose --progress
grepros -n my.bag --write postgresql://user@host/dbname \
--no-console-output --no-verbose --progress
Patterns use Python regular expression syntax, message matches if all match. '*' wildcards use simple globbing as zero or more characters, target matches if any value matches.
Note that some expressions may need to be quoted to avoid shell auto-unescaping
or auto-expanding them, e.g. linear.x=2.?5 should be given as "linear.x=2\.?5".
Care must also be taken with unquoted wildcards, as they will auto-expanded by shell if they happen to match paths on disk.
Installation
grepros is written in Python, supporting both Python 2 and Python 3.
Developed and tested under ROS1 noetic and ROS2 foxy, it should also work in later ROS2 versions; it may work in earlier ROS1 versions.
Using pip
pip install grepros
This will add the grepros command to path.
Requires ROS Python packages (ROS1: rospy, roslib, rosbag, genpy; ROS2: rclpy, rosidl_parser, rosidl_runtime_py, builtin_interfaces).
For ROS1, if only using bag files and no live topics, minimal ROS1 Python packages can also be installed separately with:
pip install rospy rosbag roslib roslz4 \
--extra-index-url https://rospypi.github.io/simple/
Using apt
If ROS apt repository has been added to system:
sudo apt install ros-noetic-grepros # ROS1
sudo apt install ros-foxy-grepros # ROS2
This will add the grepros command to the global ROS1 / ROS2 environment.
Using catkin
In a ROS1 workspace, under the source directory:
git clone https://github.com/suurjaak/grepros.git
cd grepros
catkin build --this
This will add the grepros command to the local ROS1 workspace path.
Using colcon
In a ROS2 workspace, at the workspace root:
git clone https://github.com/suurjaak/grepros.git src/grepros
colcon build --packages-select grepros
This will add the grepros command to the local ROS2 workspace path.
Inputs
Input is either from one or more ROS bag files (default), or from live ROS topics.
bag
Read messages from ROS bag files, by default all in current directory.
Recurse into subdirectories when looking for bagfiles:
-r
--recursive
Read specific filenames (supports * wildcards):
--n /tmp/*.bag
--filename my.bag 2021-11-*.bag
Scan specific paths instead of current directory (supports * wildcards):
-p /home/bags/2021-11-*
--path my/dir
Reindex unindexed ROS1 bags before processing (note: creates backup copies of files, into same directory as file):
--reindex-if-unindexed
--reindex-if-unindexed --progress
Decompress archived ROS bags before processing
(.zst .zstd extensions, requires zstandard Python package)
(note: unpacks archived file to disk, into same directory as file):
--decompress
--decompress --progress
Order bag messages first by topic or type, and only then by time:
--order-bag-by topic
--order-bag-by type
Outputs
There can be any number of outputs: printing to console (default), publishing to live ROS topics, or writing to file or database.
console
Default output is to console, in ANSI colors, mimicking grep output.
Disable printing messages to console:
--no-console-output
Manage color output:
--color always (default)
--color auto (auto-detect terminal support)
--color never (disable colors)
Note that when paging color output with more or less, the pager needs to
accept raw control characters (more -f or less -R).
bag
--write path/to/my.bag [format=bag] [overwrite=true|false]
Write messages to a ROS bag file, the custom .bag format in ROS1
or the .db3 SQLite database format in ROS2. If the bagfile already exists,
it is appended to, unless specified to overwrite.
Specifying format=bag is not required
if the filename ends with .bag in ROS1 or .db3 in ROS2.
live
--live
Read messages from live ROS topics instead of bagfiles.
Requires ROS_MASTER_URI and ROS_ROOT to be set in environment if ROS1.
Set custom queue size for subscribing (default 10):
--queue-size-in 100
Use ROS time instead of system time for incoming message timestamps:
--ros-time-in
csv
--write path/to/my.csv [format=csv] [overwrite=true|false]
Write messages to CSV files, each topic to a separate file, named
path/to/my.full__topic__name.csv for /full/topic/name.
Output mimicks CSVs compatible with PlotJuggler, all messages values flattened
to a single list, with header fields like /topic/field.subfield.listsubfield.0.data.1.
If a file already exists, a unique counter is appended to the name of the new file,
e.g. my.full__topic__name.2.csv, unless specified to overwrite.
Specifying format=csv is not required if the filename ends with .csv.
html
--write path/to/my.html [format=html] [overwrite=true|false]
[template=/path/to/html.template]
Write messages to an HTML file, with a linked table of contents, message timeline, message type definitions, and a topically traversable message list.
Note: resulting file may be large, and take a long time to open in browser.
If the file already exists, a unique counter is appended to the name of the new file,
e.g. my.2.html, unless specified to overwrite.
Specifying format=html is not required if the filename ends with .htm or .html.
A custom template file can be specified, in step syntax:
--write path/to/my.html template=/my/html.template
postgres
--write postgresql://username@host/dbname [format=postgres]
[commit-interval=NUM] [nesting=array|all]
[dialect-file=path/to/dialects.yaml]
Write messages to a Postgres database, with tables pkg/MsgType for each ROS message type,
and views /full/topic/name for each topic.
Plus table topics with a list of topics, types with message types and definitions,
and meta with table/view/column name changes from shortenings and conflicts, if any
(Postgres name length is limited to 63 characters).
ROS primitive types are inserted as Postgres data types (time/duration types as NUMERIC), uint8[] arrays as BYTEA, other primitive arrays as ARRAY, and arrays of subtypes as JSONB.
If the database already exists, it is appended to. If there are conflicting names (same package and name but different message type definition), table/view name becomes "name (MD5 hash of type definition)".
Specifying format=postgres is not required if the parameter uses the
Postgres URI scheme postgresql://.
Requires psycopg2.
Parameter --write can also use the Postgres keyword=value format,
e.g. "host=localhost port=5432 dbname=mydb username=postgres connect_timeout=10".
Standard Postgres environment variables are also supported (PGPASSWORD et al).
A custom transaction size can be specified (default is 1000; 0 is autocommit):
--write postgresql://username@host/dbname commit-interval=NUM
Updates to Postgres SQL dialect can be loaded from a YAML or JSON file:
--write postgresql://username@host/dbname dialect-file=path/to/dialects.yaml
More on SQL dialects.
Nested messages
Nested message types can be recursively populated to separate tables, linked to parent messages via foreign keys.
To recursively populate nested array fields:
--write postgresql://username@host/dbname nesting=array
E.g. for diagnostic_msgs/DiagnosticArray, this would populate the following tables:
CREATE TABLE "diagnostic_msgs/DiagnosticArray" (
"header.seq" BIGINT,
"header.stamp.secs" INTEGER,
"header.stamp.nsecs" INTEGER,
"header.frame_id" TEXT,
status JSONB, -- [_id from "diagnostic_msgs/DiagnosticStatus", ]
_topic TEXT,
_timestamp NUMERIC,
_id BIGSERIAL,
_parent_type TEXT,
_parent_id BIGINT
);
CREATE TABLE "diagnostic_msgs/DiagnosticStatus" (
level SMALLINT,
name TEXT,
message TEXT,
hardware_id TEXT,
"values" JSONB, -- [_id from "diagnostic_msgs/KeyValue", ]
_topic TEXT, -- _topic from "diagnostic_msgs/DiagnosticArray"
_timestamp NUMERIC, -- _timestamp from "diagnostic_msgs/DiagnosticArray"
_id BIGSERIAL,
_parent_type TEXT, -- "diagnostic_msgs/DiagnosticArray"
_parent_id BIGINT -- _id from "diagnostic_msgs/DiagnosticArray"
);
CREATE TABLE "diagnostic_msgs/KeyValue" (
"key" TEXT,
value TEXT,
_topic TEXT, -- _topic from "diagnostic_msgs/DiagnosticStatus"
_timestamp NUMERIC, -- _timestamp from "diagnostic_msgs/DiagnosticStatus"
_id BIGSERIAL,
_parent_type TEXT, -- "diagnostic_msgs/DiagnosticStatus"
_parent_id BIGINT -- _id from "diagnostic_msgs/DiagnosticStatus"
);
Without nesting, array field values are inserted as JSON with full subtype content.
To recursively populate all nested message types:
--write postgresql://username@host/dbname nesting=all
E.g. for diagnostic_msgs/DiagnosticArray, this would, in addition to the above, populate:
CREATE TABLE "std_msgs/Header" (
seq BIGINT,
"stamp.secs" INTEGER,
"stamp.nsecs" INTEGER,
frame_id TEXT,
_topic TEXT, -- _topic from "diagnostic_msgs/DiagnosticArray"
_timestamp NUMERIC, -- _timestamp from "diagnostic_msgs/DiagnosticArray"
_id BIGSERIAL,
_parent_type TEXT, -- "diagnostic_msgs/DiagnosticArray"
_parent_id BIGINT -- _id from "diagnostic_msgs/DiagnosticArray"
);
sqlite
--write path/to/my.sqlite [format=sqlite] [overwrite=true|false]
[commit-interval=NUM] [message-yaml=true|false] [nesting=array|all]
[dialect-file=path/to/dialects.yaml]
Write an SQLite database with tables pkg/MsgType for each ROS message type
and nested type, and views /full/topic/name for each topic.
If the database already exists, it is appended to, unless specified to overwrite.
Output is compatible with ROS2 .db3 bagfiles, supplemented with
full message YAMLs, and message type definition texts. Note that a database
dumped from a ROS1 source will most probably not be usable as a ROS2 bag,
due to breaking changes in ROS2 standard built-in types and message types.
Specifying format=sqlite is not required
if the filename ends with .sqlite or .sqlite3.
A custom transaction size can be specified (default is 1000; 0 is autocommit):
--write path/to/my.sqlite commit-interval=NUM
By default, table messages is populated with full message YAMLs, unless:
--write path/to/my.sqlite message-yaml=false
Updates to SQLite SQL dialect can be loaded from a YAML or JSON file:
--write path/to/my.sqlite dialect-file=path/to/dialects.yaml
More on SQL dialects.
Nested messages
Nested message types can be recursively populated to separate tables, linked to parent messages via foreign keys.
To recursively populate nested array fields:
--write path/to/my.sqlite nesting=array
E.g. for diagnostic_msgs/DiagnosticArray, this would populate the following tables:
CREATE TABLE "diagnostic_msgs/DiagnosticArray" (
"header.seq" INTEGER,
"header.stamp.secs" INTEGER,
"header.stamp.nsecs" INTEGER,
"header.frame_id" TEXT,
-- [_id from "diagnostic_msgs/DiagnosticStatus", ]
status "DIAGNOSTIC_MSGS/DIAGNOSTICSTATUS[]",
_topic TEXT,
_timestamp INTEGER,
_id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
_parent_type TEXT,
_parent_id INTEGER
);
CREATE TABLE "diagnostic_msgs/DiagnosticStatus" (
level SMALLINT,
name TEXT,
message TEXT,
hardware_id TEXT,
-- [_id from "diagnostic_msgs/KeyValue", ]
"values" "DIAGNOSTIC_MSGS/KEYVALUE[]",
_topic TEXT, -- _topic from "diagnostic_msgs/DiagnosticArray"
_timestamp INTEGER, -- _timestamp from "diagnostic_msgs/DiagnosticArray"
_id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
_parent_type TEXT, -- "diagnostic_msgs/DiagnosticArray"
_parent_id INTEGER -- _id from "diagnostic_msgs/DiagnosticArray"
);
CREATE TABLE "diagnostic_msgs/KeyValue" (
"key" TEXT,
value TEXT,
_topic TEXT, -- _topic from "diagnostic_msgs/DiagnosticStatus"
_timestamp INTEGER, -- _timestamp from "diagnostic_msgs/DiagnosticStatus"
_id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
_parent_type TEXT, -- "diagnostic_msgs/DiagnosticStatus"
_parent_id INTEGER -- _id from "diagnostic_msgs/DiagnosticStatus"
);
Without nesting, array field values are inserted as JSON with full subtype content.
To recursively populate all nested message types:
--write path/to/my.sqlite nesting=all
E.g. for diagnostic_msgs/DiagnosticArray, this would, in addition to the above, populate:
CREATE TABLE "std_msgs/Header" (
seq UINT32,
"stamp.secs" INT32,
"stamp.nsecs" INT32,
frame_id TEXT,
_topic STRING, -- _topic from "diagnostic_msgs/DiagnosticArray"
_timestamp INTEGER, -- _timestamp from "diagnostic_msgs/DiagnosticArray"
_id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
_parent_type TEXT, -- "diagnostic_msgs/DiagnosticArray"
_parent_id INTEGER -- _id from "diagnostic_msgs/DiagnosticArray"
);
live
--publish
Publish messages to live ROS topics. Topic prefix and suffix can be changed, or topic name set to one specific name:
--publish-prefix /myroot
--publish-suffix /myend
--publish-fixname /my/singular/name
One of the above arguments needs to be specified if publishing to live ROS topics while grepping from live ROS topics, to avoid endless loops.
Set custom queue size for publishers (default 10):
--queue-size-out 100
console / html message formatting
Set maximum number of lines to output per message:
--lines-per-message 5
Set maximum number of lines to output per message field:
--lines-per-field 2
Start message output from, or stop output at, message line number:
--start-line 2 # (1-based if positive
--end-line -2 # (count back from total if negative)
Output only the fields where patterns find a match:
--matched-fields-only
Output only matched fields and specified number of lines around match:
--lines-around-match 5
Output only specific message fields (supports nested.paths and * wildcards):
--print-field *data
Skip outputting specific message fields (supports nested.paths and * wildcards):
--no-print-field header.stamp
Wrap matches in custom texts:
--match-wrapper @@@
--match-wrapper "<<<<" ">>>>"
Set custom width for wrapping message YAML printed to console (auto-detected from terminal by default):
--wrap-width 120
Matching and filtering
Any number of patterns can be specified, message matches if all patterns find a match. If no patterns are given, any message matches.
Match messages containing any of the words:
cpu memory speed
Match messages where frame_id contains "world":
frame_id=world
Match messages where header.frame_id is present:
header.frame_id=.*
Match as plaintext, not Python regular expression patterns:
-F
--fixed-strings
Select non-matching messages instead:
-v
--invert-match
Use case-sensitive matching in patterns (default is insensitive):
-I
--no-ignore-case
Limits
Stop after matching a specified number of messages (per each file if bag input):
-m 100
--max-count 100
Scan only a specified number of topics (per each file if bag input):
--max-topics 10
Emit a specified number of matches per topic (per each file if bag input):
--max-per-topic 20
Emit every Nth match in topic:
--every-nth-match 10 # (skips 9 matches in topic after each match emitted)
Filtering
Scan specific topics only (supports * wildcards):
-t *lidar* *ins*
--topic /robot/sensors/*
Skip specific topics (supports * wildcards):
-nt *lidar* *ins*
--no-topic /robot/sensors/*
Scan specific message types only (supports * wildcards):
-d *Twist*
--type sensor_msgs/*
Skip specific message types from scanning (supports * wildcards):
-nd *Twist*
--no-type sensor_msgs/*
Set specific message fields to scan (supports nested.paths and * wildcards):
-sf twist.linear
--select-field *data
Skip specific message fields in scan (supports nested.paths and * wildcards):
-ns twist.linear
--no-select-field *data
Only emit matches that are unique in topic,
taking --select-field and --no-select-field into account (per each file if bag input):
--unique-only
Start scanning from a specific timestamp:
-t0 2021-11 # (using partial ISO datetime)
--start-time 1636900000 # (using UNIX timestamp)
--start-time +100 # (seconds from bag start time, or from script startup time if live input)
--start-time -100 # (seconds from bag end time, or script startup time if live input)
Stop scanning at a specific timestamp:
-t1 2021-11 # (using partial ISO datetime)
--end-time 1636900000 # (using UNIX timestamp)
--end-time +100 # (seconds from bag start time, or from script startup time if live input)
--end-time -100 # (seconds from bag end time, or from script startup time if live input)
Start scanning from a specific message index in topic:
-n0 -100 # (counts back from topic total message count in bag)
--start-index 10 # (1-based index)
Stop scanning at a specific message index in topic:
-n1 -100 # (counts back from topic total message count in bag)
--end-index 10 # (1-based index)
Scan every Nth message in topic:
--every-nth-message 10 # (skips 9 messages in topic with each step)
Scan messages in topic with timestamps at least N seconds apart:
--every-nth-interval 5 # (samples topic messages no more often than every 5 seconds)
Conditions
--condition "PYTHON EXPRESSION"
Specify one or more Python expressions that must evaluate as true to search encountered messages. Expressions can access topics, by name or * wildcard, and refer to message fields directly.
# (Match while last message in '/robot/enabled' has data=true)
--condition "<topic /robot/enabled>.data"
# (Match if at least 10 messages have been encountered in /robot/alerts)
--condition "len(<topic /robot/alerts>) > 10"
# (Match if last two messages in /robot/mode have equal .value)
--condition "<topic /robot/mode>[-2].value == <topic /robot/mode>[-1].value"
# (Match while control is enabled and robot is moving straight and level)
--condition "<topic */control_enable>.data and <topic */cmd_vel>.linear.x > 0 " \
"and <topic */cmd_vel>.angular.z < 0.02"
Condition namespace:
| Name | Description |
|---|---|
msg |
current message from data source |
topic |
full name of current message topic |
<topic /my/topic> |
topic by full name or * wildcard |
len(<topic ..>) |
number of messages encountered in topic |
bool(<topic ..>) |
whether any message encountered in topic |
<topic ..>.xyz |
attribute xyz of last message in topic |
<topic ..>[index] |
topic message at position |
| (from first encountered if index >= 0, last encountered if < 0) | |
<topic ..>[index].xyz |
attribute xyz of topic message at position |
Condition is automatically false if trying to access attributes of a message not yet received.
Plugins
--plugin some.python.module some.other.module.Class
Load one or more Python modules or classes as plugins. Supported (but not required) plugin interface methods:
-
init(args): invoked at startup with command-line arguments -
load(category, args): invoked with category "search" or "source" or "sink", using returned value for specified component if not None
Plugins are free to modify grepros internals, like adding command-line arguments
to grepros.main.ARGUMENTS or adding sink types to grepros.outputs.MultiSink.
Convenience methods:
-
plugins.add_write_format(name, cls, label=None, options=()): adds an output plugin to defaults -
plugins.get_argument(name): returns a command-line argument dictionary, or None
Specifying --plugin someplugin and --help will include plugin options in printed help.
Built-in plugins:
embag
--plugin grepros.plugins.embag
Use the embag library for reading ROS1 bags.
Significantly faster, but library tends to be unstable.
mcap
--plugin grepros.plugins.mcap
Read or write messages in MCAP format.
Requires mcap, and mcap_ros1_support for ROS1 or mcap_ros2_support for ROS2.
In ROS2, messages grepped from MCAP files can only be published to live topics if the same message type packages are locally installed.
Write bags in MCAP format:
--plugin grepros.plugins.mcap \
--write path/to/my.mcap [format=mcap] [overwrite=true|false]
If the file already exists, a unique counter is appended to the name of the new file,
e.g. my.2.mcap, unless specified to overwrite.
Specifying write format=mcap is not required if the filename ends with .mcap.
parquet
--plugin grepros.plugins.parquet \
--write path/to/my.parquet [format=parquet] [overwrite=true|false] \
[column-name=rostype:value] [type-rostype=arrowtype] \
[idgenerator=callable] [nesting=array|all] [writer-argname=argvalue]
Write messages to Apache Parquet files (columnar storage format, version 2.6),
each message type to a separate file, named path/to/package__MessageType__typehash/my.parquet
for package/MessageType (typehash is message type definition MD5 hashsum).
Adds fields _topic string() and _timestamp timestamp("ns") to each type.
If a file already exists, a unique counter is appended to the name of the new file,
e.g. package__MessageType__typehash/my.2.parquet, unless specified to overwrite.
Specifying format=parquet is not required if the filename ends with .parquet.
By default, message IDs are only added when populating nested message types,
as field _id string() with UUID content. To explicitly add ID columns:
--write path/to/my.parquet idgenerator="itertools.count()"
Column type is auto-detected from produced ID values: int64/float64 for numerics,
string for anything else (non-numerics cast to string).
Supports adding supplementary columns with fixed values to Parquet files:
--write path/to/my.parquet column-bag_hash=string:26dfba2c
Supports custom mapping between ROS and pyarrow types with type-rostype=arrowtype:
--write path/to/my.parquet type-time="timestamp('ns')"
--write path/to/my.parquet type-uint8[]="list(uint8())"
Time/duration types are flattened into separate integer columns secs and nsecs,
unless they are mapped to pyarrow types explicitly, like:
--write path/to/my.parquet type-time="timestamp('ns')" type-duration="duration('ns')"
Supports additional arguments given to pyarrow.parquet.ParquetWriter, as:
--write path/to/my.parquet writer-argname=argvalue
For example, specifying no compression:
--write path/to/my.parquet writer-compression=null
The value is interpreted as JSON if possible, e.g. writer-use_dictionary=false.
To recursively populate nested array fields:
--write path/to/my.parquet nesting=array
E.g. for diagnostic_msgs/DiagnosticArray, this would populate files with following schemas:
diagnostic_msgs__DiagnosticArray = pyarrow.schema([
("header.seq", pyarrow.int64()),
("header.stamp.secs", pyarrow.int32()),
("header.stamp.nsecs", pyarrow.int32()),
("header.frame_id", pyarrow.string()),
("status", pyarrow.string()), # [_id from "diagnostic_msgs/DiagnosticStatus", ]
("_topic", pyarrow.string()),
("_timestamp", pyarrow.int64()),
("_id", pyarrow.string()),
("_parent_type", pyarrow.string()),
("_parent_id", pyarrow.string()),
])
diagnostic_msgs__DiagnosticStatus = pyarrow.schema([
("level", pyarrow.int16()),
("name", pyarrow.string()),
("message", pyarrow.string()),
("hardware_id", pyarrow.string()),
("values"", pyarrow.string()), # [_id from "diagnostic_msgs/KeyValue", ]
("_topic", pyarrow.string()), # _topic from "diagnostic_msgs/DiagnosticArray"
("_timestamp", pyarrow.int64()), # _timestamp from "diagnostic_msgs/DiagnosticArray"
("_id", pyarrow.string()),
("_parent_type", pyarrow.string()), # "diagnostic_msgs/DiagnosticArray"
("_parent_id", pyarrow.string()), # _id from "diagnostic_msgs/DiagnosticArray"
])
diagnostic_msgs__KeyValue = pyarrow.schema([
("key" pyarrow.string()),
("value", pyarrow.string()),
("_topic", pyarrow.string()), # _topic from "diagnostic_msgs/DiagnosticStatus"
("_timestamp", pyarrow.int64()), # _timestamp from "diagnostic_msgs/DiagnosticStatus"
("_id", pyarrow.string()),
("_parent_type", pyarrow.string()), # "diagnostic_msgs/DiagnosticStatus"
("_parent_id", pyarrow.string()), # _id from "diagnostic_msgs/DiagnosticStatus"
])
Without nesting, array field values are inserted as JSON with full subtype content.
To recursively populate all nested message types:
--write path/to/my.parquet nesting=all
E.g. for diagnostic_msgs/DiagnosticArray, this would, in addition to the above, populate:
std_msgs__Header = pyarrow.schema([
"seq", pyarrow.int64()),
"stamp.secs", pyarrow.int32()),
"stamp.nsecs", pyarrow.int32()),
"frame_id", pyarrow.string()),
"_topic", pyarrow.string()), # _topic from "diagnostic_msgs/DiagnosticArray"
"_timestamp", pyarrow.int64()), # _timestamp from "diagnostic_msgs/DiagnosticArray"
"_id", pyarrow.string()),
"_parent_type", pyarrow.string()), # "diagnostic_msgs/DiagnosticArray"
"_parent_id", pyarrow.string()), # _id from "diagnostic_msgs/DiagnosticArray"
])
sql
--plugin grepros.plugins.sql \
--write path/to/my.sql [format=sql] [overwrite=true|false] \
[nesting=array|all] [dialect=clickhouse|postgres|sqlite] \
[dialect-file=path/to/dialects.yaml]
Write SQL schema to output file, CREATE TABLE for each message type and CREATE VIEW for each topic.
If the file already exists, a unique counter is appended to the name of the new file,
e.g. my.2.sql, unless specified to overwrite.
Specifying format=sql is not required if the filename ends with .sql.
To create tables for nested array message type fields:
--write path/to/my.sql nesting=array
To create tables for all nested message types:
--write path/to/my.sql nesting=all
A specific SQL dialect can be specified:
--write path/to/my.sql dialect=clickhouse|postgres|sqlite
Additional dialects, or updates for existing dialects, can be loaded from a YAML or JSON file:
--write path/to/my.sql dialect=mydialect dialect-file=path/to/dialects.yaml
SQL dialects
Postgres, SQLite and SQL outputs support loading additional options for SQL dialect.
Dialect file format:
dialectname:
table_template: CREATE TABLE template; args: table, cols, type, hash, package, class
view_template: CREATE VIEW template; args: view, cols, table, topic, type, hash, package, class
table_name_template: message type table name template; args: type, hash, package, class
view_name_template: topic view name template; args: topic, type, hash, package, class
types: Mapping between ROS and SQL common types for table columns,
e.g. {"uint8": "SMALLINT", "uint8[]": "BYTEA", ..}
adapters: Mapping between ROS types and callable converters for table columns,
e.g. {"time": "decimal.Decimal"}
defaulttype: Fallback SQL type if no mapped type for ROS type;
if no mapped and no default type, column type will be ROS type as-is
arraytype_template: Array type template; args: type
maxlen_entity: Maximum table/view name length, 0 disables
maxlen_column: Maximum column name length, 0 disables
invalid_char_regex: Regex for matching invalid characters in name, if any
invalid_char_repl: Replacement for invalid characters in name
Template parameters like table_name_template use Python str.format() keyword syntax,
e.g. {"table_name_template": "{type}", "view_name_template": "{topic}"}.
Time/duration types are flattened into separate integer columns secs and nsecs,
unless the dialect maps them to SQL types explicitly, e.g. {"time": "BIGINT"}.
Any dialect options not specified in the given dialect or built-in dialects, will be taken from the default dialect configuration:
table_template: 'CREATE TABLE IF NOT EXISTS {table} ({cols});'
view_template: 'DROP VIEW IF EXISTS {view};
CREATE VIEW {view} AS
SELECT {cols}
FROM {table}
WHERE _topic = {topic};'
table_name_template: '{type}',
view_name_template: '{topic}',
types: {}
defaulttype: null
arraytype_template: '{type}[]'
maxlen_entity: 0
maxlen_column: 0
invalid_char_regex: null
invalid_char_repl: '__'
Notes on ROS1 vs ROS2
In ROS1, message type packages do not need to be installed locally to be able to read messages from bags or live topics, as bags and topic publishers provide message type definition texts, and message classes can be generated at run-time from the type definition text. This is what rosbag does automatically, and so does grepros.
Additionally, each ROS1 message type has a hash code computed from its type definition text, available both in live topic metadata, and bag metadata. The message type definition hash code allows to recognize changes in message type packages and use the correct version of the message type.
ROS2 does not provide message type definitions, neither in bagfiles nor in live topics. Due to this, the message type packages always need to be installed. Also, ROS2 does not provide options for generating type classes at run-time, and it does not have the concept of a message type hash.
These are serious limitations in ROS2 compared to ROS1, at least with versions up to ROS2 Humble and counting, and require extra work to smooth over. Without knowing which version of a message type package a bag was recorded with, reading bag messages with changed definitions can result in undefined behaviour.
If the serialized message structure happens to match (e.g. a change swapped
the order of two int32 fields), messages will probably be deserialized
seemingly successfully but with invalid content. If the serialized structure
does not match, the result is a run-time error.
Because of this, it is prudent to always include a snapshot archive of used message type packages, when recording ROS2 bags.
grepros does provide the message type hash itself in ROS2 exports, by calculating the ROS2 message type hash on its own from the locally installed type definition.
All command-line arguments
positional arguments:
PATTERN pattern(s) to find in message field values,
all messages match if not given,
can specify message field as NAME=PATTERN
(supports nested.paths and * wildcards)
optional arguments:
-h, --help show this help message and exit
-F, --fixed-strings PATTERNs are ordinary strings, not regular expressions
-I, --no-ignore-case use case-sensitive matching in PATTERNs
-v, --invert-match select non-matching messages
--version display version information and exit
--live read messages from live ROS topics instead of bagfiles
--publish publish matched messages to live ROS topics
--write TARGET [format=bag|csv|html|postgres|sqlite] [KEY=VALUE ...]
write matched messages to specified output,
format is autodetected from TARGET if not specified.
Bag or database will be appended to if it already exists.
Keyword arguments are given to output writer.
commit-interval=NUM transaction size for Postgres/SQLite output
(default 1000, 0 is autocommit)
dialect-file=path/to/dialects.yaml
load additional SQL dialect options
for Postgres/SQLite output
from a YAML or JSON file
idgenerator=callable callable or iterable for producing message IDs
in Parquet output, like 'uuid.uuid4' or 'itertools.count()';
by default only nesting uses IDs
message-yaml=true|false whether to populate table field messages.yaml
in SQLite output (default true)
nesting=array|all create tables for nested message types
in Parquet/Postgres/SQLite output,
only for arrays if "array"
else for any nested types
(array fields in parent will be populated
with foreign keys instead of messages as JSON)
overwrite=true|false overwrite existing file in bag/CSV/HTML/SQLite output
instead of appending to if bag or database
or appending unique counter to file name
(default false)
template=/my/path.tpl custom template to use for HTML output
--plugin PLUGIN [PLUGIN ...]
load a Python module or class as plugin
(built-in plugins: grepros.plugins.embag,
grepros.plugins.mcap, grepros.plugins.parquet,
grepros.plugins.sql)
Filtering:
-t TOPIC [TOPIC ...], --topic TOPIC [TOPIC ...]
ROS topics to scan if not all (supports * wildcards)
-nt TOPIC [TOPIC ...], --no-topic TOPIC [TOPIC ...]
ROS topics to skip (supports * wildcards)
-d TYPE [TYPE ...], --type TYPE [TYPE ...]
ROS message types to scan if not all (supports * wildcards)
-nd TYPE [TYPE ...], --no-type TYPE [TYPE ...]
ROS message types to skip (supports * wildcards)
--condition CONDITION [CONDITION ...]
extra conditions to require for matching messages,
as ordinary Python expressions, can refer to last messages
in topics as {topic /my/topic}; topic name can contain wildcards.
E.g. --condition "{topic /robot/enabled}.data" matches
messages only while last message in '/robot/enabled' has data=true.
-t0 TIME, --start-time TIME
earliest timestamp of messages to scan
as relative seconds if signed,
or epoch timestamp or ISO datetime
(for bag input, relative to bag start time
if positive or end time if negative,
for live input relative to system time,
datetime may be partial like 2021-10-14T12)
-t1 TIME, --end-time TIME
latest timestamp of messages to scan
as relative seconds if signed,
or epoch timestamp or ISO datetime
(for bag input, relative to bag start time
if positive or end time if negative,
for live input relative to system time,
datetime may be partial like 2021-10-14T12)
-n0 INDEX, --start-index INDEX
message index within topic to start from
(1-based if positive, counts back from bag total if negative)
-n1 INDEX, --end-index INDEX
message index within topic to stop at
(1-based if positive, counts back from bag total if negative)
--every-nth-message NUM
scan every Nth message within topic
--every-nth-interval SECONDS
scan messages at least N seconds apart within topic
--every-nth-match NUM
emit every Nth match in topic
-sf FIELD [FIELD ...], --select-field FIELD [FIELD ...]
message fields to use in matching if not all
(supports nested.paths and * wildcards)
-ns FIELD [FIELD ...], --no-select-field FIELD [FIELD ...]
message fields to skip in matching
(supports nested.paths and * wildcards)
-m NUM, --max-count NUM
number of matched messages to emit (per each file if bag input)
--max-per-topic NUM number of matched messages to emit from each topic
(per each file if bag input)
--max-topics NUM number of topics to emit matches from (per each file if bag input)
--unique-only only emit matches that are unique in topic,
taking --select-field and --no-select-field into account
(per each file if bag input)
Output control:
-B NUM, --before-context NUM
emit NUM messages of leading context before match
-A NUM, --after-context NUM
emit NUM messages of trailing context after match
-C NUM, --context NUM
emit NUM messages of leading and trailing context
around match
-pf FIELD [FIELD ...], --print-field FIELD [FIELD ...]
message fields to print in console output if not all
(supports nested.paths and * wildcards)
-np FIELD [FIELD ...], --no-print-field FIELD [FIELD ...]
message fields to skip in console output
(supports nested.paths and * wildcards)
-mo, --matched-fields-only
print only the fields where PATTERNs find a match
-la NUM, --lines-around-match NUM
print only matched fields and NUM message lines
around match
-lf NUM, --lines-per-field NUM
maximum number of lines to print per field
-l0 NUM, --start-line NUM
message line number to start printing from
(1-based if positive, counts back from total if negative)
-l1 NUM, --end-line NUM
message line number to stop printing at
(1-based if positive, counts back from total if negative)
-lm NUM, --lines-per-message NUM
maximum number of lines to print per message
--match-wrapper [STR [STR ...]]
string to wrap around matched values,
both sides if one value, start and end if more than one,
or no wrapping if zero values
(default "**" in colorless output)
--wrap-width NUM character width to wrap message YAML output at,
0 disables (defaults to detected terminal width)
--color {auto,always,never}
use color output in console (default "always")
--no-meta do not print source and message metainfo to console
--no-filename do not print bag filename prefix on each console message line
--no-console-output do not print matches to console
--progress show progress bar when not printing matches to console
--verbose print status messages during console output
for publishing and writing
--no-verbose do not print status messages during console output
for publishing and writing
Bag input control:
-n FILE [FILE ...], --filename FILE [FILE ...]
names of ROS bagfiles to scan if not all in directory
(supports * wildcards)
-p PATH [PATH ...], --path PATH [PATH ...]
paths to scan if not current directory
(supports * wildcards)
-r, --recursive recurse into subdirectories when looking for bagfiles
--order-bag-by {topic,type}
order bag messages by topic or type first and then by time
--decompress decompress archived bagfiles with recognized extensions (.zst .zstd)
--reindex-if-unindexed
reindex unindexed bagfiles (ROS1 only), makes backup copies
Live topic control:
--publish-prefix PREFIX
prefix to prepend to input topic name on publishing match
--publish-suffix SUFFIX
suffix to append to input topic name on publishing match
--publish-fixname TOPIC
single output topic name to publish all matches to,
overrides prefix and suffix
--queue-size-in SIZE live ROS topic subscriber queue size (default 10)
--queue-size-out SIZE
output publisher queue size (default 10)
--ros-time-in use ROS time instead of system time for incoming message
timestamps from subsribed live ROS topics
Dependencies
grepros requires Python 3.8+ or Python 2.7, and the following 3rd-party Python packages:
- pyyaml (https://pypi.org/project/PyYAML)
- ROS1: rospy, roslib, rosbag, genpy
- ROS2: rclpy, rosidl_parser, rosidl_runtime_py, builtin_interfaces
Optional, for decompressing archived bags:
- zstandard (https://pypi.org/project/zstandard)
Optional, for faster reading of ROS1 bags:
Optional, for Postgres output:
- psycopg2 (https://pypi.org/project/psycopg2)
Optional, for Parquet output:
- pandas (https://pypi.org/project/pandas)
- pyarrow (https://pypi.org/project/pyarrow)
Optional, for MCAP input-output:
- mcap (https://pypi.org/project/mcap)
- mcap_ros1_support (https://pypi.org/project/mcap_ros1_support)
- mcap_ros2_support (https://pypi.org/project/mcap_ros2_support)
Optional, for generating API documentation:
- doxypypy (https://pypi.org/project/doxypypy;
needs latest master:
pip install git+https://github.com/Feneric/doxypypy)
Attribution
Includes a modified version of step, Simple Template Engine for Python, (c) 2012, Daniele Mazzocchio, https://github.com/dotpy/step, released under the MIT license.
License
Copyright (c) 2021 by Erki Suurjaak. Released as free open source software under the BSD License, see LICENSE.md for full details.
CONTRIBUTING
Repository Summary
| Checkout URI | https://github.com/suurjaak/grepros.git |
| VCS Type | git |
| VCS Version | master |
| Last Updated | 2023-03-30 |
| Dev Status | DEVELOPED |
| CI status | No Continuous Integration |
| Released | RELEASED |
| Tags | No category tags. |
| Contributing |
Help Wanted (0)
Good First Issues (0) Pull Requests to Review (0) |
Packages
| Name | Version |
|---|---|
| grepros | 0.6.0 |
README
grepros
grep for ROS bag files and live topics.
Searches through ROS messages and matches any message field value by regular expression patterns or plain text, regardless of field type. Can also look for specific values in specific message fields only.
By default, matches are printed to console. Additionally, matches can be written to a bagfile or HTML/CSV/MCAP/Parquet/Postgres/SQL/SQLite, or published to live topics.
Supports both ROS1 and ROS2. ROS environment variables need to be set, at least ROS_VERSION.
In ROS1, messages can be grepped even if Python packages for message types are not installed. Using ROS1 live topics requires ROS master to be running.
Using ROS2 requires Python packages for message types to be available in path.
Supports loading custom plugins, mainly for additional output formats.
API documentation available at https://suurjaak.github.io/grepros.
- Example usage
- Installation
- Inputs
- Outputs
- Matching and filtering
- Plugins
- SQL dialects
- Notes on ROS1 vs ROS2
- All command-line arguments
- Dependencies
- Attribution
- License
Example usage
Search for "my text" in all bags under current directory and subdirectories:
grepros -r "my text"
Print 30 lines of the first message from each live ROS topic:
grepros --max-per-topic 1 --lines-per-message 30 --live
Find first message containing "future" (case-sensitive) in my.bag:
grepros future -I --max-count 1 --name my.bag
Find 10 messages, from geometry_msgs package, in "map" frame, from bags in current directory, reindexing any unindexed bags:
grepros frame_id=map --type geometry_msgs/* --max-count 10 --reindex-if-unindexed
Pipe all diagnostics messages with "CPU usage" from live ROS topics to my.bag:
grepros "CPU usage" --type *DiagnosticArray --no-console-output --write my.bag
Find messages with field "key" containing "0xA002", in topics ending with "diagnostics", in bags under "/tmp":
grepros key=0xA002 --topic *diagnostics --path /tmp
Find diagnostics_msgs messages in bags in current directory, containing "navigation" in fields "name" or "message", print only header stamp and values:
grepros --type diagnostic_msgs/* --select-field name message \
--print-field header.stamp status.values -- navigation
Print first message from each lidar topic on host 1.2.3.4:
ROS_MASTER_URI=http://1.2.3.4::11311 \
grepros --live --topic *lidar* --max-per-topic 1
Export all bag messages to SQLite and Postgres, print only export progress:
grepros -n my.bag --write my.bag.sqlite --no-console-output --no-verbose --progress
grepros -n my.bag --write postgresql://user@host/dbname \
--no-console-output --no-verbose --progress
Patterns use Python regular expression syntax, message matches if all match. '*' wildcards use simple globbing as zero or more characters, target matches if any value matches.
Note that some expressions may need to be quoted to avoid shell auto-unescaping
or auto-expanding them, e.g. linear.x=2.?5 should be given as "linear.x=2\.?5".
Care must also be taken with unquoted wildcards, as they will auto-expanded by shell if they happen to match paths on disk.
Installation
grepros is written in Python, supporting both Python 2 and Python 3.
Developed and tested under ROS1 noetic and ROS2 foxy, it should also work in later ROS2 versions; it may work in earlier ROS1 versions.
Using pip
pip install grepros
This will add the grepros command to path.
Requires ROS Python packages (ROS1: rospy, roslib, rosbag, genpy; ROS2: rclpy, rosidl_parser, rosidl_runtime_py, builtin_interfaces).
For ROS1, if only using bag files and no live topics, minimal ROS1 Python packages can also be installed separately with:
pip install rospy rosbag roslib roslz4 \
--extra-index-url https://rospypi.github.io/simple/
Using apt
If ROS apt repository has been added to system:
sudo apt install ros-noetic-grepros # ROS1
sudo apt install ros-foxy-grepros # ROS2
This will add the grepros command to the global ROS1 / ROS2 environment.
Using catkin
In a ROS1 workspace, under the source directory:
git clone https://github.com/suurjaak/grepros.git
cd grepros
catkin build --this
This will add the grepros command to the local ROS1 workspace path.
Using colcon
In a ROS2 workspace, at the workspace root:
git clone https://github.com/suurjaak/grepros.git src/grepros
colcon build --packages-select grepros
This will add the grepros command to the local ROS2 workspace path.
Inputs
Input is either from one or more ROS bag files (default), or from live ROS topics.
bag
Read messages from ROS bag files, by default all in current directory.
Recurse into subdirectories when looking for bagfiles:
-r
--recursive
Read specific filenames (supports * wildcards):
--n /tmp/*.bag
--filename my.bag 2021-11-*.bag
Scan specific paths instead of current directory (supports * wildcards):
-p /home/bags/2021-11-*
--path my/dir
Reindex unindexed ROS1 bags before processing (note: creates backup copies of files, into same directory as file):
--reindex-if-unindexed
--reindex-if-unindexed --progress
Decompress archived ROS bags before processing
(.zst .zstd extensions, requires zstandard Python package)
(note: unpacks archived file to disk, into same directory as file):
--decompress
--decompress --progress
Order bag messages first by topic or type, and only then by time:
--order-bag-by topic
--order-bag-by type
Outputs
There can be any number of outputs: printing to console (default), publishing to live ROS topics, or writing to file or database.
console
Default output is to console, in ANSI colors, mimicking grep output.
Disable printing messages to console:
--no-console-output
Manage color output:
--color always (default)
--color auto (auto-detect terminal support)
--color never (disable colors)
Note that when paging color output with more or less, the pager needs to
accept raw control characters (more -f or less -R).
bag
--write path/to/my.bag [format=bag] [overwrite=true|false]
Write messages to a ROS bag file, the custom .bag format in ROS1
or the .db3 SQLite database format in ROS2. If the bagfile already exists,
it is appended to, unless specified to overwrite.
Specifying format=bag is not required
if the filename ends with .bag in ROS1 or .db3 in ROS2.
live
--live
Read messages from live ROS topics instead of bagfiles.
Requires ROS_MASTER_URI and ROS_ROOT to be set in environment if ROS1.
Set custom queue size for subscribing (default 10):
--queue-size-in 100
Use ROS time instead of system time for incoming message timestamps:
--ros-time-in
csv
--write path/to/my.csv [format=csv] [overwrite=true|false]
Write messages to CSV files, each topic to a separate file, named
path/to/my.full__topic__name.csv for /full/topic/name.
Output mimicks CSVs compatible with PlotJuggler, all messages values flattened
to a single list, with header fields like /topic/field.subfield.listsubfield.0.data.1.
If a file already exists, a unique counter is appended to the name of the new file,
e.g. my.full__topic__name.2.csv, unless specified to overwrite.
Specifying format=csv is not required if the filename ends with .csv.
html
--write path/to/my.html [format=html] [overwrite=true|false]
[template=/path/to/html.template]
Write messages to an HTML file, with a linked table of contents, message timeline, message type definitions, and a topically traversable message list.
Note: resulting file may be large, and take a long time to open in browser.
If the file already exists, a unique counter is appended to the name of the new file,
e.g. my.2.html, unless specified to overwrite.
Specifying format=html is not required if the filename ends with .htm or .html.
A custom template file can be specified, in step syntax:
--write path/to/my.html template=/my/html.template
postgres
--write postgresql://username@host/dbname [format=postgres]
[commit-interval=NUM] [nesting=array|all]
[dialect-file=path/to/dialects.yaml]
Write messages to a Postgres database, with tables pkg/MsgType for each ROS message type,
and views /full/topic/name for each topic.
Plus table topics with a list of topics, types with message types and definitions,
and meta with table/view/column name changes from shortenings and conflicts, if any
(Postgres name length is limited to 63 characters).
ROS primitive types are inserted as Postgres data types (time/duration types as NUMERIC), uint8[] arrays as BYTEA, other primitive arrays as ARRAY, and arrays of subtypes as JSONB.
If the database already exists, it is appended to. If there are conflicting names (same package and name but different message type definition), table/view name becomes "name (MD5 hash of type definition)".
Specifying format=postgres is not required if the parameter uses the
Postgres URI scheme postgresql://.
Requires psycopg2.
Parameter --write can also use the Postgres keyword=value format,
e.g. "host=localhost port=5432 dbname=mydb username=postgres connect_timeout=10".
Standard Postgres environment variables are also supported (PGPASSWORD et al).
A custom transaction size can be specified (default is 1000; 0 is autocommit):
--write postgresql://username@host/dbname commit-interval=NUM
Updates to Postgres SQL dialect can be loaded from a YAML or JSON file:
--write postgresql://username@host/dbname dialect-file=path/to/dialects.yaml
More on SQL dialects.
Nested messages
Nested message types can be recursively populated to separate tables, linked to parent messages via foreign keys.
To recursively populate nested array fields:
--write postgresql://username@host/dbname nesting=array
E.g. for diagnostic_msgs/DiagnosticArray, this would populate the following tables:
CREATE TABLE "diagnostic_msgs/DiagnosticArray" (
"header.seq" BIGINT,
"header.stamp.secs" INTEGER,
"header.stamp.nsecs" INTEGER,
"header.frame_id" TEXT,
status JSONB, -- [_id from "diagnostic_msgs/DiagnosticStatus", ]
_topic TEXT,
_timestamp NUMERIC,
_id BIGSERIAL,
_parent_type TEXT,
_parent_id BIGINT
);
CREATE TABLE "diagnostic_msgs/DiagnosticStatus" (
level SMALLINT,
name TEXT,
message TEXT,
hardware_id TEXT,
"values" JSONB, -- [_id from "diagnostic_msgs/KeyValue", ]
_topic TEXT, -- _topic from "diagnostic_msgs/DiagnosticArray"
_timestamp NUMERIC, -- _timestamp from "diagnostic_msgs/DiagnosticArray"
_id BIGSERIAL,
_parent_type TEXT, -- "diagnostic_msgs/DiagnosticArray"
_parent_id BIGINT -- _id from "diagnostic_msgs/DiagnosticArray"
);
CREATE TABLE "diagnostic_msgs/KeyValue" (
"key" TEXT,
value TEXT,
_topic TEXT, -- _topic from "diagnostic_msgs/DiagnosticStatus"
_timestamp NUMERIC, -- _timestamp from "diagnostic_msgs/DiagnosticStatus"
_id BIGSERIAL,
_parent_type TEXT, -- "diagnostic_msgs/DiagnosticStatus"
_parent_id BIGINT -- _id from "diagnostic_msgs/DiagnosticStatus"
);
Without nesting, array field values are inserted as JSON with full subtype content.
To recursively populate all nested message types:
--write postgresql://username@host/dbname nesting=all
E.g. for diagnostic_msgs/DiagnosticArray, this would, in addition to the above, populate:
CREATE TABLE "std_msgs/Header" (
seq BIGINT,
"stamp.secs" INTEGER,
"stamp.nsecs" INTEGER,
frame_id TEXT,
_topic TEXT, -- _topic from "diagnostic_msgs/DiagnosticArray"
_timestamp NUMERIC, -- _timestamp from "diagnostic_msgs/DiagnosticArray"
_id BIGSERIAL,
_parent_type TEXT, -- "diagnostic_msgs/DiagnosticArray"
_parent_id BIGINT -- _id from "diagnostic_msgs/DiagnosticArray"
);
sqlite
--write path/to/my.sqlite [format=sqlite] [overwrite=true|false]
[commit-interval=NUM] [message-yaml=true|false] [nesting=array|all]
[dialect-file=path/to/dialects.yaml]
Write an SQLite database with tables pkg/MsgType for each ROS message type
and nested type, and views /full/topic/name for each topic.
If the database already exists, it is appended to, unless specified to overwrite.
Output is compatible with ROS2 .db3 bagfiles, supplemented with
full message YAMLs, and message type definition texts. Note that a database
dumped from a ROS1 source will most probably not be usable as a ROS2 bag,
due to breaking changes in ROS2 standard built-in types and message types.
Specifying format=sqlite is not required
if the filename ends with .sqlite or .sqlite3.
A custom transaction size can be specified (default is 1000; 0 is autocommit):
--write path/to/my.sqlite commit-interval=NUM
By default, table messages is populated with full message YAMLs, unless:
--write path/to/my.sqlite message-yaml=false
Updates to SQLite SQL dialect can be loaded from a YAML or JSON file:
--write path/to/my.sqlite dialect-file=path/to/dialects.yaml
More on SQL dialects.
Nested messages
Nested message types can be recursively populated to separate tables, linked to parent messages via foreign keys.
To recursively populate nested array fields:
--write path/to/my.sqlite nesting=array
E.g. for diagnostic_msgs/DiagnosticArray, this would populate the following tables:
CREATE TABLE "diagnostic_msgs/DiagnosticArray" (
"header.seq" INTEGER,
"header.stamp.secs" INTEGER,
"header.stamp.nsecs" INTEGER,
"header.frame_id" TEXT,
-- [_id from "diagnostic_msgs/DiagnosticStatus", ]
status "DIAGNOSTIC_MSGS/DIAGNOSTICSTATUS[]",
_topic TEXT,
_timestamp INTEGER,
_id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
_parent_type TEXT,
_parent_id INTEGER
);
CREATE TABLE "diagnostic_msgs/DiagnosticStatus" (
level SMALLINT,
name TEXT,
message TEXT,
hardware_id TEXT,
-- [_id from "diagnostic_msgs/KeyValue", ]
"values" "DIAGNOSTIC_MSGS/KEYVALUE[]",
_topic TEXT, -- _topic from "diagnostic_msgs/DiagnosticArray"
_timestamp INTEGER, -- _timestamp from "diagnostic_msgs/DiagnosticArray"
_id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
_parent_type TEXT, -- "diagnostic_msgs/DiagnosticArray"
_parent_id INTEGER -- _id from "diagnostic_msgs/DiagnosticArray"
);
CREATE TABLE "diagnostic_msgs/KeyValue" (
"key" TEXT,
value TEXT,
_topic TEXT, -- _topic from "diagnostic_msgs/DiagnosticStatus"
_timestamp INTEGER, -- _timestamp from "diagnostic_msgs/DiagnosticStatus"
_id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
_parent_type TEXT, -- "diagnostic_msgs/DiagnosticStatus"
_parent_id INTEGER -- _id from "diagnostic_msgs/DiagnosticStatus"
);
Without nesting, array field values are inserted as JSON with full subtype content.
To recursively populate all nested message types:
--write path/to/my.sqlite nesting=all
E.g. for diagnostic_msgs/DiagnosticArray, this would, in addition to the above, populate:
CREATE TABLE "std_msgs/Header" (
seq UINT32,
"stamp.secs" INT32,
"stamp.nsecs" INT32,
frame_id TEXT,
_topic STRING, -- _topic from "diagnostic_msgs/DiagnosticArray"
_timestamp INTEGER, -- _timestamp from "diagnostic_msgs/DiagnosticArray"
_id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
_parent_type TEXT, -- "diagnostic_msgs/DiagnosticArray"
_parent_id INTEGER -- _id from "diagnostic_msgs/DiagnosticArray"
);
live
--publish
Publish messages to live ROS topics. Topic prefix and suffix can be changed, or topic name set to one specific name:
--publish-prefix /myroot
--publish-suffix /myend
--publish-fixname /my/singular/name
One of the above arguments needs to be specified if publishing to live ROS topics while grepping from live ROS topics, to avoid endless loops.
Set custom queue size for publishers (default 10):
--queue-size-out 100
console / html message formatting
Set maximum number of lines to output per message:
--lines-per-message 5
Set maximum number of lines to output per message field:
--lines-per-field 2
Start message output from, or stop output at, message line number:
--start-line 2 # (1-based if positive
--end-line -2 # (count back from total if negative)
Output only the fields where patterns find a match:
--matched-fields-only
Output only matched fields and specified number of lines around match:
--lines-around-match 5
Output only specific message fields (supports nested.paths and * wildcards):
--print-field *data
Skip outputting specific message fields (supports nested.paths and * wildcards):
--no-print-field header.stamp
Wrap matches in custom texts:
--match-wrapper @@@
--match-wrapper "<<<<" ">>>>"
Set custom width for wrapping message YAML printed to console (auto-detected from terminal by default):
--wrap-width 120
Matching and filtering
Any number of patterns can be specified, message matches if all patterns find a match. If no patterns are given, any message matches.
Match messages containing any of the words:
cpu memory speed
Match messages where frame_id contains "world":
frame_id=world
Match messages where header.frame_id is present:
header.frame_id=.*
Match as plaintext, not Python regular expression patterns:
-F
--fixed-strings
Select non-matching messages instead:
-v
--invert-match
Use case-sensitive matching in patterns (default is insensitive):
-I
--no-ignore-case
Limits
Stop after matching a specified number of messages (per each file if bag input):
-m 100
--max-count 100
Scan only a specified number of topics (per each file if bag input):
--max-topics 10
Emit a specified number of matches per topic (per each file if bag input):
--max-per-topic 20
Emit every Nth match in topic:
--every-nth-match 10 # (skips 9 matches in topic after each match emitted)
Filtering
Scan specific topics only (supports * wildcards):
-t *lidar* *ins*
--topic /robot/sensors/*
Skip specific topics (supports * wildcards):
-nt *lidar* *ins*
--no-topic /robot/sensors/*
Scan specific message types only (supports * wildcards):
-d *Twist*
--type sensor_msgs/*
Skip specific message types from scanning (supports * wildcards):
-nd *Twist*
--no-type sensor_msgs/*
Set specific message fields to scan (supports nested.paths and * wildcards):
-sf twist.linear
--select-field *data
Skip specific message fields in scan (supports nested.paths and * wildcards):
-ns twist.linear
--no-select-field *data
Only emit matches that are unique in topic,
taking --select-field and --no-select-field into account (per each file if bag input):
--unique-only
Start scanning from a specific timestamp:
-t0 2021-11 # (using partial ISO datetime)
--start-time 1636900000 # (using UNIX timestamp)
--start-time +100 # (seconds from bag start time, or from script startup time if live input)
--start-time -100 # (seconds from bag end time, or script startup time if live input)
Stop scanning at a specific timestamp:
-t1 2021-11 # (using partial ISO datetime)
--end-time 1636900000 # (using UNIX timestamp)
--end-time +100 # (seconds from bag start time, or from script startup time if live input)
--end-time -100 # (seconds from bag end time, or from script startup time if live input)
Start scanning from a specific message index in topic:
-n0 -100 # (counts back from topic total message count in bag)
--start-index 10 # (1-based index)
Stop scanning at a specific message index in topic:
-n1 -100 # (counts back from topic total message count in bag)
--end-index 10 # (1-based index)
Scan every Nth message in topic:
--every-nth-message 10 # (skips 9 messages in topic with each step)
Scan messages in topic with timestamps at least N seconds apart:
--every-nth-interval 5 # (samples topic messages no more often than every 5 seconds)
Conditions
--condition "PYTHON EXPRESSION"
Specify one or more Python expressions that must evaluate as true to search encountered messages. Expressions can access topics, by name or * wildcard, and refer to message fields directly.
# (Match while last message in '/robot/enabled' has data=true)
--condition "<topic /robot/enabled>.data"
# (Match if at least 10 messages have been encountered in /robot/alerts)
--condition "len(<topic /robot/alerts>) > 10"
# (Match if last two messages in /robot/mode have equal .value)
--condition "<topic /robot/mode>[-2].value == <topic /robot/mode>[-1].value"
# (Match while control is enabled and robot is moving straight and level)
--condition "<topic */control_enable>.data and <topic */cmd_vel>.linear.x > 0 " \
"and <topic */cmd_vel>.angular.z < 0.02"
Condition namespace:
| Name | Description |
|---|---|
msg |
current message from data source |
topic |
full name of current message topic |
<topic /my/topic> |
topic by full name or * wildcard |
len(<topic ..>) |
number of messages encountered in topic |
bool(<topic ..>) |
whether any message encountered in topic |
<topic ..>.xyz |
attribute xyz of last message in topic |
<topic ..>[index] |
topic message at position |
| (from first encountered if index >= 0, last encountered if < 0) | |
<topic ..>[index].xyz |
attribute xyz of topic message at position |
Condition is automatically false if trying to access attributes of a message not yet received.
Plugins
--plugin some.python.module some.other.module.Class
Load one or more Python modules or classes as plugins. Supported (but not required) plugin interface methods:
-
init(args): invoked at startup with command-line arguments -
load(category, args): invoked with category "search" or "source" or "sink", using returned value for specified component if not None
Plugins are free to modify grepros internals, like adding command-line arguments
to grepros.main.ARGUMENTS or adding sink types to grepros.outputs.MultiSink.
Convenience methods:
-
plugins.add_write_format(name, cls, label=None, options=()): adds an output plugin to defaults -
plugins.get_argument(name): returns a command-line argument dictionary, or None
Specifying --plugin someplugin and --help will include plugin options in printed help.
Built-in plugins:
embag
--plugin grepros.plugins.embag
Use the embag library for reading ROS1 bags.
Significantly faster, but library tends to be unstable.
mcap
--plugin grepros.plugins.mcap
Read or write messages in MCAP format.
Requires mcap, and mcap_ros1_support for ROS1 or mcap_ros2_support for ROS2.
In ROS2, messages grepped from MCAP files can only be published to live topics if the same message type packages are locally installed.
Write bags in MCAP format:
--plugin grepros.plugins.mcap \
--write path/to/my.mcap [format=mcap] [overwrite=true|false]
If the file already exists, a unique counter is appended to the name of the new file,
e.g. my.2.mcap, unless specified to overwrite.
Specifying write format=mcap is not required if the filename ends with .mcap.
parquet
--plugin grepros.plugins.parquet \
--write path/to/my.parquet [format=parquet] [overwrite=true|false] \
[column-name=rostype:value] [type-rostype=arrowtype] \
[idgenerator=callable] [nesting=array|all] [writer-argname=argvalue]
Write messages to Apache Parquet files (columnar storage format, version 2.6),
each message type to a separate file, named path/to/package__MessageType__typehash/my.parquet
for package/MessageType (typehash is message type definition MD5 hashsum).
Adds fields _topic string() and _timestamp timestamp("ns") to each type.
If a file already exists, a unique counter is appended to the name of the new file,
e.g. package__MessageType__typehash/my.2.parquet, unless specified to overwrite.
Specifying format=parquet is not required if the filename ends with .parquet.
By default, message IDs are only added when populating nested message types,
as field _id string() with UUID content. To explicitly add ID columns:
--write path/to/my.parquet idgenerator="itertools.count()"
Column type is auto-detected from produced ID values: int64/float64 for numerics,
string for anything else (non-numerics cast to string).
Supports adding supplementary columns with fixed values to Parquet files:
--write path/to/my.parquet column-bag_hash=string:26dfba2c
Supports custom mapping between ROS and pyarrow types with type-rostype=arrowtype:
--write path/to/my.parquet type-time="timestamp('ns')"
--write path/to/my.parquet type-uint8[]="list(uint8())"
Time/duration types are flattened into separate integer columns secs and nsecs,
unless they are mapped to pyarrow types explicitly, like:
--write path/to/my.parquet type-time="timestamp('ns')" type-duration="duration('ns')"
Supports additional arguments given to pyarrow.parquet.ParquetWriter, as:
--write path/to/my.parquet writer-argname=argvalue
For example, specifying no compression:
--write path/to/my.parquet writer-compression=null
The value is interpreted as JSON if possible, e.g. writer-use_dictionary=false.
To recursively populate nested array fields:
--write path/to/my.parquet nesting=array
E.g. for diagnostic_msgs/DiagnosticArray, this would populate files with following schemas:
diagnostic_msgs__DiagnosticArray = pyarrow.schema([
("header.seq", pyarrow.int64()),
("header.stamp.secs", pyarrow.int32()),
("header.stamp.nsecs", pyarrow.int32()),
("header.frame_id", pyarrow.string()),
("status", pyarrow.string()), # [_id from "diagnostic_msgs/DiagnosticStatus", ]
("_topic", pyarrow.string()),
("_timestamp", pyarrow.int64()),
("_id", pyarrow.string()),
("_parent_type", pyarrow.string()),
("_parent_id", pyarrow.string()),
])
diagnostic_msgs__DiagnosticStatus = pyarrow.schema([
("level", pyarrow.int16()),
("name", pyarrow.string()),
("message", pyarrow.string()),
("hardware_id", pyarrow.string()),
("values"", pyarrow.string()), # [_id from "diagnostic_msgs/KeyValue", ]
("_topic", pyarrow.string()), # _topic from "diagnostic_msgs/DiagnosticArray"
("_timestamp", pyarrow.int64()), # _timestamp from "diagnostic_msgs/DiagnosticArray"
("_id", pyarrow.string()),
("_parent_type", pyarrow.string()), # "diagnostic_msgs/DiagnosticArray"
("_parent_id", pyarrow.string()), # _id from "diagnostic_msgs/DiagnosticArray"
])
diagnostic_msgs__KeyValue = pyarrow.schema([
("key" pyarrow.string()),
("value", pyarrow.string()),
("_topic", pyarrow.string()), # _topic from "diagnostic_msgs/DiagnosticStatus"
("_timestamp", pyarrow.int64()), # _timestamp from "diagnostic_msgs/DiagnosticStatus"
("_id", pyarrow.string()),
("_parent_type", pyarrow.string()), # "diagnostic_msgs/DiagnosticStatus"
("_parent_id", pyarrow.string()), # _id from "diagnostic_msgs/DiagnosticStatus"
])
Without nesting, array field values are inserted as JSON with full subtype content.
To recursively populate all nested message types:
--write path/to/my.parquet nesting=all
E.g. for diagnostic_msgs/DiagnosticArray, this would, in addition to the above, populate:
std_msgs__Header = pyarrow.schema([
"seq", pyarrow.int64()),
"stamp.secs", pyarrow.int32()),
"stamp.nsecs", pyarrow.int32()),
"frame_id", pyarrow.string()),
"_topic", pyarrow.string()), # _topic from "diagnostic_msgs/DiagnosticArray"
"_timestamp", pyarrow.int64()), # _timestamp from "diagnostic_msgs/DiagnosticArray"
"_id", pyarrow.string()),
"_parent_type", pyarrow.string()), # "diagnostic_msgs/DiagnosticArray"
"_parent_id", pyarrow.string()), # _id from "diagnostic_msgs/DiagnosticArray"
])
sql
--plugin grepros.plugins.sql \
--write path/to/my.sql [format=sql] [overwrite=true|false] \
[nesting=array|all] [dialect=clickhouse|postgres|sqlite] \
[dialect-file=path/to/dialects.yaml]
Write SQL schema to output file, CREATE TABLE for each message type and CREATE VIEW for each topic.
If the file already exists, a unique counter is appended to the name of the new file,
e.g. my.2.sql, unless specified to overwrite.
Specifying format=sql is not required if the filename ends with .sql.
To create tables for nested array message type fields:
--write path/to/my.sql nesting=array
To create tables for all nested message types:
--write path/to/my.sql nesting=all
A specific SQL dialect can be specified:
--write path/to/my.sql dialect=clickhouse|postgres|sqlite
Additional dialects, or updates for existing dialects, can be loaded from a YAML or JSON file:
--write path/to/my.sql dialect=mydialect dialect-file=path/to/dialects.yaml
SQL dialects
Postgres, SQLite and SQL outputs support loading additional options for SQL dialect.
Dialect file format:
dialectname:
table_template: CREATE TABLE template; args: table, cols, type, hash, package, class
view_template: CREATE VIEW template; args: view, cols, table, topic, type, hash, package, class
table_name_template: message type table name template; args: type, hash, package, class
view_name_template: topic view name template; args: topic, type, hash, package, class
types: Mapping between ROS and SQL common types for table columns,
e.g. {"uint8": "SMALLINT", "uint8[]": "BYTEA", ..}
adapters: Mapping between ROS types and callable converters for table columns,
e.g. {"time": "decimal.Decimal"}
defaulttype: Fallback SQL type if no mapped type for ROS type;
if no mapped and no default type, column type will be ROS type as-is
arraytype_template: Array type template; args: type
maxlen_entity: Maximum table/view name length, 0 disables
maxlen_column: Maximum column name length, 0 disables
invalid_char_regex: Regex for matching invalid characters in name, if any
invalid_char_repl: Replacement for invalid characters in name
Template parameters like table_name_template use Python str.format() keyword syntax,
e.g. {"table_name_template": "{type}", "view_name_template": "{topic}"}.
Time/duration types are flattened into separate integer columns secs and nsecs,
unless the dialect maps them to SQL types explicitly, e.g. {"time": "BIGINT"}.
Any dialect options not specified in the given dialect or built-in dialects, will be taken from the default dialect configuration:
table_template: 'CREATE TABLE IF NOT EXISTS {table} ({cols});'
view_template: 'DROP VIEW IF EXISTS {view};
CREATE VIEW {view} AS
SELECT {cols}
FROM {table}
WHERE _topic = {topic};'
table_name_template: '{type}',
view_name_template: '{topic}',
types: {}
defaulttype: null
arraytype_template: '{type}[]'
maxlen_entity: 0
maxlen_column: 0
invalid_char_regex: null
invalid_char_repl: '__'
Notes on ROS1 vs ROS2
In ROS1, message type packages do not need to be installed locally to be able to read messages from bags or live topics, as bags and topic publishers provide message type definition texts, and message classes can be generated at run-time from the type definition text. This is what rosbag does automatically, and so does grepros.
Additionally, each ROS1 message type has a hash code computed from its type definition text, available both in live topic metadata, and bag metadata. The message type definition hash code allows to recognize changes in message type packages and use the correct version of the message type.
ROS2 does not provide message type definitions, neither in bagfiles nor in live topics. Due to this, the message type packages always need to be installed. Also, ROS2 does not provide options for generating type classes at run-time, and it does not have the concept of a message type hash.
These are serious limitations in ROS2 compared to ROS1, at least with versions up to ROS2 Humble and counting, and require extra work to smooth over. Without knowing which version of a message type package a bag was recorded with, reading bag messages with changed definitions can result in undefined behaviour.
If the serialized message structure happens to match (e.g. a change swapped
the order of two int32 fields), messages will probably be deserialized
seemingly successfully but with invalid content. If the serialized structure
does not match, the result is a run-time error.
Because of this, it is prudent to always include a snapshot archive of used message type packages, when recording ROS2 bags.
grepros does provide the message type hash itself in ROS2 exports, by calculating the ROS2 message type hash on its own from the locally installed type definition.
All command-line arguments
positional arguments:
PATTERN pattern(s) to find in message field values,
all messages match if not given,
can specify message field as NAME=PATTERN
(supports nested.paths and * wildcards)
optional arguments:
-h, --help show this help message and exit
-F, --fixed-strings PATTERNs are ordinary strings, not regular expressions
-I, --no-ignore-case use case-sensitive matching in PATTERNs
-v, --invert-match select non-matching messages
--version display version information and exit
--live read messages from live ROS topics instead of bagfiles
--publish publish matched messages to live ROS topics
--write TARGET [format=bag|csv|html|postgres|sqlite] [KEY=VALUE ...]
write matched messages to specified output,
format is autodetected from TARGET if not specified.
Bag or database will be appended to if it already exists.
Keyword arguments are given to output writer.
commit-interval=NUM transaction size for Postgres/SQLite output
(default 1000, 0 is autocommit)
dialect-file=path/to/dialects.yaml
load additional SQL dialect options
for Postgres/SQLite output
from a YAML or JSON file
idgenerator=callable callable or iterable for producing message IDs
in Parquet output, like 'uuid.uuid4' or 'itertools.count()';
by default only nesting uses IDs
message-yaml=true|false whether to populate table field messages.yaml
in SQLite output (default true)
nesting=array|all create tables for nested message types
in Parquet/Postgres/SQLite output,
only for arrays if "array"
else for any nested types
(array fields in parent will be populated
with foreign keys instead of messages as JSON)
overwrite=true|false overwrite existing file in bag/CSV/HTML/SQLite output
instead of appending to if bag or database
or appending unique counter to file name
(default false)
template=/my/path.tpl custom template to use for HTML output
--plugin PLUGIN [PLUGIN ...]
load a Python module or class as plugin
(built-in plugins: grepros.plugins.embag,
grepros.plugins.mcap, grepros.plugins.parquet,
grepros.plugins.sql)
Filtering:
-t TOPIC [TOPIC ...], --topic TOPIC [TOPIC ...]
ROS topics to scan if not all (supports * wildcards)
-nt TOPIC [TOPIC ...], --no-topic TOPIC [TOPIC ...]
ROS topics to skip (supports * wildcards)
-d TYPE [TYPE ...], --type TYPE [TYPE ...]
ROS message types to scan if not all (supports * wildcards)
-nd TYPE [TYPE ...], --no-type TYPE [TYPE ...]
ROS message types to skip (supports * wildcards)
--condition CONDITION [CONDITION ...]
extra conditions to require for matching messages,
as ordinary Python expressions, can refer to last messages
in topics as {topic /my/topic}; topic name can contain wildcards.
E.g. --condition "{topic /robot/enabled}.data" matches
messages only while last message in '/robot/enabled' has data=true.
-t0 TIME, --start-time TIME
earliest timestamp of messages to scan
as relative seconds if signed,
or epoch timestamp or ISO datetime
(for bag input, relative to bag start time
if positive or end time if negative,
for live input relative to system time,
datetime may be partial like 2021-10-14T12)
-t1 TIME, --end-time TIME
latest timestamp of messages to scan
as relative seconds if signed,
or epoch timestamp or ISO datetime
(for bag input, relative to bag start time
if positive or end time if negative,
for live input relative to system time,
datetime may be partial like 2021-10-14T12)
-n0 INDEX, --start-index INDEX
message index within topic to start from
(1-based if positive, counts back from bag total if negative)
-n1 INDEX, --end-index INDEX
message index within topic to stop at
(1-based if positive, counts back from bag total if negative)
--every-nth-message NUM
scan every Nth message within topic
--every-nth-interval SECONDS
scan messages at least N seconds apart within topic
--every-nth-match NUM
emit every Nth match in topic
-sf FIELD [FIELD ...], --select-field FIELD [FIELD ...]
message fields to use in matching if not all
(supports nested.paths and * wildcards)
-ns FIELD [FIELD ...], --no-select-field FIELD [FIELD ...]
message fields to skip in matching
(supports nested.paths and * wildcards)
-m NUM, --max-count NUM
number of matched messages to emit (per each file if bag input)
--max-per-topic NUM number of matched messages to emit from each topic
(per each file if bag input)
--max-topics NUM number of topics to emit matches from (per each file if bag input)
--unique-only only emit matches that are unique in topic,
taking --select-field and --no-select-field into account
(per each file if bag input)
Output control:
-B NUM, --before-context NUM
emit NUM messages of leading context before match
-A NUM, --after-context NUM
emit NUM messages of trailing context after match
-C NUM, --context NUM
emit NUM messages of leading and trailing context
around match
-pf FIELD [FIELD ...], --print-field FIELD [FIELD ...]
message fields to print in console output if not all
(supports nested.paths and * wildcards)
-np FIELD [FIELD ...], --no-print-field FIELD [FIELD ...]
message fields to skip in console output
(supports nested.paths and * wildcards)
-mo, --matched-fields-only
print only the fields where PATTERNs find a match
-la NUM, --lines-around-match NUM
print only matched fields and NUM message lines
around match
-lf NUM, --lines-per-field NUM
maximum number of lines to print per field
-l0 NUM, --start-line NUM
message line number to start printing from
(1-based if positive, counts back from total if negative)
-l1 NUM, --end-line NUM
message line number to stop printing at
(1-based if positive, counts back from total if negative)
-lm NUM, --lines-per-message NUM
maximum number of lines to print per message
--match-wrapper [STR [STR ...]]
string to wrap around matched values,
both sides if one value, start and end if more than one,
or no wrapping if zero values
(default "**" in colorless output)
--wrap-width NUM character width to wrap message YAML output at,
0 disables (defaults to detected terminal width)
--color {auto,always,never}
use color output in console (default "always")
--no-meta do not print source and message metainfo to console
--no-filename do not print bag filename prefix on each console message line
--no-console-output do not print matches to console
--progress show progress bar when not printing matches to console
--verbose print status messages during console output
for publishing and writing
--no-verbose do not print status messages during console output
for publishing and writing
Bag input control:
-n FILE [FILE ...], --filename FILE [FILE ...]
names of ROS bagfiles to scan if not all in directory
(supports * wildcards)
-p PATH [PATH ...], --path PATH [PATH ...]
paths to scan if not current directory
(supports * wildcards)
-r, --recursive recurse into subdirectories when looking for bagfiles
--order-bag-by {topic,type}
order bag messages by topic or type first and then by time
--decompress decompress archived bagfiles with recognized extensions (.zst .zstd)
--reindex-if-unindexed
reindex unindexed bagfiles (ROS1 only), makes backup copies
Live topic control:
--publish-prefix PREFIX
prefix to prepend to input topic name on publishing match
--publish-suffix SUFFIX
suffix to append to input topic name on publishing match
--publish-fixname TOPIC
single output topic name to publish all matches to,
overrides prefix and suffix
--queue-size-in SIZE live ROS topic subscriber queue size (default 10)
--queue-size-out SIZE
output publisher queue size (default 10)
--ros-time-in use ROS time instead of system time for incoming message
timestamps from subsribed live ROS topics
Dependencies
grepros requires Python 3.8+ or Python 2.7, and the following 3rd-party Python packages:
- pyyaml (https://pypi.org/project/PyYAML)
- ROS1: rospy, roslib, rosbag, genpy
- ROS2: rclpy, rosidl_parser, rosidl_runtime_py, builtin_interfaces
Optional, for decompressing archived bags:
- zstandard (https://pypi.org/project/zstandard)
Optional, for faster reading of ROS1 bags:
Optional, for Postgres output:
- psycopg2 (https://pypi.org/project/psycopg2)
Optional, for Parquet output:
- pandas (https://pypi.org/project/pandas)
- pyarrow (https://pypi.org/project/pyarrow)
Optional, for MCAP input-output:
- mcap (https://pypi.org/project/mcap)
- mcap_ros1_support (https://pypi.org/project/mcap_ros1_support)
- mcap_ros2_support (https://pypi.org/project/mcap_ros2_support)
Optional, for generating API documentation:
- doxypypy (https://pypi.org/project/doxypypy;
needs latest master:
pip install git+https://github.com/Feneric/doxypypy)
Attribution
Includes a modified version of step, Simple Template Engine for Python, (c) 2012, Daniele Mazzocchio, https://github.com/dotpy/step, released under the MIT license.
License
Copyright (c) 2021 by Erki Suurjaak. Released as free open source software under the BSD License, see LICENSE.md for full details.